Provence: efficient and robust context pruning for retrieval-augmented generation
Authors: Nadezhda Chirkova
Thibault Formal
Vassilina Nikoulina
Stéphane CLINCHANT
Venue: ICLR 2025 Poster (2025)
Keywords: retrieval-augmented generation
context pruning
question answering
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose Provence, an efficient and robust context pruner for question answering, which dynamically detects the amount of irrelevant information in contexts and prunes it out with negligible to no drop in performance

Abstract: Retrieval-Augmented Generation improves various aspects of large language models (LLMs) generation,  but suffers from computational overhead caused by long contexts, and the propagation of irrelevant retrieved information into generated responses. Context pruning deals with both aspects, by removing irrelevant parts of retrieved contexts before LLM generation. Existing context pruning approaches are limited, and do not present a universal model that would be both _efficient_ and _robust_ in a wide range of scenarios, e.g., when contexts contain a variable amount of relevant information or vary in length, or when evaluated on various domains. In this work, we close this gap and introduce Provence (Pruning and Reranking Of retrieVEd relevaNt ContExts), an efficient and robust context pruner for Question Answering, which dynamically detects the needed amount of pruning for a given context and can be used out-of-the-box for various domains. The three key ingredients of  Provence are formulating the context pruning task as sequence labeling, unifying context pruning capabilities with context reranking, and training on diverse data. Our experimental results show that Provence enables context pruning with negligible to no drop in performance, in various domains and settings, at almost no cost in a standard RAG pipeline. We also conduct a deeper analysis alongside various ablations to provide insights into training context pruners for future work.

Published as a conference paper at ICLR 2025
PROVENCE: EFFICIENT AND ROBUST CONTEXT PRUNING FOR RETRIEVAL- AUGMENTED GENERATION
Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, Stéphane Clinchant Naver Labs Europe
Grenoble, France {nadia a chirkova, thibault. forma f weeaeeeeaabbb....ee com
ABSTRACT
Retrieval-augmenttd generation improves various aspects of large language mod- els (LLMMs) generation, but suffers from computational overhead caused by long contexts as well as the propagation of irrelevant retrieved information into gener- ated responses. Context pruning deals with both aspects, by removing irrelevant parts of retrieved contexts before LLM generation. Existing context pruning ap- proaches are however limited, and do not provide a universal model that would be both efficient and robust in a wide range of scenarios, e.g., when contexts contain a variable amount of relevant information or vary in length, or when evaluated on various domains. In this work, we close this gap and introduce Provence (Pruning and Reranking Of Ved relevaNt ContExts), an efficient and robust context pruner for Question Answering, which dynamically detects the needed amount of pruning for a given context and can be used out-of-the-box for various domains. The three key ingredients of P Provence are formulating the context pruning task as sequence labeling, unifying context pruning capabilities with con- reranking, and training on diverse data. Our experimental results show that P Provence enables context pruning with negligible to no drop in performance, in various domains and settings, at almost no cost in a standard RAG pipeline. We also conduct a deeper analysis alongside various ablations to provide insights into training context pruners for future work. Our mode and code? are publicly available.
INTRODUCTION
Retrieval- Augmented Generation (RAG) has become a widely-used paradigm for improving factu- ality, attribution, and adaptability of Large Language Models (LLMs) (Das al. 2019 Asai et al.) 2024| Seo et al., 2019; Lewis et al. 2020 Mallen et al.] 2023a Min et al. 2023). ). Augmenting
given user's 's query with retrieved relevant contexts helps to avoid the generation of untruthful infor- mation and enables the provision of references used to generate the answer. Furthermore, using a domain-specifie datastore may enable access and reasoning over a previously unknown knowledge without fine-tuning the LLM. One additional advantage of the RAG approach is the easy plug-and- play architecture (LangChain): practitioners may choose components (retrievers, generator LLMs, context granularity etc.) which best suit their particular cases to maximize the final performance.
At the same time, the use RAG adds computational overhead due to both retrieval latency and the increased input length for the LLMs. It may also propagate irrelevant information present in retrieved contexts into generated responses. These issues can solved by developing more efficient and robust LLMs either by making architectural changes to process long contexts more efficiently (Nawrot et al., 2024 Dao 2024 Chevalier et al., (023) or increasing the diversity of the tuning data to improve processing of irrelevant contexts (Lin et al.) 2024). However, tuning the LLM can be highly resource- consuming, or even impossible to apply for proprietary (closed) I.LMs. An alternative solution consists in pruning retrieved contexts by removing context parts irrelevant to the user's 's query which reduces context lengths and therefore speeds up generation. Such context
OOr model: htw.:///ww//gggggeeee///////eeee
Our http:///pp..to.caaaaaaaa/////e/a

---

Published as a conference paper at ICLR 2025
Inference
passages
Training
pronge
Purtplinn
Eatsebovn
Engilo?
Patiowar
Conteeal: [1] Irfeecee of French Enplish 2
shrengthen your immune system and prose one uye health. Howeree, you not too much, you may experience
remurier
LLM
The most natadale toere of French has
provided Question using only information
you
Road
information provided, must ourput "No arswwer If some parts the Contect are used so answer you must ols the ccomespendingg sentences using symbola coneenetien
pumple
Ulor
sag?
PROVENCE
pruner
Contect
pruin use uye heatt. Howaseer il you not too mush, you may experience
help aberagtter inmture syalem
sumplin
Froid weids
Gnglan?
contont
Inpat
Nurpet
Sanget
Prined contont
Peruma seare
reranker
Train quezton
PROVENCE
decerrted sugueees
PROVENCE
LUM
You, busl nol lso mucis
(BOS] How many... int once of The most...
Figure I: Illustration of inference (left) and training (right) of Provence.
Table I: Analysis of existing approaches for context pruning. Violet / Orange highlight practical / less-practical solutions.
Approach
Query- dep.
Cramularity
Output
Type
Base urch.
Multi- domain testing
Model re-
lease
Selective Context
No
token-lovel
% of tokens
extr.
Llama-7B/ / GPT2
Yos
Yos
LLMLingu
No
token-level
%8 of tokens
extr.
Alpaca-7BB GPT2
Yes
Yos
Yes
token level
% of tokens
Llama 2 BB-chal
Yes
Yes
MT ingun2
No
token-level
% of rokens
extr.
RoAERTa /RERT
Yes
Yos
RECOMP
Yes
sent.-level
sentences
BERT
Yos
RECOMP abstr.
Yes
sent.-level
> O sentences
abstr.
TS-L
No
Yos
FilCo
level
sentence
abstr.
XL Llama 2:7B
COMPACT
Yes
sent.-level
sentences
abstr.
Mistral-7B
No
Yos
Provence (ours)
Yes
sent.-level
sentences
DeBERTa
Yos
Yos
pruning module can be used in plug-and-play manner with any generator LLM, featuring both easy use and better transparency in the RAG pipeline.
Despite initial efforts on developing context pruners for RAG, none of the existing solutions pro- vide a model ready be used out-of-the-box in practice. First, many approaches are designed for simplified setting, e.g., with the assumption that only one sentence per context is relevant to the input query (Wang et al. 2023 Xu et al. 2024), or that the compression ratio is fixed (Jiang et al.) 2023
Pan et a 2024). However, in practice contexts may contain various portions of relevant in- formation, from empty to full relevant context, and pruners should detect it in an adaptable fashion. Second, many works introduce context pruners that are not efficient enough to be used in practice.
This includes using billion-sized LLMs as base models for pruners (Jing et al., 2024 Pan et al., 2024
Wang et al.) 2023), or designing abstractive context compressors which require sequential autoregressive generation of the final context (Wang et al.] 2023 Xu et al. 2024). We argue that a more practical and efficient setting consists in fine-tuning a small-size model such DeBERTa (He) eet al., 2021b a). as an extractive pruner, i.c., with a lightweight prediction head for selecting relevant context parts. Third, most of the existing works train context pruners for each dataset individually and do not target nor test pruners robustness to various data domains.
Subble[ssmmarizz the properties of various existing methods along specified dimensions and shows that none of them satisfy all listed criteria. The table also includes a dimension of pruning granular- ity, i.e., token-level ys sentence-level pruning. In this work, we focus on query-dependent sentence- level pruning, which prunes out semantic units (sentences) that are deemed not relevant to generate the answer. An alternative approach is token-level pruning which prunes out low-level grammatical units such as articles or interjections, usually in a query-independent fashion. The two approaches are orthogonal and could potentially be combined.
To address listed limitations, we introduce P Provence (Pruning and Reranking Of red rel- evaNt ContExt), an approach for training an adaptable, efficient and robust sentence-level context pruner for Question Answering, which can be used out-of-the-box across various domains and set- tings. To achieve this, we formulate context pruning as binary sequence labeling so that the binary mask predicted by the pruner determines sentences (from zero all) which are relevant to the query,
2

---

Published as a conference paper at ICLR 2025
and train our pruner from lightweight DeBERTa model diverse data. Furthermore, notice that context pruning and reranking (i.e., the second step in effective retrieval pipelines) bear a strong resemblance. We therefore propose to unify these two models into single one, completely elimi- nating the cost of context pruning in the RAG pipeline.
More specifically, contributions are follows:
We propose approach for training adaptable, robust, and efficient context pruner for QA and release our trained model and code. Three key ingredients of our approach are formulating context pruning sequence labeling, unifying context pruning and reranking in a single model, and training on diverse data.
We test Provence on various QA domains and show its out-of-the-box applicability to prune contexts with negligible to no drop performance and almost no cost, substan- tially outperforming baseline approaches. We also demonstrate Provence capabilities in detecting the number of relevant sentences at any positions in the context and robustness to various context lengths.
We conduct multiple ablations to demonstrate which techniques are essential for training robust context pruners, to provide insights for future context pruners development.
Definitions. A typical RAG pipeline consists of (0) a user's question, or query; (1) a datastore, i..., a collection of documents (pieces of text) to be retrieved from, (2) an efficient retriever which enables fast retrieval from large datastore (typically a dual-encoder model, where queries and passages are encoded independently), (3) more expensive cross-encoder reranker which further reduces and reorders set of retrieved passages (cross-encoding means encoding passage together with a query); and (4) generator LLM which outputs the final response based on the user's 's query and the relevant passages. Such pipeline can be represented retrieve >> rerank >> generate. Context pruning can be incorporated before generation, i.c., retrieve >> zerank >> prune >> generate. In our work, also propose incorporate context pruning into reranking, an essential and already present component in RAG (Rau et al.) (2024)): retrieve >> rerank+prume >> generate. This enables context pruning almost zero cost.
2 RELATED WORK
Context pruning, RECOMP (Xu et al. 2024) focuses on context pruning for RAG and proposes both extractive and abstractive context pruners. The extractive RECOMP approach independently encodes sentences in the context and then selects top sentences with embeddings closest to the query embedding. Such an approach limits context understanding, due to independent processing sentences and queries. The method also requires specifying the amount of sentences to keep as a hyperparameter which usually unknown practice and should depend on each particular passage. The abstractive RECOMP summarizes key information from the passage relevant to the query (including zero relevant information) by training on silver summaries generated by GPT-3.5. However, it requires inefficient autoregressive generation of the final context, and eventually hallucinate facts not present in the input context. FilCo (Wang et al., 2023) similarly proposes to generate contexts autoregressively but trained on extractive targets, i.c., one sentence from the context selected by one of three criteria. The drawbacks are again inefficiency and the simplified assumption of relevant sentence per context. recent approach, COMPACT (Yoon al. 2024). also proposes to generate filtered contexts autoregressively hence inefficiently and introduces an iterative approach for gradually updating the relevant context after processing new portion of retrieved passages. In contrast to all listed efforts, P Povence dynamically detects the amount of relevant information in the context from zero to all sentences in an extractive and efficient way. Furthermore, we propose novel approach of integrating context pruning into reranker.
both
Concurrently to our work, DSLR (Hwang et al. 2024) performs extractive sentence-level pruning, by encoding sentences one-by-one, together with the query, using existing rerankers. Similarly to Provence, DSLR keeps sentences with scores higher than a threshold and preserves the original order of sentences. However, in contrast to Provence, DSLR processes sentences in context inde- pendently, which could lead to misunderstanding of coreference between sentences and consequent pruning errors.

---

Published as a conference paper at ICLR 2025
An orthogonal line of work proposes extractive token-level pruners. LLMLingua (Jiang al. 2023) and Selective Context (Li et al. 2023) use LLMs to remove tokens with high generation probabil- ities, independently of the query. LLMLingua2 (Pan et al. 2024) is a small BERT-based model finetuned to eliminate redundant tokens, also independently of the query. LongI.LMMingge. (Jiang et al., 2024) proposes query-dependent LLMM-baed token pruning based contrastive perplexity. Listed approaches remove tokens in a way that it does not break context understanding for the LLM hence they are not capable of removing semantic parts of the context. LLMLingua models also have many hyperparameters in the interface which are hard to tune in practice. These approaches can however also be combined with sentence-level pruning.
Retrieval granularity. Alternatively to context pruning, one can reformulate datastore content into atomic units, e.g., propositions as in Dense-X retrieval Chen et al. (2024c) or decontextualized sen- Iences (Choi et al. 202T.. Such preprocessing is expensive and can lead to some information loss.
Passage filtering. Another related and orthogonal line of works focuses on filtering entire pas- sages if they are deemed irrelevant for given question; such an approach can be straightforwardly combined with Provence. A simple method consists in introducing a threshold on the (re)ranking score. LongLLM LLiinngaa reranks passages based on the probability of question given the passage. (Yoran et al. 2024) use natural language inference models to filter out passages that do not entail question-answert pairs, but report that this approach sometimes filters out relevant passages too.
Improving context processing in LLMs. While context pruners aim to remove context parts ir- relevant to the user's query, another line of work aims to process contexts more efficiently and effectively in LLMs. Efficient context processing could be achieved through efficient attention im- plementations (Dao 2024 Anagnostidis al. (023), KV cache compression (Nawrot al.. 2024).
encoding retrieved passages in parallel (Zhu et al. (024)), or compressing contexts into one or more context embeddings (Chevalier et al., 2023] Ge et al.) 2024] Rau al. 2024b, Louis et al. 2025). Other works aim to make LLMs more robust, by exposing them to noisy contexts during training or finetuning (Izacard al. 2022 Lin al., 2024). All such approaches usually require LLM
adaptation which may complicate application to an arbitrary picked LLM.
PROVENCE
The high-level overview of our proposed approach is illustrated in Figure Our first contribution is to pose the context pruning problem as sequence labeling task. We fine-tune a DeBERTa model to encode the query-context pair and output binary masks which are used to filter out irrelevant context parts. The labels for training generated by LLama-3-8B 3-n-s----- (AI@ Meta 2024); we call them silver labels since they are generated automatically. Such an approach solves several limitations of existing context pruners: by construction, the model is able to deal with varying noise in contexts and select an appropriate pruning ratio; (2) queries are encoded together with context sentences (cross-encoding), providing richer representations compared for instance extractive RECOMP which encodes query and context sentences independently; (3) using a lightweight encoder makes our approach more efficient than LLMd-baeed abstractive methods.
Our second contribution consists in unifying reranking and context pruning instead of considering these steps distinct in the RAG pipeline. In Provence, reranking and pruning can be done in single forward step, thus eliminating the computational overhead due to context pruning - making Provence almost "free".
Training data. Our approach requires a set of training questions and retrieval datastore. Spefi- cially, we rely on the train set of the MS MARCO document ranking collection which includes 370k queries (Nguyen et al. 2016). The MS MARCO collection is domain-divers datastore of 3.2M documents crawled from the Web - which is required for the final model's robustness to various domains and is often used to train retrievers and rerankers. In ablations, we also consider the train set of Natural Questions which contains 87k queries Kwiatkowski et al. 2019).
Data processing. We create a retrieval datastore by splitting MS MARCO documents into passages consisting of N consecutive sentences - N being a random integer E 1..10. This is to enable the pruner's robustness to variable retrieved context lengths. We also prepend page titles to each pas- sage. For each question, we retrieve top-5 relevant passages using a strong retrieval pipeline (Rau et al. 2024a) consisting of SPLADE-v3 retriever (Lassance et al.] 2024) and DeBERTa-v3
4

---

Published as a conference paper at ICLR 2025
reranker (Lassance & Clinchant 2023). The resulting set of retrieved passages is naturally diverse w.r.t. relevance or irrelevance to the question, due to imperfections in retrieval.
Silver labels generation. Given question and retrieved passage (context), we split the passage into sentences Sa and prompt Llama-3-8B-Ban 8-Intrttttttt to select sentences relevant to the given question. One approach would be to use a straightforward prompt such "Output indexes of sentences rel- evant to the given question". ". However, we decided to utilize the strong LLMs' capabilities of ac- tually answering questions while citing relevant context sentences. We therefore instruct the LLM to answer the given question using only information provided in the given context, and output "No answer" in case no relevant information is provided. We also specify the easy-to-pane citation for- mat [i] and number sentences with the same marker in the context. Our prompt can be found in Appendix - Table [6] we use greedy decoding and parse cited sentences using regular expressions. We also compare different prompting strategies in the ablation study.
We found that Llama-3-8B is well capable of answering only based on a given context in most cases and of outputting citation 90% of the time. We filter out cases when no citations are produced and "No answer" is not present the LLM's output, as these are the cases when the context actually contains relevant information but the LLM "forgot" to cite it. The final labels distribution (number of selected sentences per context, their positions) is shown in Appendix - Figure 5
Training of Provence. Our context pruner receives input the concatenation of a question and a retrieved context, and outputs per-token binary labels denoting whether each token (defined by the pretrained model's tokenizer) should be included in the selected context. In Section 4.4 (Ab- lations), we also consider an approach where a special token is inserted at the beginning of each sentence, and labels are predicted per-sentence based on the representations of those tokens. We train Provence as binary per-token classifier with ground truth labels coming from the silver data labeling, and the model can be used a standalone pruner, i.e., retrieve >> rerank >> Provence (standalone) >> generate.
Unifying compression and reranking. We note that cross-encoder rerankers (Nogueira & Cho 2020) share both the same architecture and inputs (pairs of question-pasages) as Provence. Ad- ditionally, the task of context pruning (selecting parts of contexts that are useful for generating the answer to the question) intrinsically bears similarity with re-ranking (estimating the relevance of a context w.r.t. the question)- - and we hypothesize the possibility of knowledge transfer between these two related tasks. We therefore propose to unify both approaches in a single model, with two different task heads. More specifically, the reranking head outputs a scalar prediction for the BOS token while the pruning head outputs per-token predictions for the passage tokens, as illustrated in Figure [ To ease training, we propose to further fine-tune a pretrained reranker on our labeling objective, while adding ranking "regularizer" to preserve initial reranking capabilities. The regu- larizer is a Mean Squared Error loss on the reranking scores from the initial reranker. This can be viewed as a straightforward pointwise score distillation process, where the initial model serves as the teacher- - a method that has demonstrated great effectiveness in Information Retrieval Hotstätter al. (202T). The final loss function as follows:
N
Im log P(yn, + A(sn - 2n.0)²
Zn = Provence(ax))
(I)
n=1
where N is the number of datapoints (query-passage pairs), x. is a sequence of L,, + 1 input tokens (concatenated query, passage and BOS at the 0-th position), zn is a sequence of L., + predictions output by the model, Yn is a sequence of Ln, target binary labels for context pruning, 8,, is the teacher score (initial reranker), Zn,a is the ranking score predicted from the BOS representation.
In the case of the unified model, re-ranking and context pruning need a single forward step from the encoder, i.e., retrieve >> Provence (w/ re-ranking) >> generate - making context pruning almost free in terms of execution time.
Inference with Provence. At inference, we feed a concatenation of a question and retrieved pas- sage through P Provence, which outputs probabilities of including each token in the final context, as well as the passage score the case of the unified model. We simply use a threshold 7 to binarize
using the nitk. sent_tokeniitin ze function:
www.k/wwwkkkk.mmmm /ttp///.op/ittkcoco.eok..k ze.
sent sent_tokenize htmml
5

---

Published as a conference paper at ICLR 2025
Natural Questions (wiki)
Hotpot QA (wiki)
TyDi QA (wiki)
Pop QA (wiki)
100
100
100
100
compression rate compression rate
75
75
25
25
25
0
0
70
45
70
55
60
Syllabus QA (syllabi)
BioASQ (Pubmed)
RGB (news)
Full context
100
100
Provence (w/ reranker)
75 X
75
Provence (standalone)
LLMLingua2
50
50
50
LongLLMLingua
RECOMP (ext)
25
25
RECOMP (ext++thr)
RECOMP (abs)
DSLR
40
50
70
80
90
95
Dense-X retrieval
LLM-Eval
LLM-Eval
LLM-Eval
Figure 2: Main results for various QA domains, comparing Provence and baseline models. Gen- erator: LLama-2-7B, retriever: SPLADE-v3 v3, reranker: DeBERTa-v3 (or Provence in the unified setting). Plot titles denote "Dataset name (datastore type)". z-axis denotes QA performance eval- uated with LLM-as-a-judg;; y-axis denotes the context compression ratio. For both metries, the higher the better: the best model would be closest to the top right corner. Numerical scores are presented in App. Tables 8 Main conclusion:
Provence consistently lies on the Pareto front.
the token probabilites (keep or not) - which has a direct effect on the compression rate. As shown in the experiments section, the choice of threshold is generally transferable across various datasets, making the model flexible to be used out-of-the box in various QA applications
We note that our model outputs token-level predictions despite the sentence-level labeling task. We found that probabilities of including tokens into the final context are naturally clustered on the sentence level see example Appendix Figure 6 due to the sentence-level targets used in training. However, in rare cases we could still have partial sentences being selected. To avoid this phenomenon, we apply a "sentence rounding" procedure: for each sentence, we check the ratio of kept tokens (predicted label= 1), and select the entire sentence only if it is higher than 0.5.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Provence training details. We train Provence on the MS Marco data processed as described in Section 3 using PyTorch (Paszke al. 2019) and HuggingFace transformers (Wolf al. 2020). We use DeBERTa-v3 (He et | 2002 la) as our pretrained model for training the standalone P Provence. For the unified approach, we start training from an already trained cross-encoder, also based on DeBERTa-v3 (Lassance & Clinchant 2023). Note that in the latter, we initialize the ranking head from its fine-tuned version, and train the separate pruning head from scratch.
After preliminary experiments, we set the learning rate to 3 x 10-6, 6 the batch size to 48 and train models for epoch. For joint training, there is slight trade-off between pruning and reranking. We set the reranking regularization coefficient À to 0.05, chosen as the minimal value that does not substantially degrade reranking performance on the MS MARCO development set.
Evaluation datasets. We test Provence on a diverse set of QA datasets. First, we consider com- monly used datasets relying on Wikipedia datastore: Natural Questions (Kwiatkowski al. 2019)
Note that tuning the threshold per dataset could of course further improve results.

---

Published as a conference paper at ICLR 2025
Table 2: Time/MFLOPS required for context pruning. 50 samples, with batch size set I (1 sample consists of a query and top-5 retrieved documents).
Table 3: Speed up in generation due to compression (Provence, 49%
compression), Batch sizes or 256.
Pruner
Time (s)
FLOPS
M.ng.....ningua rate=0.55
194
5.400e+16
Generator
bs 1
bs 256
LLMLingua2, rate=0.5
38
9.822c+15
RECOMP extr., top=2
12
1.037e+15
LLama-2- 2-7B
x1.2
X2
RECOMP abstr.
499
5.357e+14
LLama-2 2aaa----318 131
x1.4
x2
DSLR, th=0.5
26
4.391c+15
SOLAR -10.7B
x
x1.9
Provence (standalone)
25
7.901e+14
TyDi QA (Clark et 2020), PopQA (Mallen al. 2023b) (all three datasets include single-hop questions), and HotpotQA YYang et al. with datastores from various domains: BioASQ (Nentidis al. 2023) (biomedical questions with Pubmed as datastore), SyllabusQA (Fernandez et al. 2024) (questions about educational course logistics, with courses syllabus datastore); and RGB Chen et al. (2024b) (questions about news with Google-searched news as contexts). Further details can be found in Appendix
2018) (multi-hop hop questions). Second, we consider datasets
Evaluation settings. We conduct experiments using BERGEN (Rau al., 2024a). benchmarking library for RAG, using the recommended experimental setting. For each query, retrieve top- relevant passages using a strong and robust retrieval pipeline: SPLADE-v3 >> DeBERTa-v3 reranker (except for RGB, for which Google-searched passages are already provided). We then pass (Touvron et al. 2023 to generate answers; other retrieval-get -enerraor settings reported in Appendix. Each evaluation dataset comes with short keyword answers, which we use to evaluate responses using LL.M-based evaluation (L..Meval in Rau al. 2024a); match-based metrics also reported in Appendix. We additionally measure compression as a portion of the context which was pruned out.
queries prepended with relevant document (full length or pruned) into
We compare Provence publicly available context pruning models listed in Table 1 except LLMLingua and Selective Context which were shown to underperform LLMLingua2 (Pan et al., 2024 For all context pruners (except abstractive RECOMP for which is not available), we enforce the selection of the first (title) passage sentence, to avoid ambiguity in understanding the context by the generator. For extractive RECOMP, we use the model trained on NQ, consider using top- sentences, and prepend the passage title to each sentence. For the LLML family, we vary the compression rate (0.25, 0.5, 0.75) and use code provided on the official repository We use the XLM-RoBERTa model for LLMLingua2. For Provence, we use T 0.1 and T = 0.5. We also compare our method to DSLR based on the same reranker as ours, i.e, DeBERTa-v3.
4.2 MAIN RESULTS
Context pruners are often only tested on limited domain data, e.g., with Wikipedia datastore, and important aspect of our work is evaluating context pruning series of QA domains. Figure reports the trade-off between compression (efficiency) and LLM-evaluated performance (quality), for various QA datasets and context pruning methods. We choose to report figure per dataset to better assess the Pareto front of existing solutions, rather than comparing methods with different compression rates in the same table. Figure 7 Appendix further reports similar results with match- based metric, and Appendix Tables I1 show examples of context pruning with various methods.
First, we observe that Provence achieves the highest performance across pruning methods, for similar compression ratios. Second, it is noteworthy that Provence outperforms methods requiring more computations such as LLMLingua models, showing that efficiency is not traded for effective- ness. Furthermore, Provence is the only method capable of achieving high compression levels without (or with negligible) performance drops, on all datasets. Moreover, for some datasets, egg. PopQA, pruning with Provence leads to performance improvements due to noise filtering.
'For main experiments, we chose a "wcaker" generator which relies more on contexts, to create a more challenging setting for context prumers; results with stronger generators are reported in Appendix - Figure https:: / http://////o.coroffff///././/// ngua

---

Published as a conference paper at ICLR 2025
Avg number of selected
2
NO
(T+0.3)
Context granulazity
LLMeval LLMeval Comp- (pruned entx)
(fall entx)
ression
1-
25
Natural questions
47
15
26
53
25
12
21
21
29
sents
66.8
66.1
2%
sents
68.3
68.1
50%
1.0
(T-0.3)
BIASS (T-0.1)
10 sents
68.8
67.5
64%
100 words
00.5
65.6
48%
Match
0.5
Hotpot QA
1-sent "needle'
9.7
sents
52.8
52.8
8%
0.0
2-sent "needle"
42
1.1
38
11
81
211
037 2.5
9.3
17
10 sents
sents
54.6
55.7
54.0
55.1
68%
55%
Position text
FATT derts by Provence
n.m. FANTS Provence
100 words
51.4
51.2
53%
Figure 3: Analyses. (Left) Needle-in-the boellein-the--haystack test allowing the control of the position of the ground truth sentence(s) in the context. (Middle) Comparison of the number selected sentences by the silver predictor (LLaMA-3-8B-Innnmee
and Provence. Heatmaps are normalized by rows: a cell in position (é. indicates which percentage of contexts that were pruned into sentences by the silver predictor, were pruned into j sentences by Provence. (Right) Testing Provence in settings with different context lengths. All experiments are done with unified Provence, T 0.1.
The effect of threshold. An important aspect in the out-of-the-box applicability of context pruners how much effort is needed to select the suitable values of byperparameters. For P Provence, only consists in setting the pruning threshold T. In Figure 2 (for which T 0.1 and T 0.5), we observe that Provence pruning ratio automatically varies from 50% to 80%, depending on the dataset, which demonstrates that the same values for T work well for all considered domains mak- ing Provence robust to the choice of hyperparameters. If necessary, users can still tune it further for their datasets and/or needs. We note that some models specify the desired compression ratio as hyperparameter, e.g., LLMLingua models or extractive RECOMP (through top-N sentences). While it may seem convenient to estimate inference cost, the "optimal" compression ratio (without losing performance) is specific to each particular question-context pair. Thus, using threshold as hyperparameter more appropriate for this task. We also experimented with specifying a threshold in extractive RECOMP (shown on the same plot) and found that it often leads to lower performance (compared to tVp The reason is that different queries have different ranges of similarity scores.
Efficiency. We compare Provence with other pruning methods in terms of efficiency. Table 2 reports compression time and FLOPS required by different pruning methods. As expected, LongLLMLingua (based LLama-2-7B-chh on smaller modles. RECOMP abstr. requires less FLOPS compared to P Provence, but its au- toregressive nature makes it very slow in practice. Note that in the case of the unified Provence model, pruning is almost free - as it's part of the re-ranking step. Table 3 reports speed-up gains due compression with Provence model (~ 50% compression rate). All runs were performed on single Tesla V100- 1100-SXM2-32GB GPU with vllm Kwon et al. (2023). With large batch sizes, we systematically observe 2x speed-ups at inference. while smaller batch sizes lead lower gains (especially for smaller models). We assume this is mostly due to the CPU/GPU communication bottleneck, which masks inference gains due to compression.
is much slower compared to other pruners based
4.3 ANALYSIS
This section reports more fine-grad fine-grained evaluation to better understand Provence properties.
Robustness the position of relevant information in the context. We design needle-in-the- haystack experiment which allows us to check the performance of Provence on simple toy example and to evaluate its robustness w.r.t. the position of the relevant information in the input
We use hittps://oe thwbtthb.bbb/.////// ctco..cco/aaaa comm/clig 1111999f11111 ops pree-profllee to report FLOPPS required by each pruner. We that FLOPS do always align with real inference time.
8

---

Published as a conference paper at ICLR 2025
Table Effectiveness of reranking top-50 documents retrieved by SPLADE-v3. DeBERTa-v3 is the "baseline" (initialization point for Provence, which we aim to preserve performance). We report the R @5 on two RAG datasets (NQ and HotpotQA), MRR@ @ on MS MARCO passages (dev set), nDCG @ 10 on TREC 1L'19 (Craswell et al. 2020). and mean nDCG@ @ 10 on the 13 open datasets Table 7 in Appendix reports the full results.
from the BEIR benchmark (Thakur et al. 2021
Dataset
Model
NQ
HotpotQA
MS
TREC19
BEIR
DeBERTa-v3
83.0
70.4
40.5
77.4
55.4
Provence
84.4
70.5
40.6
77.2
55.9
Provence (NQ)
84.5
70.3
40.2
77.5
55.1
context. We write questions and answer and insert answers ("needles") random positions between sentences, in a subset of 100 passages sampled from the Wikipedia datastore. Ideally, Pr Provence should only select the "needle" sentences and filter out all other sentences in contexts. We plot the number of selected sentences and percentage of cases when the pruned context contains the "needle" (Figure B] (Left)). We consider two settings: with 1- and 2-sentence "needles". We observe that Provence correctly selects "needle" sentence(s) in most cases, except at leftmost and rightmost positions
most cases P Provence does not select any irrelevant sentences. The results similar for both simpler (1 (I-sentence) and harder (2-sentence) "needles" showing Provence's flexibility in detecting the number of relevant sentences, discussed below in more details.
Adaptability to the variable number of relevant sentences. To evaluate the capability of Provence to dynamically detect the number relevant sentences in the context, we compare the number of sentences L selected by Provence and by silver oracle. A silver oracle is easy to construct for L = 0, by pairing questions with randomly sampled contexts. For L > 1, we use the labeling produced by Llama-3-8B-Instruet. Figure 3 (Middle) shows that the number of relevant sentences detected by P proence is close to the silver oracle value in most cases, for all considered datasets. In contrast, extractive RECOMP would always select prespecified number of sentences.
Robustness w.r.f. context granularity. Figure (Right) shows Provence performance for two datasets, with Wikipedia datastores made of contexts of various granularity. Here, each considered datastore is produced by splitting Wikipedia pages into chunks of N sentences, N E {2, 6, 10), or 100 words, and prepending the page title to each chunk. Provence shows high performance in all cases the performance with pruned contexts being close to the performance obtained using original contexts. As could be expected, the compression ratio higher for longer contexts.
Reranking effectiveness. compares reranking performance between our reranking baseline and unified Provence - whose training starts from the former. We can see that our joint training procedure (on both pruning and ranking tasks) makes it possible to learn context pruner that pre- serves initial reranking capabilities. We further include as a comparison point results from a model trained in similar conditions NQ. Overall, results similar further highlighting the robustness of P prvence w.r.t. training data. We further discuss such aspects in Section 4.4 (Ablations).
Applicability in different settings. Figure 8 (App.) demonstrates the applicability of P Provence in variable striivall-enneatoo settings - achieving similar results as the ones reported in Figure
4.4 ABLATIONS
In this Section we analyze various design choices made in Provence development, to provide insights into training context pruners for future works (results shown in Figure All models in this section are standalone context pruners, trained with the same amount parameter updates.
"Which library was used in the experiments?", answer: "Experiments were conducted using the Bergen library". Example reformulation into a 2-sentence answer: "Experiments conducted using library. Its name is Bergen."
'Example:
"The reason for the drops in the left-most and right-most positions is that training data has little examples of the corresponding types of relevant sentences, see e.g., statistics for the rightmost position in the App. Figure (Right). We leave further improving processing of these positions to future work.

---

Published as a conference paper at ICLR 2025
NQ (wiki)
Hotpot QA (wiki)
Syllabus QA (syllabi)
compression rate
80
80
80-
Daterta-lame
Anchor
NS-Harca (ooxnommple))
NS-Harcz
40
stru ghd, orece
20
20
wett: atwling
inteaet ande
hul cottect
70
72
52
54
56
40
45
50
LiMeval
LLMevvll
LLMeval
Figure 4: Ablation results. All models are single-component modifications of the anchor model, which is base-size model, trained on NQ, with the answer oracle and token-level labeling. Numeric scores match-based results are presented in Appendix Table TO/ Figure II correspondingly.
Model size. We first observe that DeBERTa-large slighly increases the compression rate when comprared DeBERTa-base. All other ablations are tuned from DeBERTa-base model, for ef- ficiency reasons. Note that the final Provence is trained from a DeBERTa-large model (or its equivalent reranker).
Data mixtures. We compare training on NQ (87k queries), MS MARCO downsampled to the same size, and full MS MARCO (370k queries). Despite the observation that using the MS MARCO type of data leads to lower results than NQ with equal number of queries we also find that using larger data (i.e., full MS MARCO) improves results. Our final models are trained on the full MS MARCO further ablations are conducted on the NQ data, for efficiency reasons.
Labeling strategies. As described Section 3 we can train the pruner either to perform token- level labeling (with sentence rounding at inference) former case sentence representations are richer but the model also needs to learn to output similar predictions for tokens inside one sentence. In the latter case sentence content must be represented in a single embedding which may limit representation expressivity. In practice we observe close per- formance, with the token-level strategy slightly outperforming the sentence-level one some datasets. In all other experiments we use the token-level strategy.
perform sentence-level labeling. In the
Oracle prompts. We compare three options for prompting an oracle LLM generate silver label- ing: (/) answer oracle: asking to answer the given question from the given context, citing corre- sponding sentences; (2) relevance oracle: asking list any relevant information in the context to the question, citing corresponding sentences; (3) straightforward oracle: asking to output indexes of sentences which answer the given question. We found that the behavior of the straightforward ora- ele varies on different prompts, while the use of the answer oracle makes answers more consistent. The motivation for the relevance oracle that often contexts contain distantly relevant information to the query and it could be reasonable to select the corresponding sentences. Comparing the listed prompts, we observe that the relevance oracle underperforms the answer oracle, and the straight- forward oracle performs similarly slightly lower than the answer oracle.
Unification with reranker. In Figure we compare Provence trained as a standalone model and model unified with reranker, and find that both strategies lead to similar results although the former relies on two separate inference steps (re-ranking and pruning) RAG pipeline.
CONCLUSION
In this work, we present Provence, robust, adaptable, and efficient context pruner for Question Answering either unified in a single model with reranking capabilities or available as lightweight standalone model. In contrast to previous extractive approaches, Provence dynamically detects the needed pruning ratio for given context and can be used out-of-the-box for various QA domains. In extensive experiments, we demonstrate that Provence prunes contexts with negligible to no drops in performance and in some cases even brings performance improvement due to removing context noise. We also show Provence capabilities in correctly detecting the number of relevant sentences in contexts, located at any position, and with contexts of various lengths. Finally, the ablation study highlights the importance of using a large training data and the appropriate prompt in the silver oracle.
10

---

Published as a conference paper at ICLR 2025
Limitations. Despite Provence being ready to use in various settings, demonstrated in the paper, it is focusing only on QA applications, with a single passage processed at a time, and is trained on English-only data. Future work could consider extending it to other tasks, multi-passage contexts, and languages beyond English.
REFERENCES
Al@Meta Llama 3 model card. 2024. 1lama3/blobmm M/MODEL, CARD md
t htth://mmaaalammm:a 1amma/
Sotiris Anagmostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurélien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transform- ers. ArXiv, 15802, 2023. URL https:/ stw.ai..scsmmtmcccooososs.ss httpo//stt////aoooeaaaaaaaaooa.. org/
Corpus1D:258888224
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannanch Hajishirzi, and Wen-tau Yih. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint arXiv:2403.03187, 2024.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionaliiy,, multi-granularity text embeddings through self-knowledge dis- tillation, 2024a. URL bRttps::://axii erg. abs /tw.////2222222222 2402. 03216
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmennted generation. Proceedings of the AAAI Conference Artificial Intelligence, 638216.29728. URL https://ojs.
38116)17774417722 Mar. 2024b. doi: 10.
Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense X retrieval: What retrieval granularity should use? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15159-15177, Miami, Florida, USA, November 2024c. Association for Computational Linguistics. URL https: /ots://acc http//acllaee h//t/aaachtttooooooooooooo 2024. www.pmmain.eeeeemmmmmmmmmmmmamwm 845
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3829-3846, Singapore, December 2023. Association for Computational Linguisties. doi: 10.18653/v 1/2023. emnlp-main.233 2322. URL wttph:tt/////c /ttLtttts///acatttotootte
02g/2023....0 emnlp main 232
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael
Collins. Decontextualization: Making sentences stand-alone. Transactioos of the Association for Computational Linguistics, 9:447-461, 2021. doi:
URL
https:
7/ //aclanthology htw.aachhhooooo.......///...aaaa
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information information-seekkngg question answering in typologically diverse languages. Transactions of the Association for Computational Lingustiies, 8:454-470, 2020. doi: 10. 1621ta/ a 003117. URL https: //ccl tacl- 1.330
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the trec 2019 deep learning track, 2020. URL BtRLhhtts://aaxii wttp:////w.aab...o.rrr./
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. Ms marco: Bench- marking ranking models in the large-data regime. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21 pp. 1566-1576, New York, NY, USA, 2021. Association for Computing Machinery. ISBN doi: 10.1145/340483534422200
978 9783450383379.
URL /attps://ec..rg////
844 3462804
11

---

Published as a conference paper at ICLR 2025
Tri Dao. Flashattention-22 Faster attention with better parallelism and work partitioning. In
The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview net/ 2ott://oormiiimmm22222mm..
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. Multi-step retriever- reader interaction for scalable open-domain question answering. In International Confer- ence on Learning Representations, 2019. URL phttps:///ppeeeee www.eweiiw........ ew. net./ / fo forum? ?
HkfPSh05K7
Nigel Fernandez, Alexander Scarlatos, and Andrew Lan. SyllabusQA: A course logistics ques- tion answering dataset. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10344-10369, Bangkok, Thailand, August 2024. Association for Computational Lin- htt.
guistics. doi: 10. all ac Long.577
URLhttps: http:///aaaaatlhoggggee org.2044..
Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. The Twelfth International Confer- ence Learning Representations, 2024. URL uREjG2uGJE
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2021a.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. (DEBERTA): (DECODING)-
(enhanced) (bert) [with] } [disentangled) (attention). In International Conference on Learning Representations, 2021b. URI[https: /wwpnroepeeeeeeeooooooompe..e.ee net/forum? wtwffrrrmmmdxPPPeeaeeee www.ddd
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with architecttee knowledge distillation, 2021.
URL htttps://aaxtttt org/ abs/ www.//21100 2010. 02666
Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, and Graham Neubig. Ragged: Towards informed design of retrieval augmented generation systems. arXiv preprint arXiv:2403.09041 2024.
Taeho Hwang. Soyeong Jeong, Sukmin Cho, SeungYoon Han, and Jong Park. DSLR: Document re- finement with sentence-level re-ranking and reconstruction to enhance retrieval-augmented gen- eration. In Wenhao Yu, Weijia Shi, Michihiro Yasunaga, Meng Jiang, Chenguang Zhu, Han- nanch Hajishirzi, Luke Zettlemoyer, and Zhihan Zhang (eds.). Proceedings of the 3rd Work- shop on Knowledge Augmented Methods for NLP, pp. 73-92, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:
http3//w1112224401leeeeeeee 1.6. URL org 2024 http//222...oweeeeeeeeee -wwwledgenlpp1...666
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Y Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot Learning with Retrieval Augmented Language Models, November 2022. URL
abs/ /bss/200..0 03299 urXiv:2208.03299 [cs].
Huiqiang Jiang, Qianhui Wu. Chin-Yew Lin. Yuqing Yang. and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 13358-1333 13376, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10. 10.18653/w122233.mmmm www.eeemmmmmmmn...mmm 825. URL https:
wwwaantholoo...or///000 emn.p. main. 825
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua. Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1658-1677, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL httpps://ack antho ogy. wgw..gg........... acl long. 91
12

---

Published as a conference paper at ICLR 2025
Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang. Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. Solar 10.7b: Scaling large language models with simple yet effective depth up-scaling, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 46653-4666260 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023.
LangChain. LangChain Documentation. https://pythor: hwtp:/hhho...gncniiiiiinii www.chaainnccomm
Carlos Lassance and Stéphane Clinchant. Naver labs europe (splade) @ trec deep learning 2022, 2023.
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. Splade-v3: New base- lines for splade, 2024.
Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmennte vances in Neural Information Processing Systems, volume 33, pp. 9459-9474. Curran Asso- ciates, Inc., 2020. URL httpt:ttt//cccm...... htttt:::://rroceedingg/o neurips wwtpppapee//2////eeeeeeeeeeeeeee 6b4 6b4932230255 780e wtt.bc22644552 7487e47744111 Abst ract. html
Generation for Knowledge- Intensive NLP Tasks. In Ad-
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context enhance in- ference efficiency large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Pro- Singapore, December 2023, Association for Computational Linguis- tics. doi: 1001166531112023mmmmmmo
cessing, pp.
URLhttps: http:///aaaaatthooggyo. org.oog0...g/22....
emmn 1 p-ma in. 391
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '21, pp. 2356-2362, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379. doi: 10. N0.1145/340483534462222
URLpttts:://dat wrw.:o..gg../o. 1ttp:/10.1144.0 1145/ 00445//444855.. 3204448853344333 3463238
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. RA-DIT: Retrieval-augmennte dual instruction tuning. In The Twelfth International Confer- ence on Leurning Representations, 2024. URL hnttps://openreee/ 220Tbutug9
net/ forum?id-
Maxime Louis, Hervé Déjean, and Stéphane Clinchant. Pisco: Pretty simple compression for retrieval-augmenntd generation, 2025. (RRpsttts://
abs /alss/251...m 12550.16755)
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non- parameeriiic memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802 9822, Toronto, Canada, July 2023a. Association for Computational Linguis- 1015533/112222.llllllllo
tics. doi: acl ong. 6
URL https: http//aaaanntttcccccccca org/2033.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not trust language models: Investigating effectiveness of parametric and non-parametric
13

---

Published as a conference paper at ICLR 2025
memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 9802-9822, Toronto, Canada, July 2023b. Association for Computational Linguis- doi: 111665/////2223aalllooo acl ong.ong.50.. 546
tics.
URL. https: /ttp///aaaa/thot///
org/2023.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation, 2023. URL
1425T]
Piotr Nawrot, Adrian La' ncucki, Marcin Chochowski, David Tarjan, and E. Ponti. Dynamic memory compression: Retrofitting Ilms for accelerated inference. ArXiv, abs/2403. 09966, 2024. URL
https://app
Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Salvador Lima-López, Eulalia Farré-Maduell, Luis Gasco Sanchez, Martin Krallinger, and Georgios Paliouras. Overview of BioASQ 2023: The Eleventh BioASQ Challenge Large-Scale Biomedical Semantic In- dexing and Question Answering, pp. 227-250. 09 2023. ISBN 978-3-031-42447-2 doi: 10000/997-3-001--22400
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and
Li Deng. Ms marco: human generated machine reading comprehension dataset. Novem- ber 2016. URL httpt::// http://wwww.... wttw://wwwwwoottttt////////ttttt
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert, 2020.
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang. Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings ofe the Association for Computa- tional Linguistics ACL 2024, pp. 963-981, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https: /aaaaaaaaaatttooyo.aaaa wtg.o/00...g/222/.
Eirdings-acl.55
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, high-perrforancce deep learning library. Curran Associates Inc., Red Hook, NY, USA, 2019.
David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Vassilina Nikoulina, and Stéphane Clinchant. Bergen: A benchmarking library for retrieval-augmented generation, 2024a. htthhttps:/ //arxiv. org/ httg/abb...
abs/ / 2400 01102
David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. Context embeddings for efficient 09252
answer generation in rag, 2024b. http:tt::://arxiit
Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. Real-time open-domain question answering with dense-sparse phrase index. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 4430-4441, Florence, Italy, July 2019, Association for URL
Computational Linguistics. doi: 10. 10.18653/v1/P19 1436.
orq//P99 10t/P1991100 1436 1436
Xiao Shitao, Liu Zheng, Shao Yingxia, and Cao Zhao. Retromae: Pre-training retrieval-oriietted language models via masked auto-encoder. In EMNLP. 2022. URL httts::/arxiv..or
/205. 12035
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fift Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. htt.stttt:////..m http:::///pperrvvieeee wtt.ee/eneeooooooooooooaoootrrrr
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to filter context for retrieval-augmented generation, 2023. URL https://arxiv..or
2311.0833
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. emplp-demos.6. URL htw.h:tp///// http:p/:////aaaahooooooosomo org
/20gg/22000 2020 www.eeennlo...omm 2rm..//.e.me.....mmmmde..os....
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In The Twelfth International Confer-
ence on Learning Representations, 2024. URL /ttpps://openrevlew hwtpeeewwwwwwweeeeeeeeeeeeeeeeee wtwpee/wwwwweeefoouumumm.a.
mIJLVigNHp
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop ques- tion answering. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v URL htt.thtts///clattthhtt http://aclammmoooooo.orr////1/1/
pp.
logy. wrg...gg//.....o.oo 1259
Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Com-
pact: Compressing retrieved documents actively for question answering, 2024. URL https:
//axxiv.orr//as//220....
Ori Yoran, Tomer Wolfson, Oni Ram, and Jonathan Berant. Making retrieval-augmente language models robust to irrelevant context. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL
4 4k2pH
Yun Zhu, Jia-Chen Gu, Caitlin Sikora, Ho Ko, Yinxiao Liu, Chu-Cheng Lin, Lei Shu, Liangchen Luo, Lei Meng, Bang Liu, and Jindong Chen. Accelerating inference of retrieval-augmeentd generation via sparse context selection. ArXiv, 161188 2024. URL lhtpp:: //api. semanticscholar. httpr//ccc.ootr.....r/oouusurooo
15

---

Published as a conference paper at ICLR 2025
A CODE AND EXPERIMENTS DATA
We release our code at /ttps::/tggtee /ttp:b//gghuu.......//.e
provence In this page, we also make available zip archive with experimental results for the main Figure 2 e.g. it includes model generations and scores computed with various evaluation metrics (LLMEval, Match, Recall etc).
B DATA
Evaluation datasets.
We consider the following datasets:
Datasets with Wikipedia as datastore:
Natural Questions (Kwiatkowski et al. 2019). We use a test set of 2.8k ques-
tions, distributed part of the KILT collection (fttps://hugginggfeat co/
datasets http///aaeeooooooooooeee
tasks);
HotpotQA (Yang al. of the KILT collection (https: http://ggggggaaaeocoo////t/tataa
2018). We use test set of 5.6k questions,
part
kilt_tasks):
PopQA (Mallen et al. dataset authors.
2023b). We use a test set of 14k questions distributed by the
Datasets with individual datastores:
BioASQ (Nentidis et al. 2023). We use version of the dataset provided by (Hsia et al. 2024), with 3.8k queries. We only use queries from categories "yes/no", "fac- toid". and "list".
-Syllabus QA (Fermnndez et al. by the authors;
2024 We use the test set of 1.1k questions distributed
RGB (Chen et al. 2024b). We use the test set of 200 questions distributed by the authors.
All datasets provide short answers (keywords) for each query, which we use to evaluate both match- based metrics such as Recall and LLM--bssed metries Rau et al. 2024a
the MS MARCO document collection (Craswell et al. 2021). We split each document into overlapping chunks of N sentences, where N is random E 1..10 with higher probability for longer contexts train Provence on various context lengths. Each chunk is prepended with page title. The resulting datastore contains 34.M passages. We also process the Wikipedia datastore similar fashion, for ablation experiments. We download 2024 Wikipedia dump and process it using scripts provided by Pyserini (Lin et al. 2021 [T We also prepare versions of this Wikipedia datastore with passages of N sentences with overlaps N/2 sentences, for testing Provence robustness to various context lengths. All other evaluations on Wikipedia Wikipedia-based datasets including main evaluations are conducted on the Wikipedia datastore provided at pattps::: : h/thghgggggaaaeeeeeeeeee/ co/ dataaattssssssse wwwasstss/astoorteee castorini/ bdqa-wikk coroiii-ooorraaa We use a version with passages of 6 sentences with a 3-sentence overlap making 9M passages in total.
Datastores.
For training Provence,
For Pubmed, we use the version of the dataset provided by (Hsia et al. 2024 at https:
//buggingface.. co/datasets/ wttp///aaaaetteeessssaa///a/aaaa
It consists of 58M passages, extracted from Pubmed abstracts. Each passage (chunk) is prepended with the page's title.
For SyllahusQA, split each syllabus (provided by the authors) into passages of 100 words. For RGB, Google-retrieved contexts are provided by the authors; for each query we compose a retrieved context from relevant and irrelevant documents.
Using 10.77 Kim 2023
At
thub http://hhh..ooccctttto/////aaaaa
experi mwnt
16

---

Published as a conference paper at ICLR 2025
C MODELS
We list in Table 5 all the main models used to conduct experiments for P Provence.
Model
Checkpoint
SPLADE-v3
naver/splade-v3
RetroMAB
DeBERTa-v3
micrcsaett/debberra-llaggg
DeBERTa-v3 (RR)
BGE-M3
L-Lmma2-7B-chhat
LLaMA-3-88 In-stucttttt
Eestral--B-instruee
SOLAR-10.75 10.7B-Instruct-vl.
Table 5: List of all the models used in the experiments with their corresponding HuggingFace check- points.
0.6
Dalaset
Dotaset
0.5
MS Marco
0.20
Wiki
Frequency
0.15
0.3
0.10
0.1
Beginning
Number selected sentences
Position of selected sentence in context
Figure 5: Statistics of the silver contexts labeled by LLaMA-3-8D-Instrus (Left) the distribution of the number of sentences in silver contexts. (Right) the distribution of the position of the selected sentences in contexts.
many franch words there english language lucnce Franch English The notable
Inflience of French on English has bean its extensive consribution to the English loc icon It has beon estimated that
about third of the words English are French origin lingu ist Henri ette Walter claims that this total may
be high two thirds ingu Anthony Lac oud hes estimated thai over 40 000 English words come
6 sugh directly from French and may be underssooel withouu orth egrsphical chenge by Frenchi speakers Albert
and Thomas Cable nota that although this influx Franch words was brought about by the victory of the Conquer or and by the political and social consequences of that victory It was neither sudden nor Immediataly epparent Rather begen slowly and continued with verying tempo long time
COLORS: 0.99
Figure 6: Example visualization of per-token probabilities of being selected in the final context.
17

---

Published as a conference paper at ICLR 2025
Table 6: Prompt used for generating silver labeling with LLaMA-3-8B-Inse The sentence cita- tions in the response are parsed using regular expression.
Question: {question
Context: [1] {sentencel] } [2] {sentence2] [3] (sentence3}
Answer the Question, using ONLY information provided in the Context. If useful information is provided, you MUST output "No answer". If some parts of the Context are used to answer, you MUST cite ALL the corresponding sentences. Use the symbols from sentence in the context, e.g [0] for fact from sentence O. You should only answer the given question and should not provide any additional information.
indicate when fact comes
Natural Questions (wiki)
Hotpot QA (wiki)
TyDi QA (wiki)
Pop (wiki)
100
100
100
100
75
75
75
75
comp.
25
0
Syllabus (syllabi)
BioASQ (Pubmed)
RGB (news)
100
100
100
Full context
Provence reranker)
75
75
75
(standalone)
LLMLingua2
comp.
LongLLMLingua
RECOMP (ext)
RECOMP (ext+thr)
RECOMP (abs)
Recall 100
Recall
Recall
Figure 7: Main results for various QA domains, comparing Provence and baseline models, metric: Recall. Generator: LLama-2-77 retriever: SPLADE-v3, reranker: DeBERTa-v3 (or Provence in the unified setting). Plot titles denote "Dataset name (datastore type)". r-axis denotes QA per- formance evaluated with Recall; y-axis denotes the context compression ratio. For both metrics, the higher the better: the best model would be closest the top right corner.
Table nDCG@ 10 on the 13 open BEIR datasets.
Corpus
DeBERTav3
Provence
TREC-COVID
88.3
NFCorpus
37.5
37.8
NQ
66.7
66.5
HotpotQA
74.5
74.9
FIQA-2018
47,8
47.6
ArguAna
29.8
33.2
Touché-2020
33.5
33.4
Quora
84.8
85.4
DBPedia
48,9
49.2
SCIDOCS
19.2
19.6
FEVER
86.6
87.9
Climate-FEVER
27.4
28.1
SciFact
75.8
75.3
average
55.4
55.9
18

---

Published as a conference paper at ICLR 2025
Natural Questions (wiki)
Hotpot QA (wiki)
TyDi QA (wiki)
Pop QA (wiki)
80
compression rate
75
60
60
60
50
40
40
20
20
60
80
50
55
60
80
60
Syllabus QA (syllabl)
BioASQ (Pubmed)
RGB (news)
80
compression rate %
60
60
40
40
Splade deberta + Ulma-3-8b
Splade deberta Mittal-Bb
20
20
BM25 rernn 1-7ma-2-70
0
Retromae bga-m3 LLama-2-76
50
60
75
80 LLMeval
85
85
90
95
LLMeval
L.Meval
Figure 8: Testing Provence in various RAG settings (retrieval, re-ranking, generator).
Natural Questions (wiki)
Hotpot QA (wiki)
TyDi QA (wiki)
Pop QA (wiki)
compression rate %6
100
100
100
100
75
75
75
75
50
50
50
25
25
25
25
60
70
45
50
55
65
70
50
Syllabus QA (syllabi)
BioASQ (Pubmed)
(news)
100
100
100
context
compression rate
I
LLMLingua2
RECOMP (ext)
50
50
50
RECOMP (abs)
0
0
0
35
LLM-Eval
LLM-Eval
LLM-Eval
Figure 9: Comparing Provence subset of baselines with retriever: RetroMAE (Shitao al.
2022
reranker: BGE-M3 (Chen et al. 2024a), generator:: L.Lama---7BB 2-aa--2-B-chht.
19

---

Published as a conference paper at ICLR 2025
Natural Questions (wiki)
Hotpot (wiki)
TyDi QA (wiki)
80
compression
20
20
0
0.
75
Pop QA (wiki)
Syllabus QA (syllabi)
BioASQ (Pubmed)
40
40
comp
20
20
top
top-20
52.5
55.0
57.5 LLMeval
60.0
62.5
77.5
82.5
85.0
LLMeval
LLMeval
Figure 10: Testing Provence with different top-k documents provided to the generator. The setting the the one in Figure
Table 8: Numerical scores corresponding to Figure 2
NQ, Hotpot QA, Tydi QA, and Pop QA.
NQ
HotPot QA
Tydi QA
PopQA
LLM-
Comp. rate %
LLM- Eval
Comp. rate %
LLM- Eval
Comp. rate %
LLM- Eval
Comp.
Eval
rate %
Full context
71.8
0.0
57.0
0.0
73.9
0.0
57.8
0.0
Provence
72.4
62.2
56.7
66.4
70.5
63.0
59.3
68.6
(w/ reranker)
72.6
76.0
56.0
82.4
73.6
76.2
59.5
75.8
Provence
72.3
64.1
56.6
69.5
70.9
65.8
59.0
69.9
(standalone)
70.6
77.3
54.8
84.1
70.2
78.1
58.8
76.1
LLMLingua2
59.5
74.0
47.1
74.4
57.7
73.9
42.9
75.0
67.5
45.4
52.9
45.8
67.3
45.0
52.5
46.3
70.3
25.0
55.0
24.9
70.0
24.8
55.2
25.1
LongL
61.3
69.1
52.6
68.5
56.6
69.5
49.5
65.5
68.5
47.9
55.6
46.5
65.5
47.8
54.5
43.6
71.3
28.7
56.9
26.8
69.1
28.8
57.6
23.9
RECOMP
70.6
43.6
55.5
40.9
68.6
46.4
56.9
41.1
(cxt)
68.2
59.8
53.4
57.5
67.0
63.0
55.7
57.4
66.2
77.1
50.1
75.7
64.5
79.7
52.3
74,9
RECOMP
69.0
59.5
50.9
62.9
70.9
52.4
54.8
56.0
(ext+thr)
72.9
11.8
56.4
14.4
72.3
6.9
58.5
12.4
RECOMP (abs)
66.9
94.5
53.1
94.4
66.4
95.2
54.4
92.8
DSLR
71.7
44.9
52.9
72.7
58.6
70.5
54.9
50.7
83.4
69.8
55.6
58.7
58.1
70.4
61.4
49.3
87.0
70.7
62.0
58.8
63.7
67.7
72.0
45.2
91.7
67.5
72.9
58.5
71.9
77.7
43,2
67.5
78.1
76.0
Dense-X
62.7
69.0
49.6
67.7
66.4
71.5
52.0
68.5
trieval
20

---

Published as a conference paper at ICLR 2025
Table 9: Numerical scores corresponding to Figure
Syllabus QA, BioASQ, and RGB.
Syllabus QA
BioASQ
RGB
LLM- Eval
Comp. rate %
LLM-
Comp. rate %
LLM- Eval
Comp.
Eval
rate %
Full context
52.9
0.0
80.7
0.0
93.5
0.0
Provence
49.8
60.6
80.6
49.0
94.4
60.5
(w/ reranker)
51.0
76.5
80.3
67.4
96.3
69.3
Provence
50.7
64.1
80.6
51.3
95.8
61.6
(standalone)
47.8
76.6
80.1
68.9
96.3
69.4
LLMLingua2
37 .
73.4
72.6
73.6
78.6
74.3
43.4
45.4
77.7
45.2
93.5
46.1
26.3
LLngga
42.3
71.3
72.2
72.9
71.6
73.9
45.1
48.5
50.9
29.2
78.7
31.3
92.1
32.1
RECOMP
42.0
(ext)
42.7
61.4
78.4
54.8
94.9
52.1
39.1
71.1
76.3
69.7
94.4
63.2
RECOMP
45.5
35.7
76.6
51.2
92.1
51.4
(ext+thr)
52.8
7.7
80.2
6.5
97.7
13.4
RECOMP (abs)
38.1
98.9
68.2
96.1
90.7
95.7
DSLR
49.6
33.2
80.1
29.9
97.2
41.6
49.1
46.4
79.6
40.1
97.7
46.9
47.2
55.4
79.2
47.3
96.3
49.7
44.2
70.6
60.2
54.1
40.7
78.2
75.4
68.0
95.8
56.9
NQ (wiki)
Hotpot QA (wiki)
Syllabus QA (syllabi)
80
80
oobert--large
60
60
60
MS-Marca (downsample)
MS-Marco (full)
comp.
40
40
straight.
relevant oracle
20
sent-level labeling
0
0
0
full context
70
72
74
50
52
54
42
44 Recall
46
Recall
Recall
Figure 11: Ablation results with Recall imatch-based based metric).
21

---

Published as a conference paper at ICLR 2025
Table 10: Numerical scores corresponding to Figure
NQ
HotPot QA
Syllabus QA
Thresh.
LLM- Eval
Comp. rate
LLM- Eval
Comp. rate
LLM- Eval
Comp.
rate
Anchor
0.01
72.2
46.9
56.0
46.7
51.1
33.1
0.1
71.6
66.6
55.6
70.8
51.1
65.5
0.5
70.6
79.6
52.9
85.8
44.0
78.7
Deberta-large
0.01
72.5
59.4
56.7
63.7
49.9
50.7
0.1
72.1
69.2
55.9
75.6
49.6
68.3
0.5
70.9
78.0
53.2
84.7
46.1
77.0
MS-Marco
0,01
72.9
27.1
57.2
25.9
52.5
11.6
(downsample)
0.1
72.8
57.5
56.2
61.6
50.5
49.3
0.5
68.7
79.4
52.3
84.5
43.0
78.4
MS-Marco
0.1
72.3
64.1
56.6
69.5
50.7
64.1
(full)
0.5
70.6
77.3
54.8
84.1
47.8
76.6
straight. oracle
0.01
73.1
31.1
56.8
32.8
$2.9
25.6
0.1
72.5
51.4
55.9
56.8
51.7
54.7
0.5
72.0
70.2
54.8
76.8
45.5
75.3
relevant oracle
0.01
72.8
16.1
57.4
22.6
49.4
11.1
0.1
72.6
38.8
55.7
55.1
49.6
41.4
0.5
71.3
66.9
51.6
83.9
46.8
73.3
sent-level labeling
0.01
73.0
51.2
56,2
54.5
51.5
47.2
0.1
71.0
66.4
55,0
72.5
48.3
69.2
0.5
71.2
78.2
53.0
85.3
40.4
78.8
full context
0.01
71.8
0.0
57.0
0.0
52.9
0.0
22

---

Published as a conference paper at ICLR 2025
Table 11: Example of context pruning with various approaches. Provence selects one sentence about the Shepard's s pie and removes sentences about other similar dishes, which is RECOMP (ext) is not capable of by design. RECOMP (abs) correctly generates summary; LongLLMLingua removes the part relevant to the Shepard's pie, and LLMLLnguu2 uniformly removes no-informative tokens.
Question
what goes on the bottom shepherd's 's pic
Original context
Shepherd's pie. History. In early cookery books, the dish was a means using leftover roasted meat of any kind, and the pic dish was lined on the sides and bottom with mashed potato, as well as having a mashed potato crust on top. Variations and similar dishes. Other potato-topped pies include: The modern "Cumberland pie" is version with either beef or lamb and layer of bread- crumbs and cheese on top. medieval times, and modern-day Cumbria, the pastry crust had filling of meat with fruits and spices.. In Quebee, a varia- tion on the cottage pie is called "Pâté chinois". It is made with ground beef the bottom layer, canned corn in the middle, and mashed potato on top.. The "shepherdess pie" a vegetarian version made without meat, or a vegan version made without meat and dairy.. In the Netherlands, a very similar dish called "philosopher's stew" () often adds ingredients like bcans, apples, prunes, apple saucc.. In Brazil, dish called in refers the fact that manioc puree hides layer of sun-dried meat.
Provence
Shepherd's pie. In early cookery books, the dish was means of using leftover roasted meat of any kind, and the pie dish was lined on the sides and bottom with mashed potato, as well as having a mashed potato crust on top.
RECOMP top-3)
(ext,
In early cookery books, the dish was a means of using leftover roasted meat of any kind, and the pie dish was lined on the sides and bottom with mashed potato, as well as having mashed potato crust on top. The "shepherdess pic" is vegetarian version made without meat, or vegan version made without meat and dairy.. In the Netherlands, very similar dish called "philosopher's stew" O often adds ingredients like beans, apples, prunes, or apple sauce..
RECOMP (abs)
Shepherd's pic is dish made with leftover roasted meat and mashed potato on the bottom, with variations including a version with beef or lamb and a layer of breadcrumbs and cheese on top. The dish has history of being used as a means of using leftover roasted meat and has variations on the toppings.
LLMLingua2
Shepherd's pie History early cookery books dish leftover roasted meat lined mashed potato mashed potato crust top Variations similar dishes potato-topped pies include modern "Cumberland pie" beef or lamb breadcrumbs cheese me- dieval modern-day Cumbria pastry crust filling meat fruits spices Quebec vari- ation cottage pie "Pâté ground beef bottom canned corn middle mashed potato top "shepherdess pie" vegetarian without meat vegan version without meat dairy Netherlands similar dish "philosopher's stew" adds ingredients beans apples prunes apple sauce Brazil dish manioc puree hides sun-dried meat
(comp.50%)
LongLLMMingua (comp. 50%) ) Processes all pas- sages together and dynamically decides the compression ratio of each passage.
Shepherd's 's pie. Other potato-topped pies include: The modern "Cumberland pie" a version with either beef or lamb and layer of breadcrumbs and cheese on top. In medieval times, and modern-day Cumbria, the pastry crust had a filling of meat with fruits and spices.. In Quebee, a variation on the cottage pic is called "Pâté chinois". It is made with ground beef on the bottom layer, canned corn in the middle, and mashed potato on top.. The "shepherdess pie" is vegetarian version made without meat, or vegan version made without meat and dairy.. In the Netherlands, very similar dish called "philosopher's stew" O often adds ingredients like beans, apples, prunes, or apple sauce.. In Brazil, dish called in refers to the fact that a manioc puree hides layer of sun-dried meat.
23

---

Published as a conference paper at ICLR 2025
Table 12: Example context pruning with various approaches. Provence correctly detects that the entire passage is relevant to the query, same LongLLMLingua, while RECOMP (ext) is by design not capable making such a decision.
Question
where does the sweetness of fruit come from
Original context
Sweetness. A number of plant species produce glycosides that are sweet at concentrations much lower than sugar. The most well-known example is gly- eyrrhizin, the sweet component of licorice root, which about 30 times sweeter than sucrose. Another commercially important example is stevioside, from the South American shrub "Stevia rebaudiana". It is roughly 250 times sweeter than sucrose. Another class of potent natural sweeteners are the sweet proteins such thaumatin, found in the West African katemfe fruit. Hen egg lysozyme. an antibiotic protein found chicken eggs, is also sweet.
Provence
Sweetness. A number of plant species produce glycosides that sweet concentrations much lower than sugar. The most well-known example is gly- cyrrhizin the sweet component of licorice root, which is about 30 times sweeter than sucrose. Another commercially important example is stevioside, from the South American shrub "Stevia rebaudiana". It is roughly 250 times sweeter than sucrose. Another class of potent natural sweeteners such as thaumatin, found in West African katemfe fruit. Hen egg lysozyme, an antibiotic protein found in chicken eggs, is also sweet.
the sweet proteins
RECOMP
(ext,
roughly 250 times sweeter than sucrose, Another commercially important example is stevioside, from the South American shrub "Stevia rebaudiana". number of plant species produce glycosides that are sweet concentrations much lower than sugar.
top-3)
RECOMP (abs)
[emply context]
LLMLingua2
Sweetness plant species produce glycosides sweet lower sugar glycyrrhizin sweet licorice root 30 times sweeter sucrose stevioside South American shrub "Stevia 250 times sweeter sucrose sweeteners sweet proteins thaumatin West African katemfe fruit Hen egg lysozyme antibiotic protein chicken eggs sweet
(c0
LongLLMLingua
Sweetness. number plant species produce glycosides that are sweet concentrations much lower than sugar. The most well-known example is gly- cyrrhizin, the sweet component of licorice root, which is about 30 times sweeter than sucrose. Another commercially important example is stevioside, from the South American shrub "Stevia rebaudiana". It is roughly 250 times sweeter than sucrose. Another class of potent natural sweeteners are sweet proteins such as thaumatin, found in the West African kalemfe fruit. Hen egg lysozyme, an antibiotic protein found in chicken eggs, is also sweet.
(comp. 50%)
24

---

Published as a conference paper at ICLR 2025
Table 13: Example of context pruning with various approaches. Provence selects one most relevant sentence, which is also ranked first by RECOMP (ext). RECOMP (abs) decides that no information relevant to the query, while LongLLMMingua on the contrary keeps the entire input, dropping some punctuation marks. LLMLingua2 removes too many tokens which makes text hardly under- standable.
Question
what was the tower of london originally used for
Original context
Tower of London, In the 16th century, the Tower acquired an enduring repu- tation as grim, forbidding prison. This had not always been the case. As a royal castle, was used by the monarch imprison people for various reasons, however these were usually high-status individuals for short periods rather than common citizenry as there were plenty prisons elsewhere for such people. Contrary to the popular image of the Tower, prisoners were able to make their life easier by purchasing amenities such better food or tapestries through inciden- would have been the case for any castle there
the Lieutenant of the Tower. As holding prisoners was originally role of the Tower was no purpose-built accommodation for prisoners until 1687 when a brick shed, a "Prison for Soldiers", was built to the north-west of the White Tower. The Tower's reputation for torture and imprisonment derives largely from 16th- century religious propagandists and 19th-century romanticists.
Provence
Tower of London. As royal castle, it used by the monarch to imprison people for various reasons, however these were usually high-status individuals for short periods rather than common citizenry as there were plenty of prisons elsewhere for such people.
RECOMP (ext,
As a royal castle, it was used by the monarch to imprison people for various reasons, however these were usually high-status individuals for short periods rather than common citizenry as there were plenty of prisons elsewhere for such people. This had not always been the case. The Tower's reputation for torture and imprisonment derives largely from 16th-century religious propagan- dists and 19th-century romanticists.
sorted top-3 sents)
RECOMP (abs)
Jempty context]
LLMLingua2
Tower London 16th century grim prison royal castle monarch high status com- mon citizenry prisoners amenities food Lieutenant Tower no- built accommoda- tion until 1687 "Prison for north-west White Tower reputation torture imprison- ment 16th-century propagandists 19th-century romanticists
(comp.25%)
LongLLMMingua (comp. 25%)
Tower of London In the 6th century, the acquired an enduring reputation as grim. forbidd prison. This had always been the case As royal castle, it by the to imprison people for various reasons however these were usually high- status individuals for short rather than common citizenry as there were plenty of prisons clsewhere for such people. Contrary popular of the Tower, prison- ers were able to make their life easier purchasing amenities such better food tapestries through Lieutenant of the Tower. holding prisoners was originally incident role of the- would have been the case for any was purpose- accom- modation for prisoners until 167 a "Prison for Sold", was to thewest of White Tower. The's reputation torture imprisonment derives largely from 6th- reli- gious propagandists and ttthure romanticists.
25


Reviews:
[{"review_id": "u8PidSOLJ2", "reviewer": "Reviewer_5r2e", "summary": "This paper presents an encoder-based context pruning approach to enhance retrieval-augmented generation.\nThe proposed framework aims to identify and eliminate irrelevant portions of retrieved context to improve the accuracy of generated responses. \nUnlike previous methods that typically rely on LLM-like decoder models, the proposed approach leverages an encoder model to frame the problem as a sequence labeling task, similar to POS tagging and NER.\nThe authors generate training data (silver-standard datasets) using another LLM, such as LLaMA-3.2-8B, and evaluate the performance of context pruning techniques with LLMs serving as evaluators.\nThe experimental results demonstrate that the proposed method is both effective and efficient. \nA comprehensive set of analyses further validates the impact of the approach.", "questions": "- Given the importance of both performance and compression rate, exploring a dynamic trade-off between the two could be beneficial. For instance, while one user may prioritize performance over compression, another might value compression more highly. A straightforward way to enable this dynamic trade-off could involve adjusting the threshold that determines the relevance of tokens. By fine-tuning this threshold, the model could flexibly adapt to user-specific preferences, optimizing either performance or compression as needed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The draft is well-structured, making it easy for readers to grasp the core concepts. I genuinely enjoyed reading it.\n- Unlike previous approaches that employ LLMs, the proposed method is specifically optimized for the target task, allowing it to achieve both effectiveness and efficiency.\n- A series of diverse experiments highlight the merits of the proposed approach from various angles, instilling confidence in readers about the method’s reliability.", "weaknesses": "- Training and evaluation procedures rely heavily on the use of LLMs. This is a bit concerning for me, for the following reasons:\n    - Firstly, if we already have a well-performing LLM capable of effective context pruning, why should we develop a separate model for this task? The strong performance of the existing LLM could suggest that it inherently possesses the ability to filter and focus on crucial information from the context. Rather than adding an external module, we might leverage advanced prompting techniques to enhance this functionality.\n    - Secondly, the process of training and evaluation lacks human oversight. Can we be confident that a model trained and evaluated in this way will align well with human preferences? Although LLMs are initially trained on human-generated data, I wonder if there’s a risk of a “hacking” effect, where the model is tuned to satisfy only the evaluation metrics instead of genuinely aiming to achieve the intended goal.\n- Figure 2’s main results are intuitive; however, this presentation style may open the door to different, subjective interpretations among readers. For instance, if a reader prioritizes compression rate, they may see RECOMP (abs) as the most appealing option. Thus, it may be helpful to introduce a unified metric that incorporates both compression ratio and LLM evaluation performance, potentially using a measure akin to AUROC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rxm8FoGlmk", "reviewer": "Reviewer_7vVg", "summary": "The work presents a context pruning technique, where during RAG, each sentence in the context is individually labeled as relevant or irrelevant to the query. The training is conducted by distilling from Llama-3-8B, where the model is instructed to output an extractive summary on the sentence-level of the context given some query. Note that though the labeling is on the sentence level, each token in the sentence will inherit the sentence's label during training. During inference, only sentences where +50% tokens are kept are considered kept/positive. In another variant, the pruning model is used to both output a binary judge mask and serve as a reranker by outputting a relevance score, though for this case the model needs to be initialized from pretrained reranker. The proposed model achieves competitive results on diverse QA tasks, most on par with strong baselines like RECOMP but with more compression ratios.", "questions": "Suggestions:\n\n• 181: using \"silver labels\" without defining the term first. From the context it seems to mean \"auto-generated labeled data\".", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "• Evaluated on diverse QA tasks, making the results/improvements less likely to be cherry-picked", "weaknesses": "• It is questionable for a non-instruction-tuned model like DeBERTa to understand slightly more complex relation definitions. For example, if the query asks DeBERTa to find sentences that support some point vs. refute the same point, chances are that DeBERTa is going to keep the same sentences.\n\n• Results figure (i.e., Figure 2) is not clear. Not clear what x- & y-axis represent. The caption should clearly state that, e.g., y-axis is compression ratio, etc...\n\n• Overall results figures are all hard to read, esp. Figure 4 where all symbols are superimposed on top of each other. Consider enlarging them or changing them to more informative tables.\n\n• I don't quite get the model choices here. The silver labelings are obtained with Llama-3 models, where the generator is chosen to be Llama-2? Is there any justification for the versioning discrepancy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NV4zy4Fj4B", "reviewer": "Reviewer_99vA", "summary": "The paper proposed a context pruning method for retrieval augmented generation tasks by considering the context pruning task as passage reranking. The authors used a Deberta reranking model which is significantly smaller than large generative models to improve both efficiency and performance of retrieval augmented language generation.\n\nExperiments show that when the compress rate is higher than 50%, the proposed method outperforms other strong baselines. The authors also provided comprehensive analysis and ablation studies.", "questions": "what is the performance of the proposed method when the compress rate is less than 50%?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method shows a good adaptation ability and efficiency on different tasks. When the compression rate is higher than 50%, the method still achieves a reasonable performance.", "weaknesses": "This work misses some relevant citations and does not compare to these methods, for example:\n\n[1] DSLR: Document Refinement with Sentence-Level Re-ranking and Reconstruction to Enhance Retrieval-Augmented Generation.\n\n[2] Dense X Retrieval: What Retrieval Granularity Should We Use?\n\n[3] Phrase Retrieval for Open-Domain Conversational Question Answering with Conversational Dependency Modeling via Contrastive Learning\n\n[4] Decontextualization: Making Sentences Stand-Alone\n\nI'll adjust my score if the authors settles my concerns about missing these comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fk6vjTwMDZ", "reviewer": "Reviewer_sJXd", "summary": "This paper introduces Provence, a context pruner for question answering that formulates context pruning as a sequence labeling task. Specifically, the authors fine-tune a pre-trained re-ranker (DeBERTa) using silver labels generated by Llama-3-8B. The claimed benefits of this approach are that the proposed approach operates without assuming a fixed compression ratio and can be applied out-of-the-box to various datasets. This approach improves the efficiency of RAG for QA without performance degradation.", "questions": "- Why is it useful to unify compression and reranking? In which parts of a RAG pipeline can rerank scores be effectively used? Does Provence outperform existing approaches in reranking?\n\n\n-----\nRaised my score 6-> 8 as author responses cleared most of my doubts.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Provence is a novel approach that enables context compression without performance drop, outperforming baseline models.\n- It offers practical advantages, including query-dependent, sentence-level, dynamic compression, and adaptability across multiple domains.\n- The authors conduct extensive analysis to understand the properties of Provence and explore various design options to enhance its effectiveness.", "weaknesses": "- Some hypotheses are presented without supporting results. For example: (1) Provence does not select any irrelevant sentences, and (2) leftmost and rightmost sentences are usually not deemed relevant and no explanation is provided for why catching “needle” sentences in these positions might be an exception.\n- There is insufficient information to indicate that the experiments were designed for a fair comparison. The authors do not consider an adequate retrieval pipeline for comparison with existing approaches (as Splade + DeBERTa may introduce bias favoring Provence), and the impact of the hyperparameter k in retrieving the top-k relevant passages is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)