SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction
Authors: Lu Dai
Yijie Xu
Jinhui Ye
Hao Liu
Hui Xiong
Venue: ICLR 2025 Spotlight (2025)
Keywords: information retrieval
metric
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large Language Models (LLMs) have demonstrated improved generation performance by incorporating externally retrieved knowledge, a process known as retrieval-augmented generation (RAG). Despite the potential of this approach, existing studies evaluate RAG effectiveness by 1) assessing retrieval and generation components jointly, which obscures retrieval's distinct contribution, or 2) examining retrievers using traditional metrics such as NDCG, which creates a gap in understanding retrieval's true utility in the overall generation process. To address the above limitations, in this work, we introduce an automatic evaluation method that measures retrieval quality through the lens of information gain within the RAG framework. Specifically, we propose Semantic Perplexity (SePer), a metric that captures the LLM's internal belief about the correctness of the retrieved information. We quantify the utility of retrieval by the extent to which it reduces semantic perplexity post-retrieval. Extensive experiments demonstrate that SePer not only aligns closely with human preferences but also offers a more precise and efficient evaluation of retrieval utility across diverse RAG scenarios.

Published as a conference paper at ICLR 2025
SePer: MEASURE RETRIEVAL UTILITY THROUGH THE LENS OF SEMANTIC PERPLEXITY REDUCTION
Lu Dai Yijie Xu', Jinhui Ye! Hao Liu 122 Hui Xiong!
'The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
The Hong Kong University of Science and Technology, Hong Kong SAR, China
CCanegie Mellon University, Pittsburgh, USA
1daiae@ connect. ust hk.
wwwsthutuuua... edu, on
jinhui Sonnuiiaandree cmu edu
Liuheust.... hk
.k
ABSTRACT
Large Language Models (LLMs) have demonstrated improved generation per- formance by incorporating externaly retrieved knowledge, process known -aumeetl- generation (RAG). Despite the potential of this approach, ex- isting studies evaluate RAG effectiveness by assessing retrieval and generation components jointly, which obscures retrieval's distinct contribution, or 2) exam- ining retrievers using traditional metrics such NDCG, which creates a gap in understanding retrieval's 's true utility in the overall generation process. To address the above limitations, in this work, we introduce an automatic evaluation method that measures retrieval utility through the lens of information gain within the RAG framework. Specifically, we propose Semantic Perplexity (SePer), a metric that captures the LLM's internal belief about the correctness of the generated answer. We then quantify the utility of retrieval by the extent to which it reduces semantic perplexity post retrieval. Extensive experiments demonstrate that SePer not only aligns closely with human preferences but also offers a more precise and efficient evaluation of retrieval utility across diverse RAG scenarios.
INTRODUCTION
Retrieval plays a crucial role in satisfying information needs across various interactive systems. With the rapid advancement of Large Language Models (LLMs) and various downstream applica- tions Xu et al.] (2025); Ye et al. (2025): Guo et al. (2025). retrieval has become deeply interwoven with generation processes Lewis et al. (2000) Guu et al. (2020). This integration not only enhances the accuracy and faithfulness of generated content Chen et al. (2017) but also enables handling more complex applications such as multi-hop reasoning Trivedi et al. (2022a), visual information seeking Hu et al. (2024), and task completion Yao et al. 2023) Zhang et al. (2023).
To evaluate and enhance these retrieval-augmennce systems Salemi & Zamani (2024), key chal- lenge lies in measuring the contribution retrieved information to the overall performance, i.e., the utility of retrieval. For instance, reasoning process may require different pieces of information at different steps to infer the final answer Yang et al. (018)); Talmor & Berant (2018); Gu et al. (2024) shown in Figure I However, most evaluation methods fail respond middle-step information, which may not directly match the ground truth text span. Besides, while a RAG workflow or agentic task might trigger retrieval multiple times within a single interaction cycle Asai et al. (2023); Jiang eet al. 2202)). it's difficult to quantify which retrieval effort brings in the most rewards. The lack of evaluator's sensitivity partial information also results in discontinuous scoring of retrieved infor- mation Schaeffer et al. (2023), hampering the development of more efficient retrieval mechanisms.
Unlike the independent evaluation of retrievers, the utility of information retrieval (IR) hinges not only on the quality of the information but also on the prior knowledge of the recipient (c.g.. LLM or human), their capacity to integrate external inputs and the way it interacts with the retriever Yoran et al. (2024): (Shi et al. (2024). For example, widely-know fact would bring no knowledge gain to LLMs, although it is both relevant and correct. In the meantime, an irrelevant long document may undermine LLM's performance even though it does not contain false facts and is harmless in general. Therefore, traditional IR metrics that evaluate the retriever as an independent module
Corresponding authors.

---

Published as a conference paper at ICLR 2025
cannot reflect its actual helpfulness from a systemic perspective. Recent works also use well well-trained LLMs such as GPT-4 as generalizable judges to evaluate different aspects of RAG systems, such as relevancy, correctness, and faithfulness Es et al. (2024). However, these methods are often expensive and inefficient, especially for large-scale datasets and complex RAG workflows.
In this work, propose the perspective measure retrieval utility based the knowledge gain of LLMs [Belkin ( (1980); [elkin et al.] (1982a b). Ideally, an effective measure of information retrieval utility should reflect the satisfaction of the recipient Cooper 1973).
Historically, this approach has been hypo- thetical because the knowledge gain in real humans is intangible in computation. How- ever, when LLMs act as information recip- ients, it is possible to estimate the shift in the LLM's knowledge distribution and use it as an indicator of retrieval effectiveness.
pevelour
Retrieval Utility
ONe FRapOrd
AlAec)
0
Model Belief
Yer
Along this line, we define Semantic Perplex- ity (SePer), a sampling-bases basen method to es- timate LLM's belief conditioned on an input query. Specifically, we first sample multiple responses and cluster them based on their se- mantic meanings and re-aggregate their like- lihoods following the concept of semantic entropy Kuhn al. (2023). In this way, we can compute probabilities in the seman- tic meaning space to obtain more accurate estimation of the belief distribution, as opposed to vocabulary space. We then compute the cross- entropy between the estimated semantic distribution and ground truth distribution, the exponential form of which is defined as Semantic Perplexity (SePer). By doing so, we make it tangible to esti- mate the knowledge distribution shift of LLMs and use it to quantify retrieval utility. In summary, contributions are three-fold:
-QuI> Waacdbberr Ase oillogges
-Dee 4Ex Dasii
<Teo ats tilee Joko a appiinal Trieatos
RAG Process
Princeets
1903
An illustration of retrieval utility in the multi- step RAG process. Unlike previous methods that only eval- uate the final retrieval outcome, our approach can assess the utility of intermediate retrieval steps, even when the in- formation retrieved is incomplete.
Figure 1:
We introduce ASePer, an assessment method for evaluating retrieval utility based on shifts in LLMs' knowledge distributions. This approach not only aligns closer with human annotations but is also more consistent with inference time due to its sampling-
nature.
We conduct theoretical analysis and extensive experiments to demonstrate that ASePer pro- vides more accurate, fine-gran grained, and efficient evaluation of retrieval utility. It is also gen- eralizable across a broad range of RAG scenarios. Furthermore, we augment the evaluation of various RAG systems with our SePer metric for the reference of future research, which is maintained in /ntt::::////
com wwt.s/ppeeeeerteeeeeeeeeeeeeeeee riec/sperr/ee
By utilizing SePer, we quantify the retrieval needs across different scenarios. Our findings offer valuable insights for data selection and budget allocation in practical RAG systems.
2 RELATED WORKS
Evaluation of Information Retrieval. The effectiveness of retrieval is often evaluated from two aspects: Direct content evaluation, which scrutinizes the relevance of the retrieved content itself, and response-based evaluation, which gauges the quality of the responses to reflect the effec- tiveness of in-the-middle modules Salemi & Zamani (0244). However, each method suffers from several shortcomings individually. Direct context evaluation treats retrieval as an independent mod- ule, which cannot reflect the impact on the receiver model. Response-based methods can be further divided into two categories: model-based and reference- -baed methods. Model-based methods re- quire knowledgable models, such as human and GPT-4 Liu et al.] (2023), to evaluate whether the LLM output desired response to a given query. Reference-based methods require a reference answer and evaluate the LLM outputs by their matching score to the reference, such as BLEU Pa- pineni et al. (2002). ROUGE Lin (2004), and BERTSCORE Zhang et al. 2020). However, these methods mainly reflect lexical information and cannot capture the nuanced relationship in semantic meaning. More recent works also leverage LLM judges, such as ARES Saad-Falcon et al. (2024), PROMETHEUS Kim al. (2024), and AUTO-J Li et al. (2024) assess the distance between ref- erence and output. These 4LM--ased methods are often more accurate due to the language under- standing and L.MMs comprehensive world knowledge. However, they are often slow and expensive.
2

---

Published as a conference paper at ICLR 2025
Knowledge Estimation in LLMs. There is view regarding LLM knowledge base Petroni et al. (2019); Geva et al. (2021), and generation can be viewed as retrieval from internal parametric memory Jiang al. (2220). From this perspective. injecting external knowledge into the latent knowledge of [.LM via in-context learning is equivalent to fine-tuning Dai et al. (023). Xie et al.
(2022) also explains in-context learning in Bayesian inference framework and shows that prompts influence LLM output by shifting its latent concept distribution. While the internal knowledge latent distribution is unobservable, many studies have managed estimate the knowledge state of LLMs from other signals, such as probing model internal states Ribeiro et al. (2016); Adi et al. (201 7); Meng et al. (2022) and prompted responses Zhong et al.) (2021). Studies also find that uncertainty in LLM is highly correlated with knowledge correctness, i.e., hallucination Kuhn et al. (2023); Cheng et all. 2024), which provides analysis on how knowledgeable LLM by observing uncertainty.
QUANTIFY RETRIEVAL UTILITY
In this section, we justify our quantification of retrieval utility as computable belief distribution shift in the information receiver model. This approach is grounded in the Bayesian framework, where new evidence updates prior beliefs. However, unlike traditional Bayesian updating that aims to learn model parameters. our focus is on evaluating the utility of the retrieved information in terms of its impact on the model's previous belief. We begin by envisioning several properties that retrieval utility should possess. We then formally define retrieval utility the change of belief in ground truth answers and prove that it satisfies the desirable properties. Finally, we instantiate this formulation in the RAG and introduce the al algorithm details to compute SePer and retrieval utility.
3.1 NOTATIONS
Throughout this paper, we use the following notations. Let M denote a well-trained language model (the information receiver) capable of generating answers to queries q. The correct answer set the query is denoted by A - {a") }. The retrieved result is denoted by DD where D is a set of n atomic information d,, i.e., D=(d1, , ..., d We denote by (a(()) the likelihood model M assigns to answer a without retrieval, and by Par 'D) the likelihood after incorporating D.
3.2 RETRIEVAL UTILITY AS BELIEF REVISION
To provide an intuitive conceptualization of retrieval utility U(M, d), we incorporate insights from cognitive information retrieval theories. These theories emphasize the dynamic interplay between information, the user's knowledge state, and the context of information retrieval. Consequently, we envision that an effective retrieval utility metric UM, d) should satisfy the following properties: Property 1. The retrieval utility U(M d) depends both the retrieved information d and the information receiver M. According to Cooper (1971 1973 Dervin 1999); Ingwersen (199), the effectiveness of retrieval system is contingent upon both the user's cxisting knowledge and the relevance of the retrieved information. This perspective necessitates the introduction of dependence property, which considers both the information receiver and the retrieved content in evaluation.
Property 2 (Zero Utility). For given query q, the retrieval utility U(M, d) should be zero if the information retrieved is either irrelevant to q or if the model M already possesses the requisite knowledge to address q effectively without d.
Belkin et al. (1982a) posits that information is sought to resolve an anomaly in the user's s knowledge state. Thus, if the retrieved information does not address this anomaly or if the user's 's knowledge is already sufficient, the information holds no utility, thereby justifying the zero utility property.
Property 3 (Monotonicity). Given an information receiver M and an unsatisfied information need q. the retrieval utility U (M, d) is a monotonically increasing function of the relevance of the re- trieved information d to q.
According to Ingwersen ( (196), UMM, d) depends on information relevance and the user's 's cognitive space. With cognitive space fixed, increasing the relevance of retrieved information enhances utility, supporting the monotonicity property that U (M, d') increases with the relevance of to q.
Intuitively, the retrieval utility quantifies how much the retrieved information d shifts the model's belief toward the correct answer a". Accordingly, we define the retrieval utility as follows:
Definition 1 (Retrieval Utility). The retrieval utility is defined as the change in the model's belief about the correct answer a" due to the retrieved information d:
U(M d) Par(a" |d) Pas (a").
(1)
3

---

Published as a conference paper at ICLR 2025
We demonstrate that Definition satisfies the properties listed above.
The retrieval utility UM, d) depends explicitly on both d and M through the probabilities PM (a d) and (ar(
Proof of Property|
Proof of Property We discuss the two distinct scenarios in the property separately:
1) Irrelevance of q: When d irrelevant, fails contribute any new information relevant
to the correct answer a". Consequently. the conditional probability of a" given d equals the prior probability, Par(a" d) = Pas (a"). Thus, the utility UM, d) Par("
( =
2) Redundancy of d for M: If M already knows a". the probability Par(a' 1. Since probabilities cannot exceed 1, Par Here, since d adds value, Pa [a" d) Par (ba). and thus UMM, d) remains 0.
d) also cannot exceed 1, implying UMM, d) Par(a" d) - 0.
2) Redundancy of d for M: Let be the knowledge scope of model M. a" is known to means
(as Pas K) 1. For redundant knowledge d IK, (M U {d)) (as K). Thus, d) = Par(a (a" d) Par(a") O.
Proof of Property 3 Following Dai et al. 2024). let d + a" denotes that entails a", we define the relevance of the retrieved information the query q as:
1, if
Rel(d, q) 1, otherwise.
(2)
When Rel(d, q) = 0, according to Property 2 the retrieval utility U(M,d) = 0. When Rel(d, q) = 1, since d + a", assuming the receiver M can effectively utilize d, we have (Mr(a" | Pas(a"). Therefore, the retrieval utility is positive:
U(M,0 d) = PPM(a" [d) - Par(aa) O.
Thus, as Rel (1. q) increases from 0 to 1, UMM, d) increases from 0 to positive value, demonstrating the monotonicity property under this binary definition of relevance.
For more general cases where Rel(d, d., q) is ordinal or continuous - for example, in multi-step rea- soning where d partially contributes to a* and < Rel(d, q) < we empirically demonstrate that our belief change based utility metric exhibits significantly higher correlation with human- annotated context utility compared other methods Table
Knowledge
Knowledge bistribution
Semontic clastering
Estimated Distribution
Prier
Peur tangrain
Ladtiig
Pook
Retriovel
Pae Masc
Ground Truth Dietribution
U(M.D) Pa(A"(D) Pv6A")
Dr
Dep rind Frrncieds
Equivalenee
Pa(a"ID) E, kic,,"")) Ere; p(rlD)
① Monte-Carlo Sampling
2 Semontic Clustering
L Semontic Perplexity (SePer)
C Unility Score Colculation
Figure 2: SePer: Estimating retrieval utility in multi-step retrieval-augmented generation (RAG) processes by mcasuring changes in model belief. SePer consists of four key steps: C Probing the model's belief through Monte-Can Carlo Sampling, where the LM generates N responses to the query using temperature parameter. @ Estimating the belief distribution semantic perplexity by comparing the estimated belief distribution with the ground truth distrihution. @ As- sessing the unity of partial retrieval by measuring the change semantic perplexity hefore and after retrieval.
possible answers using semantic clustering. 3 Calculating the model's
3.3 BELIEF ESTIMATION THROUGH SAMPLING
Estimating model belief Par(c) is challenging due the vast output space of the language model. Moreover, the model's outputs are in the vocabulary space, whereas our belief probabilities are defined over the semantic space. For example, "Peter", "Peter Bergmann" and "Ludwig Planck" are equally correct answers to the question "Who did Einstein work with in 1933?" and should be considered the same event in the probabilistic space Kuhn et al. (003)..
4

---

Published as a conference paper at ICLR 2025
Extending semantic entropy Farquhar et al. (2024). calculate the model's likelihood on a" based on the distribution of semantic meanings.
Definition 2 (Semantic Equivalence). Two texts and y are semantically equivalent, denoted y if x and I- x, where means entailment.
Practically, the entailment relationship is computed from a function X X - [0, 1]. where X' is the of all possible texts. Given two texts and y, E(x. y) outputs score representing the degree to which & entails y. The entailment relation holds if E(x, y) exceeds a predefined threshold We also experimentally demonstrate that using the entailment model define semantic equivalence Yao & Barbosa (2024) aligns much more accurately with human-annotated semantic cquivalence, compared to traditional rexical-matching methods and trained LLM-judges, almost on par with GPT-4 evaluation. Results are shown in Table [
Given set of responses {r.). semantic clustering the process of grouping responses into clusters C = } such that all responses within a cluster are semantically equivalent:
C = {rr r, ry. Vry C+}.
(4)
The original semantic entropy makes the entropy computable in the sampled distribution:
p(r o)) ~ Σ -I logp(C, lg). Lec
[C]
(-(1)==- C
[E p(r a) FEC
log
(5)
Since the first equation is not computable due to infinite sentence space, the expectation is estimated using Monte-Cane integration over sampled semantic cluster C. By the Law of Large Numbers, the number of samples N + oo, the frequency of responses converges to the model's belief distribution. We use similar approximation in making SePer computable. But unlike semantic entropy, we estimate model belief shift the reference answer instead of the output uncertainty originally for hallucination detection. We will detail the computation of SePer in the following part.
3.4 ASePer: SEMANTIC PERPLEXITY REDUCTION RAG
To estimate model belief on reference answers P((a") }), instead of computing the entropy of mantic clusters, we further determine which clusters are semantically equivalent to any of the a in {a*}. For clarity, we begin with the special case where there is only one ax.
Thus, P(a") is calculated by:
P(a" ( x)== E(C,,""
=0)==<(c,0 a²) Ep(r (9),
(6)
PEC
where r = (11,122200 12,,2..... 4-11 and p(r 1) the output sequence likelihood from M: |rl
p(r 4) = II p(t: | f4.f2.....4411999)) i=1
(7)
k(C', aa) is kernel function to measure the distance between the meaning of semantic cluster C and a". We tested two different implementations of iCC a*) the entailment model score F(x,, y) and Indicator function a(C,a"):
1, if i = a" I(C,a')
{ otherwise.
(8)
Finally, the SePer score is calculated by:
SePerm(q,a") P
PM (g)
E a" a(C,,a")p(C, 1)).
(9)
C.Ec
Similar to 5 the right approximate equation is an unbiased estimator the left one. This naturally extends to the general cases where there are multiple ground-truth a" provided:
SePeru A)
Σ a'EA
((C,a")p(e (1)
(10)
A
LC,EC
5

---

Published as a conference paper at ICLR 2025
Algorithm SePer ASePer
Require: Model M, reference answer a", entailment model E, threshold 7, number of samples N.
1: Sample responses: T; ~ (T), for i I....,N.
2: Compute likelihoods: l, = (ra).
3: SePerH:
SePers:
Semantic clustering: Group responses T3 into clusters
Compute entailment scores:
C= [Ck] such that E(r,, ry) 2 within each cluster. Identify C.-: matching reference answer with semantic cluster, where Vr E Ca Ta a".
h; F(ri,
Compute soft belief:
PMr(a*) = l hr-
3. Compute semantic perplexity: PM((*)= Er, l.
4: Repeat steps with retrieved information to obtain PM(a" 1 D).
5: Compute utility: UMM, D) ASePer = PM(a" (D) - PMr("))
Lastly, retrieval utility UMM, D) is calculated by ASePer, i.e., semantic perplexity reduction:
UM, D) = ASePer PMr(a () Pa 1)).
(11)
Through Monte-Carlo sampling and semantic clustering, ASePer quantifies the extent to which receiver M's belief shifts towards ground-truth answer after retrieval, i.e., how much beneficial information gain the information pieces brought to the model. Based on two different kernel function choices, we implemented SePers and SePer separately. The incorporation of kernel-based soft matching provides a more nuanced and continuous evaluation Nikitin et al. (2024). The SePer and ASePer algorithms are fully described in Algorithm| [
EVALUATION OF SePer
In this section, we conducted experiments to prove the validity and reliability of the proposed SePer metric Xiao al. (003);; ; Jacobs & Wallach (202T); Wagner al. (2021). For validity testing, we first show experiments in Section 1.1.11............ to prove that SePer is more accurate and fine-grained indicator for reference-based response evaluation and then demonstrate its better correlation and alignments with human judgments about retrieval utility in Section 4.1.2] For reliability testing, we test the robustness of the performance of SePer on different aspects, including varying datasets, repeated computation, and the number of samples used. Results in Section Section[4.]]shon that under our default setting, SePer achieved high reliability and stability with less cost in time and money. We also add ablation results about more hyperparameters in A to prove the robustness of SePer.
4.1 VALIDITY OF SePer
4.1.1
VALIDITY OF DEMANTIC-BASEE ANSWER SCORING
First, evaluate the basic component of computing SePer, i.e., using the entailment model to calculate the semantic similarity between the generated answer and the ground-truth answer.
Datasets. We use EVOUNA A Wang et al. (2024), a Question Answering (QA) benchmark to evaluate QA evaluators' reliability, Based on NATURAL QUESTIONS (NQ) Kwiatkowski et al. (019) and TRIVIAQA Joshi et al. (017). EVOUNA augmented the datasets with LLM-generated responses and asked humans to annotate whether a response is semantically equivalent to the golden answer.
Baselines. We include two types of baselines: Matching- based evaluation, such as lexical match and BERTSCORF, and I.M judge evaluators, such as AUTO-J and PROMETHEUS.
He et al. (2021) model fine-tuned on the MNLI dataset to assess the entailment relationship between two text pairs following the setting of Kuhhn et al. (202)). which is far more efficient than API-based entailment judgment Yao & Barbboa (2024) without significant performance drop. our implementation, we further leverage the entailment score get a probabilistic estimation of the likelihood of semantic equivalence.
Model. We use the
Results. As shown in Table I the NLI-based evaluation in SePer demonstrates significantly higher alignment with human judgment compared to traditional matching matching-based response evaluation by surpassing the baselines by 2% 6% F1 Fl-sc across various generators and datasets. Notably, is close to or minorly surpasses the response evaluation performance of GPT-4 in this benchmark. BERTSCORE, while capturing semantic meaning, may fail to capture the relationships in QA tasks. At the same time, trained LLM judges did not demonstrate edge over traditional methods this
https://hugginn htt.:aaac..c.....oooooooaaaaa.a.
co/

---

Published as a conference paper at ICLR 2025
benchmark, with reference-free free judges falling far behind reference- basence--bsed methods. These results show that computing the rmnantic-qquivllnne score based on the entailment model is both an effi- cient and reliable method. Its high correlation score in matching responses and answers also set solid step for the next stage of computation of ASePer.
Natural Questions
TriviaQA
Exaluator
DPR-FiD F1/Ac2
InstmicGPP Fl/Ace
CharGPT F1/Ac
GPT4 F1/Acc
BingChot Fl/Acc
DPR-FiD
InstaccGiPT F1/Acc
ChaiGT
GPT-4
BingChat Fl/Ace
Fl/Ace
Fl/Ace
Fl/Acc
Matching-
Lexical Match
92.089.7
86,984488
8500003
87.682.5
87.8/2.3
91 8A44,
94.892.3
95.28223
94.89111
94,1/89.8
BERTSCORE
83.5/75.1
77.669.5
81,2/728
84.3/76.
77.566.5
75.1665.
84.1/75.7
88,4800.8
90.5/93.5
88.3/80.4
Entail (SePer)
96.6/95.3
92.0/90.1
91.2/87.8
93.1/89.7
91.4/8700
97.6/96.1
97.5/96.0
97.9/96.4
98.5/97.2
96.2/93.2
L.I.-as---Jum Judge
Auto-J
57.8/54.2
71.9/62.1
76.4/66.5
75.4/65.1
22.8/22.2
76.3/667
80.8/11.5
81,4/713
80.4/68.7
83.0733.0
PROMETHEUS
83.8/77.8
81,1/70,5
86.4/77.7
89.3/81.5
89.5/82.3
89.4783.1
90.0/83.2
93.0/87.7
94.7/90.2
95,4911.
Human-leve level
GPT-4
96.0/94.5
93,29910
93.7/90.6
95.1/92.0
94.7/91.4
98.3/97.3
98.4/97.5
98,5/97.5
98.8/97.8
98.1
Human
97.4/96.3
97.8/66.8
96.5/95.6
97. 67.99966
97.2/95.5
100/100
99 6.99.99.4
99.29988
99.2/99.8
99.9/99.
Table 1: Correlation of entailment-bass
answer scoring (SePer) with human answer scoring. We use the F1-score and Accuracy to measure the degree of correlation. As shown the table, SePer achieves the highest accuracy in answer scoring and is on par with human-level judgments.
4.1.2 VALIDITY OF ASePer ON QUANTIFYING RETRIEVAL. UTILITY
Secondly, we prove that using ASePer in measuring retrieval utility is highly correlated with hu- man annotations with a larger margin than baseline methods. We test our method in two different settings: I) Simple question-answering tasks, which generally require single document for answer generation, and 2) reasoning-involve several steps of partial information to correctly solve problem.
question answering, which requires collecting and integrating
Datasets. In the simple open QA setting, we use three representative datasets: NQ. MS MARCO Bajaj al. (2016), and SQUAD Rajpurkar et al. (2016). Each of these datasets has annotations of questions, golden answers, and human-annotanl answering the question requires only one passage, we first attach the positive passages with util- ity score of and the negative passages with a utility score of O. Since positive passages can not bring utility to LLMs based on their known knowledge, we then filter out those cases in which LLM has already succeeded in each dataset to eliminate the baseline effect. Through this preprocessing, got the utility label passages from real-human annotations. In the reasoning-involved QA setting, we use four typical Multihop-QA datasets, 2WIKIMULTHOOQQA Ho et al. (2020), HOT- POTQA Yang et al. (018).. IIRC Ferguson al. (2020), and MUSIQUE Trivedi et al. (2022b), which contains annotations of positive passages in the middle steps of a reasoning chain. Since each step only contains incomplete information pieces, we make a natural assumption that the overall information utility is uniformly assigned to each step and thus attach a ground-trute utility score of 1/Mseps for each middle-step passage. While not perfect, we find this assumption reasonable since these datasets are mostly collected by means of question composition, as detailed in Appendix !|
positive/negative passages. Since
Metrics. We use the Pearson correlation score to measure the correlation between our ASePer score and ground-truth utility score. We use t-test to assess the significance of the observed correlation coefficient, with statistic / computed with = rx /n 2) r²). We then map the to p-value using the Student's 's i-distribution table. As a result, all the Pearson correlation coefficients in Table have corresponding p-values less than 0.01, providing strong evidence against the null hypothesis and indicating a high level of statistical significance.
Baselines. We choose various methods that can be used to estimate the LLM's knowledge and use their difference to measure retrieval utility. exxical-matching-bar ROUGE, and BLEU,, which measure the response correctness score through matching text spans. BERTSCORE matches predicted answers and ground truth through embedding similarity. Another category of baselines is uncertainty measures, such as perplexity, entropy, and semantic entropy. Unlike the matching-baser method, these uncertainty measures do not require golden answers in computation. While they not directly defined on the correctness dimension, recent literature also shows that uncertainty is correlated with knowledge capabilities Farquhar et al. 2004) Cheng et al. (0024 in calibrated L.MMs. We also included L.L.-judges similar to Table
methods include EM,
include them as
7

---

Published as a conference paper at ICLR 2025
Implementation details. We the same semantic equivalence scoring algorithm tested in Table [ For each query, we sample = 10 times and obtain the response along with sequence likelihood. All baseline methods
sampled for the same k. and the final score comes from mean h as the generator L.M. We also tested
aggregation. In Table [ we use I.l amaa 22ma--2-13
other sizes and showed tendency results in Figure
Results. We show the result in Table 2 ASePer scores show significant improvement on other metrics. Specifically, ASePers has marginal improvement ASePerh across different datasets, which may indicate that soft probability mass assignment can capture more nuanced meanings, especially in free-form generations. Comparing simple and reasoning QA tasks, the scores of almost all different metrics are lower by 10%, indicating the challenging nature of reasoning-basee QA. Even though, ASePer can achieve Pearson correlation score greater than 0.5 across almost all datasets, showing its great potential to act as an automatic evaluation metric for retrieval utility.
Simple
Reasoning
Method
NQ
MS MARCO
SQUAD
2WIKIMHQA
HOTPOTQA
HRC
MUSIQUE
Exact Match
0.454
0.197
0,422
0.307
0.392
0.303
0.298
ROUGE
0.691
0.443
0.808
0.482
0.578
0.399
0.489
BLEU 0.188
0.353
0.298
0.197
0.206
0.126
0.163
BERTSCORE
0.592
0.322
0.564
0.361
0.451
0.197
0.392
PERPLEXITY
0.008
0.005
0.024
0.005
0.013
0.009
0.011
Entropy
0.431
0.142
0.557
0.226
0.276
0.292
0.203
Semantic Entropy
0.491
0.171
0.621
0.262
0.339
0.342
0.258
AUTO-J 0,421
0.022
0.406
0.243
0.183
0.096
0.169
PROMETHEUS
0.639
0.307
0.707
0.502
0.508
0.383
0.464
0.752
0.512
0,904
0.559
0.634
0.446
0.543
ASePers
0.769
0.533
0.905
0.584
0.660
0.461
0.559
Table 2: Pearson correlation between different evaluation metrics and ground-trute truth retrieval utility with p-
value < 0.01 for ASePer. As shown in the table, both ASePery and ASePers significantly outperform other baselines in measuring retrieval utility in simple and reasoning-type tasks, with the soft version leading an edge.
4.2 RELIABILITY OF SePer
We further test the reliability of SePer from different aspects, i.e., whether SePer produces consistent and stable evaluation across different datasets, random repetitions, and number of samples.
We tested ASePerg and ASePers with different numbers of samples across four datasets: NQ, HOTPOTQA, MS MARCO, and SQUAD, and the results are shown in Figure 3 We choose the number of pled responses n at [1,5 5, 10, 15, 20} for abla- tion purposes, extending the default choice of = 10. We find that the conclusion is con- sistent in different datasets: As n increases, the correlation of ASePer with ground truth also increases, indicating better accuracy. Be- sides, the variance generally becomes smaller as n increases, indicating better robustness. Generally, the elbow point appears at n 5 to n = 10, with n = 10 having less than a 1% performance drop compared to 20. Thus, using 10 in ASePer computation would be an effective choice.
062
MUMARGO
10207
Mumber Gf Used
Mamron of Jned
Figure 3: Influence of the number of samples and re- peated calculation of SePer four datasets.
https://hugginngface.coom 11ma 1.11am- 7b-chat-hh
8

---

Published as a conference paper at ICLR 2025
The shadow area and error bar in Fig shows that the fluctuation of SePer's quality among repeated calculations is less than 1 %, indicating high stability according to measurement theory. More exper- iments and ablation about the robustness of SePer can be found in Appendix
5 FINDINGS BASED ON SePer
In this section, we apply SePer to different modules in the RAG pipeline and exhibit our findings through the new lens of SePer. In general, RAG pipelines use techniques such reranker, refiner, and control flow to improve generation quality. Through the unique lens SePer, we can get a more fine-gre ene-ggaieed and accurate view of how these factors affect the overall performance of RAG. We benchmark current RAG workflows in Appendix[C] brief introduction of these components in RAG and experiment details can be found Appendix D
Simple QA
Multi-hop QA
L4740:5 01P
Liera
User 11h Dsey
Figure 4: Results about applying SePer on different RAG settings. The areas represent the positive and negative diffcrences between SePer for generation w/ and w/o retrieval, respectively. The solid blue line indicates ASePer, i.c., the utility of retrieval. The red dashed line indicates the zero point of the differences.
5.1 EXPERIMENT GOALS
We aim to address the following main research questions (RQs) through the lens of SePer. RQI- aims to observe the utility of RAG components on the final performance, including retrieval, reranker, and prompt compression. Specifically, RQ1 and RQ2 look closer at the retrieval utility and what impact on RAG can be brought by varying numbers retrieved items and choice of generator models of different sizes. These RQs are all designed to provide evidence and guidance on designing more efficient and effective RAG pipelines:
RQI: What is the utility of retrieval LLMs of different sizes?
RQ2: How does the number of retrieved items influence overall RAG performance?
RQ3: How do different prompt compression methods influence the overall RAG performance?
RQ4: How does the reranking phase influence the overall RAG performance?
5.2
RETRIEVAL UTILITY FOR GENERATOR MODELS OF VARYING SIZES (RQ1)
In figure 4 evaluate how LLMs of different sizes benefit from retrieval. Our experiments conducted on both simple QA and multi-hop QA datasets, and more implementation details can be found in Appendix D We observed that 1) for both scenarios of QA tasks, models of different sizes generally make positive use of retrieved information to produce better answers for most datasets. exception is MS MARCO, which we attribute Wikipedia corpus we uniformly used. 2) According to our experiments, medium-size models benefit most from retrieved information. This could be due to 1) their weaker initial knowledge without retrieval and 2) their better ability to absorb retrieved in-context information as compared to smaller models.
source corpus being different from the
5.3 UTILITY DIFFERENT NUMBERS OF RETRIEVED ITEMS (RQ2)
The number in-context retrieved items used in prompts, noted as k. can significantly impact the model's generation results. Figure a) shows the experiments about the impact of k on the overall RAG performance, with set at [I, 5, 10). For most datasets, retrieving more information pro- gressively positively affects answering questions. However, increasing the number of in-context

---

Published as a conference paper at ICLR 2025
Figure 5: Differences in SePer across various retrieval and generation settings. Panel (a) illustrates the dif- ferences in SePer between generations w/ and w/o retrieval, analyzed under different retrieved items. Panel (b) highlights the effect of prompt compression methods SePer differences compared generation w/o trieval. Panel (c) examines the impact of the reranker on SePer differences relative to generation w/o retrieval.
items from 5 to 10 only brings marginal improvement and sometimes even slightly hurts the gener- ation performance, as the AmbigQA dataset. This might be due the extra noise brought by an increasing number of retrieved documents.
5.4 UTILITY OF PROMPT COMPRESSION METHODS (RQ3)
The prompt compression module is used to reduce the prompt length to lower the inference cost while preserving or facilitating the RAG performance.
Figure b)b) illustrates the results of our experiments. We test the utility of two major prompt com- pression works, SELECTIVE-CONTETT ei al. (2023) and LONGLLMLINGUA Jiang et al. (024). Although both prompt compression methods slighlyy reduce SePer compared to no compression, both methods can reduce the prompt by about 40%, thus lowering the inference costs about third the original. Additionally, we note that the LONGL.MMINGIAA maintains a relatively higher SePer than selective-context, becoming preferred choice for balancing performance and inference cost. More details about prompt compression methods can be found in Appendix D.I
5.5 UTILITY RERANKER (RQ4)
While retrieval can quickly gather candidate items from large document collections to aid genera- tion, it often lacks precision in small k; which leaves out important information and brings in many noises. To this end, the reranker module is introduced to the RAG pipeline, which not only se- lects relevant documents into prompt contexts with better accuracy but also ce-arranges them in the best order for overall generation quality Liu et al. (024). More details about the lines work on reranker can be found in Appendix D.2
Figure (i) shows experimental results comparing ASePer with and without rerankers from the implementation of bogerzerakke- www.erraneeeaarrrreeeeeeeeaaaaee Xiao al. (2024). We set top-k values of 20 for retrieved items and 5 for reranked items in reranked scenarios while keeping a constant top-k of in non-reranked scenarios. Results from ASePer are consistent with the conclusions of works in the field that reranker, in general. brings significant improvement to the RAG pipeline by removing noises and reordering contexts. However, in NQ and AMBIGQA (which are also derived from NQ), it seems that the reranking process has a negative impact on answer quality. This might indicate that simply putting more relevant contexts at an earlier position may not be the best strategy. How the ordering of contexts influences the final generation results is open for exploration.
DISCUSSION
This study introduces Semantic Perplexity (SePer) and then ASePer, a novel metric that evalu- ates the utility of information retrieval by measuring the knowledge gain in large language models (LLMs). SePer provides a more nuanced understanding of retrieval effectiveness beyond mere ac- curacy, aligning closer with real-world inference needs.
Our findings demonstrate that ASePer can effectively quantify retrieval needs across various sce- narios, aiding data selection and resource allocation in RAG systems. This metric can enhance the optimization of RAG systems for both efficiency and effectiveness, promising improved perfor- mance in complex AI applications. Future work will focus on extending SePer's applicability to more diverse and challenging RAG scenarios.
httpps: https://hugginggf co/ BAAI rerankerrreeeee--rrr large
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
This work supported in part by the National Key R&D Program of China (Grant No. (023YFF0725001), in part by the National Natural Science Foundation of China (Grant No.92370204), in part by the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2023B 151522055), in part by GuangZhou-HKKUSTGGG Joint Funding Program (Grant No. 2023A0310008), Education Bureau of Guangzhou Municipality. Yijie Xu acknowledges the support from the modern matter laboratory, HKUST(GZZ.

---

Published as a conference paper at ICLR 2025
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In ICLR, 2017.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflectin In EMNLP, 2023.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. NeurIPS, volume 30, 2016.
N. Belkin. Anomalous states knowledge as a basis for information retrieval. The Canadian Journal of Information Science, 1980.
N. Belkin, R. N. Oddy, and H. M. Brooks. Ask for information retrieval: Part i. background and theory. Journal of Documentation, 1982a.
J. Belkin, R. N. Oddy, and H. M. Brooks. Ask for information retrieval: Part ii. results of design study. Journal of Documentation, 1982b.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-
domain questions. ACL, 2017.
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. Can ai assistants know what they don't know? In ICML, 2024.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACI, 2019.
William S Cooper. A definition of relevance for information retrieval. Information Storage and Retrieval, 1971.
William S Cooper. On selecting measure of retrieval effectiveness. Journal of the American Society for Information Science, 1973.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma. Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent meta-optimizers. In ACL, 2023.
Lu Dai, Hao Liu, and Hui Xiong. Improve dense passage retrieval with entailment tuning. In EMNLP, 2024.
Brenda Dervin. On studying information seeking methodologically: the implications of connecting metatheory to method. Information Processing & Management, 1999.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint www.::224000028.... 2024.
Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In EACL, 2024.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 2024.
James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. lire: A dataset of incomplete information reading comprehension questions. In EMNLP, 2020.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. -atiivall-ugmentee generation for large language models: A survey. arXiv preprint 231. 2023.
12

---

Published as a conference paper at ICLR 2025
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In EMNLP, 2021.
Zhouhong Gu, Lin Zhang, Xiaoxuan Zhu, Jiangjie Chen, Wenhao Huang, Yikai Zhang, Shusen Wang, Zheyu Ye, Yan Gao, Hongwei Feng, and Yanghua Xiao. DetectBench: Can large language model detect and piece together implicit evidence? In ACI, 2024.
Weiyu Guo, Ziyang Chen, Shaoguang Wang, Jianxiang He, Yijie Xu, Jinhui Ye, Ying Sun, and Hui Xiong. Logic-in-frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding, 2025.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In ICML., 2020.
Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. Wikiasp: A dataset for multi-domain aspect-based summarization, TACI, 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhancee bert with disentangled attention. In ICLR, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In COLING, 2020.
Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David Ross, Cordelia Schmid, and Alireza Fathi. Avis: Autonomous visual information seeking with large language model agent. NeurlPS, 2024.
Peter Ingwersen. Cognitive perspectives of information retrieval interaction: elements of a cognitive ir theory. Journal of Documentation, 1996.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. TMLR, 2022.
Abigail Z Jacobs and Hanna Wallach. Measurement and faimess. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMMingua: Accelerating and enhancing LLMs long context scenarios via prompt compression. In ACL, 2024.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL, 2020.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. EMNLP, 2023.
Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmenned generation research. CoRR, abs/2405. 13576, 2024.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. TBD, 2019.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal. and Peter Clark. How much coffee was consumed during EMNLP 2019? fermi problems: A new reasoning challenge for AI. In EMNLP, 2021.
13

---

Published as a conference paper at ICLR 2025
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. In ICLR, 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. TACL, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmenned gener- ation for knowledge-intensivy nlp tasks. NewrlPS, 2020.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Pengfei Liu, et al. Generative judge for evaluating alignment. In ICLR, 2024.
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In EMNLP, 2023.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization
Branches Out, 2004.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. TACL, 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu G-eval: Nlg evaluation using gpt-4 with better human aligmment. EMNLP, 2023.
Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. TPAMI, 2018
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. ACL, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. NeurIPS, 2022.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In EMNLP, 2020.
Alexander V Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kerel language entropy: Fine-grained uncertainty quantification for LLMs from semantic similarities. In NeurlPS, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In EMNLP-IJCNLP, 2019.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In EMNLP Findings, 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In KDD, 2016.
14

---

Published as a conference paper at ICLR 2025
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Faanook-Beealleve Mike Gatford, et al. Okapi at tree-3. Nist Special Publication Sp. 1995.
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An automated evalu- ation framework for sarrieval-augmentee generation systems. In NAACL, 2024.
Alireza Salemi and Hamed Zamani. Evaluating retrieval quality in retrieval-augmenned generation. In SIGIR, 2024.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? NeurIPS, 2023.
Zhihong Shao, Yeyun Gong, Minlie Huang, Nan Duan, Weizhu Chen, et al. Enhancing retrieval- augmented large language models with iterative retrieval-generation synergy. In EMNLP, 2023.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco, Richard James, Mike Lewis, Luke Zetteemoer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. In NAACL, 2024.
Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In NAACI, 2018
James Thore,, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: large- scale dataset for fact extraction and VERification. In NAACI, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interlcaving trieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In ACL, 2022a.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. musique: Multi- hop questions via single-hop question composition. TACL, 2022b.
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kıcıman, Noshir Contractor, and Tina Eliassi-Rad. Measuring algorithmically infused societies. Nature, 2021.
Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang. Xiangkun Hu, Zheng Zhang, and Yue Zhang. Evaluating open-qa evaluation. NeurIPS, 2024.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint
2022.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C-pack: Packed resources for general chinese embeddings. In SIGIR, 2024.
Ziang Xiao, Susu Zhang, Vivian Lai, and Q Vera Liao. Evaluating evaluation metries: framework for analyzing nig evaluation metrics using measurement theory. In EMNLP, 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ICLR, 2022.
Yijie Xu, Aiwei Liu, Xuming Hu, Lijie Wen, and Hui Xiong. Mark your Imm: Detecting the mis- use of open-source large language models via watermarking. In The 1st Workshop on GenAl Watermarking, ICLR, 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
Peiran Yao and Denilson Barbosa. Accurate and nuanced open-QA evaluation through textual en- tailment. In ACI., 2024.
15

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023.
Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyza- guirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding. In CVPR, 2025.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrivaal-augmnnte language models robust to irrelevant context. The Twelfth ICLR, 2024.
Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. Augmentation-adapeee adapted retriever improves generalization of language models as generic plug-in. In ACL. 2023.
He Zhang, Ying Sun, Weiyu Guo, Yafei Liu, Haonan Lu, Xiaodong Lin, and Hui Xiong. Interactive interior design recommendation via coarse-to-fine multimodal reinforcement learning. In ACM MM, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In ICLR, 2020.
Zexuan Zhong. Dan Friedman, and Danqi Chen. Factual probing is [mask]] Learning vs. learning to recall. NAACL, 2021.
16

---

Published as a conference paper at ICLR 2025
Part I
Appendix
Table of Contents
Details of Experiments
18
A Dataset Statisties
18
A.2 Robustness of SePer in Repeated tests
18
Extended Results
19
B.
Qualitative analysis
B.2
Efficiency analysıs
20
B.3
Circumstances of negative utility
21
B.4
Ablation of the consistency of SePer with different entailment models
22
C
Benchmarking retriever and workflow
24
C.I
Benchmarking Retriever Quality and Utiliyy
24
Benchmarking Workflow Utility
24
D
Experiment Details in Section
28
D.1
Prompt Compression
28
D22
Rerankers
29
D.3
Dataset Selection
30
D.4
HyperParameter setting
30
D.5
Prompt Design Implementation
31
D.6
System Specifications for Reproductivity
31
17

---

Published as a conference paper at ICLR 2025
A DETAILS OF EXPERIMENTS
A. DATASET STATISTICS
This section presents the datasets used in our experiments, detailing their names, sizes, sources, key characteristics, and availability, which are categorized by task type (Single QA, Multi-hop QA, Fact Verification, Multiple-choice QA, and Summarization). The table below also includes links the original papers and dataset repositories.
Size
Source
Key Characteristics
Availability
Single
MS MARCO Bajai ctal.
2016
101,093
Bing
Search engine querics, web passages
Paper
Datasel
PorQA Mallen et al.
2023
14,267
Wikipedia
Focuses on populz culture questions
Paper
Dalaset
TRIVIAOA
11,313
Wikipedia Web
Trivis questions
evidence documents
Paper
Datasel
Rarpurkar
Wikipedia
Standard reading comprehension
Paper
NQ Kwinkowsk di
2019
3,610
Wilkipedia
Real user queries, diverse topics
Paper
Dniasel
Bool Qlark Clarkee mh
2019
3,270
Wikipedia
YouNo questions, requires inference
Paper
Dnlasel
AMBICOA Min
2020
2,002
Wikipedia
Question ambiguity with multiple answers
Paper
Datasel
FERMI Kalyin
Wikipedia
Estimation--ues reasoning
Paper
Datasel
Multi-hop QA
HorroTQA Yang et al.
2018
7,405
Wikipedia
Requires multi-hop reasuning
Paper
Datasel
2WIsMULTTHHFOAA
2020
12,576
Wikipedia
Cross document multi hop reasoning
Paper
Datasel
MuSIQUE Trivedi
2.417
Wikipedia
Multi-hpp reasoning. complex questions
Paper
BAMBOOGLE Press
1725
125
Wikipedia
Compositional reasening challenging Google
Paper
Dalasel
Fact Verification
FEVER Therne al.
2018
10,444
Wikipedia
Fact verification
PPape
Dataset]
Multiple-choice QA
Hendrycks
14,042
NA
Multipl--choice, general knowledge
[Pape Dataset]
Summarization
WIKIASP Hayashi Hayashi et al.
2021
37,368
Wikipedia
Opea-domain domain summarization
Paper
Dataset
Table 3: Summary of Datasets Used in Our Experiments
A.2 ROBUSTNESS OF SePer IN REPEATED TESTS
Coefficient Variance
Dataset
Repetition
Samples
SePer
Correlation
SePer
Correlation
NQ Kwiatkowski 2019
5
10
0.053
0.002
0.028
0.002
MS MARCO Bajaj 2016
5
10
0.055
0.003
0.003
0.005
HorrotQA Yang el al.
5
10
0.045
0.001
0.037
0.001
SQUAD Rajpurkar ei al.
2016
5
10
0.056
0.004
0.063
0.004
Table 4: Extended experimental details on the robustness test of SePer. As shown in the table, SePer demonstrates stability upon repeated testing in our default choice ten samples.
Table shows that the standard deviation(a) and coefficient variation in calculating SePer less than 10%, which means that SePer produces the result of low variance in repeated tests. We also calculated the variation in the degree correlation with human judgments in repeated tests. Results show that the fluctuation of the correlation score is less than 1 1%. All these experimental signs prove that the proposed SePer is reliable and stable measurement according to the theory of statistics.
A.2.1 P-VALUES WITH THE PEARSON CORRELATION ASePer
Dataset NQ
MS MARCO
SQUAD
HoTPoTQA
2WIKIMHQA
MUSIQUE
IRCC
v-luu
2.86 x 10 207
2,88 x 10 10-218
Table 5: p-values across different datasets. The p-value 0 does not necessarily mean zero, but below the numerical precision, which indicates a number very close
18

---

Published as a conference paper at ICLR 2025
B EXTENDED RESULTS
B.1 QUALITATIVE ANALYSIS
In this section, we analyze two RAG cases qualitatively to demonstrate the effectiveness of the proposed ASePer in measuring retrieval utility.
We two cases from two typical scenarios: one for simple RAG, in which the answer is contained in a single document, and another for reasoning- instoningintensive RAG, in which the answer should be reasoned over multiple documents.
Question:
Who sings does he love me with reba?
Reference Does:
Docl: "Does He Love You" a song written by Sandy Knox and Billy Stritch, and recorded as duct by American country music artists Reba McEntire and Linda Davis. It was released in August 1993 as the first single from Rebaš album "Greatest Hits Volume Two". It is one of country musicš several songs about love triangle. "Does He Love You" written 1982 by Billy Stritch. recorded with a which he performed at time, because wanted a song that could be sung by the other two members.
Ground Truth Answer
Linda Davis.
Retrieved Does
Model Answer (x10)
GT Answer
ASePer
Amount of information
Doci
Reha McEntire: 10
0
0
Linda Davis
Linda Davis:
1.0
Table 6: Case #1 of simple RAG task: ASePer on single retrieved doc. ASePer accurately reflects
the utility of retrieved documents.
Results Analysis of Case #1: Results in Table 6 shows that when useful document is provided (the Xmeans the retrieved document is irrelevant), the model consistently fails to answer the question correctly, even with ten times' sampling. At this time, the calculated ASePer is 0, accurately indi- cating the zero utility of irrelevant information. When a positive document is retrieved, the model successfully generates the correct answer. At this time, the calculated ASePer is accurately indi- cating the utility of useful information.
Question:
Are the Laleli Mosque and Esma Sultan Mansion located in the same neighbor- hood?
Reference Docs:
Docl:
The Laleli Mosque (Turkish: 'Laleli Camii', Tulip Mosque) is 18th-century Ottoman imperial mosque located in Laleli, Fatih, Istanbul, Turkey.
Doc2: The Esma Sultan Mansion (Turkish: 'Esma Sultan Yahısi), a histori- cal yalı (English: waterside mansion) located on the Bosphorus in the Ortaköy neighborhood of Istanbul, Turkey, named after its original owner, Esma Sultan, now used as cultural center after redevelopment.
Ground Truth Answer
No.
Retrieved Docs Doel
Model Answer (x10)
GT Answer
ASePer
Amount of information
Doc2
X
Yes: 10, No: 0
0
0
Yes: 7, No:
0.15
1/2
No
Yes: No:
0.1
1/2
Yes: 3, No:
0.7
Table 7: Case #2 of reasoning- based RAG task: ASePer multiple retrieved docs. ASePer reflects
the utility of retrieved information in a fine-grained way, successfully responding to partial informa- tion.
19

---

Published as a conference paper at ICLR 2025
Results Analysis of Case #2: Results in Table demonstrate that when relevant retrieval is provided, the model consistently gives the incorrect answer, "Yes," which indicates it is unable to infer the relationship between the two locations. When only piece of contextual information is made available, the model's answers begin to vary. This outcome suggests that partial information, although not sufficient for producing the correct answer consistently, causes the model to reconsider its initially confident but incorrect response. Traditional evaluation methods often assess retrieval utility based solely on whether the retrieved information directly enables the model provide correct answer and overlook the intermediate benefits that partial information can offer. In contrast, our ASePer successfully responds even partial information, providing a fine- aneegra evaluation of information utility.
B.2 EFFICIENCY ANALYSIS
We conduct a latency and cost analysis using widely available commercial LLM APIs. Specifi- cally, we evaluate multiple APIs with varying pricing structures from providers, including OpenAI, Authropic. Google, and Deepseek.
In the Direct Evaluation setting, we prompt the LLM with given question, context, and answer and request the model to assess the contribution of the context to the overall response by assigning integer score between 1 and 10. While in the Reduction Evaluation setting, first query the LLM with the combination of query, context, and answer to evaluate the correctness of the answer. Subsequently, query the LLM with only the query and answer assess the correctness without the context. The difference between these two scores is computed to determine the ASePer.
For experiments, utilize the Natural Questions Kwiatkowski et al. 2019) dataset, selecting 10 questions with corresponding references collected from passages in the Wikipedia corpus using the ES Wang al. (2022) model. We report the average prompt length and the average time consumed across the ten questions. We list the user prompts used in the experiment as follows:
Prompt for Direct Evaluation in LLM APIs
Evaluate the contribution of the given context to the provided answer for the specified ques- tion.
Your evaluation should be based on how effectively the context supports or justifies the answer.
Provide your assessment using integer rating between (minimal or no contribution) and 10 (critical or complete contribution).
Do not output any other information context.
Question: {question}
Context: (context)
Answer: { [answer)
Your evaluation:
Prompt for Reduction Evaluation LLM APIs with context
Evaluate the correctness the given answer based on the question and provided context. Rate correctness using integer between (completely incorrect) and 10 (completely cor- rect).
Only provide the rating your output.
Question: {question]
Context: (context)
Answer: [answer]
Your rating:
20

---

Published as a conference paper at ICLR 2025
Prompt for Reduction Evaluation L.M APIs without context
Evaluate the correctness of the given answer based solely on the question. Ignore any external information and rate correctness using an integer between (completely incorrect) and 10 (completely correct).
Only provide the rating your output.
Question: {question}
Answer: fanswer)
Your rating:
the average time latencies and costs across different API providers. We checked the latest pricing on the official websites of each API and did not enable any potential batching mechanisms. Additionally, it demonstrates the advantages of our proposed SePer and ASePer in
terms of time and economic costs.
Direct Evaluation
Reduction Evaluation
Models
Company
Time (s)
Cost ($)
Time (s)
Cost ($)
CHATGPT-40-LATEST
OpenAl'
4.22
0.0077
6.13
0.0080
GPT-4-TURBO
OpenAl!
2.01
0.0155
3.80
0.0163
GPT-3.5-TURBO
OpenAl'
3.34
0.0008
3.89
0.0008
Anthropic
7.45
0.0046
30.30
0.0052
-HAUDE--HAIKU-NX
Anthropic
5.63
0.0003
19.21
0.0004
DEMINI-1.5-PRO
Google
3.29
0,0192
6.97
0,0203
1EMINI-I.5-FLLCN
Google
3.56
0.0001
6.26
0.0001
DEEPSEEK-CHAT
Deepseek
0.88
0,0000
1.68
0,0000
SePer & ASePer
N/A
0.12
Free
0.24
Free
Table 8: Latency (L) and Cost (4) Comparison of Various LLM Models in Direct and Reduction Evaluation Settings. The table shows the average response time (in seconds) and cost (in USD) for each model in Direct and Reduction evaluation tasks.
B.3 CIRCUMSTANCES OF NEGATIVE UTILITY
Additionally, our experiments, conducted according the settings outlined in Figure revealed that some retrieved items negatively impact question-answering performance. We extended our tests to additional datasets to further investigate this phenomenon, and the datasets involved are listed in Table 3
We first observed that in certain datasets, the retrieved items hindered the model's question- answering ability. For instance, the MMLU dataset, which is a multiple-choice dataset with relatively straightforward questions, the model can often rely on its knowledge to answer rectly. In such cases, the retrieved items proved detrimental. For the MS MARCO dataset, we attributed the performance issue distribution differences, the corpus source (Bing) differs from the source of other datasets (Wikipedia). For more complex datasets like MUSIQUF, 2WIKIMUL- THHOPQA, and FERMI, which require multi-step reasoning and logical chains, small number of retrieved items could not provide all the necessary information. However, when enough items were retrieved, they offered a more comprehensive information set, thereby assisting the model mak- ing correct inferences. Additionally, In the FEVER dataset, which focused on Fact Verification, an excessive number retrieved items disrupted the model's ability to verify facts effectively.
Regarding prompt compression methods, excluding datasets that already showed negative effects even without compression (as discussed in the previous paragraph), the FERMI dataset-rich in
httpes://geeaal.cc//pp////ddddit
httss://ail.coogleede////////// tttp:///pi--ocs.eeppee..kk
21

---

Published as a conference paper at ICLR 2025
mathematical logits particularly affected by incorrect token compression, resulting in errors. Similarly, in the 2WIKIMCLTHHOPQA dataset, the compression of logical chains was identified as major issue.
Despite these challenges, we still found that the use of the reranker consistently improved perfor- mance across these datasets, further validating its robustness.
Figure 6: Differences in SePer under various retrieval and generation settings. Same as Figure 5 Panel (a) shows the differences in SePer for generation with and without retrieval under different numbers of retrieved information. Panel (b) illustrates the impact of using prompt compression and which compression method on the differences in SePer compared to generation without retrieval. Panel (c) demonstrates the reranker's effect compared generation without retrieval.
B.4 ABLATION OF THE CONSISTENCY OF SePer WITH DIFFERENT ENTAILMENT MODELS
In this paper, choose deberta- seatttavvvxxxaggee wlaarg--mnn' He et al. 2021) following Farquhar et al. (024) for deciding the entailment relationship, which strikes a balance between accuracy and efñ- ciency. For the sake of the ablation study, we tested the influence of the choice of entailment models on the result of SePer in Figure We choose seven mainstream models used for NLI classification tasks with varying sizes and architectures and compute the Pearson correlation of SePer produced from these models. Specifically, the entailment models we used are listed as follows:
Model
Developer
(Purameters)
# Layers
Hidden Size
Architecture
Microsoft
86M
12
768
Encoder-only
DeBERra-Large-Bie MNLL2
Microsoft
350M
24
1024
Encoder-only
DeBERTa-XLazge-eee -2a "
Microsoft
700M
48
1024
Encoder-only
Microsoft
710M
24
1536
Encoder-only
Microsoft
1.3B
48
1536
Encoder-only
MaBERTe-Targe-MM1..
Facebook
355M
24
1024
Encoder-only
BART-L.arge-M1..1
Facebook
406M
12+12
1024
Encoder-Decoder
Table 9: Details of different entailment models for ablation study on the robustness of SePer. We
choose main-stream models used in the field of NLI, covering different sizes and architectures.
We tested 7 NLI models 9 datasets and 3 different choices for (number of items retrieved and used in-context prompting) in computing SePer.
According to the results in Figure all of the Pearson correlation scores between the results of model pairs are above 0.7. Specifically, for simple-type QA tasks (NQ, AMBIGQA, MSMARCO, SQUAD, TRIVIAQA, PorQA), the entailment judgments even consistent, with all scores above 0,85 and most scores above 0.9. For reasoning-type QA tasks (HotpotQA, 2WikiMultihoOAA, MuSiQue), the entailment scores all above 0.7, with most scores above 0.9.
https:/ http://///ggggeeccccoo..oeeddde.
https: http:///eaiiieeecooooccoo////ee2
22

---

Published as a conference paper at ICLR 2025
Based on these high correlation scores among different entailment models, safely draw the conclusion that SePer is robust in computation in terms of entailment model choice, and still effective and reliable small models (86M) when high efficiency is required.
Figure 7: Consistency of SePer computation among different entailment models.
23

---

FLARE Jiang et al. Published as a conference paper at ICLR 2025
C BENCHMARKING RETRIEVER AND WORKFLOW
For more detailed information on benchmarking the retriever and the entire workflow, please visit the following link: SePer Benchmarks
C.I BENCHMARKING RETRIEVER QUALITY AND UTILITY
The majority of dense retrievers in dense retrieval paradigms are based on transformer architectures such as BERT. This opens the possibility for further fine-tuning with more extensive and higher- quality datasets, as well as more advanced algorithms. Consequently, wide variety of retrievers have emerged in the community. To evaluate their performance, we present benchmarks based on multiple datasets (described in Table B and multiple retrievers, utilizing SePer the evaluation framework. These benchmarks aim to assess both the quality and utility of different retrievers.
For quality evaluation, provide benchmark comparing the performance of retrievers stan- dard retrieval tasks with various numbers of retrieved items. For utility evaluation, we propose a A-SePr--base benchmark. In this setup, the ASePer is computed by taking the difference between the SePer scores achieved using question-answer pairs and those obtained using question-retrieval contexi-answer pairs.
We evaluate six dense retrievers: AARaNCE Yu et al. (2023). AAR AARcontrieve Yu et al. 2023
BGE Xiao et al. (2024), Contriever Izacard et (2022), DPR Karpukhin al. (2020), and B5 Wang et al. (2022), alongside the classical sparse retriever BM25 Robertson et al. (1995). The
results of the quality and utility benchmarks are presented in Tables 10 and respectively.
C.2 BENCHMARKING WORKFLOW UTILITY
Naive RAG strictly follows the retrieval-genin generation paradigm, which limits its ability to utilize retrieved information for further retrieval. This limitation critical for complex reasoning tasks, such multi-hop question answering. Therefore, recent research has proposed several workflows that enable the entire RAG pipeline to perform multiple retrievals and integrate information, which may enhance the reasoning ability of large language models.
We benchmarked four RAG 2023), IRCOT Trivedi al. (2022a), and ITER- Shao al. (2033 on multiple datasets using varying numbers of retrieved items. Since these methods may involve multiple rounds of retrieval, existing retrieval metrics, such as retrieval recall, are no longer suitable. Thus. we only report the ASePer metric. Additionally, RETROBUST only provides LoRA checkpoints for Llama Touvron et al. (2023) 13B, so results for the 7B model are marked as "N/A." The results of the benchmark are shown in Table
RETROBUST Yoran et al. (024).
24

---

Published as a conference paper at ICLR 2025
0.647
0.754
0. 0.800
0.562
0.632
0.722
0.835
0.698
0.776
0.795
0.739
0.734
0.786
0.818
0.348
0.419
0.467
0.457
0.335
0.299
0.467
0.428
0.538
0.639
0.364
0.440
0.438
0.610
0.726
0.645
0.766
0.458
0.508
0.593
0,790
Table 10: SePer Benchmark Across Retrievers and Parameter Sizes. The colors highlight the best performing retrievers under each dataset.
Retrieval Recall
N/A
0.574
0.670
0.740
0.463
0.339
0.662
0.774
0.632
0.725
0.7.3
0.688
0.659
0.740
0.779
0.300
0.336
0.405
0.390
0.2.3
0.244
0.384
0.367
0.465
0.573
0.496
0.359
0.376
0.534
0.662
0.579
0.704
0.437
0.446
0.532
0.729
0.331
0.391
0.507
0,248
0,272
0.424
0.570
0,444
0.548
0.580
0.508
0.427
0.555
0.635
0.175
0.189
0.216
0.244
0.13
0.150
0.223
0.205
0.287
0.326
0.31
0.198
0.229
0.307
0.431
0.403
0.510
0.286
0.281
0.355
0.528
70B
0.533
0.593
0. 0.609
0 0.505
0.344
0-586
0.621
0. 0.708
0.76
0.750
0.729
0.720
0.745
0.783
0.429
0.452
0.453
0. 0.460
0.435
0. 0.399
0.465
0.474
0.519
0.543
0.335
0.479
0.454
0.545
0.551
0.504
0.566
0.427
0.442
0.465
0.590
13B
0,461
0.516
0.551
0.431
0.472
0.510
0.565
0.671
0.723
0.711
0.658
0.688
0.709
0.739
0.403
0.414
0,420
0.434
0.407
0.368
0.422
0.364
0.409
0.4.8
0.44
0.376
0.361
0.442
0.490
0.452
0.515
0.366
0.386
0.425
0.535
7B
0 0.46
0.497
0.535
0.409
0.448
0.490
0.560
0.626
0.702
0.697
0.659
0.659
0.682
0.730
0.380
0.3 0.398
0.408
0
0.285
0.362
0.402
0.347
0.419
0.436
0430
0.369
0.350
0.428
0.478
0.431
0.477
0.358
0.382
0.397
0.512
70B
0. 0999
0.563
0.598
0.457
0.501
0.560
0.620
0,
0.76
0.750
0.729
0.720
0.745
0.783
0.404
0.412
0.431
0.428
0.403
0.370
0.427
0.507
0.537
0.535
0.457
0.440
0.527
0.513
0.477
0.550
0.379
0.408
0.445
0.57
SePer
13B
0.424
0.496
0.537
0.377
0,421
0.482
0.563
0.635
0.699
01690
0000
0,056
0,688
0.727
0.358
0.375
0.391
0.398
0.380
0.34]
0.393
0.394
0,423
0.427
0.341
0.340
0.414
0.480
0.435
0.513
0.339
0.367
0.409
0.528
7B
0.413
0.481
0.530
0.367
0.409
0.469
0.557
0.605
0.684
0,675
0,6,2
0,034
0,663
0.714
0.371
0.383
0.403
0.392
0.373
0.352
0.389
0.339
0.402
0.423
0.422
0.352
0.335
0.421
0.467
0.418
0.489
0.334
0.356
0.396
0.512
70B
0.383
0.445
0.517
0.347
0.368
0.448
0.555
0.640
0.6.89
0.684
0,684
0,656
0,067
0.736
0.344
0.348
0.371
0.378
0.353
0.309
0.361
0.365
0.445
0.464
0.468
0.380
0.360
0.460
0.415
0.394
0.474
0.301
0.325
0.344
0.486
131
0.321
0.376
0.449
0.270
0.309
0.389
0.489
0.577
0.629
0.624
0.607
0.5 79
0,607
0.667
0.341
0.349
0.350
0.367
0.349
0.322
0.354
0.295
0.351
0.378
0.361
0.305
0.301
0,373
0.394
0.372
0.454
0.282
0.291
0.321
7B
.3323
0.388
0.464
bm25 0.282
0.301
0.403
0.504
0.350
0.627
0,620
0.605
0.34
0.587
0,677
0.348
0.357
bge 0.368
bm25 0.374
0.333
0.326
0.306
0.297
0.376
0.392
0,383
0,309
0.314
0.378
0.395
0.375
0.456
0.287
0,297
0.330
0.474
AAR contriever
AAR AAR-COOEE wwwtrreeeeeeec................ee eeer
AAR contrieven reverr
AAR cont. contrreeee lever
AAR contrieve leverr
Retriever
AAR ANCE
contriever
AAR ANCE
contriever
AAR ANCE
cottriever
AAR ANCE
contriever
AAR ANCE
contrieve
bge
dpc
e5
big
bom25
ape
e5
dpc
e5
bgg
km25
dpr
65
bge
lm25
dec
05
Metric
Top-k
2WIKIMULTHOOOA
Dataset name
TRIVIAQA
HOTPOTQA
POPQA
NQ
25

---

Published as a conference paper at ICLR 2025 -
10
0.647
0.754
0,300
0.562
0.632
0.722
0.833
0.698
0.776
0.795
0.739
0.734
0.786
0.818
0.348
0.419
0.467
0.457
0.335
0.259
0.467
0.428
0.538
0.639
0.364
0..40
0,438
0.610
0.72 0.726
0.645
0,705
0.488
0.508
0.593
0.790
Table 11: ASePer Benchmark Across Retrievers and Parameter Sizes. The colors highlight the best- best-performing retrievers under each dataset.
Retriieal
N/A
0.574
0.670
0.740
0.463
0.539
0.062
0.774
0.632
0.725
0.738
0.688
6.699
0.740
0.779
0.300
0.356
0.405
0.390
0.273
0.244
0.384
0.367
0.465
0.573
0.496
0.359
0.376
0.534
0.662
0.379
0.704
0.437
0.446
0.532
0.729
0.351
0.39
0.507
0.248
0.272
0.424
0.5
0.444
0.548
0.580
0.500
0.427
0.555
0.635
0.175
0.189
0.216
0,244
0.144
0,150
0.223
0.205
0,287
0.326
0.31
0. 97
0.229
0.307
0.43
0.403
0.500
0.286
0.281
0.355
0.528
70B
0.049
0. 109
0. 25
0.02
0.060
0.102
0.137
00005
0.019
0,015
-0,05
000008
0.010
0.034
0,063
0,087
0.088
0.095
0.070
0.033
0.100
0.070
0.115
0.138
31
0.075
0.050
0.1 1404
0.1 191
0.1 0.147
0.209
0.069
0.085
0.108
0.233
10
133
0.072
0.1 0.127
0.162
0.042
0.083
0.121
0.176
0.035
0.082
0.075
0.051
0.05
0.073
0.103
0.070
0.081
0.087
0.101
0.074
0.033
0.091
0.067
0.1
0.151
0.143
0.079
0.004
0.145
0.206
0.168
0.231
0.082
0.102
0.14]
0.251
7B
0.090
0.141
0.178
0.053
0.091
0.13
0.204
0.017
0.035
0.015
0.040
0.022
-0000
0.033
0.017
0.035
0.045
0.040
0,022
-0.001
0.039
0.053
0.115
0.142
0.156 136
0.075
0.056
0.134
0.207
0.160
0.200
0.087
0.111
0.127
0.241
701
0.009
0.079
0.1
-0.027
0.017
0.076
0.136
-0.44
0.006
-0.005
40,027
-0,035
-0.01
0.028
0.039
0.047
0.065
0.062
0.038
0.005
0.062
0.040
0.103
0.132
0.130
0,053
0.035
0.122
0.156
0.119
0.193
0.022
0.051
0.087
0.214
A.So
13B
0.036
10008
0.149
-0.012
0.055
0.093
0.174
-0.00
0.003
0,054
0.125
0,020
0,052
0.091
0.025
0.012
0.058
0.065
0.047
0.008
0.060
0.048
0.997
0.126
0.130
0X044
0.045
0.117
0.196
0.131
0.239
0.055
0.083
0.125
0.244
7B
0,077
0.125
0.174
0,010
0,055
0.11
0.025
0,104
0.09
0,052
0,053
0,085
0.1.3
0,008
0,020
0.040
0,030
00010
10.011
0.027
0.045
0.108
0.129
0.129
0,058
0.041
0.127
0.196
0,147
0228
0.065
0.086
0.125
0.241
70B
00.10
-0.039
0.033
-0.137
-0.166
-0035
0.071
-0.166
-00067
-0.071
-0,072
+0,099
+0,088
+0.020
00002
-0.017
0.006
0,013
-0013
-00.6
-00004
-0039
0.041
0.000
0.064
00024
-0444
0.056
0.058
0.037
0.117
40.057
-0,032
-0.013
0. 128
13B
0.008
00012
0.000
41118
40079
0.000
0.100
00.59
-0.007
-0.012
40,029
-0,05
-0.029
0.031
0.008
0.015
0.016
0.034
0.016
11001
0.021
40,002
0.054
0.081
0.064
0.009
0.004
0.076
0.110
0.088
0-170
-0,002
0.007
0.037
0.182
7B
0.033
0.029
0.108
40.044
+0.05
0.047
1448
00.33
0.046
0.049
0.025
4,022
0.007
0.097
0.015
40.000
0.005
0.011
40.009
0.037
0.003
0.003
0.083
0.098
0,089
0X015
0,020
0.084
0.124
0.104
0.185
0.0
0.026
0.059
0.203
AAR -oon-oon C riever
AAR con contrli rlever
rever
AAR cont klever
AAR con S r. lever
Retriever
AAK- ANCS
AAR AAR- -o--cont
bge
FIR
contrieve ápx
83
ANCE
bge
bm.25
contrie dpr
05
XAR ANCE
bge
bm.25
contr verrreeee eever
dpr
es
AAR ANCE
bge bm
contr contrieve lever
dpz
e5
AAR ANCE
bge
2m
cont contrieve.
dpr e5
Metric
Top-k
AIIKIMULTTHPPAA
Dataset nane
TRIVIAQA
HOtpOrQA
POPQA
NQ
26

---

Published as a conference paper at ICLR 2025
Metrie
ASePer
Top-k
5
10
Dataset name
Workflow
7B
13B
7B
13B
7B
13B
NAIVE
0.366
0.354
0.389
0.393
0.402
0.424
RETROBUST
N/A
0.623
N/A
0.644
N/A
0.700
2WIRIMULTTHOPQA
FLARE
0.360
0.358
0.369
0.336
0.374
0.344
IRCOT
0.339
0.365
0.364
0.388
0.386
0.405
İTER RETGEN
0.371
0.346
0.420
0.413
0.435
0.446
NAIVE
0.378
0.373
0.421
0.414
0.428
0.442
RETROBUST
N/A
0.537
N/A
0.575
N/A
0.589
HOTPOTQA
FLARE
0.283
0.286
0.290
0.271
0.299
0.277
IRCOT
0.362
0.418
0.409
0.452
0.442
0.466
ITER- RETGEN
0.396
0.380
0.447
0.447
0.465
0.486
NAIVE
0.087
0.089
0.116
0.125
0.128
0.137
RETROBUST
N/A
0.456
N/A
0.464
N/A
0.485
MUSIQUE
FLARE
0.11O
0.140
0.113
0.133
0.115
0.135
IRCOT
0.143
0.131
0.161
0.157
0.176
0.164
[TER RETGEN
0.109
0.106
0.143
0.159
0.156
0.178
NAIVE
0.504
0.489
0.557
0.563
0.560
0.565
RETROBUST
N/A
0.580
N/A
0.605
N/A
0.594
NQ
FLARE
0.333
0.234
0.343
0.222
0.340
0.235
IRCOT
0.457
0.500
0.493
0.534
0.510
0.515
ITER- RETGEN
0.497
0.476
0.540
0.547
0.561
0.559
NAIVE
0.474
0.466
0.512
0.528
0.512
0.535
RETROBUST
N/A
0.493
N/A
0.553
N/A
0.521
PorQA
FLARE
0.328
0.248
0.347
0.244
0.343
0.247
IRCOT
0.426
0.451
0.478
0.496
0.483
0.493
TTER-RETGES
0.462
0.447
0.499
0.512
0.486
0.523
NAIVE
0.677
0.667
0.714
0.727
0.730
0.739
RETROBUST
N/A
0.778
N/A
0.815
N/A
0.811
TRIVIAQA
FLARE
0.555
0.504
0.558
0.486
0.568
0.490
IRCOT
0.597
0.688
0.671
0.722
0.704
0.730
ITER- RETGEN
0.688
0.677
0.730
0.734
0.740
0.751
Table 12: ASePer Benchmark Across Workflow and Parameter Sizes. The colors highlight the best- performing retrievers under each dataset.
27

---

Published as a conference paper at ICLR 2025
D EXPERIMENT DETAILS IN SECTION 5
To demonstrate that our proposed SePer and ASePer effectively integrate with various RAG pipelines, we conduct extensive experiments in Section S
We also aim to show that SePer and
ASePer are mrodule-agnostic within RAG pipelines.
Following the taxonomy proposed by Jin et al. (2024); Gao et al. (022)), modern modular RAG systems consist of various interchangeable and combinable modules, including refiner and reranker. These modules can be adapted or replaced to better target specific downstream tasks, providing greater flexibility and task-specific optimization. We additionally selected the prompt compression (a kind of refiner) and reranker modules for benchmarking and aim to provide a detailed explanation of their mechanisms and roles here.
D.1 PROMPT COMPRESSION
Prompt compression shortens the prompt by filtering redundant and low-value content while ensur- ing the context fits within the model's context length.
Given a large language model (LLM) and an input prompt x let the response generated by the model be denoted as LLM(x). The goal of prompt compression is find compressed prompt x' such that:
D(PLM(())) PLLM(E)) <,
(12)
where PLLM(r) and PLAM(Z)) represent the distributions over the model's responses when prompted with x and x', respectively. These distributions reflect the stochasticity introduced by sampling methods (e.g.. temperature scaling, top-k, or nucleus sampling) during text generation.
Here, ((::)) denotes divergence metric, such as KI. divergence, computed in a semantically mean- ingful space. Since the responses are text, we embed them in a suitable representation space (c.g., sentence embeddings) where these metrics can effectively measure differences meaning and style.
The compression requirement is formalized as:
len(x') Ien(x).
(13)
This ensures that x' retains the semantic and functional equivalence of x, while reducing token length.
We will also present the technical details of the two prompt compression methods we employed.
SELECTIVE-CONTEE Li et al. (2023) ranks and filters lexical units (e.g.. tokens, phrases, or sen- tences) based on their informativeness. Informativeness is measured using self-information, defined for token ze
[(x,) logz P(r. xt-1).
(14)
In practice, self-information is calculated with smaller models for efficiency. Tokens with higher self-information are considered more informative, while redundant tokens have lower scores. To avoid disjoint filtering, tokens are grouped into larger lexical units (e.g., noun phrases or sentences). The self-information of each unit computed by summing the scores of its tokens. Units are ranked by their scores, and percentile-based threshold of p is applied to retain the most informative con- tent.
LONGLLMLINGUA Jiang et al. (2024) aligns closely with RAG use cases, decomposing prompt compression into modular steps:
Coarse-Grained Compression: Documents are ranked by relevance using perplexity condi- tioned the question: F'k N. Σ N. - logp(a,)) que, restrict xdne), ), where higher 7*k values priori-
tize relevant documents.
Fine-Grained Compression: Token-level relevance is evaluated with contrastive perplexity:
s; = perplexity(zz (2x - perplexity(a; 20 xxxx), highlighting critical tokens based on their importance the query.
28

---

Published as a conference paper at ICLR 2025
Adaptive Compression Ratio: Compression budgets are dynamically allocated using: T
max min ((1 21(rx) )ôT 7 11,1 ,0): where higher-ranked documents (I(ra)) receive
lower compression ratios.
Subsequence Recovery: Ensures content integrity by identifying the longest matching sub- string Pley, in the LLM's response, 2) matching it with the maximum common subsequence wi in the original prompt, and 3) replacing response tokens with the original prompt's subse- quence.
Optimization Objective: The overall objective balances output accuracy and compression: ming Da y) A|o|,o, where D& measures the divergence between the original and com- pressed prompts' outputs, and Ả controls the compression tradeoff.
This approach ensures compressed prompts remain concise and informative, optimizing both effi ciency and effectiveness for long-context scenarios.
D.2 RERANKERS
To improve the precision and relevance of retrieved results, our pipeline employs rerankers to reorder coarse retrieval outputs. Below, we describe the underlying mechanisms of rerankers and their role in our system. To ensure clarity, we also briefly outline the retriever's principle and contrast it with the reranker.
We only consider dense retrieval here. The retriever uses a dual-tower architecture, wherein:
Retriever.
Query Encoder: Encodes the query into dense embedding q Rd.
Document Encoder: Encodes each document into a corresponding dense embedding d e Idd.
The similarity between query and a document d; is computed using a dot product:
score(q, d;) q d;, i= L,....,.A
(15)
The retriever selects the top-k documents with the highest scores as candidates. This coarse retrieval process is efficient and scalable because document embeddings can be pre-computed independently and stored, allowing for rapid approximate nearest neighbor (ANN) searches in vector space Douze al. (2024); Johnson et al. (2019); Malkov & Yashunin (2018), which is ideal for large-scale re- trieval. However, this independence of query and document encoding also makes the retriever less sensitive to context, as it cannot fully capture the nuanced interactions between queries and docu- ments.
Reranker. Rerankers are employed to refine the results of coarse retrieval by reordering and filtering the candidate documents based on relevance. To overcome the drawbacks stated above, rerankers use a cross-encoder architecture to jointly encode the query and document, capturing their semantic interactions.
The reranker operates as follows:
Input Preparation: Each query-document pair (q, d,) is concatenated into single sequence, i.c.: [[CLS) q. [SEP], , d, [SEP]] ), where [CLS] and [SEP] are special tokens for encod- ing in Transformer-basee models. This is typical setup for cross-encoder architectures.
Contextual Encoding: The concatenated sequence is input into a transformer (c.g., BERT), which computes joint representation of the query and document. This step enables the model to capture rich contextual interactions, which are absent in retrievers due to their independent encoding process.
Relevance Scoring: A relevance score is computed to quantify the alignment between the query and the document. In a standard cross-encoder setup, the output corresponding to the [CLS] token is passed through a scoring head (e.g., linear layer):
score(g, d,) (h (cLss)
(16)
29

---

Published as a conference paper at ICLR 2025
where reis)] represents the contextual representation of the [CLS] token. Alternatively, some architectures may use pooling methods (e.g., mean or max pooling) over all token representa- tions token-level interactions derive the relevance score.
Reordering and Selection: Based on the computed relevance scores, the candidate documents are reordered, and the top-k items are selected for downstream processing.
The key differences between retrievers and rerankers are summarized in Table 13 While retrievers efficient and suitable for coarse retrieval over large document collections, rerankers excel in precision by capturing query-document interactions.
Module
Retriever
Reranker
Architecture
Dual-tower (independent)
Cross-encoder (joint)
Input
Separate query and document inputs
Concatenated query-document pair
Output
Dot-product score for similarity
Relevance score for each pair
Efficiency
High efficiency, scalable corpus
Costly, only for candidate sets
Interaction
No interaction between pairs
Captures rich semantic interactions
Use Case
Coarse-graine candidate selection
Fine-graing reordering and filtering
Table 13: Comparison between retriever and reranker mechanisms.
D.3 DATASET SELECTION
We select commonly used Single QA and Multi-hop QA datasets for inference to evaluate the per- formance of SePer in different scenarios. The dataset selection guided by the need cover variety of QA tasks, ensuring a more comprehensive evaluation. Wherever possible, we perform inference on the test set; the test set unavailable, we use the dev set instead. We re-sample the datasets, and for datasets with more than 1000 instances, we randomly select 1000 examples for inference. Figure in the appendix presents the basic information of our utilized datasets.
D.4 HYPERPARAMETER SETTING
We conduct experiments using the Llama 2 model series Touvron et al. (2023) from the Meta Llama family, specifically 1.amm-2-11 7b--hhthhhh L1ama-2- 1- and 1ammaa---0bccatt-a20
chat generation, choose the chat versions of these models. To generate various and complete answers of vari- ous kinds for SePer computation, we set the temperature parameter of each model to 1.0, enabled do_sample, and set the maximum tokens for generation 512. For the retrieval corpus, we the DPR version of the Wikipedia December 2018 dataset as our retrieval corpus, following the config- uration we utilize in the RAG framework FA Jin et al. 2024). We experiment with the set of top-k values for retrieval being {1, 5, 10), and follow each method's ' official implementation for the hyper-parameters of different prompt compression methods. For reranker usage, we set the reranker model as BAAT /bge- reraeernneeerree rgee. We set the initial top-k value for retrieval to 20 and then apply the set as (1, 5, 10) for the reranker to choose items, leveraging the reranker's ability to both rank and filter out irrelevant content. We enable half precision when calculating SePer.
Considering that the task involves instruction-followee
30

---

Published as a conference paper at ICLR 2025
D.5 PROMPT DESIGN AND IMPLEMENTATION
The selection of prompts crucial for enabling large language models (LLMs) to understand tasks and produce responses that align with the desired style and requirements. In this work, present two types of prompts: those that generate responses directly without retrieval and those that include references for -aggeeeeed generation. Specifically, leverage the prompts introduced in Jin et al. (2024), which are listed as follows:
Prompt for naive generation
Answer the question based on your own knowledge. Only give the answer and do not output any other words. Question: {question}
Prompt for RAG
Answer the question based on the given document. Only give me the answer and do not output any other words. The following are given documents.,refereeee
Question: {question}
D.6 SYSTEM SPECIFICATIONS FOR REPRODUCTIVITY
Our experiments were conducted on nigh-performance servers, each equipped with either an In- tel(R) Xeon(R) Platinum 8378A CPU 3. 3.00GHz or an Intel(R) Xeon(R) Platinum 8358P CPU 2.60GHz, 1TB of RAM, and 4/6 NVIDIA A800 GPUs with 80GB memory. Machines with 4 GPUs are configured with the SXM4 version, while those with GPUs use the PCIe version. The software environment included Python 3.11, PyTorch 2.4, and NCCL 2.21.5 for reproductivity.
31


Reviews:
[{"review_id": "4qVFxuqvnD", "reviewer": "Reviewer_tcfG", "summary": "This paper proposes a measure of semantic perplexity reduction (SePer) to evaluate the utility of retrieved documents in RAG. The measure reflects the change in the probability of the ground truth answer when given the retrieved document. The authors evaluate the correlation of this measure with respect to machine and human judgments, showing higher correlation than other measures.SePer is then used to evaluate the effect of different factors: model size, number of retrieved document, prompt compression and document reranking.\nThe experiments suggest that SePer can be used to estimate whether a retrieved document is useful.", "questions": "1. How could SePer be used concretely in RAG?\n2. SePer is compared to retrieval annotations to estimate its correlation to the latter. Would it be possible to test if SePer correlates well with the final answer generation, i.e. when a document with higher SePer is provided to the generation process, the generated answer tends to be better?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper uses a well motivated idea to measure the utility of a retrieved document. Similar idea has been used in IR literature quite a long time ago, but it is rarely used in recent studies. Reusing this idea with the current approaches to RAG is interesting.\n2. The demonstration that SePer could be a reasonable measure for document utility is interesting (however, some unclarities - see weaknesses).", "weaknesses": "1. While the general idea is well motivated, its demonstration and tests have could be described more clearly. Some unclear presentation hampers the understanding. For example, the tables and figures are usually not well explained. E.g. In Table 1, it is unclear what the numbers correspond to. SePer reduction is a central concept of the paper. However, there is not a clear definition of it. One may guess that it corresponds to Equation 9 (?), but a clear explanation would help.\n2. Some important details are not well described in the paper. For example, reranker is a means to improve RAG. However, it is unclear how reranking is done. The single sentence that explains it is unclear. The method used for prompt compression may also be more detailed.\n3. The evaluation is limited to testing if SePer correlates to human judgments and how different factors influence SePer measure. It would be interesting to design a method to use SePer in RAG, e.g. to select retrieved documents according to SePer. This would be a more direct evaluation of the utility of SePer.\n4. The paper bears some similarity to Farquhar et al. (2024). This has not been explicitly stated in the paper.\n\nNote after rebuttal: The authors have addressed the above problems. More explanations are added, making the description clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wE76VrNtJD", "reviewer": "Reviewer_E3Wb", "summary": "The authors propose a method for estimating the value of a document in a RAG setting according to its ability to reduce semantic perplexity in the downstream model. The method is compared to ground truth utility and human preferences using measured correlation.  The method is further studied in the context as an evaluation metric.", "questions": "The authors have addressed the above weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "* **Interesting new work in relevance estimation.** Determining the relevance of an item for RAG is an exciting new area of research that moves beyond classical notions of relevance from information retrieval.", "weaknesses": "* **Metric validation.** When proposing a new metric, various criteria must be met. These include validity (e.g., construct, content, convergent, discriminate), reliability, and sensitivity.  Although Section 4.1 provides some evidence of validity, we need more analysis to understand if this would be a strong metric (e.g., reliability, sensitivity).  The remaining experiments in Section 5 are difficult to connect and are relatively informal.  The authors are strongly encouraged to review the literature in metric design, some of which are listed below,\n   * Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 acm conference on fairness, accountability, and transparency, FAccT '21, 375--385, New York, NY, USA, 2021. , Association for Computing Machinery.\n   * Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kıcıman, Noshir Contractor, and Tina Eliassi-Rad. Measuring algorithmically infused societies. Nature, 595(7866):197--204, 2021.\n   * Ziang Xiao, Susu Zhang, Vivian Lai, and Q. Vera Liao. Evaluating evaluation metrics: a framework for analyzing NLG evaluation metrics using measurement theory. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 conference on empirical methods in natural language processing, 10967--10982, Singapore, December 2023. , Association for Computational Linguistics.\n* **Related work in information retrieval.** The authors seem to dismiss much of the information retrieval work that looks at the cognitive process of searching and its connection to relevance.  Since they do cite earlier work (e..g, [Cooper 1973]), I believe this should be considered.  Please look into work from Belkin and related,\n   * Nicholas Belkin. Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, page 133-143, 11 1980.\n   * N. J. Belkin, R. N. Oddy, and H. M. Brooks. Ask for information retrieval: part i. background and theory. Journal of Documentation, 38(2):61-71, June 1982.\n   * N. J. Belkin, R. N. Oddy, and H. M. Brooks. Ask for information retrieval: part ii. results of a design study.. Journal of Documentation, 38(3):145-64, September 1982.\n* **Writing.** This is minor but the authors should consider improving the flow and exposition in the writing.  There seems to be little justification for the Properties in Section 3.2, which could benefit from discussion or citation.  The RQs in Section 5 are introduced also without much discussion of why they are important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ssxqMsy0X5", "reviewer": "Reviewer_Ls69", "summary": "This paper proposes a new measure, SePer, to measure the retrieval utility through the lens of semantic perplexity reduction.  The authors point out the current problems at evaluating RAG effectiveness: assessing retrieval and generation components jointly, which obscures retrievals distinct contribution, or examining retrievers using traditional metrics such as NDCG. To address these limitations, the authors introduce an automatic evaluation method that measures retrieval quality through the lens of information gain within the RAG framework, by proposing Semantic Perplexity (SePer) that captures the LLM’s internal belief about the correctness of the retrieved information. The measure quantifies the utility of retrieval by the extent to which it reduces semantic perplexity post-retrieval. Extensive experiments demonstrate that SePer not only aligns closely with human preferences but also offers a more precise and efficient evaluation of retrieval utility across diverse RAG scenarios.", "questions": "The points outlined in the “weakness” should be clarified in the rebuttal.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tOverall, the motivation of the paper is justified, and the paper is clearly written.  \n\n2.\tThe idea of introducing a new measure, SePer, to evaluate the effectiveness of retrieval in RAG is justified and it seems to be necessary and innovative.\n\n3.\tEvaluation of retrieval utility based on shifts in LLMs’ knowledge distributions sounds reasonable.\n\n4.\tThe experiments show that the new measure not only aligns closer with human annotations but also is more consistent with inference-time situations.\n\n5.\tBoth theoretical analysis and extensive experiments demonstrate that SePer provides a more accurate, fine-grained, and efficient evaluation of retrieval utility. It is also generalizable across a broad range of RAG scenarios.", "weaknesses": "1.\tThe design is mainly based on the entailment model and semantic clustering.  Moreover, the semantic clustering is also based on the entailment threshold, as show in its algorithm: Grouping responses rj into clusters C = {Ck} such that E(ri, rj) ≥ τ within each cluster.  Thus, the entailment model becomes the most critical factor in this measurement.  I am not sure if further discussion is needed with different entailment inference methods and different thresholds and see how they may impact the effectiveness of the SePer measure.  I feel some discussion is needed. \n\n2.\tThe design does not conduct any concrete semantic analysis on different kinds of queries and texts to be retrieved.   It seems such study and discussion could be useful to deepen the understanding although this study (simply) tested on multiple datasets as shown in Table 2.\n\n3.\tThere are quite a few typos in the paper.  For example, “a efficient” should be “an efficient” on lines 349 and 404;line 351: “It’s high correlation score in matching responses and answers also set a solid step …” should be  “Its high correlation scores in matching responses and answers also set a solid step …”; line 399 “results are shown in 3” should be “results are shown in Figure 3”, and others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "51XODrLk6n", "reviewer": "Reviewer_FzvM", "summary": "This paper proposes SePer, an automatic evaluation metric to measure the benefit of retrieval. SePer first samples multiple responses of the input query, clusters them based on the semantic meanings, and estimates the belief distribution conditioned on the current input. It computes the CE between this belief distribution and ground truth as the final information gain. Experimental results demonstrate that SePer can evaluate retrieval utility and align with human preference.", "questions": "- How much does it cost to run SePer during evaluation? The introduction claims that SePer is efficient but there isn't any cost analysis.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is well-formulated and written.\n\n- SePer has a high correlation with human annotations, which is valuable for content evaluation.", "weaknesses": "- The assumption that retrieval utility is uniformly distributed across different reasoning steps may not hold true universally, potentially oversimplifying complex retrieval interactions.\n\n- The need for extensive Monte Carlo sampling and clustering in semantic space may not be feasible for all applications, particularly those with limited resources.\n\n- My major concern is about the experiment section. The selected metrics are evaluating semantic similarity instead of information gain or retrieval utility. Comparing SePer with the baselines does not necessarily mean that it is more competitive in measuring knowledge gain. Answers with high certainty but poor information (e.g., \"I don't know\") can still achieve high metric scores. The experimental results demonstrate that SePer is more about answer quality than information gain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)