EAReranker: Efficient Embedding Adequacy Assessment for Retrieval Augmented Generation
Authors: Dongyang Zeng
Yaping Liu
Wei Zhang
Shuo Zhang
Xinwang Liu
Binxing Fang
Venue: NeurIPS 2025 poster ()
Keywords: Retrieval-Augmented Generation
Embedding-based Reranking
Document Adequacy Assessment
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: With the increasing adoption of Retrieval-Augmented Generation (RAG) systems for knowledge-intensive tasks, ensuring the adequacy of retrieved documents has become critically important for generation quality. Traditional reranking approaches face three significant challenges: substantial computational overhead that scales with document length, dependency on plain text that limits application in sensitive scenarios, and insufficient assessment of document value beyond simple relevance metrics.  We propose EAReranker, an efficient embedding-based adequacy assessment framework that evaluates document utility for RAG systems without requiring access to original text content. The framework quantifies document adequacy through a comprehensive scoring methodology considering verifiability, coverage, completeness and structural aspects, providing interpretable adequacy classifications for downstream applications. EAReranker employs a Decoder-Only Transformer architecture that introduces embedding dimension expansion method and bin-aware weighted loss, designed specifically to predict adequacy directly from embedding vectors. Our comprehensive evaluation across four public benchmarks demonstrates that EAReranker achieves competitive performance with state-of-the-art plaintext rerankers while maintaining constant memory usage ($\sim$550MB) regardless of input length and processing 2-3x faster than traditional approaches. The semantic bin adequacy prediction accuracy of 92.85\% LACC@10 and 86.12\% LACC@25 demonstrates its capability to effectively filter out inadequate documents that could potentially mislead or adversely impact RAG system performance, thereby ensuring only high-utility information serves as generation context. These results establish EAReranker as an efficient and practical solution for enhancing RAG system performance through improved context selection while addressing the computational and privacy challenges of existing methods.

Efficient Embedding Adequacy Assessment for Retrieval Augmented Generation
EAReranker:
Yaping Liu Wei Zhang' Shuo Zhang Xinwang Liu Binxing Fang!
Dongyang Zeng'
Guangghou University, Guangzhou, China
2 National University of Defense Technology, Changsha, China zj zjzdyRe azdyde.gzhu..... edu. en, {ypliu, 1zhang18)@@zhu edu. cn
Abstract
With the increasing adoption of Retrieval- Auggmented Generation (RAG) systems for knowledge-intensive tasks, ensuring the adequacy of retrieved documents has be- come critically important for generation quality. Traditional reranking approaches face three significant challenges: substantial computational overhead that scales with document length, dependency on plain text that limits application in sensitive scenarios, and insufficient assessment of document value beyond simple relevance metrics. We propose EAReranker, an efficient embedding-hasee -aased adequacy as- sessment framework that evaluates document utility for RAG systems without requiring access to original text content. The framework quantifies document adequacy through comprehensive scoring methodology considering verifiability, coverage, completeness and structural aspects. providing interpretable adequacy classifications for downstream applications. EAReranker employs Decoder- Only Transformer architecture that introduces embedding dimension expansion method and bin-aware weighted loss, designed specifically to predict adequacy directly from embedding vectors. Our comprehensive evaluation across four pub- lic benchmarks demonstrates that EAReranker achieves competitive performance with sfate-of-the-art plaintext rerankers while maintaining constant memory usage (~550MBB regardless of input length and processing 2-3x faster than traditional approaches. The semantic bin adequacy prediction accuracy of 92.85% LACC @ 10 and 86.12% LACC@2 demonstrates its capability to effectively filter out inade- quate documents that could potentially mislead adversely impact RAG system performance, thereby ensuring only high-utility information serves as generation context. These results establish EAReranker for enhancing RAG system performance through improved context selection while addressing the computational and privacy challenges of existing methods. The source code EAReranker available in https//gtthbb.oo//zjyy/EEAAR
efficient and practical solution
Introduction
Retrieval-Augmentee Generation (RAG) [ has emerged a pivotal paradigm for enhancing Large Language Models (LLMs) in knowledge-intensive applications. By incorporating external knowledge as context, RAG systems mitigate hallucinations and enhance response accuracy. As these systems evolve, ensuring retrieved knowledge is not merely relevant but genuinely adequate in comprehensively and accurately addressing query requirements has become increasingly critical.
The canonical RAG workflow consists of retrieval and reranking stages 2
5 6 While
contemporary reranking methods effectively capture semantic relationships through cross-attention
"Conesponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Videls
Velp Leteci
Ohirp
Kaudlige for Bestnd
11
Adkquacy Asvmament Nodel tulldig
Qutt-Doa Per Dat
besate Micithes allnd boning
Adbeaty Amware Scors
Erovedi Beoet Aareurr
Ratteewe--uugrrrreee Cereration (RAG] System with Enbodding Adequacy Assstament
Duty
<->
Dary
Aánqaac Fosmou lcon Prddet Intediin
koawae
UMM
LM
Fiberwedhh vievarr inciwart
Namaned
(a) Traditional plantext-
(b) Overall Workflow of EAReranker: Embedding based Document Adequacy Assessment for RAG
based Reranker RAG
Figure 1: Traditional Reranker RAG vs EAReranker RAG.
mechanisms 7 or pre-trained language models [8. their direct dependency on original text introduces significant limitations.
These limitations manifest in three primary challenges. First, substantial and unstable computa- tional overhead arises as computational requirements scale with text length, creating deployment barriers in anbouuce-constrainee environments and performance instability with variable document lengths. Second, dependency on plain text restricts deployment in scenarios where content exposure raises intellectual property or data governance concerns, impeding wider RAG adoption. Finally, a more fundamental constraint lies in the lack of refined content value assessment. Traditional reranking primarily evaluates "relevance," whereas effective RAG systems require assessment of "adequacy," which represents the substantive value and reliability of content in addressing the query yet remains challenging to quantify.
To address these limitations comprehensively, we propose EAReranker, an efficient embedding- -based adequacy assessment model that predicts document adequacy solely from embedding vectors without accessing original text. Operating entirely in vector space, EAReranker leverages the semantic understanding capabilities of pre-trained embedding models perform sophisticated adequacy assessment and reranking.
Our contributions include: (1) An embeddinggoriented adequacy assessment architecture: EAR- cranker employs a Onll Transformer with embedding dimension expansion techniques and bin-aware weighted loss function, eliminating dependence plain text while effectively utilizing embedded knowledge. (2) A principled adequacy assessment methodology: We intro- duce multi-dimennional semantic binning approach that quantifies document utility into discrete interpretable levels, validated through multiple LLMs to ensure consistent assessment quality. (3) Empirical validation of efficiency and effectiveness: Comprehensive experiments demonstrate that EAReranker achieves comparable performance to plaintext models while significantly reducing computational requirements and maintaining consistent efficiency regardless of document length.
2
Related Work
2.1 Language Embedding Models
Text embedding methods represent a fundamental technology in modern natural language processing, mapping textual data into continuous vector spaces where semantic relationships can be captured through geometric proximity. The evolution of these techniques has significantly enhanced the capabilities of information retrieval and ranking systems.
Embedding approaches have advanced from context- context-independee independent word representations sophisti- cated contextual models. Early techniques like Vord2Vec [9] and Glo Ve [10] established foundational vector space modeling principles, while Transformer-based architectures including BERT [1] and RoBERTa 12 later introduced dynamic contextual understanding. The development of sentence- level embedding frameworks such as Sentence-BFi BERT [3 further refined semantic representation by optimizing vector spaces specifically for similarity assessment.

---

Recent architectural innovations have substantially expanded embedding capabilities. Models based on tilingual and multi-domain processing abilities. Advanced architectures including Qwen2-based [17]
[4, such as bge-m3 [T5] and jina-embeddings-v3 [16]. demonstrate robust mul-
Kal.M-embed [8] and Gemma2-baser [19] bge- age-multilingual -emmm22 [15] leverage increased
parameter scales to achieve enhanced semantic modeling and extended context support (20).
The demonstrated capability of modern embeddings effectively encode semantic content sug- gests that embedding vectors alone can sufficiently represent semantic adequacy information. Our EAReranker framework builds upon this foundation, conducting document adequacy evaluation exclusively within the embedding space while maintaining semantic fidelity.
2.2 Language Reranking Models
Document reranking serves as a critical component in RAG for refining initially retrieved can-
didates 2 3
4 5). Traditional approaches utilized lexical heuristics such as BM25 [2]] and
TF-IDF
22 which offer computational efficiency but exhibit limited semantic understanding.
Neural reranking models have introduced sophisticated semantic modeling through various archi- tectural paradigms. Encoder-based approaches, exemplified by Ige-reranker-v2-m3 [15] and jina-
batp--rr-22-bbs--muulliiggg [23], process query-document pairs using token-level cross-attention mechanisms. Decoder-Only architectures, including gegereranker-v2-gemmee [15] and Ib-reranker (8),
leverage autoregressive modeling over combined sequences to capture complex dependencies.
An alternative approach involves directly prompting large language models for ranking tasks, as demonstrated by RankGPT [6] and UPR [24]. While this method offers flexibility, it requires substantial computational resources and full text access.
Current reranking approaches face two significant limitations. First, they require direct access to query and document text, resulting in computational costs that scale with input length. Second, they primarily focus on relevance assessment rather than comprehensive adequacy evaluation for RAG.
Our proposed EAReranker addresses these limitations by operating solely on embedding vectors, enabling efficient computation and privacy preservation while maintaining competitive performance. This approach represents significant advancement in developing practical and scalable solutions for document adequacy assessment in RAG.
3 Embedding Adequacy Assessment Framework
3.1 Framework Overview
Figure connrasts traditional reranker-based RAG systems with our proposed EAReranker approach. While conventional rerankers perform detailed query-document interaction requiring computational resources proportional text length, EAReranker operates exclusively on fixed-dimension embed- ding vectors, enabling efficient adequacy assessment without accessing original text content. This design strategically addresses three critical limitations of existing approaches: (1) Computational invariance: Processing fixed-dinension embeddings decouples runtime and memory requirements. (2) Avoidance of dependence on original text: By operating solely on embeddings, our method mitigates issues arising from direct text exposure or handling. eejeccquacyccetticcc evaluation: The model learns to assess moltti-dimensionll adequacy factors rather than relying solely on relevance.
A critical distinction underlying our work is that between relevance and adequacy. Traditional retrieval systems prioritize glevance-measurinn topical alignment between queries and documents through lexical or semantic similarity. However, for RAG systems performing complex generation tasks, relevance alone is insufficient. Adequacy measures a document's functional urility enabling the generation model to produce complete, accurate, and well-structured responses. For instance, document containing only the query keywords may achieve high relevance scores but provide minimal generation value, while a comprehensive, well-structured explanation may score lower on similarity metrics yet offer substantially higher utility for answer generation. Our adequacy framework evaluates utility.
documents across four complementary dimensions to capture this
3

---

High wiiniblly conam shoors selerrriatee
Stmm
minationonn
gighte
For fam ant "ypedot
D16715
Malti Scaring Crons-Vaiidation
LiM Stom euthon
Seatibrated Adequacy Assessment
Bin- Rtaedd Adleguncy Assensment Prompt
Reranker 0.1894
T
Recamber 0.6472
Fer
Eaery
Oocamers
Sperank
Whthii fin Calibraion
Raranleer
T
0.6147
0.4201
Query- Document Pair Dosa
Roranker Model Normalized Scorina
Figure 2: Pipeline of Adequacy Assessment Annotation.
3.2 Problem Formal Characterization
Traditional document reranking is typically defined as learning a function f(g, d) that evaluates the relevance between the original query text q and document text d, and ranks documents accordingly. We redefine this task as learning a model M that predicts scalar score s' [0. 1] reflecting the "adequacy" of document for query using only their embedding vectors eg E R2d and e, E IR generated by pre-traue pre-trained embedding models:
[gg d) + (Traditional)
MM(e ed) - (Ours),
(1)
where documents {d;} are then reranked based on these predicted adequacy scores s/c-
More precisely, given query embedding ey and collection of document embeddings {ea, ) obtained during the initial retrieval phase, we aim to train a model M(ey, ed,) - 8 that approximates the true adequacy score s; with high fidelity. The final document ranking is determined by sorting the documents in descending order of their predicted adequacy scores, such that:
where (i1, i2, ..., in) represents the indices of the documents after ranking. critical property of our framework is score serretaabiii-dddcummmn with similar predicted scores should demonstrate comparable utility across query-document pairs, enabling meaningful threshold- based filtering.
To effectively assess adequacy from embeddings, our model must exhibit four essential capabilities: extraction of fine-grained semantic information from fixed-dimension vectors; exclusive operation on embeddings without requiring original text; evaluation across multiple adequacy dimensions beyond simple relevance; and computational efficiency invariant document length. These requirements guide our architectural design and implementation strategies detailed in subsequent sections, with Section presenting our annotation methodology and Section describing our model architecture.
4 Adequacy Assessment Annotation
This section establishes formal framework for evaluating document adequacy in RAG systems, transcending traditional relevance metries capture the multidimensional utility of retrieved content as generation context. We introduce a rigorous methodology for quantifying document adequacy through interpretable value classification, providing both theoretical foundations and high-quality labeled datasets for training our proposed EAReranker model. As illustrated in Figure 2 our annotation pipeline integrates semantic bin-based based standards, multi-moded evaluation, and cross-model validation to ensure consistency across diverse query moryydooumment domains.
4.1 Semantic Bin-Based Adequacy Scoring
Adequacy assessment represents fundamental advancement beyond conventional relevance eval- uation. While relevance primarily quantifies semantie proximity between queries and documents, adequacy measures the substantive utility of documents as generation context. This distinction is critical: relevance indicates topical alignment, whereas adequacy evaluates whether a document provides reliable information that comprchensively addresses query requirements.

---

Table 1: Semantic Binning Scheme for Adequacy Scoring.
Semantic
Score Range
Verifiability
Need
Evidence Completeness
Structure
Description
Bin
Coverage
Suitability
Precise
0.900 1.00]
High
High
High
High
Optimal context
Adequacy
High
[0.75, 0.90)
High
High
High
Medium
High-quality
Adequacy
text
Middle
(0.50, 0.75)
Medium
Medium
Medium
Medium
Usable context with
Adequacy
supplementation
Marginal
[0.25, 0.50)
Low
Low
Low
Low
Marginally usable context
Relevance
Weak
[0.10, 0.25)
Very Low
Low
Very Low
Negligible
value
Relevance
context
Irrelevacce
(0100,0.10)
Very Low
Unusable context
We formalize adequacy assessment through four complementary dimensions: (1) Verifiability: Quan- tifies information reliability and factual foundation. High verifiability documents present explicitly cited or well-established facts, while low verifiability documents contain predominantly unsubstan- tiated claims. (2) Need Coverage: Evaluates comprehensive addressing of query requirements. High coverage documents provide complete responses to information needs, whereas low coverage documents may contain superficially relevant terminology without addressing query intent. (3) Evidence Completeness: Assesses logical coherence and supporting evidence. Documents with high completeness present well-structured arguments with sufficient substantiation, while those with low completeness exhibit logical discontinuities insufficient support. (4) Structure Suitability: Measures alignment between document presentation and required content format. High suitability documents present information in readily utilizable formats, while low suitability documents require significant transformation.
We discretize the adequacy spectrum into six distinct bins spanning [0.1]. detailed in Table I Doc- uments in the Precise Adequacy bin ([0.90, 1.00)) represent optimal context with high performance across all dimensions. High Adequacy documents ([0.75, 0.90)) contain all required information with minimal redundancy. Middle Adequacy documents ([0.50, 0.75)) address main issues but require supplementation. Marginal Relevance documents ([0.25, 0.50)) cover only peripheral aspects of the query. Weak Relevance documents ((0.10, 0.25)) provide minimal substantive information. Irrelevance documents (I0.00, 0.10)) have no discernible relationship query requirements.
This structured binning provides clear operational thresholds for RAG systems. Documents scoring above 0.75 can be directly incorporated, while those below 0.25 should generally be excluded to prevent hallucination or inaccuracy in generated content. The granular discrimination between bins enables automated optimization of context selection, significantly enhancing generation quality.
4.2 Multi-Model Semantic Bin Scoring
Constructing high-quality training data necessitates robust annotation methodology. We employ multi-model scoring framework that leverages diverse large language models under principled cross- validation scheme to ensure reliable and consistent scoring across heterogeneous query- docummet pairs. Our framework employs a cross-validation architecture that aggregates assessments from multiple L.Mss with distinct architectures and training paradigms. This approach mitigates indi- vidual model biases and captures diverse evaluation perspectives, enhancing annotation reliability particularly for ambiguous cases.
Effective LLM-based scoring relies on carefully engincered prompts encompassing: (1) Professional role definition with explicit evaluation criteria, (2) Four-dimensional adequacy framework articulation, (3) Precise semantic bin definitions with boundary cases, and (4) Structured input/output formats. We incorporate exemplar
demonstrating assessment criteria across diverse scenarios, enhancing model understanding of nuanced adequacy distinctions.
The scoring process follows Algorithm|] I which implements hierarchical validation mechanism to identify consistent model assessments. The algorithm's 's innovation lies in its combinatorial approach, systematically evaluating all possible combinations of four model scores to find consistent
5

---

subsets. Specifically, it begins with four LLM scores and checks consistency (tolerance 0.2); if unsuccessful. additional models are consulted and all four-score combinations are evaluated find self-consistent subsets. This approach offers several advantages: flexibility in finding reliable assessments without requiring all models to agree, computational efficiency through progressive validation, and effective handling of outlier scores by identifying alternative combinations that provide consistent assessments.
Algorithm 1: Mouti- Semantic Bin Scoring.
Input: documett pair (g, d), LLM models M = {M Mm)
Output: L.M adequacy score 910m
S [Score( Mr q, d) 21, 4}};
/* Initialize with firat scores +/
2 js f- (Man(();
3 if maxxes 8s 1A| Σ 0.2 then
return jr:
for i 5 to to do
S S S UScorce dd}
7
foreach Soubset E Combinations(S, 4) do Mlubbet + Mcan(S
if maX Is Msubset < 0.2 then
return Moubeets
return MSan(S);
4.3 Within-Bin Score Calibration Methodology
While multi-model validation provides reliable bin assignments, we observed that LLMs tend to produce clustered scores within each bin, limiting intra-bin differentiation. To enhance score granularity while preserving semantic boundaries, we developed within-bin calibration methodology that incorporates auxiliary ranking signals from diverse reranking models.
The calibration process begins with normalization of reranker scores across the dataset:
SR, min(sR,)
norm
(3)
R
max ()R) mii (2,,)
These normalized scores are aggregated to produce composite reranking signal:
(s [R]
Sperank
(4)
For documents with LLM-assigned scores 8tm. within bin [4:, h.). the calibration formula applies:
Sealibrated 0.5 ((sitm 4)/(h, 12) Srevank) (h; 1,) + 4.-
This calibration methodology preserves semantic bin boundaries while enhancing intra-bin differenti- ation, transforming discrete score clusters into continuous distributions and maintaining semantic consistency with original assessments.
Through this comprehensive annotation framework, we construct large-scale training datasets with high-quality adequacy scores. The methodology's 's ability to produce consistent, fine-grained ade- quacy annotations while preserving semantic interpretability represents significant advancement in automated document assessment for RAG.
5 Embedding-Based Adequacy Assessment Model
We propose EAReranker, an efficient embedding- based model for document adequacy assessment that operates exclusively in the vector space, as illustrated Figure 3 The model leverages a specialized Decoder-Only Transformer architecture to evaluate adequacy using only query and document embed- ding, eliminating the need for access to original text while maintaining competitive performance with traditional text-based approaches. The architecture consists of two primary components:

---

RMSSNorm
Attontion
RMSSNorm
FFN With SWIGGL
GELU & LN & Drupout
FC FC.512 6112 Drapout
FC. with GELU Drapous
Sigmoiid
Projection
FC 512
Spreit
Xx
XW
(Cess cpa 6p.
Ssars Galb Sap m
Decoder
Figure 3: Overall of Adequacy Assessment Model.
Embedding Dimension Expansion: To enable rich semantic feature extraction from fixed- dimensional embeddings, we transform single embedding vectors into sequential representations through independent projection layers. Given query embedding By and document embedding ed the expansion is performed as:
%/1 (e,) W Vg bas5 e/4, (e.) W Wa b
where each projection layer L and La,; is initialized independently capture distinct semantic aspects. The expanded sequence combines these transformed embeddings with trainable special tokens embedding vectors eçLs and esEp: (ecrs, eg....... yyy,N= eSEP; e',,,.-.,,',,
Transformer Processing: The expanded sequence is processed through stacked Decoder layers with Multi-Head Self-Attention and Feed-Forward Networks. The [CLS] token output passed through three-layer MLP head to predict the final adequacy
The model is trained using bin-aware weighted loss function that emphasizes accurate prediction near semantic bin boundaries:
1 N
LBw MSE Spred)
(Struegi Spred, x)?
where weights w, are computed based on prediction deviation from bin boundaries.
This architecture effectively addresses the key challenges of embedding -ased adequacy assessment while maintaining high computational efficiency and privacy preservation.
Experiments
We present a comprehensive evaluation of EAReranker's effectiveness in embedding- adequacy assessment and reranking. Our experiments assess the model's ranking performance across public benchmarks, adequacy classification accuracy, and computational efficiency, demonstrating its ability to achieve competitive results while operating exclusively embedding vectors.
6.1 Experimental Settings
Our evaluation utilized dataset comprising 1 million query query- document pairs, partitioned into training (80%) and testing (20%) sets. Vector representations were generated using established embedding models: bge-m3 (1024-dimensional) gte-multilingual-bas (gte-base, )i8dddimensionl)) [18. and
jina-embeddings-Vy (jina-v3, 1024-dimensional) [16],
instruct-v1.5 (KaLM, 896-dimensional) [20]. We deliberately excluded ligees-dimeniional models
like NV-Embed-v2 (4096ddimensional) (25 to focus on commonly dimensions in RAG.
The evaluation framework compares EAReranker against multiple baselines: traditional lexical mod- els (BM25 (21), embedding cosine similarity methods, and plaintext- reranking models includ- ing mmultlinggul--errnneerbbb [18), bge-reranker-v22mt [15), jina-reranker-y -a----e--bbbeeeeeeeeeeeee multilingual
23]. and Ib-reranker-vl.0 [8]
EAReranker employs stacked Transformer architecture with 4 layers and an embedding dimension expansion factor of 4, balancing representational capacity with computational efficiency. Training utilized the AdamW optimizer (batch size 256, learning rate le-5) for 50 epochs with early stopping.

---

Table 2: Comparative Case Study:
Traditional Retrieval Scores Versus Adequacy Assessment.
Query
Document
bge-m3
gte-base
bge-r2
gle-rb
Adequacy
Introduce China's s capital.
Beijing City (Beijing). abbreviated as Jing.
0.7305
0.7983
2.4641
0.3608
0.9261
historically known as Yanjing and Beiping
Introduce China's 's capital.
The capital is nation's 's most important politi-
0.6684
0.7723
2.4453
0.9229
0.8749
cal center. Throughout Chinese history ..
Introduce China's 's capital.
Beijing the capital of the People's 's Republic
0.6717
0.7634
0.7334
0.2303
0.5763
of China, categorized as a super first-tier city.
Introduce China's s capital.
Beijing is the capital
0.7858
0.8427
2,4805
0.3511
0,3102
Introduce China's ss capital.
Introduce China's s capital
1.0000
1,0000
10.1562
3,2539
0.1740
The formula for multiplying
1555; 215=10; 355=15; 4
0.6137
0.6300
0.1333
0.3533
0.8137
Two numbers to 10.
Where is Egypt located in A Affica?
0.3278
0.5939
-4.6656
0.6436
0.1523
A. Northeast Southwest
What China's capital city? Beijing
0.6024
0.7688
2.8633
0.6348
0.1893
What China's cupital city?
The railway
0.3977
0.4263
-10.7734
-0.4365
0.0324
6.2 Metrics
Our evaluation framework incorporates both ranking effectiveness and classification accuracy metries:
Ranking Metrics:
Normalized Discounted Cumulative Gain (NDCG@ 10) evaluates ranking quality considering both relevance and position. Mean Reciprocal Rank (MRR) quantifies the position of the felevanl document.
Adequacy Assessment Metries: ACC25 measures the proportion of samples where the absolute error between predicted and true scores is within 0.25, directly reflecting bin classification accuracy. LACC@ 25 and LACC@ 10 evaluate binary classification accuracy at thresholds of 0.25 and 0.10 respectively, indicating the model's ability to distinguish relevant from irrelevant documents. These metrics are particularly significant for RAG systems that require effective filtering mechanisms to prevent noise contamination.
6.3 Dataset
We curated comprehensive adequacy assessment dataset by augmenting the bge-m3-data 15 com- prising diverse querumnn pairs with additional samples sourced from multiple heterogeneous collections 226 27
35 This aggregated dataset spans wide spectrum of domains and query intents, encompassing fact verification, specialized knowledge retrieval, etc.
28
29
30
32
33
34
To ensure annotation quality, we adopted multi-model scoring framework employing seven LLMs 17 40 4T with diverse architectures and training regimes. Each query-document pair scored multiple times, with cross- volidaaaiiotionn and iterative selection reduce model-specific bias and improve consistency, particularly for ambiguous boundary samples.
[36 37 38
39
Table contrastss traditional retrieval scores with our bin-based adequacy assessment across represen- tative examples. Here, bge-r2 and gte-rb denote reranking scores from bge-reranker-v2-nt [55) and gte- www.mmilingua--rerneee-----e
[18), respectively; bge-m3 and gte-base represent cosine similarity scores. The comparison illustrates how traditional methods often conflate lexical or semantic similar- ity with true informational value. For instance, exact query repetition achieves maximal similarity scores but receives low adequacy ratings, reflecting its lack of substantive content. Conversely, documents offering structured, detailed information receive high adequacy despite lower similarity scores, highlighting traditional models' inability to fully capture content utility for generation tasks.
6.4 Performance Experiments
Ranking Effectiveness. Evaluation across four public benchmarks (FEVER 30,
NFCorpus 28
DuRetrieval 42 and T2Ranking (43) reveals consistent performance patterns, as shown in Table B Traditional lexical methods demonstrate the weakest performance, with embedding similarity methods showing significant improvements in NDCG and MRR. Plaintext-based reranking models achieve superior performance. However, FAReranker maintains competitive results within 0. 0.54%-1. 1.30% of the best plaintext model. This performance is particularly noteworthy given that EAReranker

---

Table 3: Ranking Performance on Public Benchmarks (%).
FEVER
NFCorpus
DuRetrieval
T2Ranking
Type
Model
NDCG@ 10 MRR NDCG@ MRR
NDCG@ MRR NDCG@ 10 MRR
Lexical
BM25
48.09
32.97
32.08
43.96
19.47
46.34
35.89
Cosine (gte-base)
92.11
94.19
36.66
56.60
87.54
93.38
84.71
92.60
Emheediig Similarity
Cosine (bge-m3)
81.37
85.64
31.41
52.17
83,96
90.77
81.37
90.30
Cosine (jinavv3)
89.05
91.28
36.44
55,30
83.17
89.72
83,16
91.53
Cosine (KaLM)
86.34
89.65
25.92
39.31
80.23
86.49
79.78
87.61
gle-reranket-buse
94.83
96.12
38.79
59.25
89.37
94.81
87.06
94.37
hge-rerankeer-22mg 2--mm
95.26
96.78
39.41 41
60.83
90.16
95.24
87.85
95.41
Plaintext Rerankers
jina-rerunker-v2
93.74
04
58.96
88.65
93.94
86,34
93.85
1b-reranker-vL.0
95.11
96.44
39.28
60.71
89.82
94.76
87.52
94.93
EAReranker (gte- base)
94.36
14.14
93.91
EAReranker (bge-m3)
94.72
96.21
38.76
59.32
89.57
94.82
87.11
94.68
Embedding Reranker
EAReranker (jina-v3)
94.51
96.08
38.51
59.14
89.23
94.56
86.89
94.32
EAReranker (KaLM)
93.84
95.43
37.65
58.46
88.54
93.95
86.12
93.57
Table 4: EAReranker Adequacy Performance($))
Table 6: Inference Computational Efficiency Comparison.
Embedding
ACC25
LACC@25
LACC@
gase
81.98
84.17
91.32
Model (Max Length)
VRAM(MB)
Inference Time(s)
bge-m3
84.28
86.12
92.85
tieereranker-buse (8K)
1209-2225
0.1697-0.3506
jina-v3
83.52
85.42
92.27
bge-reranker-y2-ntt (8K)
2176-2527
0.1312-0.5468
KaLM
83.09
85.06
91.98
jina-reranker-22 v2 (I
547-603
0.1846 0.266-0.211 2128
Ib-reranker-vl.. (128K)
967-8441
0.3479-5.9219
Table 5: Ablation Performance on bge-m3 (%).
EAReranker (gte-base)
544
0.1111
Configuration
ACC25
LACC@ 10
EAReranker (bge-m3)
550
0.1128
EARcranker (jina-v3)
550
0.1127
Complete model
84.28
92.85
EAReranker (KaLM)
547
0.1124
w/o Dimension Expansion
80.58 (-3.70) 89 (-3.23)
w/o Bin-Aware Loss
81.61 (-2.67) 90.65 (-2,20)
w/o Score Calibration
82.95 (-1.33) 91.98 (-0.77)
processes only exed-dimensiona. embedding vectors rather than full text. Additional experimental information is provided in Appendix B.
Adequacy Assessment Capability. Table presents the model's performance adequacy classifi- cation metrics. EAReranker achieves 84.28% ACC25 using bge-m3 embeddings, with LACC@ 10, reaching 92.85% and LACC@2 25 achieving 86. 12%. These results demonstrate effective discrimina- tion between adequacy levels, particularly in identifying irrelevant or marginal documents that should be filtered from RAG contexts.
Computational Efficiency. Table 6 compares the computational efficiency of EAReranker with traditional reranking models. EAReranker demonstrates significant advantages in resource utilization, maintaining consistent memory usage (>-550MB) regardless of input length, while traditional models exhibit substantial variation (up to 8441MB for 1b-reranker-v1.00
In terms of inference speed, EAReranker processes queries 0. 1s-0. 13s, consistently faster than tra- ditional models, especially for longer documents. Unlike traditional approaches with fixed maximum sequence lengths, EAReranker's processing capability depends only on the underlying embedding model's 's input capacity, providing greater flexibility for varied document lengths.
Component Contribution Analysis. Ablation studies confirm the importance of EAReranker's architectural components, as shown in Table 5 Removing the embedding dimension expansion strategy results in significant performance degradation (3.70% in ACC25, 3.23% LACC @ 10), demonstrating its essential role in extracting fine fine-grained semantic features from compressed embed- dings. Similarly, the bin-aware weighted loss proves crucial for learning precise adequacy boundaries, with its removal causing notable performance reduction (2.67% in ACC25, 2.20% in LACC@ @ 10). Our score calibration methodology also contributes 1.33% improvement ACC25, confirming its effectiveness in enhancing within-bin score granularity.

---

The experimental results collectively validate EAReranker's effectiveness as an embedding- based adequacy assessment model for RAG systems. The model maintains competitive ranking performance while operating exclusively embedding vectors, effectively distinguishes between documents, and significantly reduces computational requirements compared to traditional approaches.
7
Conclusion
This paper introduces a novel framework for document adequacy assessment in RAG systems, shifting from traditional relevance-centric metrics comprehensive multi-dimensional evaluation framework that encompasses verifiability, need coverage, evidence completeness, and structure suitability. Through our semantic binning methodology and multi-model scoring approach, we provide a quantification of retrieved documents' intrinsic utility as generative context.
Our proposed EAReranker operates exclusively embedding vectors without accessing original text, achieving comparable ranking performance to state-of-the-eee plaintext models while significantly reducing computational overhead. This demonstrates the viability of embedding- beasd methods for efficient and plaintext-presenne
RAG deployments in snsourc--consrrinee environments.
Despite promising empirical results, several challenges remain. The current semantic bin segmen- tation and integration of adequacy dimensions rely on empirically-informee heuristic criteria that require further theoretical research and discussion. Additionally, our reliance on multi-model L.MM annotations introduces potential biases that warrant systematic investigation through more robust validation methodologies. Future research directions include refining calibration techniques for enhanced scoring granularity, and adapting the framework for domain-specific applications to im- prove generation factuality. Exploring alternative embedding expansion mechanisms and model architectures may yield further efficiency gains and performance improvements.
In conclusion, this work establishes fundamental theoretical and practical contributions toward embedding-baad based adequacy assessment, advancing the development semantically nuanced, compu- tationally efficient, and trustworthy retrieval-augmenned generation systems.
8 Acknowledgements
This work is supported by National Natural Science Foundation of China (62372124).
References
[1] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, "Retrieval-augnnnnte generation for knowledge- intensive NLP tasks," in Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020, pp. 9459-9474.
[2] N. Hossain, M. Ghazvininejad, and L. Zettlemoyer, "Simple and effective retrieve-edit-renee rerank text generation." in Proceedings of the 58th Anmual Meeting of the Association for Computa- tional Linguistics, 2020, pp. 2532-2538.
[3] G. Geigle, J. Pfeiffer, N. Reimers, I. Vulić, and I. Gurevych, "Retrieve fast, rerank smart: Cooperative and joint approaches for improved cross- modal retrieval," Transactions of the Association for Computational Linguistics, vol. 10, pp. 503-521, 2022.
2022.
[4] M. Glass, G. Rossiello, M. F. M. Chowdhury, A. Naik, P. Cai, and A. Gliozzo, "Re2g: Retrieve, rerank, generate," in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, United Association for Computational Linguistics, 2022, pp. 2701-2715.
States:
[5] S. Kongyoung, C. Macdonald, and I. Ounis, "monoqa: Multi-tase learning of reranking and answer extraction for open retrevelr conversational question answering," in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, pp. 7207-7218.
[6] W. Sun, L.. Yan, X. Ma, P. Ren, D. Chen, H. Ren, Z. Ren, and M. de Rijke, "Is chatgpt good at search? investigating large language models as re-ranking agents," in Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, 2023, pp. 14 918 1888--1930 14937.
10

---

[7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017, pp. 5998-6008.
[8] lightblue. (2025) 1b-rerankeer-.55-vv1... .0. [Online]. Available: hupp://mugginngaaee.oooiihhhhhtt
lb- 1b-reranker-O.S--v...
[9] K. W. Church, "Word2vee," Natural Language Engineering, vol. 23, no. 1, pp. 155-162, 2017. 2017. [10] J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532-1543.
[11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding,"
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), 2019, pp. 4171-4188
[12] Y. Liu, M. Ott, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized bert pretraining approach," " 2019.
[13] N. Reimers and I. Gurevych, "Sentence-bert: Sentence embeddings using siamese bert- networks," in Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2019, pp. 3982-3992.
[14] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, "Unsupervised cross-lingual representation learning at scale." " in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
2020, pp. 8440-8451 8451.
[15] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Liu, C. J. Li, and Z. Yang, "Bge m3-embedding: Multi-lingual, multi-funccionalitn multi-granularity text embeddings through self-knowledge distillation," 2024.
[16] S. Sturua, I. Mohr, M. K. Akram, A. Bhardwaj. J. Kimmig, N. Fiedler, A. Ei-Kishky, M. Seo, J. Chen, M. Sclar, S. Chandrasekar, M. Josifoski, and G. Vassilev, "jna-embeddding-v3: Multi- lingual embeddings with task lora," 2024.
[17] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang. J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan. Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, "Qwen2.5 technical report," 2024.
[18) Z. Li, X. Zhang, Y. Zhang et al., "Towards general text embeddings with multi-stage contrastive learning," " 2023.
[19] G. Team, M. Rivicre, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momehev, M. Hoffman, S. Thakoor, J.-B. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, A. Parrish, A. Ahmad, A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. WU, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Choquette-Choo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozinska, D. Herbison, E. Bandy, F. Wang, E. Noland, E. Moreira, E. Senter, E. Eltsshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, G. Martins, H. Hashemi, H. Klimczak-Plucinsee H. Batra, H. Dhand, I. Nardini, J. Mcin, J. Zhou, Svensson, J. Stanway, J. Chan. J. P. Zhou, J. Carrasqueira. J. Iljazi J. Becker, J. Fernandez, J. van Amersfoort, J. Gordon, J. Lipschultz,
J. Newlan, J. ycong Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifre, L. Heuermann, L. Lago, L. McNealus, L. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Görner, M. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. Bardoliwalla, N. Devanathan, N. Dumai, N. Chauhan, O. Wahltinez, P. Botarda, P. Barnes, Barham, P. Michel, P. Jin, P. Georgiev, P. Culliton, P. Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. Carthy, S. Cogan, S. Perrin,
11

---

S. M. R. Arnold, S. Krause, S. Dai. S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisky, T. Doshi, V. Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, 7. Gong. Z. Wei, V. Cotruta, Kirk, A. Rao, M. Giang, L. Peran, Warkentin, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev, "Gemma 2: Improving open language models at practical size,"
2024. [Online]. Available: https://ariiv.orgaab///000
[20] X. Hu, Z. Shan, X. Zhao al., "Kalm-embedding: Superior training data brings a stronger embedding model," 2025.
[21] S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford, "Okapi at TREC-3," in Proceedings of the Third Text REtrieval Conference (TREC-3). NIST, 1995, pp. 109-126.
[22] D. A. Grossman and O. Frieder, Information Retrieval: Algorithms and Heuristics, 2nd ed., ser.
The Information Retrieval Series. Dordrecht: Springer, 2004, vol. 15.
[23]
jinaai. (2024)
[Online]. Available: httpss//nugggingfacc.co/
immai/jina-reranker v2rrrakkker-2--baa -ase- uuurrmmmkr---bbb----llllllualllu
[24] D. Sachan, M. Lewis, M. Joshi, A. Aghajanyan, W.-t. Yih, J. Pineau, and L. Zettlemoyer, "Improving passage retrieval with zero-shot question generation," " in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Compu- tational Linguistics, 2022, pp. 3781-3797.
[25] C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping, "Nv-embed: Improved techniques for training lims as generalist embedding models," arXiv preprint arXiv:2405.17428 2024.
[26] H. Wachsmuth, M. Trenkmann, B. Stein, and G. Engels, "A review corpus for argumentation analysis," in Computational Linguistics and Intelligent Text Processing, ser. Lecture Notes in Computer Science, vol. 8404. Springer Berlin Heidelberg, 2014, pp. 115-127.
[27] D. Hoogeveen, K. M. Verspoor, and T. Baldwin, "Cqadupstack: A benchmark data set for community question-answering research," in Proceedings of the 20th Australasian Document Computing Symposium, ser. ADCS '15. Association for Computing Machinery, 2015.
[28] V. Boteva, D. Gholipour, A. Sokolov, and S. Riezler, "A full-text learning to rank dataset for medical information retrieval," in Advances in Information Retrieval, ser. Lecture Notes in Computer Science, vol. 9626. Springer International Publishing, 2016, pp. 716-722
[29] M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, "Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension," " in Proceedings of the 55th Annual Meeting of the Association for Computationnal Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2017, pp. 1601-1611.
[30] J. Thorne, A. Vlachos, C. Christodoulopoolls, and A. Mittal, "Fever: a large-scale dataset for fact extraction and verification," in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers). Association for Computational Linguistics, 2018, pp. 809-819.
[31] M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk et al., "Financial opinion mining and question answering," 2018. [Online]. Available: hhtpss
/sites. wtt.s/gooloooooooo/wig/////////e
[32] T. Diggelmann, J. Boyd-Graber, J. Bulian, M. Ciaramita, and M. Leippold, "Climate-fever: A dataset for verification of real-world climate claims," 2020.
[33] D. Wadden, S. Lin, K. Lo, L. I.. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi, "Fact or fiction: Verifying scientific claims," in Proceedings the 2020 Conference on Empirical Meth- ods in Natural anguage Processing (EMNLP). Association for Computational Linguistics, 2020, pp. 7534-7550.
[34] E. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, E. Kanoulas, K. Lo, J. Mott, L. Scibek, and M. Thorbllae,, "Trec-covid: Constructing pandemic information retrieval test collection," ACM SIGIR Forum, vol. 54, no. 1, pp. 1-12, 2021.
[35] Q. Works, "Quiz works." " [Online]. Available: https:///qiiz wttp:/iii/ooooossoko/a/oooo/aooe
12

---

[36] OpenAI, "Gpt-40 system card," 2024.
[37]
DeepSeek-AI, "Deepseek-v3 technical report," 2024.
[38] THUDM, "Chatglm: A family of large language models from glm-130b to glm-4 all tools,"
2024.
[39] G. DeepMind. "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context," 2024.
[40] Meta, "Llama 1.3-70b-instruct, 2024. [Online]. Available: hutps://huginnffoe.commttt Ilama/ Llama-3.3 3.3na-3.3-77 70B---00--ssttccc
[41]
Anthropic, "Introducing the next generation of claude," 2024. [Online]. Available:
[42] Y. Qiu, Li, Y. Qu, Y. Chen, Q. She, J. Liu, H. Wu, and H. Wang, "DuReader-retrieval: A large-scale Chinese benchmark for passage retrieval from web search engine." in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Gold- berg, Z. Kozareva, and Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 5326-5338.
[43] X. Xie, Q. Dong, B. Wang, F. Lv, T. Yao, W. Gan, Z. Wu, X. Li, H. Li, Y. Liu, and J. Ma, "T2ranking: A large-scale chinese benchmark for passage ranking," 2023.
[44] J. Chen, N. Wang, C. Li, B. Wang, S. Xiao, H. Xiao, H. Liao, D. Lian, and Z. Liu, "AIR-bench: Automated heterogeneous information retrieval benchmark," " in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 19 022.20022. [Online]. Available: https//accaantoooogg.og//222.222
982/
[45] T. Ko" cisky, Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette, "The NarrativeQA reading comprehension challenge." Transactions of the Association for Computational Linguistics, vol. TBD, p. TBD, 2018. [Online]. Available: httips://BD
[46] N. Houlsby, A. Giurgiu, Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly, "Parameter-efficient transfer learning for NLP," in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09-15 Jun 2019, pp. 2790-2799.
[47] R. Zhang, J. Han, C. Liu, Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y. Qiao, "Llama- adapter: Efficient fine-tuning of language models with zero-init attention," arXiv preprint wrXiv:2303.16199 2023.
[48] S. MacAvaney, F. M. Nardini, R. Perego, N. Tonellotto, N. Goharian, and O. Frieder, "Efficient document re-ranking for transformers by precomputing term representations." in Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Retrieval, ser. SIGIR '20. New York, NY, USA: Association for Computing Machinery, 2020,
p. 49-58.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction (L45-58) describe the proposed EAReranker, its benefits (efficiency, no reliance plaintext) and the key contributions (embedding oriented architecture, adequacy methodology).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper includes a dedicated discussion of limitations and challenges in the Conclusion section (Section 7, L310-317), outlining areas for future research such as refining empirical heuristics in annotation and investigating LLM annotation biases.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not center on formal theoretical results or new theorems
but rather on the design and empirical evaluation of neural architecture and annotation methodology.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Section 6.1 and Appendix provide detailed information on experimental settings, datasets used, model architecture, hyperparameters, optimizer details, training epochs, and metries, sufficient for reproducing the described experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: This research utilizes established public datasets in conjunction with open- source embedding models. The complete implementation code has been made available in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cr
Sealiicggudes/Codeffe
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.iioiiioiooooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 6.1 and Appendix B.1 provide details data splits, hyperparameters, optimizer, training epochs, and hardware/software configuration.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard deviations over runs with different random seeds for the main adequacy assessment metrics in Appendix B.6 (Table 10).
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix B.1 specifies the technical hardware (GPU, CPU, memory, OS) used for experiments (lines 521-523), and Table 6 details VRAM usage and inference time for different models, including EAReranker (lines 282-289).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte:://nneui....eee htc//ubbiic/thhhicsiiiedeiiiii
Answer: [Yes]
Justification: The work is conducted on public datasets, adheres to responsible data handling (no private text exposed). No ethical violations are apparent.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
17

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both the benefit of improved efficiency and privacy in RAG systems well as the potential risks (related to mislabeling or bias in adequacy assessment, incorrect filtration) in several sections (introduction, conclusion, and limitations).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not state that high-risk assets such powerful generative models or sensitive datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification:
The paper provides references to all used datasets and models (see References [15-41]), credits their sources, and discusses licensing and terms of each case.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssii thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces new, annotated adequacy dataset. Collection and annotation procedures, binning schemes, score distributions, and annotation protocols are given in Sections 4 and B.2, with codes in supplemental material.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmdel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethies, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
The application of multiple LLMs for semantic bin annotation is front-and-
center in the methodology (see Section 4.2, Appendix A.I-A.3), Prompt engineering, model aggregation, and LLM selection are described in detail.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-standard components.
Please refer to our LLM policy bobtts://neurre cc/Conferences// /LLM) for what should or should not be described.
20


Reviews:
[{"review_id": "Vn35xStiwj", "reviewer": "Reviewer_mAVb", "summary": "This paper introduces a novel embedding-based re-ranking method (adequacy as the score to re-rank), EAReranker, to address three challenges that traditional re-ranking approaches face: computational overhead scaling with document length, dependency on plain text, and assessment of document value beyond relevance metrics. EAReranker first expands the embeddings from an existing embedding model (such as bge-m3) using multiple independent projection layers, and then trains L stacked Transformer Decoder layers with the adequacy scores obtained from multiple LLMs through their adequacy assessment annotation framework. The effectiveness of EAReranker is demonstrated through examining its ranking performance (on 2 English retrieval datasets from BEIR and 2 Chinese retrieval datasets from C-MTEB) and its adequacy assessment accuracy (on the curated datasets based on bge-m3-data).", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The proposed method is novel and inspiring. The three challenges that EAReranker aims to address are practical in real-world scenarios.\n- Evaluation results on 4 retrieval datasets are convincing. EAReranker's ranking performance is competitive with those of superior plain-text re-rankers.\n- The paper is well-written, and the structure of the paper is clear. The figures presented in the paper are also helpful for clearly understanding the proposed method. \n\n# Weaknesses \n- The experimental settings on in-domain / out-of-domain are not presented in detail. I'm worried about the generalizability of EAReranker. Many existing works [1,2] usually use MSMARCO [3] as the training dataset and use the other 14 datasets in BEIR [4] as the evaluation datasets. I'm curious about the performance of EAReranker under this setting. **Another option** is using the well-trained EAReranker to perform evaluation on some other OOD datasets, such as AIR-Bench [5].\n- One main challenge that EAReranker aims to address is the computational overhead scaling with document length. The 4 retrieval datasets used in the experiment all have relatively short documents in their corpora. MLDR [6] and NarrativeQA [7] are two long-document retrieval datasets used in the experiments of BGE-M3 [8]. Experimental results on these datasets would increase the soundness of the paper.\n- The ablation study of the \"within-bin score calibration methodology\" introduced in Section 4.3 is missing.\n\n[1] RepLLaMA: https://arxiv.org/pdf/2310.08319 \\\n[2] Llama2Vec: https://aclanthology.org/2024.acl-long.191.pdf \\\n[3] MSMARCO: https://arxiv.org/pdf/1611.09268 \\\n[4] BEIR: https://arxiv.org/pdf/2104.08663 \\\n[5] AIR-Bench: https://arxiv.org/pdf/2412.13102 \\\n[6] MLDR: https://huggingface.co/datasets/Shitao/MLDR \\\n[7] NarrativeQA: https://arxiv.org/pdf/2310.19923 \\\n[8] BGE-M3: https://arxiv.org/pdf/2402.03216", "was_revised": true, "final_justification": "I have no further technical questions to the paper. I am inclined to rate it as a weak accept as reflected by my original score.", "rebuttal": "## W1: Generalizability on Out-of-Domain Datasets\n\nFollowing your excellent suggestion, we conducted zero-shot evaluations on AIR-Bench, a challenging benchmark for real-world retrieval systems. We evaluated on wiki_zh and healthcare_en tasks following the official BM25 initial retrieval protocol:\n\n**Table 1: Zero-shot Evaluation on AIR-Bench (NDCG@10)**\n\n| Model | wiki_zh | healthcare_en |\n|-------|-----------|-----------------|\n| bge-reranker-v2-m3 (Plaintext) | 63.51 | 53.76 |\n| gte-reranker-base (Plaintext) | 64.16 | 47.16 |\n| Cosine (bge-m3) | 63.52 | 49.05 |\n| Cosine (gte-base) | 61.86 | 47.48 |\n| EAReranker (bge-m3) | 64.76 | 52.81 |\n| EAReranker (gte-base) | 63.03 | 47.62 |\n\nThese results demonstrate EAReranker's strong generalization capability across domains and languages. Our embedding-only approach remains competitive with plaintext rerankers while preserving the efficiency and privacy advantages detailed in our paper. This addresses the generalizability concern by showing consistent performance on unseen OOD datasets.\n\n## W2: Performance on Long-Document Retrieval\n\nThank you for this critical suggestion. We have conducted additional experiments on NarrativeQA, a benchmark specifically designed for evaluating long-document retrieval:\n\n**Table 2: Ranking Performance on NarrativeQA (NDCG@10)**\n\n| Model | NDCG@10 |\n|-------|---------|\n| Cosine (bge-m3) | 48.72 |\n| EAReranker (bge-m3) | 57.41 |\n\nThese results confirm that EAReranker effectively handles long documents while maintaining its key efficiency advantage. In RAG systems, document embeddings are typically pre-computed and stored in vector databases. EAReranker leverages these existing embeddings, eliminating the need to re-process long texts for each query. Unlike plaintext rerankers that incur computational costs proportional to document length, EAReranker maintains constant inference time and memory usage regardless of the original document size, while still achieving strong ranking performance.\n\n## W3: Ablation Study for Score Calibration\n\n**Our Response:** We appreciate this observation and have conducted the requested ablation study comparing our calibrated scores ($s_{calibrated}$) against uncalibrated LLM scores ($s_{llm}$):\n\n**Table 3: Ablation Study of Score Calibration Methodology**\n\n| Training Data | ACC25 | LACC@10 |\n|---------------|-------|---------|\n| Calibrated Scores (full model) | **84.28%** | **92.85%** |\n| Uncalibrated LLM Scores | 82.95% (-1.33) | 91.98% (-0.87) |\n\nThe results confirm that our calibration methodology ($s_{calibrated} = 0.5 \\times ((s_{llm} - l_i)/(h_i - l_i) + s_{rerank}) \\times (h_i - l_i) + l_i$) provides a more effective training signal. This enables the model to better distinguish fine-grained differences in document adequacy within semantic bins, improving both accuracy and filtering capability.\n\nWe thank the reviewer again for their valuable feedback. The additional experiments have strengthened our work by validating EAReranker's generalization ability, long-document performance, and calibration methodology effectiveness. These results have been included in the appendix of our revised paper."}, {"review_id": "enkrM2xans", "reviewer": "Reviewer_oBLF", "summary": "The authors propose a methodology for ranking measuring both the relevance and adequacy of documents to be input to a reader in a RAG pipeline. They propose an annotation process and a predictor for this adequacy often finding that their approach can be more efficient while maintaining effectiveness.", "questions": "The pipeline relies entirely on LLM agreement; no human annotation is described. Was any manual spot-checking performed to confirm bin labels, especially for boundary cases?\n\nCould the authors rerun key experiments with multiple seeds and provide 95 % confidence intervals or paired tests against baselines?\n\nAlgorithm 1 accepts a subset when all four scores lie within 0.2 of the mean. Why was 0.2 chosen, and how robust is the dataset size/quality if this tolerance is tightened or relaxed?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The problem addressed is important as cutoff values presented in works become increasingly shallow.\n- Writing is clear.\n- Several benchmarks are utilized across multiple domains.\n\nWeaknesses\n- The authors should clarify their definition of adequacy, the notion of a comprehensive set of information would appear to better relate to recall. The use of accuracy doesnt appear to assess the criteria from this approach is motivated.\n- The methodology in dataset curation appears to apply a point-wise scoring method for how comprehensive each text is as opposed to each ranking, this feels largely identical to the criteria used in relevance judgements (See Umbrella[1] for automated judgements and DL-19 [2] for human criteria).\n- The authors note a multifaceted evaluation process however it is ambiguous how effectiveness is further decomposed beyond accuracy-based measures.\n- The work is effectively an adapter and thus should be compared, see preTTR [3] or more generally model adapters [4, 5]\n\nReferences:\n\n[1]: Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N. and Lin, J., 2024. \nUmbrela: Umbrela is the (open-source reproduction of the) bing relevance\n assessor. *arXiv preprint arXiv:2406.06519*.\n\n[2]: Craswell, N., Mitra, B., Yilmaz, E., Campos, D. and Voorhees, E.M., 2020. Overview of the TREC 2019 deep learning track. *arXiv preprint arXiv:2003.07820*.\n\n[3]: MacAvaney, S., Nardini, F.M., Perego, R., Tonellotto, N., Goharian, N. \nand Frieder, O., 2020, July. Efficient document re-ranking for \ntransformers by precomputing term representations. In *Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval* (pp. 49-58).\n\n[4]: Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M. & Gelly, S.. (2019). Parameter-Efficient Transfer Learning for NLP. <i>Proceedings of the 36th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 97:2790-2799 Available from https://proceedings.mlr.press/v97/houlsby19a.html.\n\n[5]: Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P.,\n Li, H. and Qiao, Y., 2023. Llama-adapter: Efficient fine-tuning of \nlanguage models with zero-init attention. *arXiv preprint arXiv:2303.16199*.", "was_revised": true, "final_justification": "I feel the authors have made a good effort to address my concerns", "rebuttal": "## W1, W2 and W3: Clarification of \"Adequacy\" versus \"Relevance\"\n\nWe thank the reviewer for raising this critical point. We acknowledge the need to better articulate the distinction between the \"relevance\" commonly measured in IR and the \"adequacy\" we propose for RAG systems.\n\nThe core distinction is that **relevance** measures *topical alignment*, while **adequacy** measures a document's *functional utility* for a downstream generation task. A document can be highly relevant but inadequate. For example, consider a medical query about a specific disease treatment. A short patient case file might be highly relevant from a keyword perspective, but a comprehensive medical guideline or a well-structured wiki page would be far more *adequate*, providing the necessary background, evidence, and detailed information for a generative model to produce a reliable and complete answer.\n\nOur concept of adequacy is designed to capture this utility. As another example, for a complex query like *\"What are the latest regulatory compliance requirements for international data transfers across EU, UK and APAC regions for financial services?\"*, a relevant document might simply discuss \"EU data transfer regulations.\" However, an *adequate* document must be verifiable, comprehensive enough to cover the multifaceted aspects of the query (EU, UK, APAC, financial services), and structured in a way that the RAG model can synthesize a correct and actionable response.\n\nOur methodology, as detailed in **Section 4.1** of our submission, operationalizes this by evaluating documents on their ability to provide verifiable, comprehensive, and well-structured information. The examples in **Table 2 (\"Comparative Case Study\")** further illustrate this. For the query \"Introduce China's capital,\" the document that simply repeats the query (\"Introduce China's capital\") achieves perfect cosine similarity (1.0) but is correctly assigned a very low adequacy score (0.1740) because it provides zero utility. Conversely, a detailed paragraph receives a high adequacy score (0.9261) even with a lower similarity score. This demonstrates that our adequacy assessment successfully moves beyond simple topical relevance.\n\n## W4: Comparison to Model Adapters\n\nWe appreciate the reviewer pointing out these valuable related works. We agree that EAReranker shares the goal of parameter efficiency with adapters and query-time efficiency with methods like preTTR. We would like to clarify the key distinctions that position our work as a novel approach.\n\n* **Comparison with general model adapters:** While conceptually similar in aiming for efficiency, our approach differs fundamentally in its architecture and point of application.\n  * **Traditional adapters** are small neural modules inserted *between the layers of a large pre-trained model* (e.g., BERT). They are trained while the base model's weights remain frozen.\n  * **EAReranker**, in contrast, is a **standalone, post-hoc module**. It operates on the *output* of a pre-trained embedding model (i.e., the embedding vectors) and does not require access to the internal architecture of the embedding model itself. This makes our approach more flexible and modular, as it can be paired with *any* black-box embedding model without modification.\n\n* **Comparison with preTTR:** preTTR is an excellent work on optimizing transformer re-rankers, and we thank the reviewer for the reference. Both preTTR and our work use pre-computation to accelerate query processing. However, they operate on different input modalities, which leads to different trade-offs:\n  * **preTTR** operates on **text**. It pre-computes contextualized representations for each *token* in a document. At query time, it loads these numerous token vectors and executes the remaining transformer layers on the combined query-document text. Its computational cost is still dependent on the length of the document.\n  * **EAReranker** operates on **embeddings**. It works with a single, fixed-size vector for the entire document. This fundamental design choice provides two key advantages, which are central to our paper's contributions:\n    1. **Constant Computational Profile:** As shown in **Table 6**, EAReranker's inference time and memory usage (∼550MB) are constant and independent of the original document's length. This is a critical feature for systems handling documents of highly variable lengths, where text-based models like preTTR would see fluctuating resource usage.\n    2. **Inherent Privacy:** By relying solely on embedding vectors, our method never requires access to the original plaintext. This makes it suitable for privacy-sensitive applications where raw content cannot be processed by a third-party reranker.\n\nWhile a direct experimental comparison is challenging due to these architectural differences, we believe this highlights the unique value proposition of our embedding-only approach. We will add a more detailed discussion in the appendix to better contextualize EAReranker with respect to these important efficiency-focused methods.\n\n## Q1: Human Annotation and Spot-Checking\n\nThank you for this important question. We apologize for the omission in our description. While our goal was to build a scalable annotation pipeline, the process was not entirely automated and did include a manual verification step.\n\nThe authors performed manual spot-checking on approximately 1,000 samples generated by the LLM scoring framework. This process was crucial for validating the quality of the labels and refining our methodology. The insights gained from this manual review helped us iterate on and improve the prompts used for the LLMs (as described in **Section 4.2**) to ensure they captured the nuances of adequacy more accurately. This human-in-the-loop step was vital for ensuring the overall quality and reliability of our final dataset.\n\n## Q2: Statistical Significance\n\nThis is an excellent suggestion to enhance the rigor of our findings. We confirm that the results reported in the paper are stable across multiple runs. To demonstrate this, we trained our best-performing model variant (EAReranker with bge-m3 embeddings) five times with different random seeds. The mean and standard deviation for our primary adequacy metrics are reported below:\n\n| Metric      | Mean (%) | Std. Dev. (%) |\n|-------------|----------|---------------|\n| **ACC25**   | 84.28    | 0.31          |\n| **LACC@25** | 86.12    | 0.28          |\n| **LACC@10** | 92.85    | 0.24          |\n\nAs shown, the standard deviations are very low, confirming that our training process is robust and the reported results are reliable and representative.\n\n## Q3: Hyperparameter in Algorithm 1\n\nThis is a very insightful question about a key hyperparameter in our data annotation pipeline. The tolerance threshold of 0.2 was chosen empirically to balance dataset quality and scale.\n\nOur semantic adequacy bins, as shown in **Table 1**, have a width of 0.15 to 0.25. A tolerance of **0.2** ensures that the scores from the four LLMs in an accepted group are highly likely to fall within the *same semantic bin*, preventing major disagreements from polluting the label. We found that a stricter tolerance (e.g., <0.1) significantly reduced the dataset size by rejecting many valid but nuanced cases, while a looser tolerance (e.g., >0.3) introduced noticeable noise from inter-bin disagreements that harmed model performance. Therefore, the 0.2 threshold represents a pragmatic choice to maintain high inter-rater agreement at the semantic bin level, which is critical for our task, without overly constraining the size of the training data.\n\nWe believe these clarifications address the reviewer's concerns and underscore the contributions of our work. We are confident that our paper presents a novel and practical solution for an important problem in RAG systems. We thank the reviewer again for their time and valuable feedback."}, {"review_id": "Ctu4due8xe", "reviewer": "Reviewer_ibrT", "summary": "The paper points to the issues of high computational overheads, dependency on plain text and insufficient assessment of document value in current RAG systems. To address these issues the authors propose to sue Decoder only architectures which takes embedding vectors as input where the embedding vectors are encodings of the input as given by a retrained embedding vector. The paper also claims to have constructed a new adequacy assessment metric.", "questions": "Ina addition to my questions posed in the weaknesses please explain exactly how your proposed architecture mitigates issues of computational overheads and privacy preservation?\n\nDescribe the adequacy metric mathematically. How does each component relate to and assure the four requirements specified in Section 4.1?", "limitations": "No", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n1. The paper considers an important problem\n2. New dataset is curated\n\n**Weaknesses**\n1. The paper keeps on talking about the high level ideas abut does not provide a concrete methodology to address the discussed issues. As an example see Section 31. which I again repetition of what was said in the Introduction but it is not clear how the goals are actually achieved. I also do not see any mathematical description of the adequacy metric are at an extremely high level. Algorithm 1 which seems to be important in calculating some metric is not even explained in the text.\n2. One of the issues considered is privacy. I do not see how just using pertained encoders to separately extract embeddings solve the issue. What is the threat model there? What if the attacker also has access to the pre-trained encoder? Why using pertained encoder with plain text is safer than the whole system?\n3. I do not see much contribution of the work except a new dataset. Using embeddings instead of plain text is straightforward. it is not even directly clear how it solves the issues of computational overhead and privacy.", "was_revised": true, "final_justification": "Based on authors' response I have slightly increased my score. Unfortunately it still remains below acceptance threshold as key details are missing from the paper. I believe after addition of these details the paper requires a new review.", "rebuttal": "We sincerely thank you for your time and for providing us with insightful and constructive feedback on our paper. Your comments have highlighted areas where further clarification can strengthen the paper. We have carefully considered each point and provide detailed responses below.\n\n## W1: Clarity of Methodology\n\nWe provide the following concrete details to clarify our methodology.\n\n* **Formal Problem Characterization:** We will move beyond the high-level comparison and provide a formal problem characterization. We define the task mathematically, shifting from the traditional reranking function $f(q, d)$ (which operates on plaintext query $q$ and document $d$) to our proposed adequacy assessment model $M(e_q, e_{d_i}) \\to s'_i$, which operates exclusively on embedding vectors $e_q$ and $e_d$. We will explicitly state the four essential capabilities our model must possess to achieve this: (1) extracting fine-grained semantic information from fixed-dimension vectors; (2) operating exclusively on embeddings; (3) evaluating multi-dimensional adequacy beyond simple relevance; and (4) maintaining computational efficiency invariant to document length.\n\n* **Detailed Explanation of Algorithm 1:** Algorithm 1 implements a hierarchical validation mechanism to ensure annotation quality:\n  * **Process:** It begins by scoring a query-document pair with an initial set of 4 LLMs. It calculates the mean score $\\mu$ and checks if all individual scores fall within a tight tolerance (e.g., $|\\text{score} - \\mu| \\le 0.2$).\n  * **Combinatorial Validation:** If consistency is not achieved, the algorithm progressively incorporates scores from additional LLMs (up to a total of `m`). The core innovation lies in its combinatorial approach: at each step, it systematically evaluates all possible subsets of 4 model scores (`Combinations(S, 4)`) to find any self-consistent group.\n  * **Rationale:** This method is more robust than a simple average, as it can identify a reliable consensus even if a minority of models produce outlier scores. This ensures the high quality and consistency of the annotated dataset used to train EAReranker.\n\n## W1 and Q2: Mathematical Description of Adequacy Metric\n\nWe provide a more specific definition and examples of our adequacy assessment framework.\n\n* **Multi-Dimensional Adequacy Framework:** We formalize adequacy through four pillars: **Verifiability**, **Need Coverage**, **Evidence Completeness**, and **Structure Suitability**. **We clarify that our approach does not score each dimension independently but rather instructs the LLMs to holistically evaluate adequacy by simultaneously considering these factors, as they are inherently interconnected in practical RAG applications.** For example, a document that has high *Need Coverage* but low *Verifiability* (i.e., it addresses the query but with unsubstantiated claims) would receive a low overall adequacy score. Table 1 links score ranges to the required qualitative levels on these dimensions.\n\n* **Mathematical Formulation of Score Calibration:** We will detail the two-stage process for generating the final ground-truth score $s_{i}$.\n  1. **Stage 1: LLM-based Binning:** A score $s_{\\text{llm}}$ is generated via Algorithm 1.\n  2. **Stage 2: Within-Bin Calibration:** To enhance score granularity, we calibrate $s_{\\text{llm}}$ using signals from a set of plaintext rerankers. This is formalized by the equation below:\n     $s_{\\text{calibrated}} = 0.5 \\times \\left( \\frac{s_{\\text{llm}} - l_i}{h_i - l_i} + s_{\\text{rerank}} \\right) \\times (h_i - l_i) + l_i$\n   \n     Here, $[l_i, h_i]$ is the score range of the bin assigned by the LLM, and $s_{\\text{rerank}}$ is a normalized, composite score from multiple plaintext rerankers. This equation demonstrates how we preserve the LLM's high-level adequacy judgment (the bin) while injecting fine-grained ranking information.\n\n## W2 and Q1: Computational Overheads and Privacy Preservation\n\n* **Explanation of Computational Overhead Mitigation:**\n  * **Constant Complexity:** Traditional cross-encoder rerankers process concatenated `[CLS] query [SEP] document` sequences. Their computational complexity and memory usage scale with document length. In contrast, EAReranker's input is a sequence of *expanded* embeddings with constant length $2N+2$, completely decoupled from the original document's length.\n  * **Empirical Evidence:** As shown in **Table 6** of our main paper, this leads to constant VRAM usage (~550MB) for EAReranker, whereas plaintext models can consume up to 8441MB for long documents. This provides clear, practical mitigation of the overhead problem.\n\n* **Explanation of Privacy Preservation:**\n  * **Threat Model & Architecture:** We consider a scenario where the inquirer and the data provider are two non-trusting parties. Our method decouples the document embedding process from the reranking service. A user can convert sensitive documents to embeddings **locally** using a public pre-trained encoder. Only the fixed-size, anonymized embedding vectors are sent to the remote reranking service.\n  * **Security Guarantees:** An embedding is a lossy, one-way representation. Reconstructing long, complex, or private text from its embedding is computationally infeasible, even if an attacker possesses the same encoder model.\n  * **New Experimental Results:** To empirically validate this, we conducted inversion attacks on our query dataset using state-of-the-art methods: **Vec2Text**, **TEIA**, and **GEIA**. The Rouge-L scores of the recovered text were **0.2873, 0.1969, and 0.3180**, respectively, indicating very low semantic overlap and failed reconstruction. This demonstrates that sensitive raw data, which would be fully exposed with plaintext rerankers, remains protected in our framework.\n\n## W3: Contribution of the Work\n\n* **Novel Architecture:** A naive embedding-based model would lack the capacity for fine-grained assessment. Our contribution is a specialized architecture with two non-trivial components:\n  1. **Embedding Dimension Expansion (Section 5):** Our technique of transforming single vectors into rich sequential representations (`e'_{q,i} = W_{q,i}e_q + b_{q,i}`) is a novel method that allows a Transformer to \"unfold\" and deeply analyze the dense information within embeddings. Our ablation study (**Table 5**) shows this is critical, with its removal causing a **3.70% drop in ACC25**.\n  2. **Bin-Aware Weighted Loss (Section 5):** Our custom loss function, $L_{\\text{BWMSE}}$, which specifically penalizes predictions crossing semantic bin boundaries, is essential for training a model that aligns with the operational needs of RAG. Its removal causes a **2.67% drop in ACC25**.\n\n* **Principled Adequacy Framework:** Our work is among the first to move beyond simple \"relevance\" to formally define and operationalize a multi-dimensional concept of \"adequacy\" for RAG. Our rigorous annotation methodology (**Section 4**) and the resulting large-scale dataset are significant contributions that enable future research.\n\n* **SOTA-Level Performance with High Efficiency:** Demonstrating that an embedding-only model can achieve performance **within 0.54%** of state-of-the-art plaintext rerankers (**Table 3**) while being **2-3x faster** with **constant memory usage** (**Table 6**) is a significant and non-obvious result.\n\nWe thank the reviewer for their critical assessment, which has helped us clarify the foundations and technical innovations of our work. We have endeavored to address each concern with concrete details and empirical evidence. We believe these clarifications substantiate the novelty and significance of our contributions to the field.\n\n----------\nReferences:\n\n[1] Morris J, Kuleshov V, Shmatikov V, et al. \"Text Embeddings Reveal (Almost) As Much As Text.\" EMNLP 2023.\n\n[2] Huang Y H, Tsai Y, Hsiao H, et al. \"Transferable Embedding Inversion Attack.\" ACL 2024.\n\n[3] Li H, Xu M, Song Y. \"Generative Embedding Inversion Attack to Recover the Whole Sentence.\" Findings of ACL 2023."}, {"review_id": "PbjAiYqJVD", "reviewer": "Reviewer_hpzk", "summary": "## Summary\n\nThe paper introduces EAReranker, an efficient embedding-based framework for assessing document adequacy in RAG. EAReranker aims to address challenges of traditional reranking approaches by evaluating document utility without accessing original text content. It uses a decoder-only architecture for predicting adequacy directly from embedding vectors. An experiment on four benchmarks shows competitive performance with existing rerankers.", "questions": "1. How is the framework for assessing adequacy developed? Are these dimensions based on prior research? What is the reasoning for choosing six distinct bins with varying ranges?\n2. Does EAReranker support non-text data?\n3. Are users expected to train their own EAReranker? Or the trained EAReranker will be available for users?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\nS1. Adequacy is an overlooked and timely research area in the applications of RAGs. I appreciate that the paper focuses on this important topic.\n\nS2. The proposed EAReranker method is sound. The proposed framework to assess adequacy looks useful to the community. The experiments are thorough, and the findings are promising.\n\nS3. The paper is well-written and easy to follow.\n\n## Major Weakness\n\nW1. The framework for assessing adequacy could be strengthened with more detailed explanations. For instance, the rationale behind the four dimensions of adequacy assessment needs clarification. Are these dimensions based on prior research? What is the reasoning for choosing six distinct bins with varying ranges?\n\nW2. The challenges of existing approaches are (1) computational overhead, (2) dependency on plain text, (3) lack of refined content value assessment. The proposed method only addresses (1) and (3). Does EAReranker support non-text data? Additionally, if users must train their own EAReranker, this could reduce efficiency compared to methods that don't require training.\n\n## Minor Weakness\n\nM1. The text in figure 1 is too small to see.\n\nM2. RAG and embedding search work on multimodal content as well. (L39)", "was_revised": true, "final_justification": "Most of my questions have been answered. Given that my original rating was already positive, I will be maintaining it.", "rebuttal": "We are grateful to the reviewer for their thoughtful feedback and positive assessment of our work. We appreciate the recognition that adequacy is \"an overlooked and timely research area\" and that our proposed method is \"sound\" with \"promising\" findings. Below, we address the questions and concerns raised.\n\n## W1: Rationale for the Adequacy Assessment Framework\n\nThank you for this important question, which highlights a core contribution of our work. The reviewer asks for a more detailed explanation of our adequacy assessment framework, specifically the rationale behind the four dimensions and the choice of six bins with varying ranges.\n\n**1. On the Rationale for the Four Dimensions of Adequacy:**\n\nOur adequacy assessment framework is designed to move beyond traditional \"relevance\" and evaluate a document's *utility* as context for a generative model. This requires a multi-faceted view. The four dimensions—**Verifiability, Need Coverage, Evidence Completeness, and Structure Suitability**—were systematically developed based on an analysis of failure modes in RAG systems and a synthesis of principles from established research in information quality and argumentation theory.\n\n* **Verifiability:** This dimension is inspired by the literature on fact-checking and information reliability [1]. An unverifiable document, even if topically relevant, poses a high risk of causing the LLM to hallucinate or generate misinformation.\n* **Need Coverage & Evidence Completeness:** These dimensions are adapted from concepts in question answering and argumentation mining [2, 3]. A document might be relevant but incomplete, addressing only a fraction of the user's query (`Need Coverage`) or presenting claims without sufficient supporting arguments (`Evidence Completeness`). These are common reasons for unsatisfactory RAG outputs.\n* **Structure Suitability:** This dimension addresses a practical but critical aspect of data quality and its impact on LLM behavior. Poorly structured documents can actively introduce misinformation; for example, erroneous OCR or incorrect parsing of multi-column PDFs can concatenate unrelated sentences, creating false statements. Furthermore, LLMs are susceptible to the style and quality of their context. When presented with long, poorly structured content, they are more likely to generate responses of similarly low quality. Conversely, a document with a structure aligned with the user's needs (e.g., a well-formed table for a comparative query) makes it easier for the LLM to produce a high-quality, well-formatted response. This is especially true for smaller, less capable LLMs, which may struggle to follow complex formatting instructions but can more easily leverage a well-structured source document.\n\n**2. On the Rationale for the Six Bins:**\n\nThe six-bin structure with non-uniform score ranges was designed to provide both **interpretable categories** and **operational utility** for RAG systems. The bin boundaries are strategically placed to create actionable thresholds, a design choice informed by empirical analysis during our dataset annotation pilot studies.\n\n* **[0.90, 1.00] (Precise Adequacy) & [0.75, 0.90) (High Adequacy):** This high-end separation distinguishes \"perfect\" context from \"excellent\" context. Empirically, documents providing fully comprehensive and perfectly structured answers are rare, justifying a narrow top bin.\n* **[0.50, 0.75) (Middle Adequacy) & [0.25, 0.50) (Marginal Relevance):** The core of the distribution lies here. The 0.5 threshold is a critical pivot point, separating documents that are fundamentally useful from those that are only peripherally relevant.\n* **[0.10, 0.25) (Weak Relevance) & [0.00, 0.10) (Irrelevance):** This low-end separation helps distinguish documents with a faint, almost useless signal from those that are completely off-topic. The 0.10 threshold acts as a \"hard filter\" to discard content that offers no value.\n\n*   The fine-grained splits at the extremes—**[0.90, 1.00] (Precise Adequacy)** and **[0.00, 0.10) (Irrelevance)**—serve a key purpose in training. By isolating the absolute best and worst cases, we provide the model with clearer, more definitive guidance signals at the poles of the adequacy spectrum.\n\n*   The Irrelevance bin (**[0.00, 0.10)**) is designated for samples that are completely unrelated, akin to the random negative samples often used in training traditional rerankers. By grouping them into this narrow, low-value range, we reduce the model's sensitivity to minute differences between them (e.g., the distinction between a score of 0.05 and 0.04 becomes less significant in the loss calculation). This encourages the model to allocate its capacity more effectively toward a more critical task: distinguishing between documents with varying degrees of utility in the higher adequacy ranges (**[0.10, 1.00]**), which often correspond to the \"hard negatives\" that challenge traditional models.\n\n## W2: Support for Non-Text Data and Training Efficiency\n\n**1. Extensibility to Multimodal Data:**\n\nAn important strength of the EAReranker framework is its **inherent extensibility to non-text (multimodal) data**. Our model operates *exclusively* on embedding vectors. Therefore, as long as a query and a document (be it an image, audio clip, or table) can be mapped to fixed-dimensional vectors by a capable multimodal embedding model, EAReranker can assess its adequacy without any architectural changes. This is a significant advantage over plaintext rerankers, which are fundamentally text-dependent.\n\nThere are two primary pathways to achieving this:\n1. **Native Multimodal Training:** One could construct a multimodal adequacy dataset using appropriate multimodal LLMs for annotation, and then train a natively multimodal EAReranker from scratch.\n2. **Zero-Shot/Few-Shot Transfer:** A more resource-efficient approach would be to leverage the existing text-based EAReranker. By using a strong multimodal embedding model (which maps text and images to a shared semantic space), the text-trained EAReranker could likely perform a form of zero-shot transfer, assessing image adequacy based on the semantic proximity of its embedding to relevant text embeddings. This could be further enhanced with a small amount of multimodal labeled data for fine-tuning.\n\n**2. Justifying the One-Time Training Effort:**\n\nThe reviewer brings up the valid consideration of the initial training cost. We see this as an upfront investment that unlocks significant and continuous gains in inference efficiency.\n\n* **A \"Train-Once, Use-Many\" General-Purpose Model:** For a given embedding model, EAReranker only needs to be trained thoroughly once. The resulting model can then be deployed as a general-purpose adequacy assessor by any user of that specific embedding model. This eliminates the need for each user to undergo the training process themselves.\n* **Amortized Costs in Production:**For organizations that require custom fine-tuning on proprietary data, the training effort is a one-time or infrequent cost. In contrast, the computational overhead of plaintext rerankers is incurred *with every query* and scales with document length and traffic volume. Over time, the consistent, low-latency, and memory-stable inference provided by EAReranker (as shown in Table 6) results in a more scalable, predictable, and cost-effective production system, meaning the initial training cost is quickly amortized.\n\n## M1 and M2: Figure Text Size and Multimodal Content\n\nRegarding **M1**, we agree with the reviewer that the text in Figure 1 was too small. We have remade the figure with a larger, more legible font in the revised paper to improve readability.\n\nRegarding **M2**, thank you for this correction. RAG is not limited to text. We have corrected the statement on Line 39 and elsewhere to reflect that embeddings and RAG systems are increasingly used for multimodal content. This correction also strengthens our argument regarding EAReranker's applicability to non-text data, as discussed in our response to W2.\n\nWe hope these clarifications and the corresponding revisions to our paper have adequately addressed the reviewer's concerns. We thank the reviewer again for their thoughtful and valuable feedback.\n\n----------\nReferences:\n\n[1] Thorne J, Vlachos A, Christodoulopoulos C, et al. \"Fever: a large-scale dataset for fact extraction and verification.\" NAACL 2018.\n\n[2] Wachsmuth H, Trenkmann M, Stein B, et al. \"Are view corpus for argumentation analysis.\" Lecture Notes in Computer Science 2014.\n\n[3] Wadden D, Lin S, Lo K, et al. \"Fact or fiction: Verifying scientific claims.\" EMNLP 2020."}]

Author rebuttal:

Meta-review: This paper introduces EAReranker, a novel and efficient embedding-based framework for assessing document adequacy in RAG systems. It effectively addresses critical challenges of traditional reranking: computational overhead scaling with document length, dependency on plain text, and insufficient assessment beyond simple relevance. Reviewers liked its effectiveness in achieving competitive performance with state-of-the-art plaintext rerankers while maintaining constant memory usage and processing 2-3x faster.

The authors were very diligent in the their rebuttal. They addressed questions about performance on out-of-domain datasets (AIR-Bench) and long-document retrieval (NarrativeQA). Authors also provided extensive new experiments demonstrating strong zero-shot generalization and effective handling of long documents, significantly strengthening these points. 

There were also requests for more detailed explanations of the algorithm, mathematical clarity for adequacy scoring, and ablation for score calibration, as well as statistical significance. Authors provided in-depth explanations, mathematical formulations, a key ablation study showing calibration methodology effectiveness, and statistical robustness tests.

There were still some open questions about clarity and some missing details. A lot of this was addressed in the rebuttal and I hope the authors carry over all of their nice work from the rebuttal into the next version of the paper.
Decision: Accept (poster)