Retrieval as a Decision: Training-Free Adaptive Gating for Efficient RAG
Authors: Yufeng Wang
Lu Wei
Haibin Ling
Venue: Accepted by TMLR ()

Abstract: Retrieval-Augmented Generation (RAG) improves factuality but retrieving for every query often hurts quality while inflating tokens and latency. We propose Training-free Adaptive Retrieval Gating (\textbf{TARG}), a single-shot policy that decides when to retrieve using only a short, no-context draft from the base model. From the draft’s prefix logits, TARG computes lightweight uncertainty scores—mean token entropy, a margin signal derived from the top-1/top-2 logit gap via a monotone link, or small-$N$ variance across a handful of stochastic prefixes—and triggers retrieval only when the score exceeds a threshold. The gate is model-agnostic, adds only tens to hundreds of draft tokens, and requires no additional training or auxiliary heads. On NQ-Open, TriviaQA, and PopQA, TARG consistently pushes the accuracy–efficiency frontier: compared with Always-RAG\footnote{\textsc{Always-RAG}: retrieve for every query; \textsc{Never-RAG}: never retrieve.}, TARG matches or improves EM/F1 while reducing retrieval by 70–90\% and cutting end-to-end latency, and it remains close to Never-RAG in overhead. A central empirical finding is that under modern instruction-tuned LLMs the margin signal is a robust default (entropy compresses as backbones sharpen), with small-$N$ variance offering a conservative, budget-first alternative. We provide ablations over gate type and prefix length and use a $\Delta$-latency view to make budget trade-offs explicit.

Published in Transactions on Machine Learning Research (04/2026)
Retrieval as a Decision: Fraining-Free Adaptive Gating for Efficient RAG
Yufeng Wang
Deportment of Computer Science Stony Brook Universily
Lu Wei
Department Computer Science Stony Brook University
Haibin Ling
Department of Artificial Intelligence Westlake University
Reviewed on OpenReview: https: /ttps://openreee wpwp///vepee.....e// net/ forum? id- 18g
Abstract
Retrioval-Augmented Generation (RAG) improves factuality but retrieving for every query often hurts quality while inflating tokens and latency. We propose Training-free Adaptive Retrieval Gating (TARG), single-shot policy that decides when to retrieve using only short, no-context draft from the base model. From the draft's prefix logits, TARG com- putes lightweight uncertainty scores-ment token entropy, margin signal derived from the lop-1/top-2 logit gap via a monotone link, or small-N variance across a handful of stochas- tic prefixes and triggers retrieval only when the score exceeds a threshold. The gate is model-agnostic, adds only tens hundreds of draft tokens, and requires no additional training or auxiliary heads. On five QA benchmarks spanning short-answer (NQ-Open, TriviaQA, PopQA), multi-hop (MuSiQue), and long-form (ASQA) tasks, TARG consistently pushes the accuracy efficiency frontier: compared with Always-RACT TARG matches or improves EM/F1 while reducing retrieval by 70-90% and cutting end-to-end latency, and it remains close to Never-RAG in overhead. A central empirical finding is that under modern instruction-tuned LLMs the margin signal is a robust default (entropy compresses as back- bones sharpen), with small-N variance offering a conservative, budget-first alternative. We provide ablations over gate type and prefix length and use a A-latency view to make budget trade-offs explicit.
1 Introduction
Large language models (LLMs) demonstrate strong performance on knowledge-intens inttenivee generation yet re- main susceptible to hallucination, producing fluent but unfounded content when parametric memory lacks the necessary facts (Huang et. al.] 2025; Farquhar et al., 2024).. Retrieval-Augmeented Generation (RAG) addresses this problem by consulting an external corpus at inference time, coupling generator with non- parametric memory to improve factuality and transparency (Lewis et al. 2020 Gun et al. 2020 Wang et al.
Izacard & Grave 2020 Yu et al. 2024 Lin et al. 2023). However, invoking retrieval for every input increases latency and token usage, and can reduce accuracy when retrieved evidence noisy or tangential. Longer prompts also impose nontrivial computational costs due to the quadratic scaling of self-attention
2023
"Corresponding Author: linghalbin@westlake.dd..c
ALways-RAG: retrieve every query; NEVER-RAC RAA:: never retrieve.

---

Published in Transactions on Machine Learning Research (04/2026)
Vaswani et al. 2017 Dao al.) 2022). and evidence placed in the middle of long contexts is often under- utilized by current models (Liu et. al. 2023). These considerations motivate selective retrieval: deciding when retrieval is unnecessary is as important as deciding what to retrieve.
Recent work explores conditional retrieval, including forward-looking active retrieval (FLARE) that antici- pates upcoming content and re-queries when confidence appears low (Jiang et al., 2023), instruction-tuned regulation with reflection tokens (Self-RAG) {Asai al., 2024), and corrective actions triggered by evidence quality scoring (CRAG) (Yan et al., 2024 Unlike methods that train control heads run multi-stage tool loops, our method (ie., TARG) aims to make single, training- freee decision from prefix logits.
We revisit selective retrieval from a simpler angle: a training-free, single-shot decision of when to retrieve that any off-the-shelf language model (LM) can make before full decoding. Our approach, named Training- free Adaptive Retrieval Gating (TARG), uses a short, no-context draft to read the model's own prefix logits and compute a lightweight uncertainty score. We instantiate three signals that require no training or auxiliary heads: (1) mean token entropy, (2) a margin-based score derived from the top-1 versus top-2 logit gap via monotone link, and (3) small-N variance from handful of stochastic prefixes. The gate triggers retrieval only when the score exceeds a fixed decision threshold, adding only tens to hundreds of draft tokens per query and integrating cleanly into existing RAG stacks.
A central empirical observation is that the choice of signal matters under modern instruction-tuned LLMs. As backbones sharpen, prefix entropies often compress and may lose discriminative power. In contrast, the top-1/top-2 margin retains dynamic range and correlates with cases where external evidence changes or stabilizes the answer; small-N variance behaves similarly but is more conservative. Across five benchmarks spanning short-answer (NQ-Open (Kwiatkowski et al. 2019 Loe ct al. 2019), TriviaQA (Joshi et al. 2017). PopQA Mallen et al., 2022). multi-hop (MuSiQue (Trivedi et al.) 2022 ), and long-form (ASQA (Stelmakh) et al. 2022)) QA, TARG with margin signal traces favorable accuracy efficiency fronts relative to both Never- and Always-RAG, achieving low retrieval rates and latency overheads close to the Never baseline. We report results with both a compact generator and a stronger Llama-3.1-BB model and adopt a A-latency view that makes budget trade-olfs explicit.
necuracy-efficiency
In summary, this work makes the following contributions:
Entropy becomes substantially less discriminative on instruction-tuned models. We empirically observe that, on modern ALHHFinstruction-tung LLMs with sharper next-token dis- tributions, entropy has much lower dynamic range as an uncortainty signal in our setting. This motivates margin-based alternatives.
Logit margin as a robust, training-free uncertainty signal. The top-1/top-2 logit gap (mapped via exponential decay) remains discriminative even under peaked distributions, provid- ing practical, training-free uncertainty criterion for selective retrieval.
Single-shot, prefix-only gating. Unlike FLARE (mid--genraation), Self-RA//CRAC (multi-step with specialized prompting), or agentic tool-use loops, TARG makes a single retrieval decision from a short prefix (k: = 20), yielding negligible overhead (~50ms) and no extra retrieval calls beyond those selected by the gate.
2 Related Work
Retrieval-augmented generation (RAG). RAG couples parametric generators with non-parametric
memory to improve factuality knowledge-intensive tasks. For example, REALM exposes external knowl- edge during pretraining via differentiable retriever (Guu et al. 2020); RAG and FiD popularize end-to-end
training of retriever-reader pipelines that condition generation on retrieved passages (Lewis ct al.] 2020: Izacard & Grave 2020). Subsequent variants enhance fusion and rationale use (Wang el al.] 2023) and strategies (Yu et al. 2024).. Recent surveys synthesize these design choices and high- light robustness and cost as persistent challenges: long contexts inflate latency, retricval noise can degrade accuracy, and budgets must be controlled (Wu et al.) [2024] Sharma 2025). These observations motivate policies that decide when to retrieve, not only what to retrieve.
adopt
2

---

Published in Transactions on Machine Learning Research (04/2026)
Adaptive and active retrieval. A grrowing line of RAG research retrieves conditionally. FLARE performs forward-looking active retrieval by anticipating upcoming content and re-querying when predicted tokens look low-confidence (Jiang et al. 2023). Self-RAG trains an LM with
/control tokens to decide when to retricve and how to critique evidence (Asai et al. 2024). CRAG scores the quality of retricved passages and triggers corrective actions when evidence is weak Yaan et al. 2024]. Very recent systems (e.g.. SeaKR, SUGAR) leverage uncertainty probes modulate retrieval frequency (Yao et al. 2024 Zubkova et al., 2025). While effective, these approaches often add supervision, special control tokens, auxiliary probers, multi-stage loops that increase engineering complexity and latency. In contrast, training-free approach, TARG, makes single, up-front decision using only prefix logits from the base model; among simple signals, the top-1/top-2 margin emerges as robust default under modern instruction-tuned LLMs, with small-N variance as & conservalive alternative.
Reasoning-acting interlcaving with tools. Frameworks such as Sclf-Ask and ReAct interleave reason- ing with tool use (Press al. 2022 Yao el. al. 2023), allowing an LM to search or relrieve on demand during multi-hop solutions DSP composes retrieval and generation with programmatic pipelines (Khattab et al. 2022). These methods can yield strong performance but typically rely multi-turn tool calls, bespoke prompting, and orchestration overhead. Our scope is orthogonal: we study a one-shot gate that decides whether to retrieve at all before full decoding. The two directions are complementary: TARG can suppress unnecessary retrieval in auggaeeedt systems, improving latency without altering downstream planners.
Uncertainty, calibration, and hallucination detection. Work on LM confidence studies whether models "know when they know," " spanning logit-based, internal-state, and consistency-based signals (Geng et al. 2023 SelfCheckGPTT, for example, detects hallucinations via sample inconsistency and hence is conceptually related to our small-N variance signal, though we probe only a short prefix rather than full generations (Manakul et al. 2023). TARG operationalizes intrinsic, low-overhead uncertainty from brief no-context draft (entropy, margin, variance) to decide whether the expected benefit of retricval outweighs its cost. Empirically, as Instruction-tuner backbones sharpen, prelix entropies compress and over-trigger, whereas the top-1/top-2 margin and small-N variance retain discriminative power: yielding selective, budget- aware retrieval that easy retrofit and orthogonal future improvements in retrievers and rerankers.
3 Method
Input Query q
Generator LLM
k Prefix Tokens
Uncertainty Gate
B (44)
No
R(4) Ø
U(g)
T
Yes
Final Results
Retriever
U(q): Uneertainty Scorc
t: Retrieve Threshold
Wiki Database D
Figure 1: Illustration of TARG methodology.
Given a user query 9 and a generator LLM Go with tokenizer T, a RAG system augments the base prompt B(q) with optional context C retrieved from corpus D, then decodes answer y. Let. B(q) C
3

---

Published in Transactions on Machine Learning Research (04/2026)
denote the final prompt (token concatenation), and let the next-token distribution at step be po(y M x). Our goal is to decide, at inference time and without training, whether to retrieve (ie., choose C' + 2) for a given as to minimize compute cost: retrieval calls, context tokens, and wall-time latency, while preserving or improving task accuracy.
3.1 Prefix uncertainty from a short draft
We exploit the model's own uncertainty on short, retrieval-free prefix to decide whether retrieval is needed. We run Gg for k tokens on the base prompt only, obtaining logits f1..., fx and probabilities Fy = softmax((4), 2k... Each gate yields a per-step score ut, aggregated as = Ei= ur.
Entropy gate Per step entropy = Ej Tti log Tej with
1 Σ Ho.
(k)
(1)
larger Vent indicates higher uncertainty.
Margin-a--umcertainty gate Let Dr = l,,11 (2) 2 0 be the top-1 vs. top-2 logit gap at step t. Map
gaps to a positive uncertainty via a strictly decreasing Φ (default d(z) = exp(-z/ temperature 3 > 0):
Unaa 3)
((g)) (0.
(2)
t=1
Because á is strictly docrcasing, thresholding Unar is order-cquivalent to thresholding the mean gap; the link is a convenience rather than a fragile choice.
Lemma (order-equivvllence) For any strictly decreasing Φ, thresholding Umar at t is equivalent to thresholding the mcan gap at some T', i.c., decisions are identical up to monotone roparamcterization. (Proof shown in Appendix.)
Small-N variance gate Sample N short stochastic prefixes (temperature T) to obtain sequences (0....,g(N) At step 1, let Pe be the empirical token distribution of {st (00) and define de = max, Pr) Then
N
(k, = de.
0 S Var I
(3)
N
1=1
Lemma 2 (boundedness). The mode frequency is 2 1/N, hence de S (N 1)/N and the stated bound. (Proof shown in Appendix.)
3.2
Gate decision and decoding
Given the scalar uncertainty score U from the prefix draft, we trigger retrieval if > т, the t is the retrieve threshold:
retrieve(g) + U(g) >T.
If retrieval is triggered, construct the context C' with dense encoder E (e.g., E5-base-v2 Wang et al. 2022 ) and FAISS inner-product index (Douze et al. 2024). Let h =
be the normalized query embedding; we return top-K passages Dx(g) by similarity (h, E(d) and format the context as
=
format format(d)[ [d][I:Lex]]
(5)
dEDx (0)
truncated to the context budget Lax tokens. The final prompt B(g) @ C; otherwise (no retrieval) we proceed with ReG using B(g)- For long generations, optional single re-check be applied
4

---

Published in Transactions on Machine Learning Research (04/2026)
Algorithm TARG inference (training-fret free gate)
Require: query y, threshold 7, prefix length k, (optional) recheck stride m, retriever R, lop-K, context
budget Letx
(q))
2: draft < DEROODEREEIX(CG:
3: U < ScoRF(draft; gate (entropy, margin, variance))
4: if U > then
5:
- RETRIEVEFORMAT((R 9, K, Letx)
x t- B(g) @C
end if
y GEN
optionally re-check every m tokens if not yet retrieved
return
every m tokens: if the munning-prefie score exceeds T and retrieval has not yet occurred, retrieve once and continue decoding.
In all main experiments in Section we use TARG its single-shot form: we decode short prefix from the base prompt B(q), compute (g) once, and then either retrieve exactly once or never retrieve. Algorithm [ also illustrates an online extension for very long generations in which the gale can be re-evaluated every tokens and retrieval can be triggered mid-generation if (g) ever exceeds In our short-answer QA setting, however, simple re-check policy with fixed 7 is overly aggressive: as shown in Appendix A.9. it drastically increases retrieval rate and latency without greatly improving EM, effectively degenerating loward Always- RAG. We therefore adopt single-shot gating for all reported results and view dynamic re-check as a design option for future long-form scenarios.
We also explored simple aggregation of the three gates (e.g,, normalizing each U to [0, and averaging). On dev split, these ensembles did not yield consistent improvements over the best single gate once t and k are tuned; the combined scores behave like re-parameterized margin gates while introducing extra weights to calibrate. For simplicity and robustness, we therefore present each gate separately and recommend MARGIN as default.
3.3 Cost, accuracy and calibration
Cost model. Let T denote the always-incurred prefix, Tex the context tokens when retrieving, and T(0), T(l) the output tokens without/wite retrieval. With retrieval rate (()) Pr(U r], the expected LM tokens per query are
[T())) Taratt (1 x()) B[Tol + x77 (Tax E(T(H)))
(6)
and analogous additive decomposition holds for wall-time latency. We report efficiency using A latency to isolate the incremental overhead vs. the zero-RAG bascline.
Accuracy model and dominance. Let (g) and (g) correctness indicators (or probabilities) for zero-RAG and with-RAG, and let A(g) = (g) - ( We assune usefulness calibration at the level of conditional expectations: there exists a threshold т. such that
E(A(g) U(1) =0) and [[A(g) U(1)>
> 2.
(7)
This is average-case modeling assumption about the joint distribution of (U, under fixed re- triever/corpus, not a pointwise guarantee and not a statement that must hold for every possible dataset or routing policy. In particular, if retrieval systematically helps when U is small and hurts when U is large (c.g., adversarial or heavily filtered traffic concentrated in such regions), a threshold r, satisfying these inequalities may not exist and our dominance argument would not apply.
Under usefulness calibration, choosing т. yields
E{A (7) E (1) E (()) 1 ((V > +) max Max(E[[AV B(A4)))
(8)
5

---

Published in Transactions on Machine Learning Research (04/2026)
Intuition. Always-RAG integrates A over all q, including negative-A regions where U S T4 Never-RAG integrates wero. Thresholding near admits only the positive region. Appendix A.7 ("Error quadrants over (U(g), (()))") provides empirical support that, under our frozen dense retriever over chunked Wikipedia, the learned uncertainty scores are aligned with this average-case assumption.
Threshold calibration and budget control. Let Fy be the empirical CDF of U on development set (denoted by dev). To hit retrieval budget E [0, 1], pick = - p). Alternatively, select 7 that maximizes dev accuracy. Because U is scalar, calibration is fast and stable. In practice we tune k (prefix length), & for the margin link, and N for variance on the same dev split.
3.4 Implementation details
We use a flat inner-product FAISS (Douze et al. 2024) index with a normalized dense encoder; and Tetx are tuned on dev. For the margin gate we default to 6(z) = exp(-z/9) () with 8=3, which preserves ordering (Lemma 1) and yields interpretable U E (0, 1]. For variance we use N=5 samples by default (upper bound 4/5). All gates reuse the same k-token prefix, so the gating step adds only few dozen to few hundred tokens per query.
4 Experimental Setup
Models, decoding, and baselines. We evaluate two instruction-tumed backbones, Qwen2.5-7BB-nnstruce [Bai et al. 2023) and Jlama-3.1-8B-Insttuuee (Grattafiori et al.) 2024), under the same decoding protocol: greedy generation (no sampling), batch size 1, identical prompts stop criteria, and the same short prefix for gating. The prefix is decoded without retrieval using the same greedy policy; for the variance gate only, we draw N short prefixes at temperature T=0.7. We compare Never-RAG (decode from B(4)), Always-RAG (retrieve once per query and decode from (C)) and our training-fres gale TARG (draft tokens without retrieval, compute scalar uncertainty U, and retrieve if U > 7). All main experiments use a single-shot gate: we decide once based on the initial k-token prefix and never perform mid-gencration re-chocks; dynamic re-check variant is evaluated in the Appendix A.9
Retriever, index, and corpus. Without losing generality regarding retriever type, which was proven in Appendix A.4 we use frozen dense dual encoder E5-BASE-V2 (Wang et al.) 2022) and follow model guide- lines by prefixing queries with "query:" and passages with "passage:" Queries and passages are encoded into 768-d vectors, fg-normalized, and stored in FAISS IndexFlatIP (Douze et al. 2024); with normal- ization, inner product equals cosine similarity. At inference, we return top- K= passages and concatenate them as the retrieval context (formatted as [title] text and truncated to fit a context budget Letx). The corpus is English Wikipedia (Foundation): articles are chunked into passages of roughly 1000 characters with 100-character overlap (minimum 200 characters) and a large subset is indexed for all experiments. Our aim is to evaluate a training-free gating policy, which emphasizes when retrieve rather than to optimize the retrieval stack; the gate is orthogonal retriever quality and can sit atop stronger encoders, rerankers, or context compression modules.
Datasets and evaluation protocol. We evaluate on NQ-Open Lee et al. 2019). TriviaQA (Joshi et al. 2017), PopQA (long-tail entities) (Mallen et al. 2022 MuSiQue (multi-hop QA) Trivedi et al. 2022). and ASQA (long-form QA) (Stelmakh et al.] 22022). 3 Results are reported for Never-/Ale Aalee//wwwssRCCCC and gated operating points; for NQ-Open, we include threshold sweeps and ablations. Prefix length is ablated over k E {10, 20, 30), with 2= used by default (best cost/qualis balance). We instantiate three training-fres signals from the k-token prefix: (i) Entropy Uent k Σ. (-E, Tt.j log m.,);; ) (ii) Margin Umar = à E, Φ(g)) where 9t is the top-1/top-2 logit gap and ((g) = exp(-g/3) with 8=3 (order-preserving link yielding U e (0, 1]); and (iii) Variance Uvar = E (1- max, pr( from N= short stochastic prefixes at T-0.7, with range [0, (N-1)/NJ (N is the number of short prefixes). For ASQA, we use the standard exact-match metric scoring correctness based on ground-truth string match with least one supporting citation mapped to the gold evidence doc-id set. We cap MuSiQue generation to 50 tokens (concise multi-hop answers) and
6

---

Published in Transactions on Machine Learning Research (04/2026)
Table 1: Selective retrieval with Qwen2.5-7B-Insstruce Representative TARG operating points (two per dataset) against non-gated baselines. The threshold is shown for gated runs; "_" denotes basclines. A Latency is the added seconds per query relative the Never-RAG baseline on the same dataset /hardware; the NEVER row shows the absolute bascline latency (s/q) in parentheses.
Retrieval Rate
A Latency
Dataset
Model
EM
F1 (%) f
(s/q)
NEVER
60.8 / 61.4
0.000
2.947 (Baseline)
ALWAYS
57.6 57.2
1.000
+3.462
TriviaQA
ENTROPY
0.80
61.8
62.2
0.028
+0.876
MARGIN
0.15
62.2
62.6
0.338
+2.174
VARIANCE
0.75
61.8
/ 62.2
0.006
+0.133
NEVER
20.0 / 20.1
0.000
2.129 (Bascline)
ALWAYS
14.6
14.6
1.000
+3.828
PopQA
ENTROPY
0.80
22.4
22.3
0.124
+1.761
MARGIN
0.35
23.0
23.1
0.124
+1.761
VARIANCE
0.40
22.8
/ 22.9
0.182
+1.847
NEVER
38.8
37.7
0.000
3.293 (Baseline)
ALWAYS
37.4
36.7
1.000
+2.922
NQ-Open
ENTROPY
0.85
39.6
39.1
0.046
+0.964
MARGIN
0.20
39.6
/ 38.8
0.304
+1.295
VARIANCE
0.60
38.6 / 38.0
0.012
+0.291
structured, carly-terminating citation format for ASQA; this keeps per-query latency bounded and prevents ASQA from being treated unconstrained long- form task.
Threshold sweeps, metrics, and reproducibility. For each gate, we sweep the decision threshold t to expose eccuracy-efficiene frontiers. We report Exact Match (FM) and F1 (standard normalization). Retrieval Rate x = Pr(U > 7], and latency extra scconds per query relative to the Never-RAG bascline the same dataset sillaset/harwwaree absolute latencies are provided once in table footnotes. Calibration of can target retrieval budget via the empirical CDF of U dev set, but in main results present grid sweeps and select representative operating points at modest budgets (~5520%%). Because decoding is deterministic (greedy) and the retriever is frozen, randomness enters only through the variance gate's short prefixes and dataset sampling. To quantify statistical reliability, we compute bootstrap 95% confidence intervals over queries for EM/F1 and, for key configurations (Never-RAG, Always-RAG, and the best TARG operating points), repeat evaluation under different RNG sceds in the Appendix A.8 All runs use batch size 1 and identical decoding parameters across modes; for variance we fix the temperature and RNG seed in main tables and vary them only in ablations.
5 Results
Table [ shows that unconditionally retrieving (ALWAys) increases latency and often depresses accuracy relative to NEVER, a hallmark of off-topic or aliased passages. Training-free gating restores precision: both MARGIN and VARIANCE match or exceed NEVER while staying far cheaper than ALWAYS. On TriviaQA, MARGIN attains the best quality by retrieving more frequently, whereas VARIANCE achieves essentially the same quality at tiny retrieval rate and negligible overhead, reflecting its conservative trigger. On PopQA, where retrieval precision is harder, both gates improve over ALWAYS and surpass NEVER with moderate added cost; the gains are modest, consistent with long-tail entity drift. On NQ-Open, small amounts of retricval help: ENTROPY ENTROPY/VARIANCE /VARIANCE achieve slight improvements at very low budgets, while MARGIN trades bit more budget for similar quality. For intermediate retrieval rates (c.g., between 0.05 and 0.3; see A Appendix A.8 for full sweeps), TARG with the MARGIN or VARIANCE gates attains higher EM/F1 than
7

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: Selective retrieval with Llama-3.1-8B-Insruus Same protocol as Table |
A Latency
reports added seconds per query over the datasct's Never baseline; absolute Never latencies (s/q) appear parentheses.
Retrieval Rate
A Latency
Dataset
Model
EM/
F1 (%)
(s/q) +
NEVER
80.8 / 80.0
0.000
10.383 (Baselinc)
ALWAYS
67.6 / 67.2
1.000
+1.069
TriviaQA
ENTROPY
0.80
74.4 / 74.1
0.524
+0.495
MARGIN
0.70
83.8 / 83.0
0.001
+0.018
VARIANCE
0.75
83.6
83.0
0.001
+0.018
NEVER
35.2 / 34.4
0.000
10.299 (Baseline)
ALWAYS
24.8 / 24.6
1.000
+1.269
PopQA
ENTROPY
0.80
28.8
28.8
0.760
+0.974
MARGIN
0.45
36.6
36.2
0.108
+0.424
VARIANCE
0.55
36.4
/36.0
0.084
+0.317 7
NEVER
53.8 / 51.7
0.000
10.299 (Basclinc)
ALWAYS
48.6 / 46.1
1.000
+1.248
NQ-Open
ENTROPY
0.95
55.4 / 53.1
0.132
+0.175
MARGIN
0.50
57.6 / 54.7
0.008
+0.012
VARIANCE
0.60
56.8 / 53.7
0.026
+0.059
both NEVER and ALWAYS while adding minimal latency, indicating that the gale is not merely imitating the better unconditional baseline but actively identifying when external evidence likely beneficial. Overall, with 7B-class model, VARIANCE is strong default when budgets are tight; MARGIN is preferable when small accuracy boost justifies higher (but still selective) retrieval.
With a stronger generator (Table 2), the full frontier shifts upward while the gate ordering becomes clearer. ENTROPY now over-triggers and lags in quality, indicating that prefix entropies compress under sharper models and lose discriminative range. In contrast, MARGIN and VARIANCE achieve near-best accuracy at vanishing retrieval budgets and essentially zero overhead (e-g., +0.012 s on NQ-Open), because the top- 1/top-2 logit gap and small-N disagreement retain spread even when the distribution globally peaked, Practically, use MARGIN by default; use VARIANCE when budgets are extremely tight. The very small retrieval rates (e.g., R 0.001) in these settings should be viewed as extreme-budget points that illustrate the shape of the accuracy efficiency frontier; more moderate budgets (e.g., π between 0.05 and 0.3, see Appendix A.8 show similarly robust gains and are more relevant for deployment.
Table A.10 (see Appendix) situates our numbers against widely reported results that vary in backbone size, trained retrieval/rorankingg corpora, and scoring. These entrics indicate headroom rather than serve baselines. Two messages follow. First, absolute scores scale strongly with backbone and retrieval engineering, Second, selective retrieval is orthogonal to those choices: calibrated, training free gate be dropped into stronger stacks and should continue to reduce unnecossary retrieval and latency.
5.1 Comparison with Expanded Baselines and Agentic Frameworks
To rigorously evaluate TARG against alternative retrieval triggers and reflective frameworks, compare it against SELF-EVAL GATE (prompt-based uncortainty), FLARE (confidenc--riggered dynamic retrieval during decoding), SELF-RAG-LITE (reflective generation), CRAG-LITE (corrective retrieval), and SKR (KNN- prober). To isolate the mating/reffectionn logic from sheer scale, we enforce a strict budget: max retrieval call and max reflection cycle per query, ensuring all methods operate within comparable latency envelopes. Since all baselines are constrained to <1 retrieval call, the average retrieval calls per query equals the reported retrieval rate.
8

---

Published in Transactions on Machine Learning Research (04/2026)
Table 3: Expanded Baseline Comparison (Qwen2.5-7B) including multi-hop and long-form tasks. We report Exact Match (EM), alongside the Retrieval Rate (%) and A Latency (s) relative to Never-RAG. Under a strict budget, TARG (MARgiN) matches or outperforms reflective loops (Self-RAG, CRAG) while triggering retrieval less often. For multi-hop (MuSiQue) and long-form (ASQA), TARG grace- fully manages retrieval tradeoffis without complex multi-step scaffolds. "MuSiQue and ASQA latencies are lower than cisely (max 50 tokens) and ASQA citations are generated using a structured format that terminates early; all conditions are evaluated under identical wall-clock measurement on the same hardware. The 1 indicates that the retrieval rate < 0.1% (rounds to 0.0% at one decimal); this reflects extreme-budget operating points where the logit gap is large enough that essentially no query is flagged for retrieval. More moderate budgets at x 0.3 are discussed in the text.
because shorter generation caps apply: multi-hop queries are answered con-
Method
NQ-Open
TriviaQA
PopQA
MuSiQue
ASQA
Never-RAG
38.8 (0%, 3.29s)
60.8 (0%, 2.95s)
20.0 (0%, 2.13s)
14.6 (0%, 0.18a)
44.5 (0%, 0.10s)
Always-RAG
37.4 (100%, 22.22.)
57.6 (100%, 13.46s)
14.6 (100%, 13.83s)
10.0 (100%,, 122:24s
38.5 (100%, 22.30s)
Self-Evvl Gate
39.4 (22.8%, +1.79k)
38.4 (15.0%, +2,39s)
22.2 (0.2%, +1.16s)
14.6 (24.6%, +5.00is
43.0 (27.0%, +6.07s)
CRAG-lite
38.0 (20.0%, +2.016)
36.6 (18.0%, +2.33s)
21.8 (21.0%, +1.46s)
14.2 (23.0%, +1.7Gs)
42.2 (25.0%, +1.3ls)
F (96.0%, +14.40s)
35.8 (97.8%, +1.17s)
10.6 (99.2%, 2%, +0.77s)
9.0 (98.8%, +17.80s)
38.5 (96.5%, +2.27s)
FLARE(0 0.5) 38.6 (99.6%, 1.100s
56.6 (100.0%, 1131ss
12.8 (100.0%6, 11.73s)
9.8 (100.0%, 14.73s)
37.5 (100.0%, 15.77s)
Sef-RAG-lite 39.0 (19.8%, 12219s)
39.8 (20. 22000% 12.21s)
21.8 (8.8%. (1.87))
13.4 (48.2%, 11.80s)
40.0 (23.0%, 1 (1330)
SKR
39.0 (1.8%, 26s)s)
62.2 (3.8%, +2.27s)
23.0 (6.2%, +1.94s)
14.2 (13.2%, +0.92s)
46.0 (7.5%, +0.78s)
TARG-Margin 39.0 (6.2%, +0.71s)
61.8 (<0..1%* +0.21s)
22.6 (<0.1%* +0.45s)
15.2 (1.6%, +0.16s)
45.5 (6.0%. +0.19s)
Table 4: Cross-Model Expanded Baselines on NQ-Open. We report Exact Match (%) and Retrieval Rate (%) across four combinations of LLM families (Qwen2.5-7B, Llama-3.1-8B) and Retrievers (BGE-M3,
E5- v2). The margin- based gate provides robust discriminative power across architectures and retrievers.
Method
Qwen-BGE
Qwen-E5
Llama-BGE
Llama-E5
Never-RAG
37.2 (0%)
38.8 (0%)
55.8 (0%)
53.8 (0%)
Always-RAG
38.2 (100%)
37.4 (100%)
46.2 (100%)
48.6 (100%)
MARG-Margin
39.0 (~15%)
41.4 (~15%)
57.0 (~15%)
57.6 (~15%)
Self-Eval Gate
39.4 (23%)
37.4 (22%)
45.6 (30%)
46.4 (32%)
FLARE (0 0.5)
38.6 (100%)
37.6 (100%)
49.6 (98%)
50.0 (99%)
SKR
39.0 (5%)
40.2 (6%)
56.0 (4%)
56.2 (5%)
Table 3 summarizes the results on Qwen2.5-7B across our evaluation suile. To ensure these gains are not bapciine-specific, Table | evaluates the expanded baselines across both Qwen and Llama families on NQ- Open. Unlike semantic probers (SKR) or explicit self-reflcction (Self-RAG-lite), the margin signal leverages the inherent sharpness of the backbone's internal probability distribution, allowing TARG to maintain ac- curacy on par with Always-RAG while sharply reducing the retrieval rate to ~5-25% depending on dataset and budget target. Methods like CRAG-lite and Self-RAG-lite are effective but entail substantial genera- tion latency for reflection tokens, whereas TARG requires only a tiny k=20 prefix, minimizing wall-clock overhead. For Table 3 we report. the best-performing operating point under a budget-matching protocol; Tables 2 illustrate representative moderate-budget points (I R 5-30%), and the extremo-budget entries (RR <0.1%) reflect. threshold settings where the logit gap is large enough that essentially no query triggers retrieval.
Across datascts and backbones, the tables reveal a consistent picture. First, ALWAYS is not a safe default: when the lop-K set contains distractors or aliases, small and mid-size generators spend compute on irrelevant context and drift, so accuracy falls while latency rises. The effect is most visible on long-tail entity queries (e.g., PopQA), where retrieval precision is intrinsically harder. Second, the choice of uncortainty signal is pivotal and interacts with backbone sharpness. As instruction-tuned LMs become more peaked, prefix ropios compress and lose ranking power; in contrast, the top-1/top-2 logit gap (MARgIN) and small-N
9

---

Published in Transactions on Machine Learning Research (04/2026)
disagreement (VARIANCE) retain dynamic range and better correlate with cases where external evidence flips or stabilizes the answer. In Appendix A.6 we quantily this by binning queries by the gate score (g) and plotting base-model accuracy, observing a clear monotone decrease as uncertainty increases for the MARGIN and VARIANCE gates. Appendix A.7 further provides a quadrant-basod error analysis over (U()): Ag)) that. highlights where retrieval is benelicial, neutral, or harmful, offering empirical support for the uscfilnsss-allibration assumption in Section 3.3
This explains why, under Llama-3.1-8B, MARGIN/ VARRANCE achieve near-best quality at vanishing re- trieval rates and essentially zero added wall-time, while ENTROPY over-triggers and underperforms. Third, efficiency should be interpreted as a budget, not just absolute time. Reporting A latency isolates the in- cremental cost of retrieval and longer prompts beyond base decoding; the strongest MARGIN/ / /VARIANCE operating points cluster near the NEVER baseline in latency yet exceed ALWAYS in accuracy, yielding a con- trollable and deployment-friendee with both Qwen2.5-7B and 1lama-3.1-8B, TARG improves the fronticr; the preferred gate depends on bud- gel and model sharpness use MARGIN by default, VARIANCE when budgets are extremely tight, and reserve ENTROPY for ablations or weaker backbones.
accuracy -ffiricccceffcicccy frontier. Finally, these gains are backbone-agnostic:
In deployment, calibrate the gate on development set to match a target retrieval budget (via the empirical CDF of the gate score). Use MARGIN by default with modern instruction-tuned LLMs; switch to VARIANCE when budgets are extremely tight. Keep ENTROPY as an ablation or for weaker backbones. Finally, report accuracy her with retrieval rate and A latency so the quality cost frontier is explicit rather than implicit.
6 Discussion
Selective retrieval vs. unconditional retrieval. Across datasets and backbones, uncouditionally retriev- ing (ALWAYS) is not safe default. When the top-K set contains distractors or aliases, the generator spends compute integrating irrelevant context; longer prompts further disperse attention, increasing the chance that salient evidence is ignored. These effects are most acute on long-tail, tail, entity-heavy queries (e.g.. PopQA), where lexical/semantic similarity can be misleading. In contrast, a training-free gate that abstains on easy, high-confidence cases and triggers on genuinely uncertain ones raises the precision of retrieval: prompts stay shorter on average, latency remains close to NEVER, and accuracy does not suffer from "anchoring on a distractor." This explains why ALWAYS pays a consistent latency tax while often underperforming NEVER.
Which uncertainty signal-and why. Backbone sharpness governs the uscfulness of diffcrent prefix signals. As instruction-tuned models become more peaked, prefix entropies compress and lose ranking power- entropy can be low even when the model is confidently wrong so ENTROPY tends to over-trigger and underperform. By contrast, the top-1/top-2 logit gap retains dynamic range: genuine ambiguity shrinks the gap, raising a margin-based score; small-N variance captures a similar phenomenon via disagreement across a few stochastic drafts. Fmppriically, MARGIN and VARIANCE achieve near-best accuracy at vanishing retrieval rates and essentially zero added wall-time under stronger backbones (e.g., Llama-3.1-8B), while ENTROPY lags. A practical rule emerges: use MARGIN by default; switch to VARIANCE when budgets are extremely tight; keep ENTROPY mainly for ablations weaker backbones.
Dataset effects and PopQA. Selective retrieval is most effective when (i) many queries are already covered by parametric knowledge and (ii) occasional retrieval injects noise or redundancy-conditions conditions that hold for NQ-Open and TriviaQA. PopQA stresses long-tail entities, aliases, and temporally sensitive facts; dense retricval can surface ncar-misses with high similarity that distract the generator. Under our deliberately training-free stack (frozen encoder, modest K), absolute headroom is limited, but the gate still improves the quality- cost frontier relative to ALWAYS/NEVER. Orthogonal ppgrades-hyyrid BM25-dense retrieval, cross-encoder reranking, entity-aware scoring-raie absolute accuracy across all modes; a calibrated gate continues to suppress wasteful retrieval on easy queries and preserves efficiency as the stack improves.
Budgets, not just time. We report A latency as added seconds per query over NEVER on the same hardware, isolating the incremental cost of prefix drafting, retrieval, and longer-context decoding from base decoding: E[At] x Edrat + ((T) fretrieval + x fderde/etxx The strongest MAARIN/VARIANN operating points cluster near the NEVER baseline in A latency yet exceed ALWAYS in accuracy, yielding a controllable
10

---

Published in Transactions on Machine Learning Research (04/2026)
and deployment-friendle eeeuracy-effiiiene frontier. In practice, calibrate the threshold to target retrieval budget by matching the empirical CDF of gate scores on development set.
Context against larger systems. Reference numbers from hravy stacks (Table A.10 pro- vide headroom, not baselines: they differ in backbone size, trained /hybrid retrieval and reranking, cor- pora/snapshots, and scoring. Our contribution dropped into stronger stacks: as retrieval/rerankinng improves, all curves shift upward while a calibrated gate continues to avoid unneccssary retricval easy queries.
mrthogonal-eccdiing when to retrieve- and can be
7 Limitations
We intentionally exclude computationally unconstrained multi-step wrappers (e.g., ReAct, multi-agent de- bates) from baseline pool. While such systems push absolute accuracy limits, their unbounded budgets (multiple retrieval calls, unbounded reasoning chains) confound direct comparison against single-shot gat- ing like TARG. Instead, we strictly bound our basclines (Seff-RAG-lite, CRAG-lite) to < reflection cycle. Similarly, we exclude method classes requiring specialized auxiliary models or fine-tuning (e.g.. SUGAR), keeping focus strictly on training-fii free mechanisms; we do, however, test training- frren prompting approaches like SELF-EVAL GATE to provide a boundod-budgot comparison. SELF-EVAL. GATE specifically provides prompt-based retrieval trigger (*do you need to search for this?") under a single-step budget, partially covering the ReAct-style decision mechanism without multi-step tool loops.
TARG uses single threshold tuned on a small development set. Although calibration is fast and stable (via the empirical CDF), thresholds may shift with models, prompts, or domains. Furthermore, suppressing retrieval could increase hallucination risk on out-of-distribution queries; the VARIANCE gate acts as a safer fallback in such settings, and an ensemble gate (MARGIN U VARIANCE) can further reduce the confidently- wrong rate to < 1% at the cost of modestly higher retrieval. We heavily scoped our evaluation to English open-domain QA over Wikipedia. Future work can apply the same training-free gate to hybrid or domain corpora and multilingual encoders, and to tasks such fact verification long-form answer drafting. We deliberately use frozen dense encoder to isolate the "when-to-retrreve" decision. Absolute headroom is bounded by retrieval precision. This is orthogonal to our method: the same gate can ride on stronger stacks (such as cross-encoder reranking).
8 Conclusion
We revisited retrieval-augmmenter generation from a simple angle: deciding when to retrieve with no additional training. The proposed TARG policy reads uncortainty from a short, retricval-freo prefix and triggers retrieval only when warranted. Among training-Tree signals, the MARGIN score derived from the top-1/top- logit gap-emerges as a robust default under modern instruction-tuned backbones; VARIANCE provides a conservative alternative when budgets are extremely tight. Framed through A latency and retrieval rate, TARG consistently shifts the cfffricyy frontier: it matches exceeds NEVER while avoiding the accuracy and latency penalties of ALWAYS, and it does so at vanishing retrieval budgets. In practice, deploying TARG requires only a lightweight one-dimensional calibration of the threshold 7 (and optionally the prefix length k) on a small development set, in contrast to methods that train auxiliary heads or control tokens; our sensitivity analysis shows a broad region of good (k, r) settings rather than a narrow optimum.
Boyond raw numbers, the results deliver two actionable insights. First, unconditional retricval is not a safe default: when top-K contains distractors or aliases, longer prompts and off-topic context hurt both quality and latency. Second, backbone sharpness dictates which uncertainty signal discriminates: prefix entropies compress under stronger models, whereas the logit gap and small-N disagreement retain dynamic range and remain predictive of when external evidence will flip or stabilize the answer. These observations turn a one-line threshold into a practical control knob: set the budget you can afford, calibrate once, and deploy.
TARG is intentionally plug-and-play neither assumes trained probers nor requires changes to the retriever; it adds only tens to hundreds of draft tokens and introduces a single scalar control. As retrieval stacks improve (hybrid search, reranking, better chunking), absolute accuracy rises while the gate continues to
11

---

Published in Transactions on Machine Learning Research (04/2026)
suppress wasteful retrieval on easy -ielding similar larger efficiency dividends at higher quality. We view training-free gating as a basic primitive for RAG systems: small, dependable mechanism that restores precision to retrieval, makes latency predictable, and is simple enough to be widely adopted.
9 Acknowledgments
The authors thank the anonymous reviewers for their constructive feedback, which helped improve the paper substantially. We also thank colleagues and collaborators for helpful discussions and feedback during the development of this work.
10 Conflicts of interest
The authors declare that there are no other interests or personal relationships that could have appeared to influence the work reported in this paper.
12

---

Published in Transactions on Machine Learning Research (04/2026)
References
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reilection. International Conference on Learning Represenlations, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijic Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. Xiv preprint urXiv:2309.1660 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-cficient exact attention with io-awareness. Advances in newral information processing systems, 35:1634441663 2022.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. aiviv preprint 0rXiv22/0100820 2024.
Mazaré,
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625-666)) 630, 2024.
Wikimedia Foundation. Wikimedia downloads. URL https: http:://dumss.iiiineiiiiaa..
wikimedia. org
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Tryna Gurevych. survey of confidence estimation and calibration in large language models. arXiv preprint arXiv:2311.0829. 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint ar 20X 2024.
Kelvin Guu, Kenton Lec, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929 3938. PMLR, 2020.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, al. survey on hallucination large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):
1 55, 2025.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint 0XXi:2007.01282. 2020.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Ncubig. Active retricval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lunguage Processing, pp. 7969-79699 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Super- vised Challenge Dataset for Reading Comprehension. arXiv e-prints, art arXiv:1705.03551. 2017.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Composing retrieval and language models for mhowledge-intensive ulp. arXiv preprinl arXin:2212. 14024, 2022.
Zaharia. Demonstrate--sarch-ppree
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. arXiv preprint Xrx: 190: 003000, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrioval-augmenned generation for knowledge- intensive ulp lasks. Advances in neural information processing systems, 33:9459 9474, 2020.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmented dual instruction tuning. Tn The Twelfth International Conference Learning Representations, 2023.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXio preprinl orXix:2307.0319 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprinl, 2022.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXie preprint arXiv:2303.0888 2023.
OpenAl. Gpt-4-0613 version, 2024.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Mcasuring and narrowing the compositionality gap in language models. arXiv preprint orXi::221000333 2022.
Chaitanya Sharma. Retrieval-nggmented generation: A comprehensive survey of architectures, enhance- ments, and robustness frontiers. arXiv preprint arXin:2506. 00054, 2025.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-War Chang, Asqa: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273 8288, 2022.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop ques- tions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Cunxiang Wang, Haofei Yu, and Yue Zhang. Rfdd: Towards rational fusion-in-decoder for open-domain question answering. arXiv preprint arXi::2305.100. 17041, 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by sealdy-supervised contrastive pre-training. arXiv preprint arXiv:2212.0553 2022.
Shangyu Wa, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei- Kuo, Nan Guan, et al. Retrieval-augmentel generation for natural language processing: A survey. arXiv preprint arXie:2407111990 13199, 2024.
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Represen- lations (ICLR), 2023.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, and Juanzi Li. Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. orXiv preprinl arXiv:2406.1990 19215, 2024.
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retrieval-augmenttr generation in 1lms. Advonces in Newral Information Processing Systems, 37:121156 121184, 2024.
Hanna Zubkova, Ji-Hoon Park, and Scong-Win Whan Lec. Sugar: Leveraging contextual confidence for smarter retrieval. In TCASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
15

---

Published in Transactions on Machine Learning Research (04/2026)
A Appendix
A.1 Proof of Lemma (order-equivalence for the margin gate)
Recall the per-step logit gaps = (1,11 (,12) 2 0 for t = ,,..., k and a strictly decreasing, continuous
margin link y R2o (0, 1] (e.g., ((() = e =/ Define
1
Umar (k; sp)
A
((t));
and retrieve if Umar >>
K 1=1
Fix any "shape" vector (61....,6y 6%) with E. 6e = 0 and consider the family of gap vectors g(u) = pul a1l+0 8, parameterized by the mean gap μ E.9t 9t-
Lemma (Tauaatio-eqqiivalnnce of the margin gate). Let Φ: Th R be strietly decrensing. Fix any "shape" vector d = (61,...,óó, 5x) E R* with Ex= 6 = 0, and define the gap sequence along the location family
g(m) (n 01 ........ E μ PER.
Define the margin-bused gate statistic
Umar (1)) =
k Σ ou + 5z).
Then Um) is strictly decreasing in p. Consequently, for any threshold R there erists (unique) value Pr. such that
Umar(H) ST μ 2T
That is, along the location fasmily g(m), thresholding Umar is order-equivalent thresholding the mean gop μ.
Proof. Fix μ1 p2. For each coordinate E [l....,k} we have μr p22 + 8G, and since is strictly decreasing,
su1 + 66) > oluz + 5.).
Averaging these & strict inequalities yields
Σ Mmn + 66)
Σ ó(uz + 5z),
i.e., (mir(Hi) > (p2). Thus Umar is strictly decreasing in
Strict monotonicity implies that for any T E the equation Umar (#) = T has at most one solution; when a solution exists, denote it by pr. Because Umar is strictly decreasing, we have
Umar (1) ST 2
which shows that thresholding Unar is equivalent to thresholding along the family g())-
Remark (Scope). The lemma establishes order-equivalence only along location shifts g(a) = μ1 + with fixed shape d. For general changes of the per-step gap shape (i.e., changing 8), Umar is still coordinatewise decreasing in each argument, but is not, in general, a function of the mean alone.
A.2 Proof of Lemma 2 (boundedness for the small-N variance gate)
Suppose we draw N short stochastic contimuation; at stop t let Pe be the empirical token distribution and define
k
d, A 1 maxpp(() E 00, 1J, (k,
de
k
16

---

Published in Transactions on Machine Learning Research (04/2026)
Lemma 2. For every t, max, pc(5) 2 Ni hence
N
Nk; N -
0 d
N
N
N
Moreover, the upper bound is tight when all N samples at step are distinct.
Proof. Among N samples step t, the modal token appears least once, so max, (e( N and di 1- Averaging preserves bounds. Tightness: if samples are distinct then max, pr(j) せ and de N-1
N
A.3 Formalizing the intuition in $3.3: accuracy and budget calibration
Let (g), ( E [0, [] denote (calibrated) correctness without/ without/with /with retrieval for query and let A(g) 2 (q) - (g) E [-,, Given a score (4) and threshold т, define the retrieval indicator R- = (q((g) > r}. The gated accuracy is
Agate(T;4) = (4) + A(g) Rx(g).
Dominance over Never-RAG under usefulness calibration. Assume there exists r" s.l.
B[A(g) (g)) S + S O, E(A(g) ((()) > + 20. 2.
Proposition 1 (Weak dominance over NEVER). With r = r",
E [[Agate(T)
2 E ]
Proof. By the tower rule;
= +EAR, E[A + PPUUU>~*))L
E [[ggte((T)) E|
U +*] > [...
When the gate also beats Always-RAG. Since E[A()] = [[A + E[A], we have
E[ [[Agat (T*)] - = E[A] = E[A I(U -T*))-
Proposition 2 (Dominance over ALWAYS under one-sided sign). If ((g) s 0 almost surely on (U(g) - +*} (retrieval never helps in the low-U region), then
E E[Agaro (T*)] 2 B[A"].
Proof. Under the stated sign condition, - EEA ST 2 0.
Budget calibration consistency. Let Für be the CDF of U. For target retricval rate p E [0, 1], set FF
Tp =F= (1- F On an i..d. development set, the empirical quantile fp satisfies f, - To almost surely, and the realized retrieval rate A(fp) + Thus quantile-basod thresholding provides statistically consistent knob for meeting latency/com ccompute budgets.
Cost Cost/Lateness decomposition. Let Tarant = be the prefix tokens, Texx the (bounded) retrieved context size, and ((0) (()) the output lengths without/wie /withh retrieval. With <( Pr(U > 7),
E [(()) Tarat (1- x()) B(T + ((() (Texx BTTHH
so the incremental overhead relative NEVER is directly governed by (T)), which is calibrated by the score quantile.
17

---

Published in Transactions on Machine Learning Research (04/2026)
A.4 Ablation Study: Retriever Sensitivity (E5-base-v2 v2 vs. BGE-m3)
We hold the generator, prompts, corpus, chunking, FAISS lype, and decoding fixed, and swap the frowen dual encoder from E5- BASE-V2 to BGE-M3. For each dataset we report NEVER, ALWAYS, and three training- free gates (ENTROPY, MARGIN, VARIANCE). A Latency is added seconds/ over the datasct's NEVER baseline for the same backbone. Thresholds are the representative settings from the main results. For simplicity, the RR in the following context is short for the Retrieve rate, and A stands for the A latency.
Based on Table S2 and ST
we observed three consistent phenomena:
Unconditional retrieval remains a poor default, irrespective of retriever. Across datasots and both backbones, ALWAYS pays a clear latency tax and frequently underperforms NEVER. With Qwen on TriviaQA, moving from E5 to BGE worsens ALWAYS (F1: 57.2 -1.77 A latency: +3..662 +4.663), highlighting that the bottleneck is retrieval precision rather than the gating policy. Similar patterns appear PopQA and NQ-Open
The ordering of training-fres gates is retriever-agnoosicc; costs remain minimal. Under both E5 and BGE, ENTROPY tends to fire more often (e.g.. Llama/TriviaQA RR = 0.524), raising overhead with only moderate gains. In contrast, MARGIN and VARIANCE achieve near-best FM/F1 at tiny retrieval rates and baselie A latency. With Slama/NQ-Opent MARGIN (E5) attains EM/F1 at RR. = 0.008 and A = +0.012s, while VARIANCE (BGE) reaches the same 17.6/54.7 EM/F1 at RR = 0.054 and A = s0.122s. On / VARIANCE delivers the best BGE quality at 7.4% RR and only +0.151s overhoad, mirroring the E5 story.
Absolute scores shift idiosyncratically with the retriever, but the frontier stays the same. Entropy with BGE on NQ-Open (Qwen) reaches a slightly higher EM/F1 than with E5 (40.9/39.5 vs. 39.6/39.1) but at a much higher budget (Retrieve rate 0.232 vs. 0.046), while MAR- GIN/ CIN/VARIANCE VARIANCE preserve the "near-NeVER latency, taater-than-AAwarr accuracy" property across retrievers. These paired columns make clear that our improvements stem from when-to-retrieve decisions, not from retriever-specific quirks.
From Table S2 and Table ST. we conclude that the deltas for ALWAYS are diagnostic: whenever retrieval precision dips (e.g., BGE on (A)en/TriviQAA), ALWAYS suffers the most, confirming that indiscriminate context is risky. Second, the behavior of ENTROPY under stronger backbones remains consistent across retrievers: peaked next-token distributions compress entropy and reduce ranking power, so entropy- based gates over-trigger (high RR) and add latency without commensurate gains. Third, MARGIN and VARIANCE retain discriminative range because the top-1/top-2 gap and small-N disagreement track instability in the prefix; that instability correlates with cases where external evidence flips or stabilizes the answer. Finally, reading efficiency through A latency shows the budget clarity of gating: the best cluster near the NEVER floor while outperforming ALWAYS, for both retrievers.
points
In short, the TARG method is retriivee-agnnsticc. Upgrading the retriever raises absolute ceilings for all modes, but a calibrated MARGIN/ CARRIINVARLLNCEE gate continues avoid wasteful retrieval on easy inputs and to dominate ALWAYS on the accuracy-efficien frontier. This directly supports the claim that training-free, budget-aware galing is portable primilive for RAG systems.
A.5 Ablation study: Prefix length k.
Table S3 indicates that mid thresholds favor 20-token prefix. At r=0.5, k=20 attains the best overall quality (EM 40.8, F1 39.4) at moderate retrieval rate (23.2%), which is also the global optimum in the sweep-suggeste 20 tokens are sufficient to stabilize the prefix-umcortiinty signal without over-triggerin retrieval. Longer prefixes raise the retrieval rate without consistent gains: at 7-0.3, k=30 improves EM/F1 by only 1-1.3 points yet drives retrieval to 66% (vs. 56% for k=20 and 32.6% for k=10),, an unfavorable quality-cost trade. Shorter prefixes under-inform the gate: with k=10 at r=0.5, quality lags (EM 39.2 F1 37.7) despite frugal retrieval (11.0%), implying ten tokens often fail to expose the uncertainty patterns that predict retrieval benefit. At higher thresholds (7=0.65-0.. quality differences narrow, but k= remains
18

---

Published in Transactions on Machine Learning Research (04/2026)
Table S1: Retriever sensitivity with Qwen2.5-7B-Instrucc. Each row reports a method; columns pair E5-base-v2 vs. BGE-m3. Trends are consistent across retricvers: MARGIN/VANEE /VARIANCE improve the accu- racy efficiecyy frontier with minimal A latency, while ALWAYS pays latency tax and often underperforms NEVER.
E5-basc-v2
BGE-m3 RR
Dataset
Method
EM/F1
RR
(())
EM/F1
(
NEVER
60.8/61.4
0.000
2.947 (abs.)
60.8/61.4
0.000
2.947 (abs.)
ALWAYS
57.6/57.2
1.000
+3.462
52.0/51.7
1.000
+4.663
TriviaQA
ENTROPY
61.8/62.2
0.028
+0.876
62.0/62.6
0.028
+0.404
MARGIN
62.2/62.6
0.338
+2.174
61.8/62.5
0.001
+0.208
VARIANCE
61.8/62.2
0.006
+0.133
62.0/62.7
0.074
+0.151
NEVER
20.0/20.1
0.000
2.129 (abs.)
20.0/20.1
0.000
2.129 (abs.)
ALWAYS
14.6/14.6
1.000
+3.828
16.8/16.8
1.000
+4.650
PopQA
ENTROPY
22.4/22.3
0.124
+1.761
22.0/22.1
0.418
+2.776
MARGIN
23.0/23.1
0.124
+1.761
22.6/22.5
0.002
+1.078
VARIANCE
22.8/22.9
0.182
+1.847
23.0/23.1
0.040
+1.520
NEVER
38.8/37.7
0.000
3.293 (abs.)
38.8/37.7
0.000
3.293 (abs.)
ALWAYS
37.4/36.7
1.000
+2.922
38.2/37.7
1.000
+4.104
NQ-Open
ENTROPY
39.6/39.1
0.046
+0.964
40.9/39.5
0.232
+1.787
MARGIN
39.6/38.8
0.304
+1.295
39.0/38.3
0.062
+0.713
VARIANCE
38.6/38.0
0.012
+0.291
39.4/37.7
0.004
+0.121
Table S2: Retriever sensitivity with Llama-3.1-$B--nstruue Same layout as Table S1
With a stronger
backbone, MARCIN/VARIANA reach near-best quality vanishing retrieval rates and near-zero A latency under both retrievers, while ENTROPY over-triggers.
E5-basc-v2
BGE-m3
Dataset
Method
EM/FI
RR
A(s)
EM/F1
RR
A(s)
NEVER
80.8/80.0
0.000
10.383 (abs.)
80.8/80.0
0.000
10.383 (abs.)
ALWAYS
67.6/67.2
1.000
+1.069
66.8/65.4
1.000
+1.581
TriviaQA
ENTROPY
74.4/74.1
0.524
+0.495
75.6/74.9
0.524
+0.435
MARGIN
83.8/83.0
0.001
+0.018
83.6/82.9
0.002
+0.034
VARIANCE
83.6/83.0
0.001
+0.018
83.4/82.6
0.001
+0.025
NEVER
35.2/34.4
0.000
10.299 (abs.)
35.2/34.4
0.000
10.299 (abs.)
ALWAYS
24.8/24.6
1.000
+1.269
26.0/25.8
1.000
+0.700
PopQA
ENTROPY
28.8/28.8
0.760
+0.974
32.4/32.4
0.760
+0.497
MARGIN
36.6/36.2
0.108
+0.424
36.2/35.6
0.001
+0.063
VARIANCE
36.4/36.0
0.084
+0.317
36.2/35.8
0.028
+0,290
NEVER
53.8/51.7
0.000
10.299 (abs.)
53.8/51.7
0.000
10.299 (abs.)
ALWAYS
48.6/46.1
1.000
+1.248
46.2/44.8
1.000
+1.670
NQ-Open
ENTROPY
55.4/53.1
0.132
+0.175
53.8/51.3
0.228
+0.505
MARGIN
57.6/54.7
0.008
+0.012
57.0/54.1
0.008
+0.028
VARIANCE
56.8/53.7
0.026
+0.059
57.6/54.7
0.054
+0.122
best or tied on EM and best on Fl while koeping retricval low (6-13%). Overall, a 20-token draft offers the best quality-cost balance; we adopt k==20 as the default and calibrate the threshold mid values (e.g., via score quantiles) to meet target retrieval budget.
19

---

Published in Transactions on Machine Learning Research (04/2026)
Table S3: Prefix-length ablation for TARG (gate fixed). Each cell shows EM / F1 [Retrieval Rate] in
%. Bold indicates the best EM (primary) at each threshold. 20-token draft (k=20) yiclds the best overall quality at mid thresholds while avoiding the high retrieval of longer prefixes.
Prefix
0=0.3
7=0.5
70665
T=0.8
k=10 10
38.6
37.5
[32.6]
39.2 / 37.7 [11.0]
39.2
38.2
[6.0]
39.4 / 38.3 [2.8]
k=20
38.2
37.2
[5600]
40.8 / 39.4
[23.2]
39.8
39.2
[13.0]
39.8 / 38.8 [6.0]
k=30
39.6
38.5
[66.0]
40.0 / 38.5
[31.2]
40.0
38.7
[16.0]
39.0 / 37.7 [7.2]
A.6
Uncertainty vs. base-model accuracy
To empirically examine whether the scalar scores U behave as proxies for parametric knowledge, we analyze base- base-model accuracy (Never-RAG) as a function of uncertainty. For each dataset and gate:
We run the zero-RAG baseline and compute the gate score U4) from the k-token prefix for every query
2. We partition queries into B bins (we use deciles, B=10) according to U(4), from lowest to highest
uncertainty.
3. Within each bin, we compute EM for the base generator under Never-RAG.
Figure ST reports EM as a function of the U-quantile for Qwen2.5-7B and Llama-3.1-8B on NQ-Open, TriviaQA, and PopQA. For the MARGIN and VARIANCE gates, base-model accuracy decreases monotonically (or moar-monotonicalll) as U increases on all datasets, indicating that these scores are informative proxies for epistemic uncertainly about parametric knowledge. The trend is strongest. under Llama-3.1-8B, where the distribution is sharper and logit gaps/ensemble disagreement retain more discriminative power. The ENTROPY score exhibits weaker trend, particularly for Llama-3.1-8) 1-1BB, consistent with our main results: prefix entropies compress under sharper models and lose ranking power relative to margin- or variance-based signals.
A.7 Error quadrants over (U()). Aq)))
We next characterize when retricval helps or hurts by analyzing quadrants over the uncortainty score U()) and the retrieval benefit A(q) introduced in Section 3.3 For each query g in development split, we compute:
(g) and (g): correctness under Never-RAG and Always-RAG:
Agg) E (-,,0,1}, defined as +1 if Always-RAG is correct and Never-RAG is wrong, -1 if the converse holds, and 0 otherwise;
the margin-gate score U(q) from the k-token prefix (same k as in the main experiments).
We then fix the same threshold - used our main TARG runs and define low vs. high uncertainty by (g)) S and (q)) r, respectively. This yields four quadrants:
(a) Low U, A 0: base model is wrong and retrieval would help ("confidently wrong, retrieval useful");
(b) High U, A 0: model is uncertain and retricval is harmful ("uncertain, retrieval harmful");
(c) High U, A 0: model is uncertain and retrieval is beneficial (ideal region for TARG to retrieve);
(d) Low U. A < 0: base model is safe to without retrieval (retrieval neutral or harmful).
20

---

Published in Transactions on Machine Learning Research (04/2026)
Table S4: Quadrant analysis over (U(g), on a dev split, using the margin gate and the same 7 as in our main experiments. Values are percentages of queries in each quadrant (rows sum 100%). Model: Qwen2.5--BBInstrucc.
Dataset
(a) Low U, A> (b) High U, << (c) High U, A (d) Low U, A 0
NQ-Open
6.0%
2.2%
4.0%
87.8%
TriviaQA 5.4%
3.8%
2.0%
88.8%
PopQA
1.2%
9.6%
7.2%
82.0%
Table S5: Quadrant analysis over (U(g), on a dev split, using the margin gate and the same T as in our main experiments. Values are percentages of queries each quadrant (rows sum 100%). Model: Llama-3.1-BB-Instruce
Dataset
(a) Low U, A (b) High U, A < (c) High U, A>> 0 (d) Low U, A S 0
NQ-Open
1.4%
10.2%
5.8%
82.6%
TriviaQA
0.6%
15.2%
2.6%
81.6%
PopQA
0.4%
20.0%
9.8%
69.8%
Tables 54 and S5
report the percentage of dev queries in each quadrant for 20wen2.5-7B-nnstrie and Llama- using the margin gate and the same thresholds T as in our main experiments.
Several trends are noteworthy. First, across all datasets and both models, the majority of queries lie in quadrant (d): low-uncertainty queries for which retrieval neutral harmful (82 89% for Qwen2.5, 70- 83% for Llama-3.1). In these cases, TARG's decision not retrieve when (q) S is aligned with the sign of ((4) and avoids unnecessary context.
Second, the problematic quadrant (a): low U but A > 0, where the gate would skip retrieval even though it improves accuracy, is relatively small on all datasets, and becomes very small once we move to the stronger Llama-3.1 backbone (c.g., 1.4% on NQ-Open, 0.6% on TriviaQA, 0.4% on PopQA). This indicates that genuinely "confidently wrong, but fixable by retrieval" cases are rare under the margin gate, especially for the sharper model.
Third, the high-uncertainty region concentrales most of the examples where retrieval meaningfully changes outcomes. On Qwen2.5, the combined mass of quadrants (b) and (c) is modest (typically 6 6-17%), reflecting that Always-RAG differs from Neve--RAG on a relatively small set questions. On Llama-3.1, a larger fraction of queries fall into (b) and (c), particularly on PopQA (29.8% total), with a substantial subset in (c) (e.g., 9.8% for PopQA), where retrieval improves accuracy precisely when the model uncertain. The quadrant (b) shows high U but A 0, which captures cases where the retriever surfaces distractors entity aliases and retrieval becomes harmful; its larger mass on PopQA and TriviaQA highlights that these datasets pose a more difficult retrieval problem.
Overall, this quadrant analysis supports the usefulness-calibration assumption of Section 3.3 while U(4) and (gg) are not perfectly aligned on every query, low-uncertainty queries are overwhelmingly those where retrieval unnecessary or harmful, and high-uncertainty queries are where retrieval most likely to change the answer (cither positively or negatively). TARG therefore concentrates retricval in the region where external evidence is most consequential, while keeping the risk of skipping beneficial retrieval (quadrant (a)) small, especially for the stronger backbone.
The quadrant analysis above is descriptive of the benchmark distributions under our specific stack (frozen dense relriever over chunked Wikipedia) and serves to check the usefulness-calibratiin assumption in Sec- tion 3.3 empirically. In particular, we observe that the problematic "low-U, A > 0" region (quadrant (a)) is small, while high-U mass (quadrants (b)+(c) concentrates queries where retrieval changes the answer. However, this behavior not universl.. In adversarial or tiered routing settings that deliberately concentrate traffic in quadrants (a) and (b), the conditional expectations in the asefuhhess-allibratin inequalities need
21

---

Published in Transactions on Machine Learning Research (04/2026)
not hold, and a single threshold need not exist. Such regimes lie outside the scope of our simple dominance argument in Section 3.3 and would require re-calibration of the gate or alternative control strategies under that shifted distribution.
A.8 Sensitivity to threshold and prefix length
To assess the cost of calibration, perform a more extensive sweep over the prefix length k and the decision threshold r for the margin gate. On NQ-Open, for each & E {5, 10, 20, 30, 40} we:
compute the margin score Umar(q) for every dev query using the k-token prefix, and
evaluate EM over a grid of thresholds т corresponding to fixed quantiles of the empirical CDF of Unar (we use equally spaced quantiles).
Figure
S2
shows EM as function of (k, for Qwen2.5-7BB Inessttt-B- Instruct and Llama-3.1 3.1-8B-Instrus Two trends
emerge:
There is broad region of good performance: for each model, wide band of (k, r) values yields EM close to the best observed value, rather than single needle-like optimum.
Moderate mis-specification of either &t 7 has limited impact on EM and retrieval rate, especially in the regime of modest retricval budgets (æ between 0.05 and 0.3).
Figure S3 provides 1D slices: EM versus T for fixed 2=20 and EM versus when is chosen to match fixed retrieval budget. These plots confirm that TARG does not require fine-grained tuning and that a simple one-dimensional grid search over on a small dev set suffices in practice.
A.9 Dynamic re-check ablation
Algorithm 1 admits an online variant which the gate can be ro-ovaluated periodically and retrieval can be triggered mid-generation. Concretely, we consider simple dynamic re-check policy on NQ-Open with Owen2.5--7BIIstrruuc and the margin gate: starting from the base prompt B(q), we decode in fixed blocks of m tokens, recompute (g)) on the rumning prefix after each block, and, if U (g) > 7 at any checkpoint and retricval has not yet occurred, we retrieve once and contimue decoding from B(q) @ C.
Figure compares this dynamic re-check variant against single-shot TARG as a function of the threshold T (from 0.1 to 0.8), reporting EM, retrieval rate, and latency overhead relative to the Never-RAG baseline Several trends emerge:
For low to moderate thresholds (r E (0.1, 0.4)), the dynamic re-check policy is extremely aggressive. For example, at r=0.3, single-shot TARG achieves EM 39.0% with retrieval rate of 6.2% and & 4.1 s/query latency, whereas dynamic re-check achieves EM = 38.8% with a retrieval rate of 88.2% and x (0.6r//uery latency, i.c., ~ 16.6s extra wall-clock time per query for essentially no accuracy gain. At 7=0.1, retrieval rate saturates at 100% and latency almost doubles relative to singlo-shot.
At higher thresholds (r 2 0,5), single-shot TARG rarely or never retrieves (retrieval rate 0- 0-10%), while dynamic re-chock continues to incur a large latency overhoad (== 16-17s/query) despite only modest changes EM (improvements al most about points). This reflects the cost. of repeatedly scoring prefixes during generation, even when retrieval rarely triggered.
Across the entire sweep, EM differences between single-shot and dynamic re-check remain small (within x +2 points), whereas retricval rate and latency can increase dramatically under dynamic re-check, especially at thresholds that are otherwise reasonable for single-shot gating.
immpleeentation details and numbers provided the released scripts.
22

---

Published in Transactions on Machine Learning Research (04/2026)
Overall, this ablation shows that naive dynamic re-check policy with fixed threshold is not cost-effective for the short-answer QA regime studied in this work: it either degenerates toward Always-RAG (very high retrieval rates and large latency overhead) or incurs substantial extra computation for marginal accuracy changes. For this reason, we adopt single-shot TARG in all main experiments and view dynamic re-check as an extension that may be better suited to long-form generation with more sophisticated, possibly adaptive thresholds.
A.10 External Reference Systems (Absolute Headroom)
Table 56 situates our numbers against widely reported results that vary in backbone size, trained re- trieval rreankking,,, corpora, and scoring. These entries indicate headroom rather than serve as baselines. Two messages follow. First, absolute scores scale strongly with backbone and retrieval engineering. Second, selective retrieval is orthogonal to those choices: calibrated, training-free gate can be dropped into stronger stacks and should continue to reduce unnecessary retricval and latency.
Table S6: External reference systems (context only). Results reported by prior work on our evaluation datasets. These systems are not direclly comporable to our training-free, frozen-retriever setup: most use larger backbones and/or trained retrievensrrerankers and may differ in corpora and scoring. Values are EM or EM/Ace, whichever available.
NQ EM
TriviaQA EM
PopQA EM
Models
Withoul Retrieval- Auginnteed Generalion
GPT-4-0613 APenAA 2024
40.3
84.8
31.3
1e1-4tuurb-20024--11
(OpenAl
2024
41.5
80.0
25.0
With Retrieval- Augmented Generation
FiD-Large IIacard & Grave
2020
51.4
61.6
RFiD-Large Waan et al.
2023
54.3
72.6
RA-DIT 65B Lin et al.
2023
35.2
75.4
Mlama3-RankHi 8B YYu
2024
50.6
82.9
57.6
Blama3-RankRAG 70B Yu et al. 2024
54.2
86.5
59.9
Qwen2.5 7B-Entropy (r 0.85, 0.8, 0.8)
39.6
61.8
22.4
Qwen2.5 7B-Margin (r = 0.2, 0.15, 0.35)
39.6
62.2
23.0
Qwen2.5 7B- "T-VVarianee (r 0.6, 0.75, 0.4)
38.6
61.8
22.8
LLAMA3.1 8B-Entropy (r 0.95, 0.8)
55.4
74.4
28.8
LLAMA3.1 8B-Margin (r 0.5, 0.45)
57.6
83.8
36.6
ILAAM3.1 Varviniecce (T = 0.6, 0.55)
56.8
83.6
36.4
A.11
Expanded Baseline Verification for Llama-3.1-8B
The trends observed in Table hold precisely for Llama-3.1-8B TARG's margin gate robustly identifies uncertain queries, pulling latency back down perfectly while hitting comparable or superior Exact Match numbers uniformly against heavy reflective cycles like Self-RAG-lite and CRAG-lite.
A.12 Robustness to Prompt Templates
To verify that the uncertainty signals are not an artifact of a specific prompt, we tested three prompt template styles on NQ-Open using Llama-3.1-8B: (1) Default: "You are a helpful assistant. Answer concisely and factually." (2) Directive: "Answer the following question in one sentence." (3) Step-by-step: "Think step by step, then give concise final answer." As shown in Table [S7] the margin gate remains discriminative across all templates. The recalibrated threshold shifts by less than 0.08 between templates, well within the broad stable region where performance varies minimally. Even when transferring the threshold tuned
23

---

Published in Transactions on Machine Learning Research (04/2026)
the default prompt zero-shot to the directive prompt, EM drops by just 0.6 points. However, extreme prompt changes (like "think step-by-step", which drastically inflates verbosity and uncertainty) naturally trigger higher retrieval rates absent recalibration, suggesting a quick recalibration on a small dev set is recommended when making major structural changes to the prompt.
Table S7: Prompt template sensitivity (NQ,-pen, Alama-3.1-8B). Using the Margin gate. We report EM at a fixed threshold (r = 0.25, tuned on Default), the resulting Retrieval Rate (RetRate), and the new threshold and EM when recalibrated to match the original retrieval budget.
Template EM (Fixed (T ( 0.25))
RetRate
Recalibrated т EM (Recalibrated T)
Default
51.6%
45.2%
0.26
52.2%
Directive
51.0%
61.4%
0.30
51.8%
Step-by-step 42.2%
75.8%
0.33
49.4%
A.13 Robustness to Decoding Strategy
Our main experiments use greedy decoding to isolate the effect of the retrieval gate. To ensure the threshold holds under sampling, we evaluated NQ-Open with Qwen2.5-7B under nucleus sampling ==70.7, 0.9). Using the 0.25 threshold calibrated under greedy decoding, the sampling run yields virtuually identical retrieval rate (12.2% vs 12.0%) and only minor change in EM (39.4% 38.8%). Recalibrating to target = 16% retrieval shifts to 0.235 and maintains EM at 39. 39.2%. This confirms that gating signals derived from greedy prefix logits remain stable predictors of uncertainty even when subsequent generation usee sampling.
A.14 Datsse Threshold Transferability
A key advantage of TARG is its training-free nature. Table 58 shows the effect of calibrating on NQ-Open and applying it zero-shot to TriviaQA and PopQA. The transferred T provides reasonable out-of-the-box performance, often keeping EM within a few points of the re-tuned optimum. We recommend a single-pass calibration on & 100 dev samples when deploying to a new domain to maximize efficiency, but the gate behaves prodictably even under zero-shot domain transfer bounds.
Table S8: Cross-datasct 7 transfcrability. We transfer the optimal 7 found on NQ-Open zero-shot to TriviaQA and PopQA, comparing the resulting "Transferred EM" to the "Re-tuned EM optimized directly on the target dataset.
Transfer Setting
Transferred EM
Re-tuned EM
A
TT-TTrivaaQA (Llama-BGE)
68.2%
83.6%
15.4
NQ PooPA (Llama-BGE)
27.6%
36.2%
-8.6
NQ-iTvvvaQAA (Qwen-BGE)
56.4%
61.8%
-5.4
NQ >PopQA (Qwen-BGE)
16.2%
22.6%
-6.4
24

---

Published in Transactions on Machine Learning Research (04/2026)
Qwen2.5-7B
Llama-3.1-8B
1.0
1.0
Margin Variance
Entrapy
0.8
0.8
NQ-Open EM
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
1.0
1.0
0.6
TriviaQ
EM
0.4
0.4
0.2
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
1.0
1.0
0.8
0.8
0.6
0.6
PopQA
EM
0.2
0.0
0.0
U-Quantile
U-Quantile
Figure S1: Base-model accuracy vs. uncertainty quantiles. For each gale and dataset, we bin queries by the score (q) and plot Never-RAG EM in each bin. Margin and variance gates show a clear decreasing trend, supporting their use as training-free proxies for parametric uncertainty.
25

---

Published in Transactions on Machine Learning Research (04/2026)
0.40
0.36
0.37
0.37
0:39
0.39
0.38
0.37
0.36
0.36
0.38
0.37
0.37
0.39
0.37
0.39
0.38
0.38
0.40
0.39
0.38
0.38
0.38
0.36
0.41
0.39
0.39
0.37
0.37
0.36
0.38
0.38
0.40
0.39
0.39
0.36
0:37
0.38
0.38
0.39
0.39
0.37
Prefix length
0.36
20
0.39
0.38
0.37
0.39
0.40
0.37
0.34
30
0.38
0.39
0.37
0.39
0.38
0.38
0.37
0.36
0.38
0.40
0.39
0.38
0.38
0.36
0.39
0.40
0.39
0.39
0.40
0.38
0.37
0:37
0.38
0.30
0.01
0.02
0.04
0.05
0.06
07
0e
0.09
0.10
0.15
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.
Threshold
Figure S2: EM as function of (k: r) for the Margin gale on NQ-Open, Owen2.5-7BB-nniiee model.
broad region of high performance indicates robustness to the choice prefix length and threshold.
5-5%
1=10%
2.2
0.25
EM
3.30
0.29
022
9.4
0.00
Prefix langth
Threshold
Figure S3: 1D sensitivity slices for Qwen 355-7BIInstrue model NQ-Open dataset, using margin gale, Left: EM versus for fixed k=20. Right: EM versus when 7 is chosen to match a fixed retrieval budget. Performance varies smoothly, supporting the claim that calibration is cheap and robust.
26

---

Published in Transactions on Machine Learning Research (04/2026)
Exact Match
42
Single-shot TARG
TARG + re-check
40
EM (%)
38
36
34
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Threshold
Retrieval Rate
100
Single-shot TARG
TARG + re-check
Retrieval rate тt (%)
80
60
40
20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Threshold
Latency Overhead
17.5
15.0
Latency (s/q)
12.5
10.0
7.5
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Threshold
Figure S4: Dynamic re-check vs. single-shot TARG on NQ-Open (Qwen2.5-7B, Margin gate). Exact Match (top), retrieval rate (middle), and latency overhead relative to Never-RAG (bottom) as a function of the threshold т. Dynamic re-check yields similar EM but dramatically higher retricval and latency at many thresholds, motivating our choice to use single-shot gating in all main experiments.
27


Reviews:
[{"review_id": "xScDstvQy0", "reviewer": "Reviewer_Bti7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes TARG (Training-free Adaptive Retrieval Gating), a method for dynamically deciding when to invoke retrieval in RAG systems. The core idea leverages uncertainty signals extracted from a short prefix (k=20 tokens) to determine whether external knowledge is necessary. The authors evaluate three training-free signals: average token entropy, the top-1/top-2 logit gap (mapped via exponential decay), and small-N variance across random prefixes. Experiments conducted on NQ-Open, TriviaQA, and PopQA using Qwen2.5-7B and Llama-3.1-8B demonstrate that gap-based signals provide the most robust criterion for retrieval decisions, achieving a 70–90% reduction in retrieval calls while maintaining or improving accuracy compared to an always-retrieve baseline.\n\nStrengths\n1. The paper addresses a critical problem with significant practical value for RAG systems.\n2. The proposed approach is intuitive and logically coherent.\nWeaknesses\n1. The experimental scope is relatively narrow, covering only QA tasks. It remains unclear how the method would generalize to other task types—particularly long-form text generation—especially given its reliance on a k-token prefix.\n2. The paper lacks comparisons with essential baselines such as FLARE, Self-RAG, and CRAG.\n3. The robustness of the threshold selection (e.g., sensitivity to different values or datasets) requires further discussion.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper lacks essential baseline comparisons and additional experimental validation.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a critical problem with significant practical value for RAG systems.", "requested_changes": "1. Supplement the paper with experimental results across additional scenarios and task types.\n2. Add evaluations against the aforementioned baselines (FLARE, Self-RAG, CRAG).\n3. Include a discussion on threshold robustness, such as sensitivity analysis or criteria for threshold selection across different domains.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oRbHSNAF3P", "reviewer": "Reviewer_jvch", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of when to trigger retrieval in retrieval-augmented generation (RAG) systems. It proposes Training-free Adaptive Retrieval Gating (TARG), a lightweight method that decides whether to retrieve based on uncertainty signals derived from a short no-retrieval prefix of the base LLM. The method uses simple statistics from prefix logits (entropy, top-1 vs top-2 logit margin, or small-N stochastic variance) and triggers retrieval when a threshold is exceeded. The authors argue that this training-free gate improves the accuracy–efficiency trade-off compared to Always-RAG and Never-RAG baselines across several open-domain QA datasets, while introducing minimal latency overhead.\n\nStrengths\n\n- Clear problem formulation: selective retrieval is practically important for cost and latency control in RAG systems.\n\n- Simple and easily deployable approach that is claimed to not require training (with caveats as discussed below) or auxiliary modules.\n\n- Empirical evaluation across multiple datasets and two backbone LLMs.\n\n- Useful discussion of accuracy–latency trade-offs and threshold calibration.\n\nWeaknesses\n\n- Contribution relative to existing conditional retrieval, uncertainty-based routing, and agentic tool-use literature is unclear.\n\n- Baselines may be insufficient for establishing meaningful progress over existing adaptive retrieval or agentic systems.\n\n- Several design choices (thresholding, hyperparameters, robustness across domains/models) remain underexplored or insufficiently justified.", "claims_and_evidence_yn": "No", "claims_explanation": "The empirical results show that the proposed gating method can improve the accuracy–efficiency frontier relative to Always-RAG and Never-RAG baselines on several QA datasets. However, the evidence is currently insufficient to fully support the broader claims about contribution, robustness, and general utility (Table 3 is not very convincing to the reviewer because there are too many variables by just reporting numbers from other people's *systems*).\n\n1. The central contribution is a training-free decision rule for when to retrieve. However, the paper does not convincingly establish contribution relative to a large body of existing work on conditional retrieval and tool-use gating. In particular, the following relevant directions appear either missing or insufficiently evaluated: FLARE, Self-RAG, and recent “uncertainty-aware retrieval” or retrieval-budget routing methods. Also, modern agentic systems already decide when to search or retrieve (e.g., ReAct-style tool use, tool-calling LLMs, planning-based agents). There're also training-free or low-resource alternatives like SUGAR and zero-shot self-evaluations. Without such comparisons, it is difficult to assess whether the proposed method advances the state of the art or mainly simplifies existing ideas (which may have value, but limited).\n\n\n2. The method relies heavily on prefix logit-derived uncertainty. While experiments suggest margin-based signals correlate with retrieval usefulness, robustness is not fully established, e.g., How stable is the method across model families? How sensitive are results to prompt design, decoding strategy, or temperature? Overall, the paper claims model-agnosticism but provides limited cross-model or cross-domain validation.\n\n\n3. The method depends on: Prefix length k, Threshold τ, Margin temperature β, Variance sample size N. While calibration is described as simple, there is limited analysis of and hyperparameter sensitivity and cross-dataset transferability.\n\n\n4. The evaluation focuses on short-answer QA. However, retrieval decisions in real systems often occur mid-generation or in multi-step reasoning.", "audience_yn": "Yes", "audience_explanation": "Selective retrieval and inference-time cost control are important topics for both research and deployment of LLM systems. A simple, training-free gating mechanism is appealing and may be useful in practical RAG pipelines.", "requested_changes": "Below is a prioritized list of requested revisions.\n\nCritical for acceptance\n\n1. Stronger positioning and related-work comparison\n\nProvide a clearer contribution statement relative to conditional retrieval, uncertainty-based routing, and agentic tool-use literature.\n\nAdd comparisons against representative baselines beyond Always-/Never-RAG:, e.g. Prompt-based retrieval triggering (e.g., ReAct-style search). Learned or heuristic retrieval routers. Existing uncertainty-based gating methods.\n\nClarify what is fundamentally new: the specific uncertainty signal, the training-free setup, or the single-shot design.\n\n2. Broader and more realistic baselines. Evaluate against at least some of: Tool-use or agentic retrieval baselines. Lightweight learned gating models. Multi-step or mid-generation retrieval methods. Also, recent \"training-free\" or \"low-resource\" alternatives like SUGAR or even simple zero-shot self-evaluations (asking the model \"Do you need to search for this?\") should be included as baselines to truly establish TARG's contribution and efficiency.\n\n3. Provide experiments or analysis showing: Sensitivity to model family and scale. Transfer across datasets/domains (e.g., The authors mention some are calibrated via a validation set. Please provide an experiment or discussion on cross-domain robustness: if calibrated on NQ-Open, how does it perform on PopQA without re-calibration? This is vital for the \"training-free\" claim to hold in practical, out-of-distribution scenarios.)\n\n4. Thresholding and hyperparameter clarity. Clarify: How τ is selected in practice? Whether per-dataset tuning is required? Sensitivity to k, β, N. \n\nImportant but not critical\n\n5. Evaluation scope: Extend or discuss applicability beyond short QA. Evaluate dynamic retrieval vs single-shot gating\n\n6. Analysis of failure modes: When does the gate fail? When does retrieval help despite low uncertainty? Interaction with retriever quality\n\n7. Practical deployment considerations, such as integration with agentic pipelines", "broader_impact_concerns": "No major ethical concerns beyond standard issues related to factuality and retrieval reliability. If deployed widely, incorrect gating decisions could suppress necessary retrieval and lead to hallucinations. A brief discussion of such risks and mitigation strategies would be beneficial.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wb8fCVod04", "reviewer": "Reviewer_xC5g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors introduce TARG - a lightweight method to decide when to retrieve in RAG systems. \n- More specifically, instead of retrieving for every query or training a specialized classifier, TARG generates a short prefix draft from the base LLM. It then computes an uncertainty score based on the logits of this draft. \n- Key finding is that entropy fails on modern instruction-tuned models because the distributions are too peaked. \n- In experiments, the authors show that the proposed method reduces retrieval volume by 70~90% while matching or improving accuracy compared to Always-RAG on multiple benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The experimental results on standard open-domain QA benchmarks are convincing.\n- The results show a clear improvement in the efficiency with maintaining the accuracy.", "audience_yn": "Yes", "audience_explanation": "- The authors show thatstandard entropy metrics fail on modern instruction-tuned models. Identifying logit gap as a robust alternative provides a practical takeaway for the broader NLP community.\n- The proposed method is completely training-free and adds negligible latency. It represents a plug-and-play solution that can be immediately adopted in diverse RAG systems to reduce costs.", "requested_changes": "- The paper compares primarily against Always-RAG and Never-RAG. It lacks direct empirical comparison with other adaptive retrieval methods such as Self-Knowledge Guided Retrieval (https://aclanthology.org/2023.findings-emnlp.691.pdf) which is also training-free. Some other example is FLARE (https://arxiv.org/pdf/2305.06983).\n- The reliance on a short prefix assumes uncertainty is detectable immediately. This heuristic works for simple QA (like the current benchmarks that the authors utilized in the paper). However, it may fail on complex reasoning or multi-hop tasks where uncertainty or hallucinations occur later in the generation. \n- Conceptually, if the gate decides to retrieve, the computed draft is discarded. While the cost is low, it is a slight computational inefficiency inherent to the design.", "broader_impact_concerns": "N/A", "additional_comments": "- Did the authors analyze cases where the model is confidently wrong? How frequent are these pathologies in the datasets tested?\n- Is the threshold tau sensitive to the prompt template used? If the prompt changes, does the calibration need to be re-run entirely?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)