Frozen Priors, Fluid Forecasts: Prequential Uncertainty for Low-Data Deployment with Pretrained Generative Models
Authors: Fernando Ruiz-Mazo
Vikas K Garg
Venue: ICLR 2026 Poster (2026)
Keywords: predictive uncertainty quantification
prequential inference
measure-valued martingales
frozen generative models
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: Provably coherent prequential UQ for frozen generative models: a decaying-weight blend with a GPU-parallel martingale posterior that yields calibrated, small-n intervals for long-run operational rates.

Abstract: Deploying ML systems with only a few real samples makes operational metrics (such as alert rates or mean scores) highly unstable. Existing uncertainty quantification (UQ) methods fail here: frequentist intervals ignore the deployed predictive rule, Bayesian posteriors assume continual refitting, and conformal methods offer per-example rather than long-run guarantees. We introduce a forecast-first UQ framework that blends the empirical distribution with a frozen pretrained generator using a unique Dirichlet schedule, ensuring time-consistent forecasts. Uncertainty is quantified via martingale posteriors: a lightweight, likelihood-free resampling method that simulates future forecasts under the deployed rule, yielding sharp, well-calibrated intervals for both current and long-run metrics without retraining or density evaluation. A single hyperparameter, set by a small-$n$ minimax criterion, balances sampling variance and model--data mismatch; for bounded scores, we provide finite-time drift guarantees. We also show how this framework informs optimal retraining decisions. Applicable off-the-shelf to frozen generators (flows, diffusion, autoregressive models, GANs) and linear metrics (means, tails, NLL), it outperforms bootstrap baselines across vision and language benchmarks (WikiText-2, CIFAR-10, and SVHN datasets); e.g., it achieves $\sim$90\% coverage on GPT-2 with 20 samples vs.\ 37\% for bootstrap. Importantly, our uncertainty estimates are operational under the deployed forecasting rule agnostic of the population parameters, affording practicable estimators for deployment in real world settings. Code available at \url{https://github.com/Aalto-QuML/Prequential/}.

Published as a conference paper at ICLR 2026
FROZEN PRIORS, FLUID FORECASTS: PREQUENTIAL UNCERTAINTY FOR LOW-DATA DEPLOY- MENT WITH PRETRAINED GENERATIVE MODELS
Fernando Ruiz-Mazo
Vikas Garg
Department of Computer Science, Aalto University; DPMMS, University of Cambridge Er 4400 cam. ac.uk
Aalto University
Yai Ltd vgarglcsail mit edu
ABSTRACT
Deploying ML systems with only few real samples makes operational metrics (such as alert rates or mean scores) highly unstable. Existing uncertainty quantifi- cation (UQ) methods fail here: frequentist intervals ignore the deployed predictive rule, Bayesian posteriors assume continual refitting, and conformal methods offer per-example rather than long-run guarantees. We introduce forecast-first UQ framework that blends the empirical distribution with a frozen pretrained generator using unique Dirichlet schedule, ensuring time-consistent forecasts. Uncertainty is quantified via martingale posteriors: a lightweight, likelihood resampling method that simulates future forecasts under the deployed rule, yielding sharp, well-calibrated intervals for both current and long-run metrics without retraining density evaluation. single hyperparameter, set by small-n minimax criterion, balances sampling variance and model--atta mismatch; for bounded scores, provide finite-time drift guarantces. We also show how this framework informs optimal retraining decisions. Applicable off-the-shelf to frozen generators (flows, diffusion, autoregressive models, GANs) and linear metrics (means, tails, NLL.) it outperforms bootstrap baselines across vision and language benchmarks (WikiText- 2, CIFAR-10, and SVHN datasets); e.g., it achieves ~900% coverage on GPT-2 with 20 samples 37% for bootstrap. Importantly, our uncertainty estimates are operational under the deployed forecasting rule agnostic of the population parameters, affording practicable estimators for deployment in real world settings. Code available at https:/ thub. com Aa to QUM.. ia
INTRODUCTION
When machine learning system deployed, must begin operating on real-world data, often with only handful of examples available. Despite this data scarcity, practitioners still need to answer key operational questions early on regarding how the model will behave in the long run. For example:
What fraction of cases will trigger alert if keep using the system? (e.g., for automated trading setup, the proportion of minutes that would hit a safety stop)
What is the expected value of particular score or metric in the long term? (e.g., for new health program or app, the long-run average outcome per person, such as "high blood pressure".)
Formally, let Y y be a future outcome and h y - IR fixed score function. Ideally, to respond to these questions, we would calculate the expected value of the score function under the true (but unknown) data distribution Y Ft:
0(F*) = hhFF.
(
This includes quantities like Pr(h(Y) 7} (a tail) and E[)(Y)) (a mean). However, with only
small sample Yi. the empirical distribution F is unstable and yields unreliable estimates.

---

Published as a conference paper at ICLR 2026
Throughout, let denote the number of real observations actually in the new deployment environment. In staged rollouts and safety-gated launches, n is intentionally small at the beginning due to human quality checks, rate limiting, and policy restrictions. Our methods are expressly designed for this small-n regime, where classical resampling often undercovers.
practical workaround (used in domains such translation, vision, and medical imaging (Sennrich) Azizi
et 2016)] Azizi et al. 2023 Yi et al. 2019) is to borrow stability from a pretrained generative model the Appendix for more details). Let Qo be the distribution defined by such a model. Importantly, we do not assume Qs F*; instead, we use it as a prior-like
trained on similar data (see section
stabilizer that fades out more real data arrives.
1.1 CONTRIBUTIONS
Summary of contributions
New forecasting method:
Prequential Blend
Section
Theoretical Analysis:
Adjusting the blend
Section
Adjusting the predictive horizon
Section
Main algorithm (Martingale Posterior adaptation)
Section
use case of the framework:
Optimal retraining of deployed generative models
Section
Experiments:
Two Moons
C.1
GPT-2
C.2
CIFAR-10 10 (ID C.3.1 and SVHN (OOD C.3.2
Section
Operationally, our target is the long-run metric under the deployed rule, 9x., rather than the population parameter ((F)).
Given this setting, how do we integrate the "synthetic" data generated by this generative model Q with the data start receiving from the real distribution F*? How do do it in way that is mathematically sound and rigorous? And even more, are there multiple ways of doing this or just one? To answer these questions,
propose prequential forecasting approach (Dawid) 1984] Fortini & Petrone 2025), where at each time step i (each new data point/batch arrives) we blend the empirical data distribution with the fixed generator one:
P1= P. Qa.
with A
a ita O.
showing in section that this Dirichlet-style blending rule is the only affine combination that ensures time-consistent forecasts: for any bounded score h. the sequence {0( (P(P:)} forms martingale. This means forecasts are stable early on (shrinking toward the model) and gradually revert to empirical behavior data accumulates.
Reality vs. Surrogate Forecasting
We do not claim that real data follows our forecasts. The true data distribution F* may differ from both the generator and the blended forecast P.. We deliberately choose this surrogate forecasting system to stabilize small-sample metrics. Our theoretical results (on coherence, martingales, and uncertainty quantification) apply this surrogate process and its operational targets O(P:), not to ((F*).
Crucially, this same mechanism enables uncertainty quantification (UQ) without retraining or likeli- hoods. By simulating future forecasts under the deployed rule and tracking only the score values, construct in section 55 martingale posterior (MP) (Fong al. 2003),, calibrated predictive distribution over the long-run metric 0z.. This yields lightweight, coherent uncertainty intervals that reflect what operations actually the forecasted score under the deployed blend.
2

---

Published as a conference paper at ICLR 2026
Furthermore, also give a principled way of choosing the hyperparameter in our blend (section 4 as well as the predictive horizon to simulate our MP resamples (section 6). Finally, we leverage our theory to analyze how our framework may be used to optimally decide when to retrain deployed generative model.
Why not employ standard Uncertainty Quantification (UQ) techniques? Frequentist intervals target population parameters of the unknown data law and ignore the deployed predictive rule, often becoming conservative at small n. Bayesian posteriors integrate parameter uncertainty and typically presume continual refitting, which conflicts with frozen deployments. Conformal prediction provides per-example coverage rather than uncertainty for long-run rates under a fixed rule (Vovk et al. 20055 Angelopoulos & Bates 2023). Raw model likelihood is unreliable under shift (e.g., flows giving higher likelthood to out of of-ssstributtion (OOD) samples than in- in-distribution (ID) ones), and likelihood- ratio fixes still rely on density modeling (Nalisnick et al., 2019 Ren et al., 2019 Kirichenko et al.) 2020). A more in-depth discussion of these techniques can be found in section A of the Appendix. These limitations of the standard methodologies motivate our approach.
Scope. The method applies to any frozen generator with a fixed sampler and evaluable score h: explicit-density models (flows, VAEs; with h(y) = - log go(y) when available), diffusion/score- based models with fixed schedules, autoregressive LMs with fixed decoding, implicit samplers (GANs), and EBMs with frozen MCMC kernel. Not for: online updates to parameters//eccodnng/// CMC/prompts (breaks coherence); targeting population parameters of F* (prefer frequentist tools); or per-example coverage (prefer conformal prediction (Angelopoulos & Bates 2023)). We assume frozen generator Qs: the parameters @ are fixed at deployment start, so Q is meassrrable for all i; any internal sampling randomness (e.g., diffusion noise or GAN latent 2) is taken conditional on Qo and is not updated online. Because we only require sampling X ~ Q2 and evaluating h(X), the procedure is architecture-agnostic and applies under both domain and semantic shifts.
2 PRELIMINARIES
At a glance. We set up the prequential (forecast-firs) framework and notation. Under (scalar) prequential coherence, the one-step forecasts (P.) make the scalar functional ((P.) h dPP, martingale for any bounded (or L.²) score h, hence O(P.) + 0.c. This justifies targeting the long-run operational metric Ox.. and using the martingale posterior (MP) lMP(- F.)) for uncertainty; a Freedman- type deviation bound later sizes simulation horizons and provides conservative drift diagnostic.
This section recalls the prequential (forecast-first) viewpoint and the minimal martingale machinery we use later. Mainly, this point of view of statistics puts our source of uncertainty in the outcomes we have not seen yet 11,+1,3 Yu+2. ...) contrary to the Bayesian approach, which puts on the parameters of our distribution; see Dawid (1984) and surveys such as Fortini & Petrone (2025) for background. For martingale posteriors see Fong et al. (2023).
Data stream and forecasts. Let (Y, be measurable outcome space and Yi, Y2,... ... the observed sequence with F; oYY..) A forecasting system is an F--adapted sequence of probability measures
P. Ω P(Y),
i 0,
meaning that for every A E B the map (A)) is -emeaaurable. Equivalently, (P.) is stochastic kernel from (2, FR) to (Y, At time i+1 P. is computed and then Y:+1 is realized; under ideal calibration one has
P.(-) Pr((141 E F...
Prequential calibration (sealar coherence). A one-step forecasting system (P.) is (scalar) coher- ent for class 76 of test functions if, for every h E H.
E h(Y+1) Fi shaP,
(1 2 0).
We will take H to contain least all integrable h.
3

---

Published as a conference paper at ICLR 2026
Linear functionals and martingales. Fix (bounded square-integrabbe score h. Define the linear functional ((F) = dF and the process
O(P.) - fhde: (i 0).
Under calibration, B(0(P,-1) F.] O(P.), so (0(P;)) is martingale. If is bounded, sup; Ef dP, < oc, Doob's 's martingale convergence theorem yields
((P.)
20 almost surely and in L²,
where 8zo is the long-run value of the operational metric under the given forecasting system (Doob) 1953).
Martingale posterior (MP). Following Fong et al. 2023), the martingale posterior for the target the conditional law of the limit:
1MMe( Fx) == Law(0> Fa).
Operationally, one approximates Пup by predictive resampling ! starting from the observed history and the forecasting system (Pi), simulate future one-step forecasts and outcomes, track the functional O(P:) forward, and use long-horizon values as draws from the MP. This keeps inference forecast- centric and likelihood-agnostic..
Deviation bounds. When h is bounded, the increments Ait1 == 0(P++)) - (P:)) are bounded,
and their conditional variances accumulate V2 = (-1 A 1+1 Fi). Freedman's inequality gives, for any E (0, 1) and all n.
(((P)) -
V2 V1 log(2/0)) 1og(2/6) with probability at least
where bounds | almost surely (see Freedman 1975). This provides generic, prequential "drift" control between the current forecasted functional and future values.
What to keep in mind. (i) Prequential analysis is forecast-first: objects of interest are the issued P/'s and functionals thereof, not parameter posteriors. (ii) Calibration implies martingale structure for ((P.), enabling convergence and uncertainty quantification via MPs. (ii) Concentration tools for martingales provide finite-tine time, deviation controls that we will use to size simulation horizons and report conservative drift terms.
3 THE PREDICTIVE RULE
At glance. Among affine empirical-model blends with predictable weights, requiring forecasts that "do not move on average" (scalar prequential coherence) forces the Dirichlet schedule X; a/ (i + a). This makes ((P.) We will later this to size MP horizons (via deviation bounds) and connect the conditional mean E[0x> FF.] to the familiar shrinkage point.
dP; a martingale for bounded (or L²) scores h, so ((P:) + 0.c
As stated, we study a simple, transparent prequential rule for the deployed forecasts P::
P. (1- + A Qsi
F
8Y
with predictable weights X; E [0. (measurable w.r.t. F; The data are generated prequentially: Pr(Y, E A Fi-1) P:- (A) DDawid 1984).
4

---

Published as a conference paper at ICLR 2026
Intuition: forecasts that don't move on average
In the prequential view, today's 's forecast should the best predictor of tomorrow's forecast. Testing only scalar quantities, this says that for every bounded score h,
E hndP F.-1 = f ddPi-
Within affine blends with predictable weights, this forces the Dirichlet/Pólya pseudo-count
schedule >; ax/ft + a)) (Blackwell & MacQueen 1973 Ferguson 1973); early forecasts
borrow from Qs and then fade to the empirical law.
Theorem (Coherence + Dirichlet; scalar consequences). Fix N, F; (Y and assume
Qo is Fn-measurable (frozen post-training). For 2 n P, = (1 - X,) + X,Q0 with X predictable, and suppose the prequential law holds for i 2 n+1.
Then the following are equivalent:
(A) Scalar coherence for all bounded h: E hd, (d. h all n+1.
(B) Dirichlet weights: X a/(i a) for some 0 and all i n.
Under either condition, for any bounded (or I score h the process ((P:) == J hdP, is martingale and
((P:) - 0xx a.s. and in I.2,
EE 00 Fa
+ h.dF + n +
n
a
I
h hdQ
Why this matters. (i) Scoped uniqueness. If you want interpretable empirical-man -model blend with predictable weights and you want forecasts that "don't tt move on average." " you must use dP; is the quantity operations care about (mean rate, tail mass, NLL). The limit 0. is its long-run value, and its conditional mean equals the familiar shrinkage point. Convergence and the mean identity follow from standard martingale theory (Doob 1953).
X; = a/ a). (ii) Direct operational payoff. For any fixed score h, ((P.)
Practical tie-in (MP and resampling). We quantify uncertainty for 0oo via the martingale posterior (the conditional law of 0xx given F,,) using predictive resampling in the spirit of Fong et al. (2023):
simulate forward under the rule A; a/(i + a) with P; (1 X²). + X.Qo. track only running values h, and use long-horizon values as MP draws (see Algorithm [ For conservative stopping rule for the simulation horizon, Section 6 for principled retraining trigger, see Section 7
What breaks if we violate scope. Fixed À (no decay) or non-predictable weights violate scalar co- herence generic histories; the martingale property fails. Updating á online breaks F, -e-amsurabillty of Q and the coefficient recursion that yields the Dirichlet schedule.
HOW TO CHOOSE a OPTIMALLY
We want a single, transparent way pick the knob a that sets the coherent shrinkage weight a/(n + a) in the estimator ((P.) (1 A) ((F,)) + x0( for the linear functional 0(F) dF when only a few observations are available. In this regime, error has two sources: the sampling noise of the empirical plug-in 6 ((F)) and the potential bias the frozen generator Q+ is misspecified for the target functional. Our plan is to make this trade-off explicit by formalizing
precisely what uncertainty we allow at small n. We assume Y1m i..d. F and place the unknown truth in ambiguity set that contains all data laws whose score variance is at most o² and whose target functional differs from the model's by at most A,
g(o2, A) ( Varp [MY)) < o², 18(Qs) - 0(F*) & A a:
This codifies the two forces at play: a controls sampling variability and A controls model-data mismatch the very quantity care about. For any fixed blend À E [0, 1), the squared error of
5

---

Published as a conference paper at ICLR 2026
the shrinkage estimate 8x (1 - ))0(F))) + 28(Qa) against ((F*) decomposes into variance piece scaled by (1 X) and a bias piece scaled by X², so it is natural to choose À by minimizing the worst-case version of this trade-off over G(o² A); doing so yields an exact small-n minimax weight, which matches the coherent prequential schedule À a/(n ( + a) and thus identifies the optimal pseudo-count a in closed form.
Theorem 2 (Exact minimax shrinkage). Fix n E and set a =²2 For X [0, 1]. define the shrinkage estimator
0₂ (1 (1-1))(FA)) + x0(Qa),
and its squared-error risk
R(A,F") == Ep. [6, - ((F'))
For every F E
R(A,F") S (1 (1- + x²
Moreover,
A
attained at Under the coherent schedule À a/(n + a) this corresponds the pseudo-count
inf XE
sup EGE
R(A,
==
a a+
a A
o² A
o'
(independent of nt).
The bound is light, achieved by a two-ppont least-fuvouruble F* with [0(Qa) - 0(F)) =A
In practice, we do not know o² or A, so we pick them in way that preserves the same conservative spirit: we estimate the score variance by the sample variance o² and we upper bound the motel-data gap by padding the observed difference with a high- cecb-bbbbbbbittty safety margin, A = (((2)
((Fw)) + tn. where for bounded |/| S H an empirical- empirical-Bernstein radius adaptively reflects the observed variability while accounting for bounded range.
20 log(2/0)
2H1
2 8
log
n
Alternatives with different regularity assumptions Hoeffding's t, H /lo((2//)) (2m) sub-Gausian/normal choice i, =1-6/2 // vn; if ((Qa) is computed by Monte Carlo with m draws, we also add a model-side margin &m inside A.
With these plug-in ins, turn the minimax prescription into concrete weight by setting & clip( 62 / Ommnn amax) and u á/(n + â) (c.g., Ominn 0max=200), where clip(æ; L, U) := min max(2; (r, L), U}. This automatically increases shrinkage toward Q & when the observed variance is larger or the inferred mismatch is smaller, and because a is n-indeppendent, the prequential weights data accrue.
(i + a) fade their A; (i
The following result shows that, with high probability, this data-driven weight essentially preserves the oracle minimax risk, up to second-order terms from estimating o² and padding A.
Proposition (Near-oracle risk with data-driven weight). Let d = 2 > 0, d A ā = d² /n, and x == (ā + d). On the high-probubility event & = ={ (10(P.)) - O(F*) < In and when clipping is
inactive,
2
(02 22-00 222 n2 A2
R(A,F*) < R + Ci E
sup
+ 2
R (orucie minimax)
for an absolute constant C.
Remark 4 (Robustness when t, A). t,, < A the excess is Ott)). Without this mild condition one obtains the same conclusion with an additional O(t²) term by replacing with (A + t,)) in the intermediate bounds.

---

Published as a conference paper at ICLR 2026
Algorithm 1 Martingale Posterior sampling
1: Inputs: observed data Yi.i frozen generator Qøi score h; pseudo-count horizon M; number of replicates B
2: For each replicate b I....,
3: Initialize: scores
sum + Ei= h(Y);; count + n
4: for m 1 to M do
advance one prequential step
5: t- n m, X 4 a/(i - 1 +
6: for b- = to B do
7:
Draw U (ernoulli())
8:
if U = then
model branch
9:
draw X ~ Qø: 2 + h(X)
else
b> empirical branch Append scores_b; sum t sum lo + 2; count_b b count_b +1
11:
sample uniformly from scores_k b (i.e., resample-wid--rppaeemm
12:
Return: MP draws (bb) + sum b count b for b I....,
Putting it all together, the implementation is single pass over observables: compute 2; h(Y)). their mean E; 2i, and the sample variance a²; evaluate 0(Qo) (adding im if it is Monte Carlo-estimated); form the conservative gap 10(2)) ž + tni set à = o² with mild clipping and X á/(n + à);; report the shrinkage estimate
((rr) (1 X) + +x(Qa).
and pass à to the MP sampler so that uncertainty quantification follows the same deployed, coherent rule.
SAMPLING THE MARTINGALE POSTERIOR (GPU-PARALLEL)
Following Fong et al. (003)) Algorithm [ we approximate the MP for the long-run value 0x = -imm J dP, by simulating the same one-step prequential rule forward and tracking only the functional state. Each replicate keeps a running sum s, a count c, and its own pool of past h-values; no inputs stored, so the memory footprint is small and the implementation is graphics processing unit (GPU)-friendly. Note that MP quantifies uncertainty for the surrogate 8.c under P,, not for 6( (**) We size the simulation horizon M using the deviation bound in Section 6
Remark (Linear metrics (in F): Dirichlet-mean shortcut for MP). Two computational regimes arise, depending on whether the operational metric is linear in F or not. For any operational metric that is linear in the distribution, O(F)
d.F. under the coherent blend P: = (1 >; P. X:Q. with X; oz/ii (i a), the martingale -ootterrr for the long-run value 0x = h dP: admits closed-form sampling scheme that avoids forward simulation. Importantly, h need not be linear;
can be any bounded (or L²) score. Let 2; h(Y)) and let Ho be the pushforward of Q by h (if
X Qo then Zo = h(X) ~ Ho). Drawing
(Wa...., ..........0.. w.) Dirichlet(a,) 1, 1,,.,,)),
1), Zo Ho-
g(b) wn Zo +
w;2;
yields i.i.d. samples ((0) Law(O,, Fx). Thus, for linear-in-F metrics this Dirichlet-mean sampler produces exactly the posterior as MP, but with substantially lower compute, while MP remains the general tool for non-linear or path-dependent objectives and policy simulations.
WHEN DO WE STOP RESAMPLING?
We want a principled way to decide how far to push the martingale- posterior (MP) resampling before additional simulation stops changing the answer in practically meaningful way. Concretely, operational target at step is the linear functional (P.,) = h dP,, and we would like to control, with high probability, how much this target can drift from its current shrinkage point 0 (PP.) any
7

---

Published as a conference paper at ICLR 2026
future value (P)) under the very same deployed prequential rule. The handle use is finite-ting deviation bound that applies whenever the score is bounded, ||h lec I H, and the forecasts follow the coherent Dirichlet schedule X; a/(i + a)) (with 0), which together yield an anytime guarantee valid simultaneously for all future times 2 n:
Theorem (Finite-tims deviation for bounded (tightened)). Assume || S H and the coherent Dirichlet schedule X =a(i + a) for 2 n (with 0). Then, for any 8 (0,1).
2 (2g() S H
2H
2
sup ((P)) - ((P.)) 22n
log
with probability at least 1 8.
n+a
3(n + +
This result tells us exactly what we need for stopping: the maximum future drift is controlled only by the score range H and the effective sample size n-aa, and it shrinks at the rate (n+aa)11/3) To turn into a practical rule, fix a confidence level 6 and read the bound as uniform certificate that, no matter how far we continue the prequential run, the gap (0(P)) ((P,)) remains below
H
2 (lg(22 a+
2H 3(++++1) log 2/ for all
We then pick the MP simulation horizon M so that this theoretical drift bound is smaller than the MP Monte Carlo error, ensuring further resampling cannot noise; and we report the same drift term alongside the MP interval for 0sc as a simple calibration diagnostic. Because the right-hand side tightens monotonically either the prequential run can only improve (never worsen) the guaranteed gap, which makes this criterion safe and transparent stopping rule.
the forecast more than sampling
increases, extending
Operational rule (horizon selection). In simple words, this tells us that we can choose M in algorithm I as the smallest value for which the right-hand side in Theorem i below the empirical Monte Carlo error of the MP quantiles you will report; also include that bound alongside the interval as drift diagnostic.
7 USING OUR FRAMEWORK TO DECIDE WHEN TO RETRAIN
We want an auditable, deployment--lignee answer to a simple question: will retraining a frozen generator pay for itself over the next H uses? The plan is to monitor the same linear key performance indicators (Klls) ((F)=] d.F under the very rule used production, summarize the current state with coherent shrinkage estimate and its MP uncertainty, and then compare the certified per-use improvement from switching models to the all-in retraining cost. We keep forecasts coherent by blending empirical evidence with the frozen model, the unique predictable affine schedule whose 3); at step n we compute
KPI forecasts are martingales (Sec.
T2
M(Y)).
a==
E (M(Y)) - 2)2. A= = In:
choose the pseudo-count
à == clip((00 Omin 0max).
and calculate the coherent shrinkage point together with MP intervals for the long-run value (Sec. 5):
((P,) = n+a = nta O(Qa).
With weights wi over KPIs and total retraining cost Crt. we then compare expected per-use loss under the deployed rule before vs. after retraining, using a pilot when available or a certified minimax proxy otherwise, and we trigger only if the //-use gain exceeds Ct::
Checklist (one pass).
1. Compute 2; = h(Y)), the mean ī = E. 2t. and the sample variance à²
2. Evaluate ((Qs); if Monte Carlo with draws, add a model-side margin E...
3. Form the conservative gap â (0(Q.)
(choose i, per Section
8

---

Published as a conference paper at ICLR 2026
4. Set â clip(0" / / Omiin Omax and X = â/(n + â). Report (P,))
=+a nta ((Qs) with MP intervals (Sec. 5)
Compare current vs. candidate model using either a pilot MP comparison or the minimax proxy below; trigger only if the H-use gain exceeds Ctt.
Decision rule (expected gain vs. cost)
With a pilot (operational target). Run two MPs (current Q4 vs. pilot Qo-), each with its own pseudo-count a set by the small-n minimax rule (Sec. 4 and trigger if
Σ We (4.trrt
Ense-(19,, |[((0..)))) 2 Cn.
No pilot (certified proxy). Use the small-n minimax risk R Ra, A) and plug-ins ins a = 8² //n, A Â (Thm.
o42 a+
with a = o²/n
Σ we R (R / (e) - R RO (o))/,,A)))))))))))())))) 2 Cit.
Here At is your planned post-retrain mismatch (from small side-pilot, historical effects, or a minimum effect size you require).
Proposition (Worst- Woorst-case--ee case-aaeee safe proxy trigger). Let 0(F) f h&F be a linear KPI and let G(o², A) be the ambiguity set of Sec. 4 Set = /'/ Suppose retraining changes the model-data mismatch on the target functional from A to A + I A while the data law F is umchanged. Consider the coherent Dirichlet-weighted es- timator with the minimax weights before/after retraining, i.e., (a, A) and X (a, A For the truth centered squared loss (0) c0 ((F*)) 2 e O, one has the right worst case bounds
sup E[L)(0)] = =cRt' (a, A), FTE
([t] = ca,
FT
where R² (a, A)
Consequently, the worst case per- case expected loss drops by c(f (a, A) R' (a, A+)), and over H cases, triggering when
aa
a+23.
H He(R" (a, A) R² (a, (t) 2 Cr.
worst-case cost-effective.
Intuition: at small n (number of observed cases) the two levers that matter are variance o² /n (noise per case, where o² the variability of the KPI scores i(Y)) under the true data) and mismatch A (the systematic gap on the KPI between the frozen model and reality, i.e., 10(Qo) ((F*)): any blend has worst-case risk (1 (1-X) a X² (variance term scaled by how much trust the data, plus squared bias scaled by how much we trust the model), where fraction of trust placed on the model forecast versus the data): this is minimized by X a+ to R (a, A) (the best achievable risk given noise level and mismatch A); retrain leaves unchanged (same sample size n, same data noise) but aims to reduce the mismatch from A to A (the planned post-retrain gap on the KPI), so the guaranteed gain is exactly Rt (a, A) Rt (a, A+); multiplying by H (the number of future uses one cares about) and comparing to Crt (the all- in retraining cost) turns this into a direct, stakeholder-frieet friendly trigger.
[0, 1] is the shrinkage weight (the
8 EXPERIMENTS
We evaluate three settings with frozen generators and linear operational metrics: (i) Language (in- distribution, ID): GPT-2 (117M) on WikiText-2; score is reacher-forees negative log likeeelldoodd (NLL) per token; target ((F) E[h]. (ii) Vision (in-distribution / out-of dittof--istributionn ID/OOD): CIFAR- 10 (ID) and SVHN (OOD) with a CIFAR-10 1--retr--ee generator; score h is CLIP-rarity (protocol in App. C target ((F) E[h). We choose strong shift (CIFAR-10- (V--NN) to stress

---

Published as a conference paper at ICLR 2026
GPT2 on Wildiexx-2:: Predictive coverage @ 90% vs
SVHN (OOD), B- carltyy Predictive coverage y 90% vs
1.0
0
80%
0.8-
0.6
Dradiction
0.4
prodectve
0.4
0.2
Exyesin Bastarap
Dapazan Eorzzes
20
(a) GPT-2 (Wi)iText-2): coverage@90% vs rig
(b) SVHN: OOD coverage@9052 vs
Figure I: Predictive coverage @90% versus the number of initial real samples no. The dashed line marks the nominal 0.90 target.
calibration small n. (iii) Toy (Two Moons): alert-rate and mean-score functionals. Baselines: nonparametric bootstrap (NPB), Bayesian bootstrap (BB), Jackknife (JK). Our methods: DWS (Dirichlet-weighee) shrinkage with the minimax a) and MP (martingale posterior). For linear 0 use the Dirichlet-mean shorteut (exact law of x.)). Full protocols and extended results are in App.
Results at glance. Figure shows coverage @ 90% versus Ao: the share of runs where the 90% predictive interval for 0x (under the deployed rule) contains large-sample reference from an independent truth pool; error bars p 1.96 consistently closest to nominal in the small-n regime (reaching Rs 0.90 by no=20 and remaining stable through 100),00, whereas NPB/JK markedly under-cover for ng S 50, and the plain Dirichlet- mean (without the minimax a)) under-covers at very small n. The right panel (Fig. lb) summarizes
p)/R. On GPT-2 (WikiText-2) (Fig. lay. DWS
the same trend across methods with error bars. Figures 2a 2b illustrate how uncertainty behaves under the deployed rule: at no=50 the MP draws place the shrinkage mean between empirical and model; by =003000 the forecast has faded to empirical behavior and the 90% band has contracted. On vision (see App. C : on CIFAR-IO (ID) methods tend to cluster with near-nominal coverage; on SVHN (OOD) DWS is best-calibrated for small n (near 90% when others under-cover). Across datasets and sample sizes, the intervals from DWS/MP decrease steadily as no grows, reflecting the increased effective sample size ta and yielding progressively sharper yet calibrated forecasts.
Why our method wins when n is small. (i) Coherent pseudo-counts: the unique predictable affine blend A; = a/(i+a) makes ((P.) a martingale, stabilizing early forecasts while ensuring fade-out. (ii) Minimax single, data-driven knob trades off sampling variance against model- data discrepancy, curbing the under-coverage typical of bootstrap at tiny n. (iii) Right target; MP/Dirichlet- quantify uncertainty for the operational limit 0xx under the deployed rule, which what operations act on.
draws PrtYs =0
MP draws for e; Prty,, >- 13 at 3000
Empirical = 0.030
Emaiiial
Model 0.061
Shrerkage mean 0.070
200
Stetitage meen 0.0M
Mode
35.0
90%, [C.033.
HP 90% (0.347. 0553
aenitty
150
100
7.5
5.0
50-
0.0
0.1c
0.14
0.0450
0.0475
0,0500
0.0525
0,0550
0.0575
0.0600
MP draws nou 550
MP draws =00000
Figure 2: Behavior of the martingale posterior (MP) for an alert-rate functional: shrinkage and wider bands at small (left), fade-out to empirical and tighter bands at large (right).
10

---

Published as a conference paper at ICLR 2026
Reproducibility Statement. All implementation details, hyperparameters, and full experimental protocols are provided Appendix [ including exact configurations for all settings (language. vision. and Two Moons). The code is available at hattps://github. com http:////.........o.oooeeraaaa// www.///eeeeaaaeerrreeeooaaoa priors_flrr fludd_forecasts
LLM Usage. We used large language model sparingly for polishing the abstract.
ETHICS STATEMENT
While this work has many societal consequences pertaining to deployment of machine learning models in the industry, we do not foresee any specific ethical concerns that must be highlighted.
ACKNOWLEDGMENTS
VG acknowledges the Research Council of Finland, Saab-WASP, and the Jane and Aatos Erkko Foundation for their support.
REFERENCES
Anastasios N. Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foun- dations and Trendso in Machine Learning, 16(4-5):494-91,, 2023. doi: 10.1561/220000000..
Shekoofeh Azizi, Simon Korbblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv, 2023. doi: 10. 08550//rXii.2204.000
Patrizia Berti, Luca Pratelli, and Pictro Rigo. Limit theorems for conditionally identically distributed random variables. The Annals of Probability, 32(3A):2029-2552, 2004. doi:
|AL224/00911700400000006
David Blackwell and James B. MacQueen. Ferguson distributions via Pólya urn schemes. The Annals of Statistics, 1(2):353-355 1973. doi: 10. 21144/os/1173342311
A. Philip Dawid. Present position and potential developments: Some personal views-statistion theory: The prequential approach. Journal of the Royal Statistical Society: Series A (General), 147 (2):278-292) 1984. doi: 10.2307/2981682 1683.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace eeduxeeffortless Bayesian deep learning. In Advances in Neural Information Processing Systems, volume 34, pp. 20089-20103, 2021.
J. L. Doob. Stochastic Processes. John Wiley & Sons, New York, 1953.
Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statisties, (1):1-26, 1979, doi: 10. 10.12144aos/1177445552 URL http::t:://
11763445521
Bradley Efron and Robert Tibshirani. An Introduction to the Bootstrap. Chapman and Hall/CRC, 1994. doi: 10.120199880222465999
Thomas S. Ferguson. A Bayesian analysis of some nonparametric problems. The Annals of Statistics, 1(2):209-230, 1973. doi: 10.1214/aos/1 17342336..
Edwin Fong, Chris Holmes, and Stephen G. Walker. Martingale posterior distributions. Journal of the Royal Statistical Society: Seriey B (Statistical Methodology), 85(5)1135-1131, 2023. doi: 10.1 11 1rsssb. 12639.
Sandra Fortini and Sonia Petrone. Prediction-basel uncertainty quantification for exchangeable sequences. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, (81(2247):202211423 2023. doi: 10. 10920/sta..222.0142..

---

Published as a conference paper at ICLR 2026
Sandra Fortini and Sonia Petrone. Exchangeability, prediction and predictive modeling in Bayesian statistics. Statistical Science, 40(1):40-66 67, 2025. doi: 10.1214/2-STSS665
Sandra Fortini, Lucia Ladelli, and Eugenio Regazzini. Exchangeability, predictive distributions and parametric models. Sankhyā: The Indian Journal of Statistics, Series A, 62(1):86-109, 2000.
David A. Freedman. On tail probabilities for martingales. The Annals of Probability, 3(I):100-111, 1975. doi: 10. 10.1214/aop/1 11144/////1179995555
Seymour Geisser. Predictive Inference. Chapman and Hall/CRC, 1993. doi: 10.1201/97802033442311
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pp.
2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, pp. 6626-6637, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pp. 876-885, 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect out-of-distribution data. In Advances in Neural Information Processing Systems, volume 33, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pp. 6402-6413, 2017.
Wesley J. Maddox, Timur Garipov, Pavel Izmailov, Dmitry P. Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, volume 32. pp. 13132-13143, 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
M. H. Quenouille. Problems in plane sampling. The Annals of Mathemattical Statistics, 20(3):355- 375, 1949. doi: 10.22199/mm//1117779999 1177729989
M. H. Quenouille. Notes bias in estimation. Biometrika, 43(3-4)3353-36/1 1956. doi: 10.1093/ biomet/43.3-4.335 URL //ttss:/doo...
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.
PMLR, 2021.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin. Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distributiin detection. In Advances in Neural Information Processing Systems, volume 32, pp. 14680- 14691, 2019.
12

---

Published as a conference paper at ICLR 2026
Donald B. Rubin. The Bayesian bootstrap. The Annals of Statistics, 9(1): 134, 1981. doi: 10.1214/aos/1176364
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86-96 96, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/ P16- 1009.
John W. Tukey, Bias and confidence in not-quite large samples (abstract). The of Statistics, 29(2):614, 1958. doi: 10. / URL T214/ aoms 1177766677
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, New York, 2005. doi: 10. (007/b106715.
Xin Yi, Ekta Walia, and Paul Babyn. Generative adversarial network in medical imaging: A review. Medical Image Analysis, 58: 101552, 2019. doi: 0..016f..ecdaa2011.11555.
A RELATED WORK
Pretrained generators as small-n stabilizers. common small-date tactic to borrow stability from pretrained generator trained on a nearby domain: trec--rrnslation in MT (Sennrich et al.) 2016), diffusion-backed augmentation in vision (Azizi et al.] 2023). and GAN- synthesis in medical imaging Yi et al. 2019). Generators are routinely used frozen across flows, diffusion models. (Papamakarios
al.) 2021 (Ho al.) 20200 Kingma & Welling Goodffllw et al. 2014). Yet explicit likelihoods can misalign with semantic OODness (notably in flows) (Nalisnick et al. 2019 Ren et al. 22019] Kirichenko et al. 2020). motivating targets
VAEs, and GANs for sampling/monitorinn
that operate the deployed predictive sequence and support density- free metrics (e.g.. CLIP rarity;
FID-style distances) while recovering NLL when available (Radford et al. 2021 Heusel et al. 2017).
Prequential viewpoint and coherent shrinkage. The prequential (forecast-firs) program evaluates one-step predictive sequences rather than parameters (Dawid) 1984; Geisser 1993). Dirichlet-P6lya predictives give canonical pseudo-count shrinkage that ensures scalar prequential coherence, for any bounded score h, (P.)
h d.,, is a martingale (Ferguson 1973 Blackwell & MacQueen
1973 Fortini et al., 2000 Berti al., (004). Martingale posteriory (MPs) operationalize inference
on functionals of the predictive limit via predictive resampling (Fong et al. 2023 Fortini & Petrone
2023). Fong et al. (Fong et al. 2023) take the predictive sequence (P.) as given and show how to construct MPs for general prequential limits. Building on this, our work (i) identifies the unique Dirichlet prequential blend for frozen generators that yields scalar-coherent forecasts, (ii) provides a small-n minimax rule and plug-in estimator for the pseudo-count a, and (iii) uses the resulting MP machinery to target deployment-aligned functionals 0.c and to derive an auditable retraining trigger.
Position relative to broader UQ. Parameter-centric UQ (ensembles, SWA/SWAG. Laplace) inte- grates or perturbs parameters [Lakshminarayanan et al. 2017 [zmailov et al.) 2018 Maddox et
2019: Daxherger et al., 2021); bootstrap captures sampling variability of the empirical law (Efron & Tibshirani 1994); conformal offers distribution- free per-sample guarantees under exchangeability
(Vovk et al.) 2005; Angelopoulos & Bates 2023). Our angle is forecast-centric: a coherent pseudo-
count blend for the deployed rule and based resampling to quantify uncertainty for operational
linear functionals.
B PROOFS
B. PROOF OF THEOREM 1
Fix and let F, o (Yv:). Let Qø be F,, measuralall (frozen after training). For i n define
F = Σ 8Ym:
P = (1 (1-x)) + Ao. d E [0. 1] predictable (F,-1 measuralea.
k=1
13

---

Published as a conference paper at ICLR 2026
Assume the prequential law Pr(Y, A Fi-1) Pa-1(A) for all Borel A and i 2 n+1 Dawid 1984).
Auxiliary identity. For i n+1,
Fi-1 = P.- + Pi-1-
(I)
Proof. For bounded measurable f.
b /d. F. = ((x)
-L)()))))))) BL(())) 1 -
((0)))
- --1 - sdfi 1 // sdPi-
by the prequential law, which is precisely equation
(B) = (A) (Dirichlet weights imply scalar coherence). and all 2 n. For any bounded h.
Suppose A a/(i ( a for some > 0
Fi-1 =1 (11-1)) -D) hdElE Fi-1l+ + 2o hdQe.
E
hdP:
Using equation
and P:-1 (1 -1-xp (1->i-1) -)--1) + Ai-
E harp Fi-1 (1 ا | hnd + (1-
(11 Xx) 2 2,1 h hdQ
direct substitution of X3 = a/(j (j+ a)) yields the coefficient identities (1 X1) --)-- 2 - : + d5 A1-1, hence
A
and (1 X,) A=-1
E
|ndd. Fi-1
hdP:- for bounded h,
i.e. scalar prequential coherence holds.
(A) (B) (scalar coherence forces the Dirichlet schedule). bounded h and é 2 n+1,
Assume scalar coherence: for every
E hhdP.
Fi-1 Fin. / h
Repeat the calculation above but regard the two coefficients in front of J h dF; and h dQo as unknowns. Because the identity holds for all bounded h, the coefficients must match:
(1 A,) - A--1.
(1 (1- No) A1-1
x = A
The second equality gives the recursion
A 1)1-11
i2 n+1
Ai-1
Let 0; == >s/(1- X²). Then i-1 0;-1, 5o = (n/i) and
0 (n/i)an
A + a; 1+ (n/i)an i+a a
with a: nan 0.
Thus the Dirichlet (pseudo-cound schedule is the unique predictable affine schedule that achieves scalar coherence, in line with the Dirichlet-Pólya predictive updates (Blackwell & MacQueen 1973.
Ferguson 1973).
Under either condition (A) or (B), scalar coherence holds, hence for any bounded (or L²) score the process ((P.) ;= ff h d., satisfies B(0(P.) [F=1]] O(P;-1) and is therefore martingale. If
h is bounded then [0(P: (P:) < |[h]]so a.s.; if h E T,2 under the predictives (our standing assumption
14

---

Published as a conference paper at ICLR 2026
sup; h² dP; 0o), then by Jensen/Cauchy- SShharr--crrr (c( < &E = dP; b.uubeddedd. By Doob's 1,2 martingale convergence theorem,
oo, (0(P.)) is
((P:) 0,
almost surely and in L?, and
= = F
(see, e.g.. Doob p.53). Finally, using Pn = -n)n) + Xo Ro with A, a/(n + a) gives
0(P..) = hdPP (
n - a ʃ hdF + aa ʃ hdQo.
n n
which identifies the stated shrinkage mean.
This completes the proof.
B.2
PROOF OF THEOREM 2
Recall the ambiguity set
g( {F= F Varr-[M(Y)] o², ((Qx)) ((F') s s4}.
and the shrinkage estimator
02 = (1-
10
with risk R(A, F*) = E(( {) under F*. Set := 02/n.
Let μc, == ((F*) and Mm := a(),), and define the functional discrepancy 8 == Jm Ma. Since
((F,)) 1/2 Σn= h(Yk) and EF. [h(Y)] = p. by linearity,
Varp-(0(F.)) 1 --aar-- (h(()))))..
EF. (0(P)l = M..
n
Therefore
-r-1. 14 -(1 (1 -1))()(())))) n.)) + *Mpmm p(m)
=(1 -))(((p))) - 14.) + Ao.
so that
R(A, F*) = E F)
(0( )1.) + + 1)18 m.]+ X282 = =1 X)2 Varp (0(F.)) X
(2)
(The cross term vanishes since F( - -14,1 =.)
Recall = o2/nn n. = 0(F*), Um == ((Qo), and 6 = pm p., By the definition of G(o? we have
a² = a
-arz-(0 (w)) = (arr-(h(V)) s
5 A.
n
Plugging these bounds into equation yields, for every F =((2,
R(A,F") = (1 (arF: + X28 < (1 (1-X)²
X) a + 2²A2
Therefore
R(A, < (1- X R()).
sup ?
This upper bound is the quantity minimize in À in the next step.
Recall a == o² /n and the upper bound R(A) = (1 - 2) + X²A" from Step 2. Define ʃ()) =
(1 - 2a + A² for À E 1]. This is a convex quadratic (strictly convex if a > 0 or A > 0). Differentiating.
(((A) 2(1 A)a 2AA2
15

---

Published as a conference paper at ICLR 2026
and the unique stationary point in [0. 1] solves (A) 0, i...
a
A
Since a, A? 2 0, we have A E [0, 1]. Evaluating,
f(x* = a aA²
A =: R.
Therefore
inf XE F)EG(O?,A)
sup
R(,,F)
S inf R(A) Re achieved at X² a A 1e
Tightness. The bound is attained by two-point F* with Var(h) = o² and - A. Hence
R(A, F*) = 22
sup AA
80 X- = a/(a + A²) is truly minimax and inf supp- R(A, P*) = Rº (a, A).
Under the coherent blend, the weight À = a/(n + a) with pseudo-count a > Equating this to the optimizer X and solving for gives
A = a(a + A?) = (n+ aaaa = a a =
=na na = a
n1 A
σ²
A
which does not depend n.
Edge cases. If = 0, then X = 1 and R 0 (trust Qa fully). If = (e.g., n - 00 or Varr- (h) 0), then X* and R (revert empirical behaviour).
B.3 PROOF OF PROPOSITION
E Let a = o² and d = A > O. For À E [0, 1] set
ad
Ja,(AA) = - (1- X2d,
X = a+d d'
R= = Sa,(>>) =
a a d
By Theorem 2 for every F E G(02, and every
R(A,F*) S fa,o(A) R" + (a + (A 2222
Let d=:
A and === a/(a + d). Then
(a d) S (d (d- dd2 d)²
a' d)²
faa(A) - R² = (a R²=
d)
a a+d
o+d
for a universal constant C > 0. On the event E := ((0(F.)) - [(F*)] < tn } (one may assume
In A by truncation), the triangle inequality yields A A s + (0(F..) 0(F*) S 21, and
A + < A. Hence
(()))))(+++) CC22
and therefore
Sa,a(N) - R C 1 on E.
(3)
Let = 8²/n and define :=
= a/(ā+ + d). For g(z) = x/ + d) we have f fa,(() = fa)((((a))
-
(a + d) (g(@) g(a))² Since is Lipschitz with (g)(E) = d/(E + 2ÿ² < 1/(g + d) - 1/(1),
(a (a+ a)+ (o(a) g(a))² < c (a a)
16

---

Published as a conference paper at ICLR 2026
Thus
(ā a)
a)²
la S Sa + C
d
By monotonicity in d, (() S (a) for all X, hence
(a s (2 S Sa C (ā-a)²
Sad(A) + C (ā=a)²
d
Using equation d2d d= A on & and - a -62
(x S R Ci + C c(0² -62p
on E.
n² A²
Finally R(A, F*) - (a) for all F E EG(o? which yields the claim.
Remark 8. If one sets Ā == a/(a + d) (no variance plug-in), Step 2 is unnecessary and the term
C(a² - 02)3 /12//nn>)) vanishes; the bound reduces to R(X, F*) S R + C on E.
B.4 PROOF OF THEOREM 6
Fix n E N. Assume | < H and A; = a/(i+ for 2 n with a > 0. Let F; = o(Vis) and define ((P.) == f h.P, and A, == 0(P:) - ((P:-1) for i 2 n+1.
Using Pi = 7ta
_ F. + + Qs and f h dF. = ا f di + (YY),, we obtain
i i- 1 +
((P)) ita h(Y,)
0 ((::1),
ita
hence
A: MYY) 0(P.
i+a
Since O(P:-1) ShdP,-1 and Y, Pi-i conditionally Fi-1 (by prequential coherence), (0(P.))zzn is a martingale and (A))2n+ are martingale differences.
Because ||/llso S and [0(P.-1)] < H.
2H
Var(h(Y;) Fi-1)
2 (i ai+
|i: <
and
B F--1l=
i ata
(i+a)2
Therefore, for f n,
c = sup neist
|
2H
V Σ
B[A F--1 $
CC
H
S
112
Voo-
n ++++
(1+a)3
n + a
Let S. == El A; = - ((P.)
The maximal Freedman inequality for martingales with
a.s. bounded increments (e.g., Freedman 1975) yields, for any E (0,
Pr sup [St] s 22 log(2/0) log((/0)) 1-6. t2n
Substitute V. S H2/(nn+ + a) and < 2H/(n 1) obtain the stated bound.
B.5 PROOF OF PROPOSITION
Fix linear KPI O(P) f dF and an initial sample size n. Let = 02/n and consider the
ambiguity sets ( and g A+) from Sec. 4 with 0 s A < A. Define the (time-n) coherent shrinkage estimators
(1 X x)))(f.)) X (QQs)
:= x ((F X ((Qx+),
where x = a+ and ++ a mp are the minimax-optin weights from Theorem (equiva-
lently, a* / and a + /2/((A ) give the same via the Dirichlet schedule a (n+a); cf. Sec.
17

---

Published as a conference paper at ICLR 2026
For any given (a, A) there exists F* G( A) attaining the bound R (a, A): let Z h(Y) under F be a two-point law with mean μ4, and variance o² (so Var Var(Zn) (n.) = a). and set 8(Qo) = p. A.
Then for any À [0,
- - = (1 1--2))aa >²
which is minimized at X* to the value R (a. A) = a+ The same construction with replaced by At attains R (a, A }.
a1
By the varianer decomposition in Theorem for every F E EG(o?, and X,
R(A, (*) = B (0-
Hence, X*,
sup
E (0- (a)
(4)
F)
Replacing A by A gives
sup F.EG(rI,At)
E (0 - -(( = Ra, At).
(5)
Let the per-case loss be the truth centered squared loss (. = c( -(F)))² with > 0. Then
(O) e cB((O ((F*)) so from equation 4 equation 5
(0] c (a, A),
[[L] = cc
sup P)
sup F'EG(O",Ar) ,A)
Therefore the worst-case per-case expected loss drops by ca, ((),))))))))))))
Over a planning horizon of H future cases, the same drop scales linearly: the worst cose case total expected loss decreases by H e (R* (a. A) (a, A ))) Trigger a retrain whenever
He (R" (a, A) Re (a, A+) 2 Cit.
which ensures the certified reduction least the all-in retraining cost Cit.
C EXPERIMENTAL DETAILS AND EXTENDED RESULTS
This Appendix documents datasets, models, metrics, protocols, hyperparameters, and complete numerical results. It also records the exact implementation of the martingale-posterior (MP) sampler and clarifies which choices were 10/SVHN and GPT-2 experiments should be read as cross-modality proof-of-coneepp demonstrations rather than limits on the architectures or scales to which the framework applies.
versus compute compute-controlled controlled, and our CIFAR-
Note (algorithmic choice). For operational metrics that linear in the distribution, 0(F) f ddF, and under the coherent blend P. = i+a shortcut yields the same posterior for the long-run quantity 0.0 while reducing runtime. For this reason, we keep the resampling variant for pedagogical elucidation in experiment C.I but we change to the Dirichlet-mean shortcut for language and vision.
1 F, + it Qo with frozen Qs. an equivalent Dirichlet- mean
Scope note (frozen sampler). All experiments assume a frozen generator Qs (fixed weights and sampling policy during each run). Randomness in sampling is conditional on this fixed Qs and independent of the observed data, which makes Qo measurabe and places in the setting covered by Theorem (scalar coherence). This "frozen" scope matches standard deployment between retraining cycles across diffusion models, GANs/VAFs and autoregressive LMs..
18

---

Published as a conference paper at ICLR 2026
Operational target and coverage metric (clarification). Our target is the operational long-rum functional
/ hdP.s 1-100
0x
lim
P.
ie., the KPI under the actually deployed prequential rule. "Coverage "00% reports the proportion of repetitions whose 90% predictive interval for 9x contains a high-precision reference computed from an independent held-out truth pool. Truth-pool sizes and Monte-Carke error: language (Wiki'Text-2 uses 1, 1,200 held-out texts (MC error ze 3%), vision (CIFAR-10/SVHN) uses 1,000 held-out images.
Baselines and canonical references. For language and vision, we benchmark against three classic resampling-based uncertainty methods: (i) Nonparametrie Bootstrap (NPB) (Efron 1979): repeat- edly resamples the observed data with replacement to approximate the sampling distribution; (ii) Bayesian Bootstrap (BB) (Rubin| 1981: assigns random Dirichlet (1..leettt...................... ..... weights the sample instead of resampling: (iii) Jackknife (JK) (Quenouilee 1949) 1956 Tukey, 1958): systematically leaves out one observation
time and combines these leave-one-out estimates assess bias and variance. Why these baselines? They assumption-light plug-and-plas with any estimator, and require only the observed sample (no model refitting), making them the standard, widely used yardstick for small-sample uncertainty when analytic formulas unavailable (Efron 1979).
Reporting Format. Predictive coverage reported as p = 1.96 SE, with SE /p(1 p)/ R. For each bold:
The method with coverage closest to 90%.
Runtime is reported as mean wall-clock time per repetition.
C.1 Toy EXPERIMENT (Two MOONS): EXPERIMENTAL. DETAILS AND EXTENDED RESULTS
Setup and Metrics. We evaluate the Two Moons toy setting using frozen RealNVP generator. Two linear operational metrics are considered:
Alert rate: 01 (F) = P
F (Y2 > r}. where T is the empirical 95th percentile of the second coordinate under the target distribution.
Mean NLL: (F) = Ey [- log 4s (Y)), the average negative log-likelihood (nats) under the frozen RealNVP model qp-
For each metric, report predictive coverage at 90% for the long-run operational target Oz and wall-clock runtime.
Methods Compared. We compare the following uncertainty quantification methods, all evaluated frozen mode:
MP: Martingale Posterior (our method),
NPB: Nonparametric Bootstrap,
PB: Parametric Bootstrap (sampling from the frozen generator).
Hyperparameters.
Number of repetitions: R = 40 per (method, rt),
MP resampling: Bmp 512 draws, horizon Maefault 1500,
Bootstrap resampling: Broat 1000.
Sample sizes: (5, 10, 20, 50, 100, 1000).
Pseudo-count tuning: normal/sub-Gaussiin plug-in radius t,, 2// / with one-sided
1.64 (approx. 90% confidence) with mild clipping [5, 200).
19

---

Published as a conference paper at ICLR 2026
MP stopping rule (anytime drift bound). The MP horizon is sized via the anytime bound
Theorem 6: we stop once the certified future drift sup ((P)) - ((P,)) falls below the
Monte-Carlo error of the reported interval. In Tables fixed = 1500 satisfies this criterion in all runs.
Stopping rule for MP For bounded scores with range H, Theorem 6 gives an anytime bound on
future drift: supt2n (P.)) ((P.) S H V 2lo((2/8) + 3(++++1) log We set the MP horizon M a+o
2/
so this certified drift is below the Monte Carlo error of the interval: for linear metries we instead use the exact Dirichlet-mean shortcut, so no forward simulation is needed.
Table 1: Two Moons (01): Predictive doverage@90%% and runtime (using Algorithm for MP).
Winners per n. coverage closest to 0.90 (bold in coverage column).
T
Method
Predictive cov (90%
Runtime (8)
5
MP
0.825 + 0.118
19.57
5
Nonparametric Bootstrap
0.175 + 0.118
8.78
5
Parametric Bootstrap
1.000 = 0.000
24.78
10
MP
0.875 + 0.102
19.42
10
Nonparametric Bootstrap
0.275 = 0.138
8.82
10
Parametric Bootstrap
1.000 = 0.000
24.80
20
MP
0.950 = 0.068
18.86
20
Nonparametric Bootstrap
0.675 = O. 145
8.68
20
Parametric Bootstrap
1.000 + 0.000
24.37
50
MP
0.925 = 0.082
19.31
50
Nonparametric Bootstrap
0.925 + 0.082
8.87
Parametric Bootstrap
0.875 + 0.102
24.91
100
MP
0.975 = 0.048
19.42
100
Nonparametric Bootstrap
1.000 + 0.000
8.89
100
Parametric Bootstrap
0.900 = 0.093
25.08
1000
0.825 1 0.118
19.07
1000
Nonparametric Bootstrap
0.925 = 0.082
8.94
1000
Parametric Bootstrap
0.625 = 0.150
25.18
Table 2: Two Moons (02): Predictive coverage 90% 90% and runtime (using Algorithm Winners per n: coverage closest to 0.90 (bold in coverage column).
for MP).
Method
Predictive cov@90%
Runtime (s)
5
MP
0.825 = 0.118
19.57
5
Nonparametric Bootstrap
0.675 = 0.145
8.78
5
Parametric Bootstrap
1.000 = 0.000
24.78
MP
0.975 = 0.048
19.42
Nonparametric Bootstrap
0.900 + 0.093
8.82
10
Parametric Bootstrap
1,000 = 0.000
24.80
0.875 1 0.102
18.86
20
Nonparametric Bootstrap
0.900 1: 0.093
8.68
20
Parametric Bootstrap
1,000 + 0.000
24.37
50
MP
0.900 +: 0.093
19.31
50
Nonparametric Bootstrap
0.925 = 0.082
8.87
50
Parametric Bootstrap
0.850 = 0.111
24.91
100
MP
0.900 = 0.093
19.42
100
Nonparametric Bootstrap
1,000 = 0.000
8.89
100
Parametric Bootstrap
0.750 = 0.134
25.08
1000
MP
0.825 + 0.118
19.07
1000
Nonparametric Bootstrap
0.875 1 0.102
8.94
1000
Parametric Bootstrap
0.825 1 0.118
25.18
Across both metrics, MP is typically the closest to the 90% target at small to medium sample sizes. T and 4ummariis MP widths across no, showing steady shrinkage as
Discussion.
20

---

Published as a conference paper at ICLR 2026
Table 3: Two Moons (01), MP interval widths across no-
MP Width
5
0.126
10
0.118
20
0.106
50
0.078
100
0.060
1000
0.017
Table 4: Two Moons (02), MP interval widths across ni-
MP Width
0.558
10
0.512
20
0,439
50
0.338
0.266
1000
0.071
grows. For 01, MP wins coverage {5, 10, 20, 50) and achieves the narrowest width from n 2 20. For 02 MP consistently yields the smallest widths, while coverage alternates slightly at some
Runtime is stable across sample sizes: NPB is fastest (==8.8). MP is mid-range (=19ss, and PB is slowest (=25s). These results confirm that MP provides calibrated predictive coverage close to nominal with sharper intervals intervals-especially for while smoothly fading to empirical behavior as n increases.
C.2 GPT-2 (WIKITEXT-2): EXPERIMENTAL DETAILS AND EXTENDED RESULTS
Setup and Metrics. We evaluate frozen GPT-2 (117M) model on the WikiText-2 validation set. The operational unit Y text sequence tokenized using GPT-2 BPE and truncated to max_text_lengee = 256. The score is the teacher-forced negative log-likelihood (NLL) per token under GPT-2, and the operational metric is the long-run mean:
((F) = h dF
which is linear in F and admits the Dirichlet-mean shortcut for the martingale posterior (sce Remark
Methods Compared. We compare the following uncertainty quantification methods, all evaluated frozen mode:
BB: Bayesian Bootstrap,
NPB: Nonparametric Bootstrap. Jackknife,
DWS: Dirichlet-weighted shrinkage (our method, using the Dirichlet-mean shortcut).
MP resampling is omitted here since 0 is linear in F.
Hyperparameters.
Number of repetitions: R = 10 per (method, nı),
Bootstrap resampling: Broa 160, Brayesian 160.
DWS sampling: Bas: 160.
21

---

Published as a conference paper at ICLR 2026
Table GPT-2 (WikiText-2, 01: NLL/token). Predictive coverage 90% and runtime. Values are mean + 1.96SE over repetitions (coverage SE from Bernoull;; runtime across runs). Winners per coverage closest 0.90 (bold in coverage column).
Method
Predictive cov@90%)
Runtime
Bayesian Bootstrap
0.000 + 0.000
0.03 + 0.00
Nonparametric Bootstrap
0.200 0.248
4.32 = 0.20
Jackknife
0.500 0.310
0.15 0.01
DWS
1.000 0.000
6.06 - 0.01
10 Bayesian Bootstrap
0,300 0.284
0.05 0.00
10
Nonparametric Bootstrap
0,300 0.284
7.27 = 0.31
10
Jackknife
0,300 0.284
0.48 + 0.02
10
DWS
0.800 0.248
6.06 = 0.01
20
Bayesian Bontstrap
0,400 = 0.304
0.10 0.00
20
NPB
0.500 0.310
15.73 0.39
20
Jackknife
0.400 0.304
2.03 0.07
DWS
0.900 0.186
6.15
50
Bayesian Bootstrap
0.500 0.310
0.24 0.01
Nonparametric Bootstrap
0,500 0.310
37.30 0.78
50
Jackknife
0,500 0.310
11.60 0.29
50
DWS
0.900 0.186
6.28 0.04
100
Bayesian Bootstrap
0.900 = 0.186
0.46 0.01
100
Nonparametric Bootstrap
0.800 0.248
72.18 0.76
100
Jackknife
0.900 0.186
15.70
100
DWS
0.900 0.186
6.48 0.01
Evaluation batch size: 16,
Validation pool size: truth truth_ppol____ size 1200,
Prequential horizon: .M = 120 (unused for Dirichlet shortcut),
Pseudo-count tuning (GPT-2): We use sub-Gaussian plug-in margin based on the calibra- tion subset:
Mmod
Mempl
+ Cmargin V /eeali.
Cyargin 1.0,
and set à clip 00//3; 5, 200) We did not include a separate model-side margin T..
because the model mean used m 200 samples and its MC error was negligible relative to the empirical term.
Model/dataset. GPT-2 ("'pt2", 117M) with GPT-2 BPE; WikiText-2 "wikitext-2rraw-vi' Sequences truncated to max text length 256, eval batch sitcch_siieeeee 16. Mixed precision is enabled on CUDA. Disjoint pools. Trial pool: first 600 validation texts; truth pool: next 1,200 validation texts; calibration subset for â: 150 texts from the test split (indices 200-349), all disjoint from trial/truth. Linear functional and exact MP shortcut. The score is teacher-forced NLL/token h(x);; for linear ((F) = dFF the Dirichlet-mean shortcut yields i.i.d. draws from the exact MP for 0. (no forward rollouts). Pseudo-count (small-n minimax) estimator. We use
â 2
(((Q)) 8(Fw) tx.
with conservative padding in (sub-Gard Gaussian/EB choice) and mild clipping [5, 200]. For GPT-2 we estimate 0 ((2)) via 200 model draws from the frozen LMM, set Cmargin =1, and compute 6² from the calibration subset. The coherent schedule A; = àz(ii + â) then fades model reliance automatically as i grows.
Discussion. At small-n, method DWS achieves the most calibrated predictive coverage: (1.000), n 10 (0.800), n = 20 (0.900), consistently closest to the nominal 90% threshold among the compared methods.
22

---

Published as a conference paper at ICLR 2026
Table GPT-2 (WikiText-2), DWS interval widths across no.
DWS Width
1.259 1 0.128
10
0.901 0.076
20
0.661
0.064
50
0.370
0.038
100
0.245 = 0.015
Table summarizes the DWS widths across no. These widths shrink steadily with n, but remain larger than bootstrap-based methods at small n, reflecting our conservative design for calibration under uncertainty.
At 50, DWS maintains best coverage (0.900), while bootstrap methods achieve narrower intervals but with undereoverage. At 100, coverage ties 0.900 across several methods, with bootstrap yielding smaller widths and lower runtime, as expected when empirical estimates stabilize.
Overall, DWS is the clear winner in the regime that matters most; scarce initial data. It provides well-calibrated predictive uncertainty under the deployed rule, while other methods either under-cover or sacrifice reliability for sharpness.
C.3 VISION
Implementation details (vision). Frozen generator Qo is the google//ddpm-ci 0- checkpoint (Diffusers). The CLIP head is bpenaa//lllp-vvi-aaass---pe
(ViT-B/32). We form 10 text prompts photo of a [airplane.....uuckk] cache text features, and score im- ages via CLIPRarity(v)
log mmaxke(1o] softmax((Sing (v), Jiea [iea())))), with standard CLIP preprocessing. Target and metric. All coverage numbers are for the operational long-run target 0x lim å dP, under the deployed rule; "coverage "90""""90"""" is the proportion of repeti- tions whose 90% predictive interval for fx contains a high-precision reference computed on an independent held-out pool (1000-1200 items for language/vision). We report p + 1.96 SE with SE Vp(1 p)/R. R.
C.3.1 CIFAR-10 (ID): EXPERIMENTAL DETAILS AND EXTENDED RESULTS
Setup and Metries. We evaluate in-distribution (ID) vision setting using frozen generator Q+ on the CIFAR-10 dataset. The operational metric is the long-run mean of the CLIP rarity score:
O(F) = E[((Y)]
where the CLIP rarity function. This metric is linear in F, and intervals report predictive uncertainty for 0xx under the deployed prequential rule. For linear metrics, method uses the mhrichlet-mean shortcut.
Methods Compared. We compare the following uncertainty quantification methods, all evaluated in frozen mode:
BB: Bayesian Bootstrap.
NPB: Nonparametric Bootstrap,
Jackknifee
DWS: Dirichlet-weighted shrinkage (our method, using the Dirichlet-mest mean shortcut).
Hyperparameters.
Number of repetitions: R. 30 per (method, n²),
Sample sizes: (5, 10, 20, 50, 100).
Predictive level: q 0.90,
23

---

Published as a conference paper at ICLR 2026
Calibration pool size: 500,
Truth pool size: 1000,
Evaluation batch size: 64,
Replicate budgets: Bbayssian 40, Booot = 40, Baws = 40, Bmp = 40.
Prequential horizon: Mpreqqentiall = 100 (unused Dirichlet shortcut),
Discrepancy radius: empirical- Bernstel
26 log(2/0)
2H
2 8
log 3n
n
used to conservatively upper-bound matel-data mismatch.
Implementation details (repro), CLIP rarity, and á. Generator. Diffusion DDPM checkpoint (ooogle/ddm-ctte
0-32 (Diffusers pipeline), frozen; images are 32x32 and normalized to [- Sampling RNG uses a fixed generator seed. CLIP head and rarity score. We use CLIP ViT-B/32 of a {class]". Images are resized to 224x224, mapped [0, 1]. and normalized with the OpenAl constants (mean [0. (0.48145466, 0,4578275, 0. 0.40021073), std (0.26862954, 0.26130258, 0.27577711)). Let p(c y) be the softmax over CLIP logits: the CLIP rarity is
httppaaii/iiiaaeeeeaeeaaeeiieeee wit-base patch32) with the 10 CIFAR-I0 prompts "a photo
h(y) log max(maxp(( y)),
clamped below at e e for numerical stability. The operational metric is ((F) = EEh(Y) (lin- ear). Pools and budgets. Calibration pool from CIFAR-I0 train (500 images); trial pool from the remainder of train; truth pool from CIFAR-10 test (1,000 images); _ss_eval= 64; replicate budgets Bbayesiun Boout = Baws- 40. Pseudo-count (EB minimax). We estimate á = 22/ / with an -erriseen--- padding
2 (2/(2/8)
log 8-0.20, H=10,
" and clip à e [5, 200). The model mean ((Q,) estimated with 512 DDPM draws. This conservative padding enlarges A when generator-data mismatch is present, shrinking â and reducing reliance on Qo small n.
In this ID regime, the pretrained generator Qo is well-aligned with the data, so the model-data discrepancy A is small. Our method DWS is designed for small-n minimax performance, balancing sampling variance and potential model mismatch via a conservative discrepancy radius in and a Dirichlet pseudo-count a. This yields over-coverage small (e.g., 1.000 n 10), resulting in wider intervals than bootstrap methods.
Discussion.
Table summmariss the DWS widths across ng. While these widths shrink as n grows, they remain larger than bootstrap-based methods at small n, reflecting our conservative design for calibration under uncertainty.
At larger bootstrap methods become more competitive in coverage and runtime, but DWS remains well-calibrated and operationally meaningful throughout.
Remark (semantic drifts within CIFAR-10). The same prequential rule and Dirichlet-mean shortcut apply to milder semantic shifts (e.g., restricting to classes 0-4 vs. 5-9). Since the discrepancy A is smaller than in CIFAR the EB padding contracts and & increases, yielding even closer-to-nominal coverage with narrower intervals at small n.
C.3.2 VISION: SVHN (OOD): EXPERIMENTAL DETAILS AND EXTENDED RESULTS
Setup and Metries. We evaluate the nut-of-distributiin (OOD) behavior using frozen generator Qo on the SVHN dataset. The operational metric is the long-run mean of the CLIP rarity score:
0( F) E[](Y))
where h(y) = s(y) is the CLIP rarity score. This metric is linear in F, and intervals report predictive uncertainty for the under the deployed prequential rule. For linear metrics, our method uses the Dirichlet-mean shorteut.
24

---

Published as a conference paper at ICLR 2026
Table 7: CIFAR-IO (ID), CCUP (mean): Predictive coverage @ 990% and runtime (mean + 1.96SE). MP omitted; Dirichlet shortcut used. Winners per coverage closest to 0.90 (bold in coverage column).
Method
Predictive cov@90%
Runtime (s)
5
Bayesian Bootstrap
0.567 + 0.177
0.011 + 0.001
5
Nonparametric Bootstrap
0.533 + 0.179
0.406 0.029
5
Jackknife
0.767 1 0.151
0.067 0.007
5
DWS
1.000 1 0.000
26.737 1 0.175
Bayesian Bootstrap
0.833 + 0.133
0.015 0.003
Nonparametric Bootstrap
0.800 + 0.143
0,510 0.086
10
Jackknife
0.833 1 0.133
0.170 + 0.051
10
DWS
1,000 1 0.000
26.630 + 0.052
20
Bayesian Bootstrap
0.867 + 0.122
0.016 0.003
20
Nonparametric Bootstrap
0.900 + 0.107
0,506 + 0.073
20
Jackknife
0,900 1 0.107
0.242 1 0.019
20
DWS
0.967 1 0.064
26.727 1 0.128
50
Bayesian Bootstrap
0.867 + 0.122
0.019 = 0.003
50
Nonparametric Bootstrap
0.933 = 0.089
0,663 = 0.040
50
Jackknife
0.933 1 0.089
0.775 1 0.018
DWS
0.967 1 0.064
26.538 1 0.021
100
Bayesian Bootstrap
1,000 + 0.000
0.032 = 0.001
100
Nonparametric Bootstrap
1.000 = 0.000
1.241 0.014
Jackknife
1.000 1 0.000
3.041 0.013
100
DWS
0.967 + 0.064
26.731 1 0.151
Table 8: CIFAR-10 (ID), DWS interval widths across nog
Tin
DWS Width
0.678 0.042
10
0.500 0.025
20
0.327 0.022
50
0.185 + 0.010
100
0.116 0.006
OOD protocol and â (frozen CIFAR-10 policy). We probe the frozen CIFAR-10 DDPM SVHN to stress-test calibration under a strong domain/semantic shift. The prequential blend and the estimator unchanged: EB padding with 8=0.20, ==10, & [5, 200), and 0(Q₂) estimated from 512 DDPM samples. Trial sets are drawn from SVHN; the truth pool uses up to 1,000 SVHN test images. This fixed nel-yy OOD setting intentionally exercises the regime where standard bootstrap under-covers at small n, while our conservative padding shrinks å and protects coverage.
Methods Compared. We compare the following uncertainty quantification methods, all evaluated in frozen mode:
BB: Bayesian Bootstrap,
NPB: Nonparametric Bootstrap. Jackknife,
. DWS: Dirichlet-werd Dirichlet-weighted shrinkage (our method, using the Dirichlet-ment shortcut).
MP resampling is omitted here since 0 is linear in F.
Hyprraaramsters.
Number of repetitions: R = 10 per (method, n)),
Sample sizes: (5, 10, 20, 50),
Predictive level: q 0.90,
25

---

Published as a conference paper at ICLR 2026
Table 9: SVHN (OOD). EJCLIP rarity]: Predictive @ and runtime (mean + 1.96SE). MP omitted; Dirichlet shortcut used. coverage column).
Winners per n. coverage closest to 0.90 (bold in
Method
Predictive cov @99%
Runtime (s)
5
Nonparametric Bootstrap
0.300 + 0.284
0.55 + 0.03
5
Bayesian Bootstrap
0.300 = 0.284
0.02 1 0.00
Jackknife
0.500 + 0.310
0.10 0.02
5
DWS
1.000 + 0.000
32.00 - 0.74
10
Nonparametric Bootstrap
0,500 0.310
0.59
10
Bayesian Bootstrap
0.500 + 0.310
0.02
10
Jackknifc
0,500 + 0.310
0.18
10
DWS
1,000 = 0.000
32.98 0.06
20
Nonparametric Bootstrap
0,500 + 0.310
0.63 0.13
20
Bayesian Bootstrap
0.400 + 0.304
0.02
20
Jackknife
0,400 = 0.304
0.31
20
DWS
1.000 + 0.000
33.30 0.29
50
Nonparametric Bootstrap
0.600 = 0.304
0.62
50
Bayesian Bootstrap
0.900 + 0.186
0.04
50
Jackknife
0.800 + 0.248
3.04
50
DWS
0.900 + 0.186
31.96 - 0.53
Table 10: SVHN (OOD), DWS interval widths across Ylg-
7A0
DWS Width
0.794 + 0.067
10
0.620 + 0.047
20
0.401 + 0.031
50
0.141 + 0.013
Calibration pool size: 500,
Evaluation batch size: 64,
Replicate budgets: Bhuyesian Bhoat Baws 40,
Prequential horizon: M 100 (unused for Dirichlet shortcut),
Pseudo-count tuning: We the empirical empirical-Bernstein padding
20a n
2 8
2H
2
log
+
log
(6 0.20, H 10),
and á = clip (02/ (Momod Hempl + tn) 5, 200)
Discussion. In the scarce-data regime (n & 50), our method DWS consistently achieves predictive coverage closest to the nominal 90% target. At (5, 10, 20). DWS reaches 100% coverage, while all baselines under-cover significantly (S. 50%). At 50, DWS and Bayesian Bootstrap both reach 90%, but DWS does so with principled minimax blend that accounts for model-data discrepancy, whereas BB relies purely resampling.
Table 10 reports the DWS interval widths across no. Computation is slower for DWS (+= 32s), which the cost of maintaining reliable calibration under distribution shift. In contrast, BB and NPB fail to provide trustworthy coverage at small n, which is precisely when uncertainty matters most.
At 100, BB matches DWS in coverage and has lower runtime, which is expected the empirical law stabilizes and the need for model-based regularization fades.
These results highlight that DWS is the most robust method for predictive uncertainty in low-data OOD deployment: avoids undercoverage, adapts to model-data mismatch, and ensures that intervals reflect the operational uncertainty under the deployed rule.
26

---

Published as a conference paper at ICLR 2026
When to update the generator: If persistent mismatch inflates Å, Section 7's certified retraining trigger (Proposition 7) recommends switching Qo only when the guaranteed worst-case improvement exceeds the stated cost, providing an auditable decision rule.
27


Reviews:
[{"review_id": "nUgnKZ8zrT", "reviewer": "Reviewer_GPLc", "summary": "This paper tackles the problem of uncertainty quantification (UQ) for long-run operational metrics, e.g., mean scores, alert rates, when a system is deployed with very few initial samples (n). The authors argue that standard UQ methods fail in this \"low-data, frozen-model\" setting: bootstrap is unstable, Bayesian posteriors assume refitting, and conformal prediction targets per-example guarantees, not long-run rates.\n\n\nThe authors propose a forecast-first UQ framework with three core contributions:\n\n(1) a prequential blend: a forecasting rule, $P_i$, that blends the sparse empirical data ($\\hat{F}\\_i$) with a frozen pretrained generator ($Q_{\\phi}$). The paper proves that a specific Dirichlet schedule ($\\lambda_i = \\alpha / (i + \\alpha)$) is the only affine blend that ensures time-consistent (martingale) forecasts.\n\n(2) a minimax hyperparameter: a data-driven method to set the blend strength $\\alpha$. It is derived from a minimax criterion (Theorem 2) that optimally balances sampling variance (trusting the noisy data) versus model-data mismatch (trusting the potentially biased model).\n\n(3) a novel UQ method: uncertainty is quantified via Martingale Posteriors (MP), a likelihood-free resampling method (Algorithm 1) that simulates the long-run value of the metric ($\\theta_{\\infty}$) under the deployed blend rule.\n\n\nThe authors validated the framwework on a range of experiments on language (GPT-2 on WikiText-2) and vision benchmarks (image generation on CIFAR-10 and SVHN datasets) Experiments on GPT-2 (for NLL) and OOD vision tasks (for CLIP rarity) show the method achieves its target 90% coverage, with as little as n=20 samples for GPT-2, where other standard methods such as bootstrap fails (37% coverage).", "questions": "This paper takes an interesting direction of study: the problem of UQ for long-run operational metrics when a system is deployed with very few initial samples. I find that the paper is strong and theoretically dense.\nThe reported results in with few samples are impressive. My main questions concern the stability of the estimators and the experiments on the vision domain.\n\nHere are a few questions and suggestions that could be potentially addressed in the rebuttal or in future versions of this work (please note that suggested experiments are not necessarily expected to be conducted for the rebuttal):\n\n1. The key hyperparameter $\\hat{\\alpha}$ is calculated from statistics computed from a tiny number of samples (e.g., 5, 10, 20, ...) with potential high-variance. How does the method remain so well-calibrated (Fig 1a) when its core parameter is based on such an unstable estimate?\n\n2. Please add additional implementation details on the vision side. What type of generator was used? How does the performance of DWS vary depending on the type of generator.\n\n3. Extension of vision OOD experiments towards more semantic drifts (e.g., use clases 0-4 as in-distribution and classes 5-9 as OOD) than domain drifts.\n\n\n4. Regarding the surrogate target, if the frozen model $Q_{\\phi}$ is bad, the surrogate $\\theta_{\\infty}$ could be far from the truth $\\theta(F^{*})$. What can be done in such cases?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Significance**\n- this work addresses the critical and practical problem of UQ for long-run metrics in the \"low-data\" initial deployment phase\n\n**Originality**\n- the prequential (forecast-first) framing seems to me a novel way to approach UQ for frozen models.\n- to me the paper provides a few strong contributions: proving the Dirichlet blend is uniquely coherent (Theorem 1), deriving the minimax criterion for $\\alpha$ (Theorem 2) towards setting the blend strength, applying the martingale posterior in a generative context.\n\n**Clarity**\n- I find that the motivation of the paper is clear, arguing why standard UQ methods (bayesian, boostratp, conformal) are not suitable for this problem of a frozen model and long-run metric.\n\n\n**Quality**\n- The experimental results show that the proposed approach does well on language and vision tasks in the low-sample regime in comparison with multiple standard baselines.\n- I find the bits useful the bit on deciding when to stop resampling and on when to decide to retrain. They are practical.", "weaknesses": "**Stability of hyperparameter estimation**\n- for choosing $\\alpha$ the $\\sigma^{2}$ and $\\Delta$ are estimated from the few available real samples. \n- I'm wondering how stable would these be for n=5 or n=10 and how doest this impact behavior and performance of the framework given that $\\alpha$ might be derived from unstable statistics? How does the method remain so well calibrated?\n\n**OOD experiments**\n- the considered experiment using CIFAR-10 as in-distribution and SVHN as out-of-distribution is interesting but quite extreme as the two distributions are far away in terms of semantics (very different classes) and overall content (objects vs. street numbers). This makes it easier to distinguish the two distributions are they come from different domains.\n- it would be interesting to see if such a model could capture semantic drifts as well. For instance the generator can be trained on classes 0-4 from CIFAR-10 and the remaining classes 5-9 could be the out-of-distribution ones.\n\n**The UQ target (surrogate vs. true)**\n- here the UQ is for the surrogate metric $\\theta_{\\infty}$ (the long-run value of the blend), not the true population metric $\\theta(F^{*})$ \n- if the frozen model $Q_{\\phi}$ is bad, the surrogate $\\theta_{\\infty}$ could be far from the truth $\\theta(F^{*})$. While  the UQ is \"operationally\" correct, it could potentially be \"factually\" wrong, which could be misleading.\n\n\n\n**Missing implementation details**\n- there are no details about the generator used for the vision tasks. Is it an autoencoder, VAE, GAN?\n- the considered CLIP rarity score is not described or referenced. Presumably it's [b]\n\n\n**[Minor] Writing and format**\n- I find that the paper is nicely written, yet in a very compact manner. Even like this, the experiments and related work are in the appendix.\n- A longer form version of this in a journal format might be easier to adopt to given the content of the work\n\n\n**References:**\n\n[a] Ahmed & Courville, Detecting semantic anomalies, AAAI 2020\n\n[b] Han et al., Rarity Score: A New Metric to Evaluate the Uncommonness of Synthesized Images, ICLR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0vfdyYz4M", "reviewer": "Reviewer_UbgZ", "summary": "This paper has introduced an uncertainty quantification framework that \"blends\" the empirical distribution with a frozen distribution (e.g. a distribution from a pre-trained generator), using a Dirichlet schedule. The main algorithm, referred to as \"Martingale posterior sampling\", is illustrated in Algorithm 1 in Section 5. Rigorous theoretical motivation and justification for this algorithm have been provided in Section 2, 3, and 4. In Section 6 and 7, this paper also discusses when to stop resampling in the algorithm and when to retrain the frozen generator. Preliminary experiment results are demonstrated in Section 8.", "questions": "- Please better explain the novelty of this paper relative to recent literature, especially [Fong et al. 2023]\n\n- Please try to address the weaknesses listed above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper seems to provide an alternative uncertainty quantification approach different from existing frequentist, Bayesian, and conformal approaches. I am not familiar with the existing literature on this approach (e.g. Fong et al. 2023), so it is a little bit hard for me to judge the novelty of this paper. However, to the best of my knowledge, the proposed approach seems novel for the machine learning/AI community.\n\n- The mathematical framework, derivation, and analysis in this paper seem to be rigorous.\n\n- The proposed algorithm (Algorithm 1), is simple and elegant.", "weaknesses": "- I very much like the mathematical rigor of this paper. However, I do feel that some sections of this paper are a little bit hard to read for machine learning/AI practitioners who are less mathematical. Thus, I recommend the authors to further polish Section 2-7 to make them easier to follow while keeping the mathematical rigor. One possible approach is to add more plain English explanations of the core ideas.\n\n- In the experiment section, the only metric this paper has considered is the predictive coverage. Can we demonstrate experiment results under some other metrics? For instance, in simple Bayesian settings where the posterior can be exactly computed, might the authors also demonstrate the KL-divergence (or other probability space metric) between the predictive distribution under each method (i.e. NPB, BB, JK, DWS, and MP) and the predictive distribution under the true posterior? I think such results will further strengthen the paper.\n\n- Section 7 (decide when to retrain) is very interesting. Might the authors also provide some experiment results (even preliminary ones) to support the theory in this section?\n\n- [Minor] in Theorem 2, where is $R(\\lambda, F^*)$ rigorously defined?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nUJMY38xgM", "reviewer": "Reviewer_CDPE", "summary": "This paper introduces the \"Prequential Blend,\" a forecast-first framework for quantifying uncertainty in machine learning models when only a small amount of real-world data is available. It is validated under language (GPT2 on WikiText), vision (CIFAR10) and toy scenarios, demonstrating significantly better performance than traditional bootstrap methods in few-sample settings.", "questions": "- Are the settings in the experiments practical usage and worth considering? Take the language modelling case for example, the proposed method is advantageous when n < 100. This seems like an insignificant number considering today's LLMs' huge throughput.\n- Could you please clarify the definition of the \"coverage@90%\" metric used in the results?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and nicely presented. \n- The proofs and implementation details are provided in detail.", "weaknesses": "Please refer to questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hN13jbVRzc", "reviewer": "Reviewer_iCc2", "summary": "This paper introduces a new forecasting method that quantifies prequential uncertainty via Martingle posteriors with a pretrained generator and a Dirichlet schedule. The proposed method requires a small-$n$ minimax criterion hyperparameter to balance sampling variance and model-data mismatch. Experimental results show the proposed method outperforms bootstrap baselines in vision and language datasets.", "questions": "1. How computationally efficient of the DWS compared to other baselines? In the case of non-GPU-parallel, do the authors have an idea to improve the sampling efficiency?\n2. In Theorem 1, the generative model $Q_\\phi$ is $\\mathcal{F}_n$-measureable. How does this assumption hold in practice with modern models (e.g., Diffusion, GAN, VAE)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is generally well-written and is clear to understand the important aspects.\n- The Prequential Blend method is solid with theoretical guarantees about how to adjust the blend and predict horizon.\n- The proposed framework is also useful in a generative model use case where we need to decide when to retrain the pre-trained model.", "weaknesses": "- Sampling the martingale posteriors requires multiple forward passes, raising a concern about the computational efficiency of the algorithm in long-horizon.\n- The proposed method requires predefining a hyperparameter $n$, which may not be trivial to select in practice.\n- Experimental results may need to be improved with a larger scale setting, such as with modern architectures and a higher-dimensional dataset (e.g., ImageNet, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)