Robust Decision-Making with Partially Calibrated Forecasters
Authors: Shayan Kiyani
Hamed Hassani
George J. Pappas
Aaron Roth
Venue: ICLR 2026 Poster (2026)
Keywords: Calibration
Decision Making
Uncertainty Quantification
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Calibration has emerged as a foundational goal in trustworthy machine learning, in part because of its strong decision theoretic semantics. Independent of the underlying distribution, and independent of the decision maker's utility function, calibration promises that amongst all policies mapping predictions to actions, the uniformly best policy is the one that trusts the predictions and acts as if they were correct. But this is true only of fully calibrated forecasts, which are tractable to guarantee only for very low dimensional prediction problems. For higher dimensional prediction problems (e.g. when outcomes are multiclass), weaker forms of calibration have been studied that lack these decision theoretic properties. In this paper we study how a conservative decision maker should map predictions endowed with these weaker (partial) calibration guarantees to actions, in a way that is robust in a minimax sense: i.e. to maximize their expected utility in the worst case over distributions consistent with the calibration guarantees. We characterize their minimax optimal decision rule via a duality argument, and show that surprisingly, trusting the predictions and acting accordingly is recovered in this minimax sense by decision calibration (and any strictly stronger notion of calibration), a substantially weaker and more tractable condition than full calibration. For calibration guarantees that fall short of decision calibration, the minimax optimal decision rule is still efficiently computable, and we provide an empirical evaluation of a natural one that applies to any regression model solved to optimize squared error.

Published as a conference paper at ICLR 2026
ROBUST DECISION MAKING WITH PARTIALLY CALI- BRATED FORECASTS
Shayan Kiyani, Hamed Hassani, George Pappas & Aaron Roth University of Pennsylvania (shayank, hassani, pappasg, aaroth) seas upenn.
ABSTRACT
Calibration has emerged as a foundational goal in "trustworthy machine learning", in part because of strong decision theoretic semantics. Independent the derlying distribution, and independent of the decision maker's s utility function, cal- ibration promises that amongst policies mapping predictions to actions, the uni- formly best policy is the one that "trusts the predictions" and acts as if they were correct. But this is true only of fully calibrated forecasts, which are tractable to guarantee only for very low dimensional prediction problems. For higher dimen- sional prediction problems (e.g. when outcomes are multiclass), weaker forms of calibration have been studied that lack these decision theoretic properties. In this paper we study how a conservative decision maker should map predictions endowed with these weaker ("partial"") calibration guarantees to actions, in a way that is robust in minimax sense: i.e. to maximize their expected utility in the worst case over distributions consistent with the calibration guarantees. We char- acterize their minimax optimal decision rule via a duality argument, and show that surprisingly. "trusting the predictions and acting accordingly" is recovered this minimax sense by decision calibration (and any strictly stronger notion of calibration), a substantially weaker and more tractable condition than full calibra- tion. For calibration guarantees that fall short of decision calibration, the minimax optimal decision rule still efficiently computable, and we provide an empirical evaluation of a natural one that applies to any regression model solved optimize squared error.
INTRODUCTION
Machine learning systems are increasingly deployed in high-stakes decision making domains such as healthcare, finance, and law. The predictive power of these models can be extraordinary, but scoring well on predictive error metrics does not directly guarantee that decisions downstream of those predictions will be correct. For predictions to be operationally useful, decision-maker must be able to treat them as reliable inputs into a downstream decision making policy. This raises two fundamental questions:
On the Model Side: What does it mean for machine learning predictions to be trustworthy in decision-makine contexts?
On the Decision Making Side: Given predictions that satisfy a particular type of "trustworthiness" how should the decision maker adapt its actions to the promised guarantees?
On the Model Side: A natural answer is that trustworthy predictions should directly support good decisions as they are. other words, the decision-maker should able to reliably best respond to the forecaster's predictions as if they were correct. Formally, let (X,Y² Y) be pair of random variables drawn from a joint distribution D, where X E X' represents the observed features and Y E [0, 1] is the outcome of interest. Let A denote the action set, and suppose the decision-maker follows policy a(-) : J0, 1]a + A mapping predictions to actions. Given a predictor f. the decision maker's performance when using policy is measured by its expected utility the underlying distribution:
((x,))~Du((al (M(o)(((((()))))

---

Published as a conference paper at ICLR 2026
where u(a, y) E R is utility function. Given a forecaster f f [0, 11 the plug-it in best response to forecast is defined as
aBn (f()) arg max u(a, f())). acA
Thus, forecaster f is trustworthy if the decision-maker's best-response policy @BB (x()) achieves higher utility than any other policy. When is this the case?
The classical answer lies in the notion of calibration. Intuitively, forecaster is calibrated if, when- ever it predicts a vector f()) = v E [0, II the empirical outcomes are consistent with that predic- tion. More formally, forecaster f' is said to be fully calibrated if for every [0, 1Jo,
Y f(x)
v.
It is well known that best responding to calibrated forecasts is the optimal decision policy among all policies that map forecasts to actions (Foster & Vohra 1997 Kleinberg et al. 2023 Noarov et al.,
2023
Roth 2022).
However, achieving full calibration is extremely difficult, both in theory-the the sample complexity of calibrating an existing forecaster without harming its accuracy grows exponentially with the out- come dimension d (Gopalan et al. (2024a-a00 and in practice, where empirical evidence shows sys- tematic deviations from calibration, ranging from neural networks to large language models (Guo et al. 2017]] Kull et al.) [219 Gupta & Ramdas [022] Plaut et al. (002)). Thus, despite the appeal- ing link between calibration and trustworthy MI.-powered decision- makinn,, this connection quickly breaks down in real-world applications.
On the Decision Making Side: Decision making from predictions admits two canonical extremes. At one end, the decision maker aggressively best responds to the forecasts, acting as if they were fully correct. At the other end, the decision maker conservatively plays minimax-safety strat- egy, arg m8x minyey u(a, v), treating the forecasts as if they carried no information about the instance.
Departing from these extremes, treat a model f and its forecast f(x) information that con- strains what the true, instance-condttional outcome distribution could be. In other words, after observing (x), the decision maker considers the set of candidate realities- outcome distributions consistent with the forecast and the available calibration guarantees. Intuitively, the "volume" of this set is governed by the strength of calibration: under full calibration, the set collapses to the forecast itself (the prediction can be treated as reality, at least in expectation), whereas as calibration weakens, the set enlarges. A principled decision rule should therefore tune its conservatism to what the reality could be, consistent with the provided guarantees. This idea, together with the fragility of full calibration in practice, leads to the central question of this paper: can we derive optimal decision makiinnnn-miiiggggg policies under weaker and more practical conditions than full calibration?
We answer this question affirmatively. We introduce a framework based on conservative decision making that nevertheless fully exploits partially calibrated forecasts. This viewpoint echoes ideas in robust optimization and control, but it has not been systematically developed for post hoc decision making with partially calibrated machine-learning forecasts.
1.1 OUR RESULTS
We consider a parameterized family of weighted calibration guarantees that have recently become popular object of study (Hébert-Johnson et al. 2018 Gopalan et al. 2022). Informally speaking, this family of guarantees constrains the residuals of predictor f to be uncorrelated with a collection of "test functions" h e H mapping the range of f to the reals. When H. consists of all such test functions, recover full calibration, but many popular variants of calibration (e.g. top label calibration, decision calibration, etc) can be expressed as instances of H--alibration under much smaller/more tractable sets H. Our contributions are as follows:
1. In Section 2 we formalize the following question: given set of test functions H and predictor f(x) that is promised to satisfy H-calibration, what decision rule a [0, 1]d + A, mapping predictions to actions, will maximize a decision maker's 's expected utility in the worst case over all joint distributions over X x Y that are consistent with the promise that f is H-calibrated?
2

---

Published as a conference paper at ICLR 2026
2. In Section we answer this question by giving closed-fome for the decision maker's 's opti- mal decision rule, in terms of the dual variables of a convex program that can be efficiently computed for any finite H.
3. In Section 4 instantiate this decision rule for various calibration guarantees of interest. Of particular note, we find that when H. corresponds to the tractable notion of decision calibration (Zhao et al., 2021 Noarov al. 2023), then the optimal decision rule is the best response decision rule (BR- just as it is for (the intractable notion of) full calibration. In fact, suffices that H. contains the decision calibration constraints any larger set also makes best response the optimal decision rule. Thus what could have been very large hierarchy of minimax optimal decision rules "collapses" to best response at the level of decision calibration. An upshot of this is that predictor can be simultaneously decision calibrated for many downstream decision makers, and for each of them, best response will be their optimal decision policy in this minimax sense. We also derive the minimax optimal decision rule for a simple self-oothogonality" calibration condition that will hold for any regression model with linear final layer trained to optimize squared loss, and hence will be commonly satisfied without any algorithmic intervention.
4. In Section we train a two-layer MLP to minimize squared loss on two regression datasets, and evaluate both the best-response decision rule and the robust decision rule that results from the self-orthogonalliy condition of squared error regression. We find that, as predicted by our theory, the robust decision rule outperforms the best-response decision rule under calibration-presse preserving distribution shift, and that the cost of this robustness is mild even under ideal conditions.
1.2 RELATED WORK
Rothblum & Yona (003)) consider setting in which both the outcome and decision maker's action are binary, and study how a decision maker should act minimize their worst case regret over distributions such that the predictor has maximum calibration error bounded by a: informally that [E[)] v| - v| S a for all v. The models f they study are (approximately) fully calibrated, which is a reasonable assumption in their setting, since they limit their study to 1-dimensional out- comes. In contrast, our interest is not (just) in quantitative measures of full calibration error, but rather qualitatively weaker calibration guarantees, as even approximate full calibration becomes in- tractable in high dimensions.
A line of recent work (Zhao al. 2021 Kleinberg et al. 2023 Noarov et al. 2023 Roth & Shi, 2024 Hu Wu 2024, Okoroafor et al. 2025) has studied the guarantees that can be given to downstream decision makers who best respond to predictions that have weaker guarantees than full calibration (and which in the cases of Zhao et al. (2021): Noarov et al. 20 Roth & Shil (2024) can be tractably guaranteed in higher dimensional outcome settings). These guarantees take the form of (external and swap) regrer bounds, which are qualitatively weaker than the kind of "trustworthiness" promised by full calibration. Informally, regret bounds promise that the decision maker could not have done better by consistently playing a fixed action (or a fixed function remapping their actions to other actions), not that they could not have done better by using different policy from predictions to actions. We show that even in high dimensions, the tractable "decision calibration" condition given by Zhao et al. (202T) recovers the same "trustworthiness" semantics of full calibration when viewed through our minimax decision making lens.
Analyzing minimax optimal decision policies is a common way of analyzing robust or risk- averse decision making guarantees, with deep roots in economics (Gilboa & Schmmidlerr 1989.
Hansen & Sargent 2001 Manski] 2000 2004 Manski & Tetenov| 20077] Manski| 201 1). statistics
[Wald]] 1950), and robust optimization (Ben-Tie Tal & Nemirovski 2002 Kuhn et al. 2019 Duchi &
Namkoong 2021). 1) For example, (2015)) adopts this lens this in the context of contract theory and Kiyani et al. (2025) and Andrews & Chen (2025) do so in the context of conformal prediction.
To the best of our knowledge, we are the first to apply this "robust" minimax lens to the problem of
partially calibrated high dimensional forecasts.
3

---

Published as a conference paper at ICLR 2026
maximally conservative
macimnally aggressive
Ravacala arg (6) -c/ you
(1)
min R. (( * (uge .. you
aun (()) ang aFd
wia. fix))
H no information
all functions
full calibration
complexity of M
Figure 1: Schematic of the interpolating property
ROBUST DECISION MAKING AND H-CALIBRATION
In this Section, define H-calibration as flexible relaxation of full calibration and then intro- duce a framework to derive minimax optimal decision making policies that are designed to act on forecasters guaranteed to satisfy H-calibration. This family of calibration guarantees has been stud- ied extensively in the recent literature on multicalibration and its extensions (Hébert-Johnson et al.) 2018 Dwork al. 202T; Gopalan et al. [2022 Deng et al., 2023) - in particular, H-calibration is special case of what Gopalan et al. (2022) call weighted multicalibration.
H-Calibration. Let H be set of functions h : [0, 1jd + TR. forecaster f is said to be H- calibrated if for every h E H.
B[M((XX) - ((x)
Equivalently, writing qu)) = E[Y f(X) o] for the true conditional expectation, H-calibration requires
E [[(((X)) (l(f(x)) ((x)) =0 Yh E H.
(3)
This definition captures spectrum of guarantees. When H contains all bounded measurable func- tions, H-calibration reduces to full calibration i.e. it requires that f(v) = q(v) == IB[Y [(X) v] almost surely. For smaller classes H. the requirement is weaker and can be seen
relaxation of calibration, enforcing consistency only with respect to a restricted set of tests. In the main body of
the paper we focus on the H-calibration defined above, but in Appendix B we also discuss scenarios in which only approximate H-calibration is available.
Robust Decision Making. Fix H--allirraedd forecaster f. Define the set
@ = { 10 11 + jo, 1 E[M((X))
=0, 4h EhE (4)
In words, Q consists of all candidate conditional expectations consistent with satisfying H- calibration. Because the perfect predictor S(X) [[[[X] satisfies H-calibration for every H. the identity map q(v) = v is always Q-but in general the set may contain many maps. From the perspective of the decision-man maker who knows and the promised calibration guarantee H, but does not know the underlying distribution, given a forecast f(x), the true expectation E[Y f(x)] is uncertain but must lic within Q. As H grows richer, Q shrinks, eventually reducing to {g(o) b} in the case of full calibration.
Faced with this uncertainty, a natural strategy is to adopt robust policy that guards against the worst-case admissible reality. Formally, the robust decision rule
Grobust
arg max min ([ula(((x)). EEQ
(5)
at =
That is, the decision-maker chooses an action policy that maximizes utility under the worst-case conditional expectation consistent with calibration guarantees.
Interpolating Property. The robust policy in Equation 5 interpollates between two classical ex- tremes (Figure []. If H contains all functions, then Q {a(e) F. and drobust reduces to the best response 4BR (.) (Equation equation []. If 7. is empty, then Q contains all functions and the policy collapses to the constant minimax strategy @Minimax (2: arg max min yoo. 1]a u(a, y). Thus, Equation 3gquation3]proveee provides principled bridge between best-responding to calibrated forecasts and adopting fully conservative policies, with the level of conservatism controlled by the richness of H.
4

---

Published as a conference paper at ICLR 2026
The central theme of the remainder of this paper is to investigate the interaction between different levels of H-calibration and the resulting optimal robust policies. Our focus is not on developing methods for achieving H-calibration itself (for which we refer the reader to rich line of recent work showing how to accomplish this in both the batch and online adversarial setting HHbbertJohnson et a [0T8] [Gopalan et al. 2022 Deng al. 2023 Noarov al. 2023] Elobus-Harris
al. 2023), consequences once such guarantees are in place. In the next section, we begin by analyzing the general problem of deriving optimal robust decision rules for arbitrary classes H. We then specialize to the important case of decision calibration, showing that this weaker and more practical notion identifies large classes of partially calibrated forecasters for which best responding remains optimal. Beyond its theoretical appeal, this result is also practically useful: when decision-mane maker can influence the design or post-processing of the forecaster, they can request decision-calibrated forecaster, to which they can then simply, reliably, and optimally best respond.
but rather on understanding the decision-make
Assumption 2.1. The utility u(a, 2) linear in its second argument v E [0. 1]d for each a A.
This assumption naturally holds in multi-class settings where is probability vector over d out- comes and the decision maker has arbitrary utilities (a, k) for each action-outcome pair. In this case, u(a,v) v) (a, Y)] = Σi_
U k), which is linear in v. Such risk-neutral expected- utility models underlie much of the calibration and decision-making literature (e.g., (Foster & Vohra 1997 Kleinberg et al.] 2023 Roth & Shi 2024)). Utilities that are nonlinear in for example, risk-averse utilities depending on outcome variance, fall outside our framework and represent an important direction for future work.
OPTIMAL DECISION POLICIES FOR FINITE DIMENSIONAL H-CALLBRATION
In this section, we characterize the optimal robust decision making policies, i.e., solutions to Equa- tion [5] Throughout this section, we assume the function class H is finite dimensional space, i.e. it can be described as span of finitely many functions. Formally, let H sp ha} be the linear class generated by measurable h, [0, 1]d R. Then the H-calibration condition equation is equivalent to the k linear moment equalities
E (fx(f( (a(f(x)) - f(x)) = 0,
i 1=
so that the ambiguity set in equation may be written as
= { [0, - [0, 1]dd
B [fx((((x) (a(f(x)) =0 for 1, 1k....,k..
Intuitively, each equality enforces that, conditional on the forecast, the forecast error has zero cor- relation with the corresponding test hy; taken together, these constraints exhaust the information provided by H-calibration criteria and hence precisely describe the admissible reality faced by the robust ciceision-mkee in equation 5)
Theorem 3.1 (Characterization of the Optimal Robust Policy). Suppose H = span(h,..... hx)
with each b4 : [0, 1]d + R. and let Q be defined as above. Then the minimax problem Equation 5 admits a saddle point (arobust, q²)) with the following structure:
There exist multipliers X* (Xi...., Xt) with each X E Rd such that for almost every forecast 2 f() the worst-case map q" (v) solves
('(v) E arg min val(() + p
Σ nn(o)se)
where val(p) max u(a, p).
aeA
Given q", the optimal robust action at u is the best response q- (v):
Grobust (v) E arg max ((a.g"(v)) a
Interpretation. Theorem 3.1 characterizes both the worst-case distribution consistent with 7--
calibration and the corresponding optimal response. For any realized forecast =a()), the theorem
5

---

Published as a conference paper at ICLR 2026
yields simple two-step procedure: compute the adversarial belief
(v) arg min (val(p) + (v)); pejo.1]a
(* Σ b,(1)a
and then take the best response arobust (v) E arg max q(a, q" (V)). Thus, the optimal policy is always a best response, not to the raw forecast f(x), but to the adversarially tilted distribution (v) allowed by the calibration constraints. Additionally, useful consequence is pointwise com- putability: evaluating 4robust at given reduces to two low-dimensional optimizations, without constructing the full mapping I Grobust (z).
From an optimization perspective, the multipliers X solve a finite-dimensional concave maxi- mization problem (see the proof of Theorem B.Ip. and 9" (v) is obtained by pointwise con- vex minimization over with provable guarantees (e.g.. projected subgradient ascent for the dual, or simple primal-dual scheme), after which one evaluates q² (V) via the pointwise minimization and takes the best response Arobust (v) = arg max. u(a, q² (v)).
[0, 1] Both stages can be carried out by standard, fast methods
In the next section, we analyze the behavior of the resulting decision rules by specializing concrete H-classes. One might expect that Theorem 3.1 induces a vast hicrarchy of policies whose form depends sensitively on H. Perhaps surprisingly, this is not the case. In particular, we show sharp transition: for each decision maker, there exists a specific test class, precisely the one associated with decision calibration, such that soon as H contains this class, the adversarial tilt collapses (q" (v) v for a.e. v) and the optimal robust rule reduces to the plug-in best response to the forecaster.
4 ROBUST POLICIES UNDER DECISION CALIBRATION AND BEYOND
In this section, we specialize the general characterization derived in Theorem 3.1 to concrete test classes H. Our core result concerns decision calibration: practically tractable guarantee under which the minimax-optimal robust policy collapses to the plug-in (best-response) rule. This identi- fies simple path to srccisinnthhorret trustworthiness that does not require full calibration.
4.1 DECISION CALIBRATION AND PLUG-IN BEST RESPONSE OPTIMALITY
Here we define the variant of decision calibration given by Noarov et al. (2023), slight strength- ening of the definition originally given by Zhao et al. (2021). Fix a single decision problem with action set A and utility function u(a, v). For each action A, let
R₂ { c [0, 1jd u(a,v) v) 2 u(a',u) for all a' E A
be the (closed, convex) decision region which is plug-in best response, The decision- calibration class is Haee = { [1R. :a a A). Here, we denote 1x = 1(x E A]. forecaster f is decision calibrated if it is -ale-bcaibbrted, i.e.
E (1xx(((x)) f(x))] = 0 for all a E A.
Compared to full calibration, decision calibration is far more statistically tractable, since its test class has size |Haeel = |AA, potentially small and fixed number of actions, rather than the large families required for full calibration.
Theorem 4.1 (Decision calibration = plug-in best response optimality). If Hade-calibbatted then the manimax-optimal robust rule in equation coincides with the plug-in in best response:
arobust (v) E arg rrggmau((a aEA
for almost every u = f(x).
Equivalently, under decision calibration, best responding to the forecaster is minimax optimal among all forecast-baa based policies.
Put differently, upon observing forecast v f(x), the decision-maker need only best respond to v: no adversarial "tilt" survives the caliiirtatiiatioonn constraints. Conceptually, this upgrades the previously known guarantees of decision calibration- that it implies swap regret (Noarov al.

---

Published as a conference paper at ICLR 2026
maximally conservative
maximally aggressive
maammin
ABR
QBR
H = 0 no information
Mai = (IR, : a AA
H { all functions full calibration
decision calibration
complexity of H
Figure 2: Schematic of the Sharp Transition
2023 to minimax optimality. Swap regret guarantees do not preclude the existence policy [0, _] A that dominates the plugin best response policy OBR only that no improved policy has the form a(v) ((aBR()) (u)) for some mapping ø A + A. using "actions as a bottleneck".. In contrast, Theorem 4. directly establishes that other policy [0, 1] - A can improve the plugin policy OBR in our minimax sense.
The preceding result assumes that the information conveyed by the forecaster to the decision-maker exhausted by the decision-calibration tests RR, JaEA. In practice, a forecaster might satisfy additional calibration equalities,
E h(f(X)) (Y - -(x)(x)]
for functions beyond the indicators In,- The next theorem shows that the plug-in optimality conclusion is stable under such enrichments. This is intuitive: if forecaster is trustworthy, then making it more calibrated (i.e., adding information) should not diminish that trustworthiness.
Theorem 4.2. Let H be any test class that contains the decision-cali indicators, Hdec {IR. : E A}. If is perfectly H-calibrated, then the minimax-optimen robust rule in equation coincides (a.e.) with the plug-in in best response:
Grobust (V) E arg max u(a, v) aeA
for a.e. == f(x).
As we make precise in the proof of Theorem calibration constraints ensure that the expected utility of the plug-in best-response policy aBR is invariant to the adversary's 's choice of q Q. For any satisfying the Hace constraints,
the "collapse" occurs because the decision-
E E]u(aEn(f(x)) (((()))) = E[u((en(((X)) f(X))
Thus, the adversary cannot reduce the utility of ann; its worst-case utility equals its nominal utility. Since AB is the optimal policy under the nominal distribution, and its performance cannot degrade under any admissible g, must also be the Linimax-optimal policy.
Sharp transition. One might initially expect a gradual shift from fully conservative to plug-in best response as is enriched. Theorems 4.1 4.2 show sharper phenomenon (Figure 2 once H contains the |A| decision tests {1n. JoEAA the adversarial tilt disappears (q"(v) v a.e.) and the robust rule collapses the plug-in in best response equation Enlarging further does not change the minimax-optimal policy.
Decision calibration is a tractable, task-specific threshold which robust decision making and plug-in best-response coincide, providing a crisp target for forecaster design and a clear requirement for downstream decision makers.
As byproduct, this leads to another practical advantage of decision calibration: a single forecaster can be made simultancously reliable for collection of downstream decision problems. Intuitively, the forecast passes the decision calibration tests of each problem, then none of the decision makers needs additional robustness, the plug-in best-response is minimax-optimal for all of them.
Corollary 4.3 (Simultaneous plug-in optimality across multiple decisions). Let u1,.... Um be m decision problems, with respective action sets A, and linear utilities u,(a, v) v E [Ó. 1j.. For each j and a E Aj. let
Ra.j (v E (0, 10. uz(a, v) uzla', v) for all a' E Aj}
7

---

Published as a conference paper at ICLR 2026
be the plug-in decision region of action in problem j. and define the combined test class
H
(1Rm, : aE
j=1
If f is H-calibrated for some H satisfying Hale H. then for every j E (l,...,m)) the minimax- optimal robust policy for problem j coincides (a.e.) with the plug- in best response:
m
Jrobust, (v) E arg max uy(a, for v= BEA,
Proof. For each problem, j. the included indicators {1 Ra. JaEA, ensure that H contains the decision- calibration tests of problem j. Theorem 4.2 then applies verbatim to each j. yielding plug-in opti- mality problem by problem.
4.2 BEYOND DECISION CALIBRATION: GENERIC H. HI-CLASSES FROM TRAINING PIPELINES
Thus far we have focused on decision calibration, which, when attainable, collapses Grobust to the plug-in best response. In practice, regimes arise. (i) If one can influence the forecaster's training pipeline, decision calibration is the natural target: is practical, and our results guarantee plug-in minimax optimality. (ii) If one cannot control training, the forecaster might not be decision cali- brated for the downstream task. Identifying its partial-calibration profile may be difficult, yet certain moment conditions arise structurally from standard training procedures. We give two examples of how to leverage such "free" structure to specify usable H's and derive the associated robust policies.
eelf-orthogonnality from squared-loss training. A ubiquitous example selfforrthgoonalit (a form of self-calibration) that follows from first-order optimality when model with linear last layer is trained to minimize mean squared error. This includes the universally adopted cases of regression with either linear model or neural network with a linear head, trained by mean squared error. This and similar guarantees for other loss functions have previously been investigated as consequences of low degree multicalibration (Gopalan et al. 2022).
Proposition 4.4 (Self-orther SSllo-orth under squared loss). Let X + zx( E IR be representation and (x) = W (X) E IR a linear last layer. Suppose = (Ø, W) is trained to first-order stationary point of the expected squared loss
- ((0) E [f -
Then the following calibration moments hold:
(Y (x) fo(x)) =0 and (Y
In particular, fo is Pl-calibrated for the test class H. = (h,(u) = =eju: j I,.... d} (and for any linear combination thereof).
Implications. Proposition 4.4 provides a generic, inplline-indueed H-calibration guarantee when- ever a linear head is trained to stationarity under squared loss. Specializing Theorem 3.1 to this setting yields simple dual. For d = 1 (e.g., one-dimensional regression) with H (h(r) v). the multiplier is a scalar X, and for each forecast v = f(z) the worst-case distribution is
q (v())= arg min (val(p) + Avp). val(p) pE[U.1]
max ma, aF
The robust action is then: (v) arg max, u(a, q" (v)). When u(a, p) is linear in p and A is finite, val is convex piecewise linear, so the inner minimization reduces to checking finitely many candidate points (endpoints and pairwise breakpoints). The dual objective
G(A) = EE[ min (val(p) + *f(X)p]] - XE(f(X) p=
concave in and can be maximized via standard one-dimensional methods (e.g., bisection on a monotone subgradient). In higher dimensions (d > 1), the correction term Aup becomes Avp for matrix of multipliers A, and the pointwise problem remains a small convex program over E (0, 1]d; for finite A and linear utilities, it is again efficiently solvable.
8

---

Published as a conference paper at ICLR 2026
Zero-bias and bin-wise calibration. A widely available source of partial calibration comes from post-hoc recalibration that many practitioners already apply (mean correction, histogram binning, held-out split). These procedures enforce generic (not task-specific)
isotomic-style step fits moment constraints that are directly usable our framework. We focus on bin-wise calibration: take partition of the forecast range into bins (Bi....; By} and enforce, for each bin,
B(LIXXXEBB) (Y [(X)) 0, j 1,.....,
This corresponds to the test class Hbin = {1g, : ,...... J}, and reduces to zero-bias when J-1 with B₁ = -0, 1|
Proposition 4.5 (Robust policy under bin-wise calibration). Let the utility be linear in the outcome and the action set A be finite. If f is Hssn -calibrrad then with
m == [[)( f(X) E B,] E f(X) B,),
the worst-case belief is piecewise constant
q'()) = m, for E B3 (a.e.)
and the robust action best-responds to the bin mean:
Grobust (v) E arg max {u(a, m;)}
for u E B, (a.e.).
aEA
Implications. Bin-wise calibration Hbin can be obtained cheaply via standard post-hoc methods (histogram binning or isotonic regression), and Proposition 4.3 yields an especially simple, closed- form characterization of the robust policy. Computing arobust. reduces to: (i) estimating m, on calibration split, and (ii) at test time, mapping to its bin B, and best-responding to my. No additional optimization is needed to compute actions. As a special case, when J = 1 we recover the global-mean constraint E[Y f(X)] = O. Then q" is constant, 9a (t) = m., with m [[ E[Y]. and the robust rule ignores e and plays arg max,eA u(a, m.) As the partition is refined, the robust rule moves from single global plug-in best response m piecewise plug-in best response at my. yielding a richer, finerggraine decision policy.
EXPERIMENTS
In this section, we evaluate the validity and practical consequences of our framework by implement- ing our methods on two real-world datasets. We compare the plug-in best response (ABR) against the robust policy (arobust). which enjoys minimax optimality guarantees under H-calibration.
We focus on two classes of metrics. Nominal performance measures average utility when the test data are i.i.d. from the same distribution as the training and calibration splits; this reflects opti- mistic regime that often degrades in practice. Adversarial performance probes the other extreme by altering the test-time outcome distribution in ways: (i) a worst case tailored to the plug-in pol- icy, and (ii) worst case induced by the robust dual, tailored to the robust policy. In both cases, the adversarial distributions respect the H-calibration constraints and are therefore indistinguishable, from the decision-maker perspective, from i.i.d. test draws given an H-calibrated forecaster.
Our theory predicts two patterns. First, by minimax optimality, the robust policy should dominate the plug-in rule when each is evaluated against its own worst-case distribution (and typically also under the adversary tuned to hurt the plug-in). Second, because (arobust, q²) forms a saddle point of equation [] when both policies are evaluated under the robust-tuned adversary, the robust policy should not underperform the plug-in rule. Under nominal i.idd evaluation, the plug-in rule may achieve higher utility, reflecting the lack of need for conservatism in that regime.
5.1
CASE STUDIES: BIKE SHARING AND CALIFORNIA HOUSING
We evaluate our framework on two regression datasets with distinct decision making interpretations.
Bike Sharing (UCI). The UCI Bike Sharing (daily) Fannsee-Ta & Gama (0014) records daily rider counts alongside calendar and weather covariates (season, month, weekday, holiday, working day, weather state. temperature, humidity, wind). The outcome Y E [0, 1) is the rescaled total rider count, and the decision-mann maker chooses a staffing/capacity multiplier from A = {0.8, 1.0, 1.2], interpretable conservative, nominal, and aggressive provisioning.

---

Published as a conference paper at ICLR 2026
Table I: Mean utility on the test set under natural i..dd evaluation and two adversarial evaluations.
Adversaries respect H-calibration (H = (h(w) = =)).
i..dd.
Worst-case for robust
Worst-case for plug-in
Dataset
Plug-in in
Robust
Plug-in
Robust
Plug-in
Robust
Bike Sharing (UCI)
0.474
0.463
0.402
0.410
0.393
0.412
California Housing
0.216
0.207
0.160
0.164
0.155
0.166
California Housing, The California Housing dataset Pace & Barry (1997) records median house values (rescaled to [0, 1)) with demographic and geographic covariates (median income, housing age, population, latiudeelongitude, etc.). Here the decision-makes chooses investment multiplier from A = {0.6, 0.75, 0.90), interpretable as conservative, nominal, and aggressive investment.
Utility specification. In both settings we adopt the utility function u(a, y) = aay- waayyCCll C(a), which is linear in y. The benefit term a y captures service or return proportional to realized outcome y, scaled by a > 0. The cost term C(a) grows a, penalizing aggressive choices via over-provisioning costs investment risk. This form tunes the mnder/over-trade-o0 without departing from linearity. For Bike Sharing we use (a, C(-)) = (0.9, {0.02, 0.05, 0.1}). while for California Housing we use (a, C()) (0.9, (0.02, 0.05, 0.20)). The qualitative conclusions of this Section remain the same under other reasonable parameter choices.
Forecasting model. In both datasets, the forecaster f is two-layer MLP regressor trained to optimize mean squared error. By the self-orthogonality property of linear heads under squared loss (Proposition 4.4). the learned forecaster approximately satisfies H-calibration with H (h(u) = v), which is the calibration constraint used to derive the robust policy Grobust- All experiments an i.i.d. rrinncaliirationntess split (60/20/20). We use the calibration data to substitute any population level expectation that needed be computed derive Grobust.
Results. Table D reports the mean utilities. The results match theory: under adversaries tailored to the robust policy, the robust rule achieves at least the plug-in performance; under adversaries tuned to harm the plug-in rule, the robust policy secures noticeably higher utility, reflecting its minimax protection. Moreover, the robust policy outperforms the plug-in best response when each evaluated against its own worst-case distribution.
6 CONCLUSION AND LIMITATIONS
We developed a becision-theoreti framework for acting on partially calibrated forecasts via a minimax-optimal robust policy over H-callibrated forecasters. We then identified a sharp transition in the behavior of these policies: for any decision problem with m actions, there exist m decision tests (the decision-calibration class) such that, once they are included H. the robust policy col- lapses to the plug-in best response. This spotlights decision calibration as a natural requirement whenever the decision-maker can influence the training pipeline. Moreover, even when decision calibration is unavailable, we showed that generic properties induced by standard training and post hoc procedures (c.g., self-orthogonallty under squared loss and bin-wise calibration) yield usable test classes H. and tractable robust policies within our framework.
Our model assumed that downstream decision makers were risk neutral i.c. their utility functions u(a, v) are linear in and A is finite; these are standard assumptions in the calibration literature, but broadening them would be interesting. We note that certain classes of non-lineat utility functions can be linearized over an appropriate basis (Gopalan al. 2024b Lu al. (202)) which would allow our results to apply - though these bases are not always low dimensional enough to be practical.
REFERENCES
Isaiah Andrews and Jiafeng Chen. Certified decisions. arXiv preprint axXiv:2502.17830, 2025.
10

---

Published as a conference paper at ICLR 2026
Aharon Ben-Tal and Arkadi Nemirovski. Robust optimization-metiooollogy and applications. Mathematical programming, 92(3):453-45 2002.
Gabriel Carroll. Robustness and linear contracts. American Economic Review, 105(2):56--663, 2015.
Zhun Deng, Cynthia Dwork, and Linjun Zhang. Happymap: A generalized multicalibration method. In 14th Innovations in Theoretical Computer Science Conference (ITCS 2023), pp. 41-1. Schloss Dagstul--LeibaizzZZentuu für Informatik, 2023.
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distribu- tionally robust optimization. The Annals of Statistics, 19(3):1378-144)) 2021.
Cynthia Dwork, Michael P Kim, Omer Reingold, Guy N Rothblum, and Gal Yona. Outcome in- distinguishability. In Proceedings of the 53rd Annual ACM SIGACT Symposium Theory of Computing, pp. 1095-1108, 2021.
Hadi Fanaee-T and Joao Gama. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, 2(2):113-133 2014.
Dean P Foster and Rakesh Vohra. Calibrated learning and correlated equilibrium. Games and Economic Behavior, 21(1-2):40-55, 1997.
Itzhak Gilboa and David Schmeidler. Maxmin expected utility with non-unique prior. Journal of mathematical economics, 18(2):141-153, 1989.
Ira Globus-Harris Declan Harrison, Michael Kearns, Aaron Roth, and Jessica Sorrell. Multicalibra- tion as boosting for regression. International Conference on Machine Learning, pp. 11459- 11492. PMLR, 2023.
Parikshit Gopalan, Michael P Kim, Mihir A Singhal, and Shengjia Zhao. Low-degree multicalibra- tion. In Conference on Learning Theory, pp. 3193-3234. PMLR, 2022.
Parikshit Gopalan, Lunjia Hu, and Guy N Rothblum. On computationally efficient multi-class cal- ibration. In The Thirty Seventh Annual Conference on Learning Theory, pp. 1983-2026. PMLR, 2024a.
Parikshit Gopalan, Princewill Okoroafor, Prasad Raghavendra, Abhishek Sherry, and Mihir Sing- hal. Omnipredictors for regression and the approximate rank of convex functions. The Thirty Seventh Annual Conference Learning Theory, pp. 2027-2070. PMLR, 2024b.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Chirag Gupta and Aaditya Ramdas. Top-label calibration and coatticlaas-to-binnne reductions. In International Conference on Learning Representations. OpenReview, 2022.
Lars Peter Hansen and Thomas J Sargent. Robust control and model uncertainty. American Eco-
nomic Review, 91(2))60-66, 2001.
Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Cal- ibration for the commutationally-ddenfifit masses. In International Conference Machine Learning, pp. 1939 1948. PMLR, 2018.
Lunjia Hu and Yifan Wu. Predict minimize swap regret for all payoff-bounded tasks. In 65th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2024, Chicago, IL, USA, October 27-30, 2024, pp. 244-263. IEEE, 2024.
Shayan Kiyani, George Pappas, Aaron Roth, and Hamed Hassani. Decision theoretie foundations for conformal prediction: Optimal uncertainty quantification for aiskreverse agents, 2025. URL
https://arxilt
02561

---

Published as a conference paper at ICLR 2026
Bobby Kleinberg, Renato Paes Leme, Jon Schneider, and Yifeng Teng. U-calibration: Forecasting for an unknown agent. In Gergely Neu and Lorenzo Rosasco (eds.), The Thirty Sixth Anmual Conference on Learning Theory, COLT 2023, 12-15 July 2023, Bangalore, India, volume 195 of Proceedings of Machine Learning Research, pp. 5143-5145. PMIR, 2023.
Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shafieezadeh-Abaddeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics, pp. 130-166. Informs, 2019.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp. Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Jiuyao Lu, Aaron Roth, and Mirah Shi. Sample efficient omniprediction and downstream swap regret for non-linear losses, In Nika Haghtalab and Ankur Moitra (eds.). The Thirty Eighth Annual Conference on Learning Theory, 30-4 July 2025, Lyon, France, volume 291 of Proceedings of Machine Learning Research, pp. 5829-38777 PMLR, 2025. URL https::/pprceeeiingg mlr. D.s.ppees//v2///nee html
Charles F Manski. Identification problems and decisions under ambiguity. Journal of Econometrics, 95(2):115442, 2000.
Charles F Manski. Statistical treatment rules for heterogeneous populations. Econometrica, 72(4): 1221-1246, 2004.
Charles F Manski. Choosing treatment policies under ambiguity. Annual Review of Economics, 3: 25 25-49, 1.
Charles F Manski and Aleksey Tetenov. Admissible treatment rules for a risk-averse planner. Econo- metrica, 75(3):715-752, 2007.
Georgy Noarov, Ramya Ramalingam, Aaron Roth, and Stephan Xie. High- High-dimensional prediction for sequential decision making. arXiv preprint 1rX1v:2310.17611 2023.
Princewill Okoroafor, Robert Kleinberg, and Michael P Kim. Near-optimal algorithms for om- niprediction. arXiv preprint arXiv:2501.17205, 2025.
Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33 (3):291-297, 1997.
Benjamin Plaut, Nguyen X Khanh, and Tu Trinh. Probabilities of chat Ilms are miscalibrated but still predict correctness on multiple-choice q&a. arXiv preprint arXiv:2402.13213, 2024.
Aaron Roth. Uncertain: Modern topics in uncertainty estimation. Lecture Notes, 1 11:30-31, 2022.
Aaron Roth and Mirah Shi. Forecasting for swap regret for all downstream agents. In Dirk Berge- mann, Robert Kleinberg, and Daniela Sabán (eds.), Proceedings of the 25th ACM Conference on Economics and Computation, EC 2024, New Haven, CT, USA, July 8-11, 2024, pp. 466-488 ACM, 2024.
Guy N Rothblum and Gal Yona. mecision-making under miscalibration. In 14th Innovations The- oretical Computer Science Conference, ITCS 2023, pp. 92. Schloss Dagstub-Lebbniz-zentruu fur Informatik GmbH, Dagstuhl Publishing, 2023.
Abraham Wald. Statistical decision functions. In Breakthroughs in Statistics: Foundations and Basic Theory, pp. 342-357. Springer, 1950.
Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predic- tions to decisions: A novel approach to multi-class calibration. Advances in Neural Information Processing Systems, 34:22313-22324 2021.
12

---

Published as a conference paper at ICLR 2026
Appendix
A MISSING PROOFS FROM THE MAIN BODY
Proof of Theorem 3.1
Proof. We begin from the robust formulation
max min B[u(a(f(x)), (((()))) a()xx qE@
(6)
where R is compact, u(;;)) is linear in second component, Q is the nonempty, convex, and compact set of measurable maps
1j - [0, 1] satisfying the linear moment equalities in equation 4 and a(-) ranges over measurable policies with values in A. The mapping (a, q) - (xx((((X) (f(())) is convex in q (since u(a, +) is linear, hence convex, in y and expectation preserves convexity), concave in a (as a pointwise maximum over linear functionals in a on the compact set A). Hence, by Sion's minimax theorem,
max min E (ooo(((())),(((())))
min max max=E|aa((((X))))((((X) qeQ a()
a() QEQ
Fix any q Q. value function
The inner maximization policies separates pointwise in e f(x), yielding the
val(p)
A
max uax p)
and
max a(-)
E Elu(a((((X)) =a(v((((((()))
Therefore the robust value equals the convex adversarial problem
min
E' val()(((X)))
gE
which will be analyzed via Lagrangian duality below.
Introduce vector Lagrange multipliers A E Rd for the d-dimensional equalities in equation and let À = (A1...., ) Define
s(v)
A
ha E R4, vE [0, 1ja.
The Lagrangian of equation
Llg. = E8 [[vaa(((((X)))
Σ A ((E(((((()) ((((())) s=1
By linearity of expectation,
L(g.A) A) va(aa((((X)) + o((()) (s(x)) -(x)-
The dual function is obtained by minimizing L(g, A) over measurable q [0, [0, 1J Since the integrand depends on q only through q(f(X)) the infimum can be taken pointwise in the forecast value e fX)::
G(A) = inf L( = inf (val(p) + [x(() - E[((X) -S(x))] pE(0.1ja
The primal problem equation is convex (convex objective, affine constraints) and feasible (e.g., q(v) = v), thereby strong duality holds. Hence,
min E val val((((())) T
max G(A), xE(Ré)E
and there exists a maximizing multiplier A Define
&
(v) 4 Σ
ha(u) X3 Ed..
13

---

Published as a conference paper at ICLR 2026
By the definition of (()) and strong duality, any primal optimizer E Q must minimize the Lagrangian at X*. Since the dependence on q is only through ((f(X)), this yields the pointwise characterization, for e f(x) almost surely,
q" ('()) E arg min (val(p) + p- p.p8p (v)} pe(0,1]e
With q² fixed, define the policy
Grabust (e) E
arg max u(a, q (v)). aEA
Then, by the definition of val and the construction of q",
= Ea = min E[ [[val( qE@
max a(-)
which shows that (drobust; 9²) is saddle point of equation 6 the outer maximization, and q- is worst-case optimal for the inner minimization, with q character- ized pointwise by the minimization problem above and determined by the dual multiplier \". This matches the statement of Theorem 3.1 and completes the proof.
In particular, dsobust is optimal for
LI
Proof of Theorem 4.1
Proof. We use the reduction
max min E [[u(a([(XX))
min min[[val(([[(X))]] gE
a() qE@
established in the proof of Theorem 3.1 Fix the decision regions
Ra { [0, 11": u(a, v) 2 u(d', Va' E EA}.
each convex. Under Haec {1R, a E A}, admissible satisfy
(f
f(x)) Va.
equivalently (whenever (X E R.) > 0),
E(g(s(X)) E R.] F[S(X) R.] =: pa E Ra.
By Jensen's 's inequality (convexity of val), for any E and any a.
E E[val(9(f(X))) f(X) e Rs] 2 val(ua).
Define the piccewise-constant q(v) = Σa Ha 1R, (v). Then q E Q and, conditionally on f(X) e Ra, we have (()())) = Ma a.s., hence the bound is attained:
E B[val(((f(X)))
Σ IP((()) Ra)) val(p.) s =[va((((((X))) Vg e e
Thus worst-case belief is 9* q. region-wise constant with q" (v) 1a Ra-
Finally, since Ma Rg. by definition of Ra we have u(a, 2 u(a', .aa) for all a', so a is best response to Pa- Therefore the robust action at Rg
Grobust (v) arg max ula', (u)) arg max u(a', pa) 3 a,
which coincides (a.c.) with the plug-in best response to v. This proves Theorem
Proof of Theorem 4.2
Recall val(p) maX u(a,p) p) and the reduction
max min ([u(a(((X)) = min ([v)()(((
a() g
vEQu.
14

---

Published as a conference paper at ICLR 2026
established earlier in the proof of Theorem 3.1 Moreover, the identity map qi((v) always lies in Qu (the perfect forecaster is consistent with every H-calibration constraint), so for any policy a(-),
min E[ula(f(x)) s ([ula(((x)) ge@u
Let (BR(1) (v) E arg maXae.4 u(a, v) be plug-in best response | the celliion- tests {IR. R. Jaca:
We show that, assuming H. contains
E[u(ann (X(X),, Write a4a :- EI ((X S(X) E R. whenever P(S(X) Ro) > 0 ((FI((( e Ra) = 0, any
E [[u(ann[ (X))), ((())))(((())
Aq 2H-
(9)
choice of (oa harmless since the corresponding terms vanish). Then
Σ Blu(a.o(f(X)) ((SOOERaa)
Σ P(F(X) E Ra) v(a, Ela(((X)) e Rol)
aEA
Σ P(f(X) Ro) ula, B[f(X) f(X) E Ral)
Σ P(f(X) E Ra) u(a,
[[l(aa,(()) ((sx)eeaa
aEAA
[[ (f(x),
Here: (4) uses that u(a,-) is linear in its second argument,
E[u(a, (((()))) S(X) E Ra] = u(a, (X)( E Ra)). (i) uses the decision-calibration equalities E[1n, ([)(f(X)) f(X) E RR.] = (X) f(X) E Ral = Na whenever (f) E Ra) > 0; and (iti)
- O, equivalently
again uses linearity: u(a, (ua) = u(a, (XJ(X) (S(X) E Ral) = E[v(a, (())) IS(X) E Ra).
Combining equation g the optimality of best response on the perceived outcomes,
(X)( ((x)) s ([x((
for all policies a(-),
and the invariance equation 9 we obtain the minimax dominance
min ([([(aB(((())) (((()))) j
([ f(X))
min Elu(a(f(x)).
for every forecast-based policy a(.). Hence the plug-in best response is minimax optimal under any H that contains the decision-calibee calibration tests, as claimed.
Proof of Proposition 4.4
Assume = < ac and Y| < oo so that all derivatives and expectations below well-defined and we may interchange expectation and differentiation by dominated convergence. Write 2 == (X) E R and f = fo X) = Wz E Rd. The squared-loss risk is
Proof.
(()) = [|||
[ = [[( 2 Y) T (Wz-Y)) -
For the linear head W E R d the gradient with respect to W satisfies the standard identity
V
IW= Y1 (Wz Y) TT Rd
Taking expectation and interchanging V with E yields
V C(0) = E - Y) =
Fix any deterministic tie-breaking so that apn and the regions R. = {u: (pn(r) = a) measurable.
15

---

Published as a conference paper at ICLR 2026
At first-order stationary point (in particular, when the gradient with respect W vanishes) have
B((f-Y) Y)- 27]
0axk-
Transposing gives
B[ (J - [ = Opxd
({=((=-))]
0sxd:
which is the first claimed moment identity.
For the second identity, observe that f = W hence
E[f( (Y - f) = E[Wz (Y - { = WY - = W Oxxd = 0dxd. Therefore both [2[ = x) (Y (V- for each coordinate = 1,....,.444 d. e fo( (o() and El 2x (Y-fo = 0, so So is H-calibrated for 7 = (1)(w) eju: = -I....,d and for any linear combination thereof. This proves the proposition.
= 0 and (Y-
0 hold. In particular,
Proof of Proposition 4.5
Proof. By the reduction established earlier (see the proof of Theorem 3.1..
3.11
the robust problem
min rEe (X))a(((X))).
a(-)
with linear utilities and finite i equivalent to the convex program val(p) max u(a,)), aEA
min I' (al(
gE
subject the Hain calibration constraints
E (((X))))) (a(s(x) - ((X)) 0,
I,...,J
Write Ej == (S(X) E B₁} and assume IP(E,) > (bins with zero probability are immaterial).
Then the constraints equivalent to
[[g(f(X)) E,] = ([f() Ej] =: mys j=
Because u(a,.) is linear in the outcome, val is the pointwise maximum of linear maps and hence convex. Decomposing by bins and applying Jensen's inequality gives, for any feasible 4.
E val (a(I(X)))) Σ E(a)) Eg]
P(B,) vall)) [E)
Σ P(E,) val(m;))
Define the piecewise-constant candidate
7(v)
Σ 1=1
mj 1B, (v).
Then ğ is feasible, since for each j. E [1E,
- ((l((x)- and it attains the Jensen lower bound because ((f())) = m, almost surely on Ey: E (vv)()(ff Ej] val(m,,)
= P(E)) (my E(((X) E2]) = 0,
Therefore q is an optimizer, and any minimizer q" can be chosen (a.c.) piecewise constant with (u) my for e By.
Finally, fixing such a 4", the robust action at forecast v E Bj solves
Grobust (v) E arg max (a, q (u)) = arg max u(a, my);
aEA
atA
which depends only on the bin index, i.c., it is the best response to the bin mean. This proves the claim.
n
16

---

Published as a conference paper at ICLR 2026
B APPROXIMATE H-CALIBRATION: STABILITY UNDER E-SLACK
This appendix extends the main results to the practically relevant regime in which H-calibration holds only approximately. Coneretely, we relax each linear calibration equality in equation 3 to an fz-ball of radius E. Throughout, we retain the standing assumptions of the main text: utilities are linear in the outcome, so there exist (ra Rd, Co IR JaEA with
u(a,p) p) P +. Ca
val(p) = max u(a. p) is convex and L-Lipschitz w.r.t. || ||2. aeA
where L = max ||rl|2. We write expectations over (X, distributed as in the main body, and f: X + [0, 1] denotes the given forecaster.
Approximate calibration constraints. Let H = spanfh)..... ha3 with measurable h; [0, 1j R bounded by [h,(v)] s 1. For candidate conditional expectation [0. 1] - [0, 1] define the (vector) calibration moments
m=(g) == E hx(s())) (a([(x)) - [(X)] E RM,
i= l....,k.
We say q is s-approximately H-calibrated if [|mx(9)]la S & for all i. The corresponding ambiguity set and robust value are
Qe = { 10:10.11- lo, 1] |mm I = 1, 1....,kk kk
Ve == min v/ gEQ.
For reference, the exact-calibration value is Vo = m E[val({(f(X)))] where Q is the cquality-based set from equation 4
Roadmap. We first show a dual penalty bound: moving from exact to a-appoxximate constraints subtracts an explicit f2-norm penalty from the exact dual objective, yielding two-sided value bounds and a linear-in-s degradation guarantee. We then quantify the robustness of decision calibration: even under e-slack, the plug-in best response is (m mem)-mimima optimal (with m==|A]). Finally, for bin-wise (histogram) calibration with e-slack, we obtain piecewise-conster worst-case belief's and tight value bounds, recovering the exact structural picture up to (() terms when there are J bins.
Policy characterization under s-slack. The primal inner problem remains convex and pointwise in v f(x), while the dual acquires the norm penalty from Theorem B. Consequently, the optimal robust policy admits the same form as in the exact case, with the unique change that the dual multiplier solves penalized maximization.
Theorem B.1 (s-robust policy via penalized dual). Let H. = spanth h,r and define G(A) as the main text. Let
X arg max {G(A)
Σ [Arlla]
(0) Σ h,(w) X
AE(R")
Then there exists worst-case belief of 10, - 10, 1jd such that for almost every forecast v = f(x).
(v) E arg min (val()) p ox2(1)) pE(O.1]
val(p) max u(a,p)-
aaA
The s-robust action is the best response to (v):
(v(v) arg o (v)).
aEA
Proof of Theorem B.7 Recall the robust formulation under linear utilities and forecast-based poli- cies reduces to the adversarial convex program
min E ((vx))(([(x))) val(p) max u(a,))
with the E a-appoxximate H--aalibration set
2. = (q: 1 10. - - (()) S8, i 1....,,kk
17

---

Published as a conference paper at ICLR 2026
Introduce slack vectors 8; R (one per test) that each constraint is rewritten as the equality
[((((((X))e((((x)) f(X)} with | S (i k.....k).
Let X E Rd be the Lagrange multipliers for these equalities and set sx == Ei= hz(D)), The Lagrangian reads
L(g, 8; A) =a(vaa Σ X (F1,0U())11((xx))-111 -
Minimizing over the slacks 8; subject to ||s:||2 contributes the support function of the b2-ball,
inf |
(~>a- 8:)
sup (Ar 8;) ||Ae|12-
Minimizing the remaining part over depends on q only through ((())) and yields, pointwise in v = f(X),
inf L(g, = inf pE[0,1]4
{val(p) + p- sx(((x))3 -
Σ 1
Therefore the dual function is
(A = EE min (val(p) + - (x -
-
Aalle
pE p
=:G(A)
ic., the exact-calibration dual G(A) penalized by = E; Aill2
The primal problem is convex (convex objective, affine moment constraints) and feasible (e.g.. q(u) = makes all moments O, which is strictly feasible when > 0), so Slater's 's condition holds; hence strong duality holds and maximizer X of Ge exists:
min E(val(((((())) = g.Q.
(G(A) - 2 [Avlla]
max
XE(RA)E
Moreover, comparing with the exact case (which corresponds to E = 0) gives the two-sided value bound
max {G(A)) Σ [alle s Ve s Vo = max G(A).
and 0 < Vo - Ve S minXgarg max E; 1111l12
By strong duality, any primal minimizer of Qc together with X forms a saddle point: L(q A) S L(g X*) < L(g. x*). The first inequality implies that o minimizes the Lagrangian at X*, which (by the pointwise structure above) yields, for almost every forecast v = f(x).
(u) E arg min (val(p) + +p- + (x) (x. - Σ M.(o)) pE|o.1]
11
With o.' fixed, the optimal robust action at solves
Grobust.e (v) E arg max u(a, 4: (v)). aEA
ie., it is the best response to the worst-case belief qf (v). This is the same best-response structure in the exact case, now using the penalized dual optimizer X (cf. the exact characterization in the main text).
Altogether, have (i) the dual penalty representation with value bounds, (ii) existence of dual maximizer X*, (iii) the pointwise form the worst-case belief qr, and (iv) the robust policy as a pointwise best response to q completing the proof.
18

---

Published as a conference paper at ICLR 2026
Computation. Algorithmically. the recipe mirrors the exact case: (i) maximize the concave objec- tive - A (e.g., projectea/suubgaddenn or bisection in ID; small-scale mirror descent otherwise); (ii) for each forecast compute of (v) by solving the convex problem in p; (iii) play (g(r) as the best response to of (v). For finite A and utilities linear in step (ii) reduces to checking small finite set of candidates (endpoints and pairwise breakpoints of val), exactly in the main text.
Decision tests contained in H. under E-slack: near-optimality of plug-in. Let R. {u u(a,v) 2 u(a', .)) Va' E A} be the plug-in region for action a, and write P. == P(f(X) R.) (regions with Pa = O are ignorable). Assume H. is test class that contains the decision indicators (In, E A}, with each test bounded by [[ bb 1. We impose approxximate H--alibration in the componentwise sense of Section B so in particular
E (f(x) (alf(x)) - f(X) 112 E, for all a E A and all E Qc.
Theorem B.2 (Plug-in is Ommii)-minnna optimal when decision tests lie in H). Let m == |A] and L = max ||rall2 as above. If H contains the decision indicators {1R, } and f is e-appoximatell H--aalibrated, then the plug-ih in rule (BR(V) arg max, u(a, v) satisfies, for any forecast-based policy a(.),
min E ((X)) 2
2 min m(na(((((((()))
m LE.
Proof. Fix any E Qe. Decompose by plug-in regions:
E B(u(aan(():9(()] = Σ If Ro].
aEA
Since u(a,)) is linear,
Eiu(a, ((f)),((f))])))) Ra] u(a, Blg(f) I E Ral)
Let pa = Elf f E Ra]. By L-Lipschitzness u(a, -) and the elskack on the indicator test, ((,,E(((() | Ral) (a, Ha) s 1 11|E(a(0)-1) Ral Therefore,
| n. (a(s) - Alll2
Pa
Pa
E(u(ABR(().9(F)) 2 Ea - E c
(lu(aBR()),F) - m Le.
Pa
Let 9 E arg min (lu((BR()), {())] Then min Elu(ann(f) = Elu(amn()).((()) 2 B[u(ana(f).f)) mLe. 9E2.
For any forecast-based policy a(-), optimality of the plug-in in action on f implies E E(u(agu(f))
Blu(a(f), f)). Moreover, since (v) is feasible for Qe, have minge Q.
E[u(a()); f)). Combining the last three displays yields the claimed inequality.
Remark. The proof uses only the e-slack constraints for the decision indicators {1 }: any H
that contains these tests (with per-test slack bounded by 2) suffices. Thus Theorem B.2 generalizes both Theorem 4.1 and Theorem 4.2
Bin-wise calibration under s-slack: value stability and structure. able partition of [0, 1]s. Assume w-bin-wise calibration:
Let (Bi) be measur-
| (r(x)EB), (a(s(x)) f(x) 1l, Se. j
Write E, := (f(X) E Bj}. Pj = P(E,). and m, = Exf(X) E,] (bins with P, = are ignorable).
19

---

Published as a conference paper at ICLR 2026
Proposition B.3 (Value stability and piecewise-constant worst-case beliefs). Under E- e-bin-wise cal- ibration,
Σ P3 val(m)) j=1
J L v min
((val((((X)))
& Σ P3 val(m;))
qEQ.
j=1
Moreover, there exists worst-case (or arbitrarily near- belief that is piecewise constant: for each one can take
& Pj arg
min 1p-myllaSe//
val(p);
By (a.e.),
and the robust action on Bj rest-responds to pj.
Proof. For any feasible
E [[val(a(f))]
Σ Pj ([va(((((()) [,]]
Σ
Ps val (B((()) E,))
f=1
by Jensen since val is convex. The slack constraint implies
[(u(() Exlll= E[ (a(l) fllla
8 Pj
S
Pj
so, using L L-iihhchitzness of val,
val [E (((()) ES)) 2 val(m;)
s
Pj
Summing over j yields the lower bound [[va(((())) 2 Σ, P3 val(m;) - Le. The upper bound holds because Q C Qr and equality is achieved at = 0 by the exact bin-wise result.
For structure, fix any feasible q. Replacing q by its conditional mean each bin,
((v) Σ (l((f) E,] LB, (v).
does not increase the objective (by Jensen within each bin) and preserves feasibility (the bin-wise moments are unchanged). Hence the minimization reduces choosing, for each bin, a point py [0, 1] subject to l/p, my ||| e/P; to minimize E; Py val(p;), which yields the stated piecewise- constant form with p e arg m my my val(p). The best-response form of the robust action on each bin is immediate from the definition of val.
20


Reviews:
[{"review_id": "Q7ruTxOcG6", "reviewer": "Reviewer_u2QY", "summary": "Summary:\nThe main contribution of this paper is demonstrating how partially calibrated forecasts can be used in decision making. The authors apply a robust optimization perspective: They assume that the forecaster $f$ satisfies a weak notion of calibration called $\\mathcal{H}$-calibration under the (unknown) true distribution. Then, they construct the set of distributions (which can be reduced to a set of candidate conditional expectations) under which the forecaster $f$ satisfies $\\mathcal{H}$-calibration. By assumption, the true distribution must lie in this set. They propose to find the decision rule that maximizes the worst-case utility over the uncertainty set – which provides robust performance guarantees under the true distribution. They provide a characterization of the optimal robust policy under the uncertainty set. They find that when $\\mathcal{H}$-calibration corresponds to notions of decision calibration, the optimal robust policy simply best responds to the forecast.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, I found this paper to be clear and well-written.\nTo the best of my knowledge, this is the first time I have seen the connection between \"partial\" calibration and robust optimization -- I believe this paper proposes a novel question and framework, and also provides a clear exploration of the solution.\nThis is a strong contribution to the calibration literature and a very nice application of robust optimization.", "weaknesses": "I have questions about how the connection to this framework and decision calibration should be interpreted: To the best of my understanding, decision calibration is the “necessary” and “sufficient” condition on probabilistic forecasts required to recover the expected utility (under the true distribution), e.g. Zhao et al, 2021, Sahoo et al, 2021. \nIs it the case if $\\mathcal{H}$ corresponds to a set of test functions for the appropriate notion of decision calibration for our decision task, then every distribution (or conditional expectation) in the ambiguity set $\\mathcal{Q}$ yields the same expected utility (which is the true expected utility)? \nFurthermore, does this explain the “collapse” of the optimal minimax decision rule to the decision rule that treats the forecasts “as-is” when $\\mathcal{H}$ corresponds to a decision calibration notion? \nIf this intuition is correct, it may be worth highlighting for readers in the main text to explain where the optimal minimax policy (for decision calibration classes) comes from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "auD3dApRSl", "reviewer": "Reviewer_ZUcV", "summary": "* The paper studies decision-making with partially calibrated predictors. Particularly, the paper studies $\\mathcal{H}$-calibration, a relaxation of full calibration, which also generalizes several existing calibration notions. \n\n* It shows that best-responding to a decision-calibrated predictor is minimax-optimal, giving a simple, practical rule for decision makers.\n\n* With weaker calibration (moment/bin-wise), it derives the tractable best-response policy.\n\n* Empirics illustrate the robust rule's behavior vs. a naive plug-in policy.", "questions": "I do not have outstanding questions, see my comments in weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* I really like the justification for decision calibration. The exponential sample complexity has been a problem for full calibration. The paper proves that, once satisfied, simple best response remains minimax-optimal. While full calibration is not efficiently achievable, this result is actionable for practice: decision makers need only best-respond; forecasters should focus on the concrete decision calibration objective. \n\n* Bridges theory and empirical: connects common training artifacts (e.g., MSE first-order moments, binning) to usable robustness guarantees.", "weaknesses": "* The paper would benefit from a sensitivity analysis. For example, how does decision calibration loss contribute to the loss of minimax-responding decision makers? I would imagine a linear bound. \n\n* The paper would also benefit from a discussion of the efficiency of decision calibration, given the motivation of the exponential sample complexity for achieving full calibration. \n\n\n* Typos and minor issues:\n  - Line 71: it's -> its\n  - Line 203, citation format should be \\citealp\n  - Line 475, Section number missing\n  - Line 132: a notable exception is Hu & Wu (2024) who do give a weaker notion than full calibration. The notion in the paper is equivalent to full calibration once the error is 0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jhnfCDgpfb", "reviewer": "Reviewer_5jx8", "summary": "The paper studies the calibration of predictive forecasts, inspired by ideas such as multicalibration.\nThey consider the idea of \"$\\mathcal{H}$-calibration\", which states that a forecaster $f : \\mathcal{X} \\to [0,1]^d$ is calibrated if $\\mathbb{E}[h(f(X)) \\cdot (Y - f(X))] = 0$ for all $h \\in \\mathcal{H}$.\nThe authors analyze optimal decision-making under $\\mathcal{H}$-calibration, for varying choices of $\\mathcal{H}$.\nA key result is that, once $\\mathcal{H}$ is \"rich enough\" -- which they characterize by a notion of \"decision calibration\", weaker than the usual full calibration -- the calibration guarantee becomes already strong enough to the point of recovering the optimal decision properties of full calibration.\nOptimal decision rules are also derived for some weaker choices of $\\mathcal{H}$.", "questions": "- Could the authors discuss the applicability of the linearity assumption, and elaborate on its necessity?\n- Could the authors clarify the novelty of the analyses on the practical choices of $\\mathcal{H}$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper tackles the important problem of calibration, and in particular that of achieving tractable yet powerful notions of calibration.\nThe result regarding the \"decision calibration\" choice of $\\mathcal{H}$ is quite interesting, though I'm unsure whether decision calibration is truly achievable in practice.\n\nOverall, I believe the results are a meaningful contribution to the field, from both a theoretical point-of-view (characterization of different $\\mathcal{H}$s and their optimal decision rules) and a practical point-of-view (identification of certain practical notions of calibration and optimal decision rules).", "weaknesses": "My major concern is regarding the assumption that the utility be linear in the outcome $y$ (stated in Remark 2.1; line 210/211). It seems really strong, and I'm slightly unconvinced on how realistic it is.\nCould the authors give examples besides simple expectations of (i) utilities satisfying this, and (ii) utilities *not* satisfying this? In particular, I am interested in whether there are any meaningful utilities other than simple expectations that will satisfy this.\nIt would also be nice to have a brief explanation on why this is necessary in the paper (as far as I can tell, this is used in the proof of Theorem 3.1 so that both Sion's minimax theorem and convexity of the adversarial problem hold; I would appreciate it if the authors could confirm or correct me). \\\n(It's also worth mentioning that perhaps this assumption should be in a `\\begin{assumption} \\end{assumption}` environment, rather than in a remark.)\n\nI am also slightly unsure as to the actual novelty of the practical choices of $\\mathcal{H}$, and in particular the general cases when (i) $\\mathcal{H}$ is a finite dimensional subspace; and (ii) self-orthogonality under squared loss.\nNonetheless, I believe the paper stands on its own even if these indeed have limited novelty.\n(In the case where $h$ has $X$ as the input, I am sure that these are not new, especially the finite dimensional case; but when $h$ has the forecast $f(X)$ as input I am less sure.)\n\nOverall, this leads me to a score of 'borderline accept'. \\\n**Why not higher:** while I'm fairly confident with the papers contribution re. decision calibration, I am less sure of how meaningful its contributions on the weaker forms of calibration actually are. I'm also somewhat concerned with the strength of the linearity assumption on the utility. \\\n**Why not lower:** The paper is sound and has interesting results on a relevant problem. Its conclusions can be of broader impact, both theoretical and practical. This makes me lean towards acceptance.\n\nThese aside, I only have some minor comments:\n\n- Line 044/045: are the observed outcomes $Y$ really in $[0, 1]^d$? I would have expected them to be in something more like $\\{1, \\ldots, d\\}$. As far as I can tell this intended to be like a slightly more general form of \"one-hot encoding\" of outcomes in $\\{1, \\ldots, d\\}$ (which would be more consistent with e.g. $E[Y | f(X) = v] = v$); am I correct?\n- Lines 324-326, \"Decision calibration is a minimal, task-specific threshold at which robust decision making and plug-in best-response coincide, [...]\": is it really clear that there is no $\\mathcal{H}$ smaller than decision calibration that still enables optimality of the plug-in best-response? I ask because of the use of \"minimal\" here.\n- Typos / writing / formatting:\n    - Line 063/064: large language models are generally neural networks, so \"from neural networks to large language models\" does not make much sense\n    - Line 088: \"informally speaking this family\" -> \"informally speaking, this family\"\n    - Line 166: \"for every $\\mathcal{H}$\" -> \"for every $h \\in \\mathcal{H}$\"\n    - Line 210: \"Throughout this Paper\" -> \"Throughout this paper\" (capitalization)\n    - Line 475: broken Section `\\ref`", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)