Risk-controlling Prediction with Distributionally Robust Optimization
Authors: Franck Iutzeler
Adrien Mazoyer
Venue: Accepted by TMLR ()

Abstract: Conformal prediction is a popular paradigm to quantify the uncertainty of a model's output on a new batch of data. Quite differently, distributionally robust optimization aims at training a model that is robust to uncertainties in the distribution of the training data. In this paper, we examine the links between the two approaches. In particular, we show that we can learn conformal prediction intervals by distributionally robust optimization on a well chosen objective. This further entails to train a model and build conformal prediction intervals all at once, using the same data.

Published in Transactions on Machine Learning Research (08/2025)
Risk-controlling Prediction with Distributionally Robust Optimization
Franck Iutzeler
wrenck.iuteeler@@iet htt.://mmwwmtt.iuoooooouuuuueee.
Institut de Mathématiques de Toulouse Université de Toulouse, CNRS, UPS, 31062, Toulouse, France
Adrien Mazoyer
adrien. mariiennaaoyermmaaahh wtgp://rrr.mmtt.....oooooonossss
Instifut de Mathémaliques de Toulouse Université de Toulouse, CNRS, UPS, 31062, Toulouse, France
Reviewed on OpenReview: https: // openreview. net/ pormf id= J9d
Abstract
Conformal prodiction is a popular paradigm to quantify the uncertainty of a model's output on a new batch of data. Quite differently, distributionally robust optimization aims al training a model that is robust uncertainties in the distribution of the training data. In this paper, examine the links between the two approaches. In particular, show that can learn conformal prediction intervals by distributionally robust optimization well chosen objective. This further entails to train model and build conformal prediction intervals all at once, using the same data.
1 Introduction
1.1 Background and contribution
During the last decade, the use of conformal inference to quantify reliability of black-box method and complex machine learning models, has been widely studied (see e.g., (Vovk et al., 2005; Angelopoulos and Bates, 2023; Fontana et al., 2023)). In particular, Split Conformal Inference (SCT) as introduced by Papadopoulos et al. (2002) is procedure that aims to estimate the reliability of pre-trained models by observing its performance on calibration data. More formally, for some imput input/outpa /output spaces x y, the aim is to construct a suitable function C:: + P(V), with output in the set of parts of y denoted by P(V), from the model f : - y and n calibration points (X,, Y in X V. In SCI, the objective is that for a new point (Xn+1, Ynt1) and miscoverage level a set by the user, the following holds
P[Y++1 E C(X 21-0.
(1)
where the above probability is taken with respect to the joint distribution of (X,, YD of SCI is that requires only exchangeability assumptions for the joint distribution of (X, Y2)+! references above for more details), and still gives finite sample guarantees.
n+1
The main advantage (sce the
We considering here related paradigm to SCI, referred to risk controlling prediction by Bates et al. (2021). The controlled risk corresponds to R(T)) = E[L(Y Ta(X where Tx X + P()) is a parameterized sel-valued predictor and L: 3/ x P()") - R_ is a loss that evaluates the quality of Tx. The objective here is to ensure that, given some 7n depending on the calibration data, the risk R(T, ) == E[ Ts., (Xx+1)) Yy), remains with high probability below some target level. More formally we aim to guarantee for levels a and chosen by the user that.
P P(T ) 2 15
(2)

---

Published in Transactions on Machine Learning Research (08/2025)
where the above probability is over the joint distribution of (X,, Y) The only requirement to get that guarantee is assuming the calibration points (X, Yi)L=ı to be independent and identically distributed (i.id.) and having access to a concentration inequality in order to bound (Tx) by some a with probability 1 - 8.
We develop in this paper the idea that an interesting alternative to concentration inequalities in this context can be obtain by the formalism distributionally robust optimization (DRO). Indeed, the principle of dis- tributional robustness to design an uncertainty neighborhood of distributions ((Pn) around the empirical distribution of the calibration data P, n 1i=10(x,,x) Then, if the distribution of (Xx+1, Yn+1) belongs to (P,)) with probability 1 - 6, then (2) holds with a = a(xx) where
a(\) = sup E (xxx)=(((XX,T(xx) QEU((P.)
(3)
is a distributionally robust version of the conformal loss. Conversely, from the user-chosen risk level a, choosing Ån so that a(Ax) S leads to the sought guarantee (2). The design effort for the upper-bound is thus differed to the construction of (P.).
Conformal methods mainly rely on exchangeability assumptions on (X,, Y.) which may be violated, typically when the calibration points are i.idd. according to a distribution different from the test point. Interestingly, the distributional robustness approach presented above does not rely on such assumptions but only the probability of the distribution of (Xn+1, Yn+1) belonging to ((P.).
In this paper, we consider the paradigm of Risk-Controlee Controlling Prediction Sets (RCPS), i.e., (2), where the upper-bound on the risk is obtained by Wasserstein distributionally robust optimization (WDRO), i.e., (3) where (,), a ball of radius p around Pn the Wasserstein distance. We demonstrate that the procedure exposed by Bates et al. (2021) remains valid provided that p is adoquately chosen and can additionally suffer shift compared to the training distribution in terms of Wasserstein distance.
In Section 2, we recall the RCPS methodology and connect it to distributional robustness. Then, in Section 3, we leverage tools from Wasserstein distributionally robust optimization to show the following result (stated below informally):
Theorem 1 (informal). Let YD ~ pea and (Xn+1, Yn+1) ~ P.. For any > O and any ô E (0, 1), there crists pln, 8) and dn (chosen that a(dn) S where al-) is defined in (3)) so that for any p 2 p(n, 6)
P ( a 2 21-8
whenever P-
This enables theoretically the use of WDRO to produce upper-bounds in RCPS. Interestingly, this method is pertinent for any value of a (see Section 3.2.3) and compatible with a joint training of the prediction and conformal prediction on the same datasct (sce Section 3.5). Finally, we illustrate its encouraging performance in practice in Section 4.
1.2
Related works
The link between classical SCT and RCPS can be made through the work of Gupta et al. (2022) who reformulated Conformal Inference (CI) in terms of nested prediction sets, as well as Conformal Risk Control (CRC) introduced by Angelopoulos et al. (2024) where the guarantee (2) is expressed in expectation, that is (TR) I a. Note that guarantee (1) is then retrieved by taking R(T,,) ) Lyx+igTs, (Xn+s): where the expression of XS depends on the calibration data (see Bates et al. (2021, App. A) for more details).
Distribution shift in CI has been recently widely studied for different type of shifting. Tibshirani et al. (2019) studied the covariate shift case (i.e., X distribution differs betwen calibration and test points, but not the Y|X distribution) by introducing the notion of weighted exchangeability. This approach has been gencralized for the CRC context by Angelopoulos et al. (2024, Sec. 4.1). Podkopaev and Ramdas (2021) used similar arguments in order to tackle the label shift case (i.e., Y distribution differs but not the X|Y distribution). Barber et al. (2023) studied the consequences of such distribution shifts, by quantifying the
2

---

Published in Transactions on Machine Learning Research (08/2025)
coverage gap in terms of total variation distance. The online time series case has also been considered in the works of Gibbs and Candès (2021; 2024). Their former work has been considered in the CRC context by Feldman et al. (2023). Cauchois et al. (2024) studied shift in terms of f-divergence, which closest context of our, and less restrictive than the other case mentioned here. It is also worth to notice that our results are also close but different from the conformal prediction with adversarial shift presented by Bates al. (2021, Sec. 6.3) as possible extension, since we consider shifts on the data distribution rather than pointwise shift in the training set. That adversarial context has actually been studied for SCI by Gendler al. (2022) followed by Yan et al. (2024).
Very recently, Aolaritei et al. (2025) proposed to address the problem of distribution shifts in conformal prediction by considering ambiguity sets of distributions (similar to what we denote by U(Pw)) based on the Lévy-Pree ProkhProkoooov pseudo-metin (where we use the Wasserstein distance). Their results are thus comple- mentary to ours.
2
Risk-controlled prediction sets and Distributional Robustness
In this soction, we focus on the RCPS paradigm. In the third part of the section, we connect this with the notion of distributional robustness.
2.1
Setup
We suppose have access a sample (X,, Y of i.id. couples of random variables with probability distribution Po on X' x y
We seek to provide conformal prediction sets around predictor f f X + 5 (the predictor output space y is usually equal to y but in some contexts such as classification, it can be equal to the simplex over )). For the moment, we will consider that the predictor is fixed (e.g., learned by an external procedure), we will get back to the predictor training in Section 3.5.
We recall here the notion of nested prediction sets as described by Bates et al. (2021). We consider class of parametric sets of the form
T: A X (A,z)
P()) (xx)
where P())) denotes the parts of set y' and A is a closed subset of R. We assume that these predictions sets are nested i.e., that
A
Ag] [Tx (x) C (x) for all X].
(4)
Typically, these sets are constructed from predictor outputs, as detailed in the examples below. Finally, we define the loss for the conformal prediction as
L x()) + R+ (y,
+ L(y.S)
which we will often use in conjunction of the parametric sets above by considering L(y, Tx(x).. Moreover, for practical convenience and in line with applications of interest, will choose it monotone in its second argument i.e.
CS (). 2 S'y for all y].
(5)
The goal in conformal prediction is to use the available samples to build prediction sets around predictor output such that the average conformal prediction loss is the low in expectation over the unknown distribution from which the samples are generated. To do 80, we rely on the following assumption.
The authors comment on the links between the Pro--Poohy pseudo-metrie and the type-oc Wasserstein distance, which very different the 2 Wasserstein distance we employ both terms statistical guarantees and computational issues.
3

---

Published in Transactions on Machine Learning Research (08/2025)
Assumption 2. The following properties hold:
i) the parametric sets (Ta satisfy the nesting property (4)
i) the conformal loss L is non-negalive, upper-semiccnlinuones and sabisfies the monolony properly (5)
tii the pointwise limit of the function (z, v)) - Ly, Tx(=)) as À + sup(A) is the wull function
in) the function (x, v) + Mly, Tx())) is upper-bounded uniformly in X.
Assumption 2-1) is the classical nesting property of conformal prediction. Item ii) is also standard and fits the vast majority of the target usecases (the upper-semicontimiiy assumption can be relaxed for most results of the paper). Note also that non-monotone losses could be also considered, using procedures described by Angelopoulos et al. (2024)². As for item iii), it is slightly different from c.g., Bates et al. (2021) which assumes the existence of some Amax such that E(x,y),, (E)7)~,_L((YT,,) [L (X)) 0 (as well as all greater À by monotony of L); we rely on this modified assumption mainly to avoid complications about distributions supports in future developments. Notice that item iv) actually not mandatory for RCPS: it can be relaxed into assuming the coefficient of variation of the empirical risk (6) is bounded, in order to consider unbounded losses (see Bates et al. (2021, Sec. 3.2)). However, it will be required when introducing the connection with Wasserstein distributionally robust optimization in Section 3. Morcover, without any loss of generality, we will assume that losses are bounded by 1.
Example 2.1 (Regression). Take x= Rd and y = = R. Typical predictors include (pretrained) linear models (fo(x) = (0;z)), neural networks, etc. A simple class of parametric sets fitting our framework is Tx(x) = {y E y : ly - fo())] A} (in this setting, 5 = ==--R) R). Doing so, the conformal loss can be defined from the 0/1 loss as Lyy, Tx(=)) (T) = 1 foo=fo(=)>>>--
Example 2.2 (Multi-class Classification). Take X = Rdd and y (1,2, Usual prediclors include (pretrained) once again linear models, neural networks, etc. However, in this setup, in order to get meaningful prediction sets, it is usual to work with predictors oulputting the probability of each class ruther than single class, so Y is the simpler over c alternatives (This is not really restrictive since this is exactly the output of a linear predictor or a neural net after the cross-entropy but before thresholding). possible class of porametrie sets is (xx) ty E : fo(x)(y) 2 1 }} (most probable classes predicted, in the sense of probability greater than the threshold), associated to loss of the form L(y, Tx(x)) [TyTTA()) fo()/))<11--
2.2 Risk-controlled prediction sets
The objective of risk-controlled prediction as described by Bates et al. (2021) is find some that gives a low loss for new point (X,Y) distributed according to some distribution P.. More precisely, we want to ensure that
R(A) = E(XY)),,, TX())
is small. Note that the expectation is taken over the distribution of the new sample, P., which is a priori but does depend
unknown. Importantly, TA (and thus (z, + L(y, Tx())) does not depend on on which is a fixed parameter. However, what follows, the choice of À may depend on (X,, Y:) which case it will be denoted by Ån to avoid confusion.
As a proxy, the empirical risk obtained from the data an entry point for computations:
(2) = (xx,))_,,114XX,,10000
1 n Σ LOY,
(6)
where P.. =
T Er denotes the empirical distribution of the calibration data.
Whenever the samples (X,, y are i...d and, even more importantly Po = P., the following rationale has been proposed by Bates et al. (2021):
2 At. cost though: this might lead larger prediction set the gap with monotonicity of the loss increases.
4

---

Published in Transactions on Machine Learning Research (08/2025)
Step A Given confidence 1 - E (0, 1), consider (random) function R2 (A) verifying
A, P(x,)))_,-' R()) - R2t (2) 21-6;
(7)
this is lypically obtained by concentration inequality (e.g., Hoeflding).
Step B Given level (0, 1), find random value Ån (depending (X,, Y.) such that
A 22i== (A) sa
(8)
by using a 1D search.
Putting it all together, we obtain that
(((m)) S a 2 8
(9)
which, in words, means that have low true risk with high probability (this last probability reflecting the fact that dn depends on (X,, Yi and thus random).
Example 2.3 (Regression (cont.)). With the notation from Ezample 2.1 and relying on the fact that Pu
P. = P, Equation (9) guaranlees that
PixY)) ( fo(x) < <in] a <
P P(X,
where we recover our double guarantee on the conformal interval: with respect to the randomness of the new point and of the data sample.
2.3 Distributionally Robust Optimisation
The empirical risk (6) can lead to over-confident decisions (lower risk for training samples than in test) and be sensitive to distribution shifts between training and application (see e.g., (Esfahani and Kuhn, 2018)). In context, this means that the empirical risk may not be valid upper bound of the true risk in the sense of (7) (hence the need of an auxiliary function R (A)). To overcome these drawbacks, gaining momentum in machine lcarning is distributionally robust optimization, which consists in minimizing the worst expectation of the loss when the distribution lives in a neighborhood of Pn.
approach
In our context, distributionally robust optimization amount to solving the problem
sup QEU,(P.)
Bxx,r>-Q14(XXTX
(10)
where the supremum taken over all probability distributions on = X' which are close to Pn i.e.,
UP,).) = (Q P(E) dist(P,,,, Q) <
where p > controls the required level of robustness around P, and dist is denotes any kind of metric on probability distributions.
Popular choices distribution neighborhoods are based on the Kullback-Leibler (KL) divergence (Laguel et al., 2020; Levy et al., 2020), kernel tools (Zhu et al., 2021a; Staib and Jegelka, 2019; Zhu al., 2021b), moments (Delage and Ye, 2010; Goh and Sim, 2010), or the Wasserstein distance (Shafieezadeh Abadeh et al., 2015; Esfahani and Kuhn, 2018).
An important motivation for distributionally robust models is that if the true distribution belongs to the robustness neighborhood, the method directly benefits from generalization guarantees. This is why a careful choice of the neighborhood is necessary. Indeed, if if E (P,) then we directly get that
R Exxy)_ (XX (xX))
sup LLY, (xx)) QEM,.(P))
(11)

---

Published in Transactions on Machine Learning Research (08/2025)
This attractive property is also highly connected with the choice of the metric on probability distributions. For instance, using the KT. divergence (i.e., taking (p(P.) {Q E P(=) (T, Q) S pt) was studied in details in the literature as a natural choice leading to tractable formulations, see e.g., Ben- Tal et al. (2013); Hu and Hong (2013); Namkoong and Duchi (2016) and Wang ct al. (2023, Rem. 5). However, it imposes that the support of P. should comprised in the one of Pn if one wants have P. E (p))p)) since the KL divergence between two distribution infinite as soon as the supports are disjoint. Thus, this approach amounts reweighting the samples with higher conformal loss so that the newly obtained empirical conformal loss is higher and can act as an upper- bound. While this approach does not grants us the same guarantees, note that the idea of reweighting samples to account for distributional shifts provides a nice connection with the notion of weighted conformal prediction for covariate shifts, see Tibshirani et al. (2019) and Cauchois et al. (2024) in light of Hu and Hong (2013, Sec. 2).
In this paper, we focus on the ambiguity sets relying the Wasserstein distance, which naturally metrizes the convergence of measures," which is a sought feature in this context. This is the topic of the next section.
3 Risk-controlled prediction sets using Wasserstein Distributionally Robust Optimization
In this section, we propose to use a Wasserstein distributionally robust version of the empirical risk as our high probability upper-bound on the true risk in the form of (7). We investigate the tractability and the statistical guarantees that can be offered by WDRO RCPS.
We recall that:
we denote by P. the unknown true distribution of (X,Y), so that the true risk R()) Exx,y>>P,IL(, (XX))
we have access to n samples (X,, YD iid. with probability distribution Po (different from P. in general)
we denote by Pn = +. (x,,x) the empirical data distribution, so that the empirical risk is (A) E(XY)~ [L(Y, (x)] L(Y, (X..))
3.1 Wasserstein distributionally robust optimization
Wasserstein distributionally robust optimization corresponds to Problem (10) with distribution neighbor- hoods formulated in terms of optimal transport distance i.e.,
U,(PP) = E : Q) s s)
where W2(ss)) denotes the (type-2) Wasserstein distance defined as
1/2
W W2(Q, Q') (rEP(Ex3).m=-.rg
inf Ble,sr [IkE -(I1)
where P(E x E) is the set of probability distributions in the product space E x E, and (resp. 22) denotes the first (resp. second) marginal of x.
The Wasserstein distance is a natural metric to compare discrete and absolutely contimnous probabil- ity distributions and its use for distributional robustness has attracted a lot of attention; see e.g., (Shafieezadeh Abadeh et al., 2015; Sinha et al., 2018; Shafieezadeh- Abadeh et al., 2019; Li et al., 2020; Kwon et al., 2020) and the review articles (Blanchet et al., 2021: Kuhn al., 2024).
While this motivation is valid and provides good intuition, we note that having P. Up(Pa) may be slightly too demanding compared verifying directly (11). We will get back to this point more details in the next sections. We focus here on the type-2 distance for simplicity. Actually, the transport cost could be casily modified in several parts of the paper accommodate for instance discrete sets classificttion problems. The results still hold without modifications, except for Lemma 4 which has be modified accordingly (see Gao and Kleywegt, (2023, Rem. 2)).

---

Published in Transactions on Machine Learning Research (08/2025)
3.2 Construction of WDRO risk-controlled prediction sets
Now, let construct risk-controlled prediction sets using Wasserstein distributionally robust optimizaliin problems using the rationale of Section 2.2.
3.2.1
Preliminaries
In this section, we will use as an upper-bound to the true risk the Wasserstein distributionally robust empirical risk defined
RM(()
sup (P..
E [xx,)--114X,Tx00))
(12)
Now, in order for this proxy to have desired properties such as finiteness, we have to introduce adequate assumptions.
Assumption 3. Assume that Rr(A) S too for all X E A and that either of the following properties hold:
a) the set X x y is bounded and the function (I. + L(y. Tx(x)) is upper-bounded uniformly in X b) the set X x y is unbounded and there crists finite constants M, M' > 0, independent of X, such that Lyy, Tx(x)) S M + (2 + +1yy|(==) for some E (0,2)
These assumptions are rather mild as the conformal losses are usually bounded (especially in light of As- sumption 2-iii)). This implies that R2, is finite by classical results of the WDRO literature (e.g., Gao and Kleywegt (2023)).
Lemma 4. Let Assumption 3 hold. Then, for all X E A, and all p 2 0, (A) < toc.
Proof. This directly follows from Gao and Kleywegt (2023, Th. 1) as Assumption 3 guarantees that (x, 8) H L(y, [x))) E L' (P.)) and that the growth rate (sce Gao and Kleywegt (2023, Def. 4)) is finite.
3.2.2
Step A
In order to build upper-bounde of the form of (7), one has to guarantee that R((A) is a valid upper-bound in our goal provided that the robustness radius is chosen adequately. Since we consider a Wasserstein distributionally robust upper bound, we can directly rely on measure concentration results in W2 distance (e.g,, Fournier and Guillin (2015)). This naturally provides candidate radius that has the merit to be valid uniformly in À (and in the base model) as it gives a probability for Po to be included in the constraint set in the definition of (A) in (12).
Lemma Suppose that EGpPO [cxp(>/][][9) < oc for some B > 2,y > 0. For any E (0, 1), there exists p(n, ()n,8)=0(0 -((n log((-11) maxi4.05 so that for any p 2 p(n, we have
P ~P YAEA, R < R((A)
2 21-8
(13)
whenever WP, S - p(n, 8). In particular, if = Pa, Eq. (13) holds as soon as 2 p(n, 8).
Proof. From Fournier and Guillin (2015, Th. 2), case (1), we have that if Es,r Eewro Ee [exp(>|[[11)] for some 3 > 2, 0, then for any x > 0
if < 4
and < 1
exp( exp(-cnz// log + 1/v2)3) exp exp(-cnz411 exp( exp(-cn2
if d 4 if i otherwise
P (P,, Po) < <x/21-C
(14)
7

---

Published in Transactions on Machine Learning Research (08/2025)
where constants and C' only depend d, B, y and Eg.r. Fix E (0, 1). Then, with
log(C/8)
if < 4
and n 2 lop(C/6)
(og(C/6)
4/d
pnn, 8)
if d
CR
log(C/6) Cn
4/3
otherwise
we have P(X.,)))_ PP) pnn, 6)] 2 In turn, if (P P.) S p(n,, 0), then with probability 1 - 8, W2(P., S and thus
A R(A) = E (4. &
sup Q:W,(Q.P.)SP
E (X, = (2)(1))
which is the claimed result.
This result. validates the use of WDRO for Step of the production of risk-controlled prediction sets. One disadvantage of the approach is that the choice of the uncertainty radius is not easy to choose as depends non-trivially on the problem's 's parameters.
Remark 6 (Conformal guarantees on the WDRO problem). By flipping the problem, the above resull means that for any choice of radius p > 0, the Wasserstein distributionally robust conformal prediction problem (12) provides natural guarantee of the form of Eq. (13) with probability 1 where C cxp(-cn p") with p [1, (n,x(d,8)/4 depending on the value of d and p (see Eq. (14)).
3.2.3 Step
Now, we want to find some dn that (8) is satisfied with our WDRO upper-bound. In words, have to make the conformal prediction set sufficiently large the robust conformal loss is arbitrarily small. We first show existence of such a An
Lemma 7 (Existence). Let Assumption hold. Then, for any a E (0,1) and p 0 there crists X satisfying
AYAΛ, 2 R S a.
Proof. Fix a E (0,1). Using Assumption 2-iii), there exists some À such that for all X 2 A, L(y, Tx(x)) S for all (I,y)= E X y.
Now, we have that for any p > O, and for all A> X,
R()(A)= =
sup QEP(E),W
E
LLY, LXY,
Sa
where we simply used that the supremum is taken over probability distributions.
The result is thus almost direct from Assumption 2-iii) nevertheless it tells us that under this assumption (which holds in most cases of interest), classical Hoeffding or Bernstein bounds (see the upcoming Fig. in Section 4.1).
find suitable Xn for any a, which is not the case with the
The above result means that shall rely data-driven values both for tractability and performance. In order to desirable property is monotonicity in X, as it enables the use of dichotomy search.
find data-independent value for Ån to get (8). Nevertheless, in practice,
Lemma 8 (Monotonicity). Let Assumption 2 hold. Then, for any p 0, Rg)()) is monotonically decrensing.

---

Published in Transactions on Machine Learning Research (08/2025)
Proof. Fix P 0. Using Assumption 2-1) and ti), we have that if A 2 X2, Th, (2)) N
Mv, Tx2(x)))) for all v) E X x y. Thus,
(Xa)
sup (EP B (x,
E Exxyy~ L(Y, Ty, (x)) = (11)
sup QEPIE)W((P..Q0Sp
hence the monotonocity.
3.2.4 Conclusion
We are now in position to state our main result on the construction of risk-controlled prodiction sets using WDRO. It states that if the radius p is well chosen, there exists Ån that gives the sought guarantees and more precisely that any that satisfies R2, (Xn) S a will do thanks to the monotonicity.
Theorem 9. Let Assumptions 2 and 3 hold and suppose that Eg E [exp(r][(11)) < co for some 3 > 2, > 0. For any a E 10, and any 5 E (0,1), there erists p(n, 5) = p p(n, 8), choosing on so that Rg (In) S a, we have
mastr.r) ) s0 that for ony
P P(X,,)) sn (R(An) sa 2 5
whenever Sp 8). In particular, if PPo the result holds as soon 2 p(n,
The result directly follows from Lemmas 5, and 8. Although quite natural once the adequate results of the WDRO literature and assumptions are put together, highlights that the the connection between WDRO and RCPS (or possibly other conformal methods) can be of interest. After discussing further the reach and limitations of this result, the remaining of the paper will thus be devoted to the computation and numerical evaluation of this methodology.
3.3 Discussion
The main parameter to tune in WDRO is the robustness radius p (one could also mention the transport cost), the same holds for the approach developed here. One point to notice is that once is chosen, the statistical guarantec of Eq. (13) is uniform in X. The sole condition on it in Theorem 9 is that it should be greater than some prescribed radius p(n, 5). Indeed, the excess p p(n, 5) can be interpreted as the amount of distribution shift. that can be handled by this approach. This point important as it means that WDRO risk-controlled prediction sets are inherently robust to some amount of distribution shift.
Nevertheless, a drawback is that p(n, 8) is not explicitly computable. In fact, the tuning of p is still an active research direction in the WDRO community. A common choice is x 1/vn, reflecting the fact that the radius should be larger in a data-poor environment. However, the amount of distribution shift between the training distribution and the true one is also difficult to estimate in general so the drawback is not really specific to the developed approach.
Another limitation that the prescribed radius p(n, in Theorem 9 (coming from Lemma 5) suffers from the curse of the dimensionality. Indeed, p(n, ō) decreases in ((1/n-/d) which can be very slow when d is large. This is due to the fact that concentrate here the full probability distribution instead of looking directly at the WDRO problem. Indeed, for WDRO problems, scaling in O(1//n) appears to be asymptotically optimal (Blanchet et al., 2022; Blanchet and Shapiro, 2023). Recent works have managed to demonst rate this formally for a wide class of WDRO objectives (see (Azizian et al., 2023b; Le and Malick, 2024)): nevertheless, the required assumptions on the loss make the results inapplicable in our setting of interest. In order to focus on the main points of the paper, we leave this question open.
3.4 Dual problem and Numerical Tractability
As problem (12) is an infinite dimensional problem over a space of measures, duality has been a central tool in both the theoretical analyses and computational schemes of WDRO from the onset. (Shafieezadeh Abadeh
9

---

Published in Transactions on Machine Learning Research (08/2025)
al., 2015; Esfahani and Kuhn, 2018). Indeed, the dual problem (Gao and Kleywegt, 2023; Sinha et al., 2018; Blanchet and Murthy, 2019) can be written as one-dimensional problem involving subproblems:
(A) = inf x0² + BxxYYxa 720
sup (K
(T(x)) - > 1'11 >11 - -1/193]
(15)
Though its expression appears simpler, the presence of a sup makes (15) involved to solve in general. Nev- ertheless, since both the optimization in and in the supremum are finite-dimensional, it open the door to easier practical implermentaton and carries important information about the solution of the primal problem.
3.4.1 Strong duality and worst-case distributions
The following theorem guarantees that we have strong duality i.c., that the optimal value of the dual problem corresponds to (A) which is the workhorse of our approach.
Theorem 10 (Strong duality). Under Assumptions 2 and 3, for all A E A,
(2) = (A)
and are finite. Purthermore, the worst case distribution for the WDRO risk attained and
Q
argsup
LLY Tx())
QEP(E):W, (P..Q))
is supported on n + 1 aloms and Q* = Ei= Ogg + +1o deg where
2tt ||,, 5/113 }. i= 1=
& E sug ((,TTx('')) 11 ||x x' 1111 (1'')))E
2
with >** the optimal dual variable in (15), in E n., Po E [0, 1],
and <tg E argsup sup(('sss/) L(y', Tx(')) - _ | - x²12 //4 |Y y/ 113
Proof. As for Lemma 4, the strong duality follows from Gao and Kleywegt (2023, Th. 1). Furthermore, we can invoke Yue et al. (2022, Th. 4) to get the existence of worst case distribution in Rg(A) and Gao and Kleywegt (2023, Cor. 2) to obtain the form of the worst case distribution.
This result gives an intuition what the worst distribution shift looks like. In words, the worst case distribution is shift of the empirical one towards the regions where the conformal loss is large.
3.4.2 A special case
To give a simple example where (15) leads to explicit solution, consider the setup of Example 2.1 where Txxx) = ty E J' ly - fe()) < A} and L(y, (x))) LugT,(") [=))2( In addition, let us suppose that only the Y are moved in the worst case distribution (this can be seen for instance as the transport cost for X being multiplied by a large constant).
The dual problem is then
1
(A(A) = inf yp
+
sup {1 [1 1000-/a(x0)2 --00000A1--111-- > - LEER
(16)
720
7
1=1
and notice that for each i:
IY, fo(X)) 2 X (the data point not in the confidence region), then
SUP " 1 - >/18 2/1²2 } 1, attained by Y;:
10

---

Published in Transactions on Machine Learning Research (08/2025)
fo(X,)) - (x() - X+
fx()))
fo(X)) +1
fo(X;) + V
Figure 1: Illustration of the value of the supremaa in (16) depending on Y, If Y; is in the plain region around fo(X;) (Case 2.a), it is sufficiently well classified (depending À but also y and thus p show after) so it is too costly to displace it, the optimal y' Y; and the supremum is worth O. If Y, in the dashed region (Case 2.b), it is well classified but can be pushed outside the confidence region, the optimal y is then (X()) + and the supremum is in (0, 1). Finally, if Y is in the dotted region (Case 1), no displacement is necessary to be out of the confidence region and thus the optimal y is Y, and the supremum is worth 1.
2. otherwise (the data point in the confidence region), in order to push y' outside of the prediction interval, it has to be distance at least À - | fx()) > 0 of Y, and thus:
a) if |X fo(X))) A- v 11 (1-1111 |Y; (x.))) > 0, the displaced point leads to positive value for the term in braces. Thus, is optimal to displace the point and the supremum is equal to 1 (A - | fo (x())))) 0;
b) otherwise, I - (X.) S X - गन + - -(A Y (xx))) S O, the displaced point leads to negative value for the term braces. Thus, it is optimal to take y' Y, the supremum is then equal to 0;
This reasoning is illustrated by Fig. 1; note that are actually exhibiting the worst-case distribution. Without loss of generality, we reorder the samples by decreasing error so that
fo 2 1Y2 - fo(Xx)) 2 2 IY, fx()))l
and let in = maxfi: IY fx()) 2 }} and d² = max(i: : IY, (x)))
} } (so that indices in [1, io]
correspond to Case 1, indices in (šo + 1, 2 correspond to Case 2.a, indices in [+] + nj correspond to Case 2.b) to have
1
(A)()) inf yp²
11 |8, 11- ((X)I
Σ 0
720
im
1=1+1
in
= inf 7p²
+
+
1- 11 - - fo(x))
(17)
720
n 72 i1
We are left with finding the optimal value of the dual variable 7. In fact, by looking at the definition of 1 and the expression of (17), it is direct that y can be sought as 1/(A - |Y, - fxx,)) for i in [ + 1, n]. By computing the value of (17) for these n - i0 1 points and denoting by i, the arg min, we obtain an optimal dual variable >- 1/(A- - |Yr. - fX( 21² and we have
1²
(A- I8 1o(x,)) i=1++1 () Y., - fo(,,,)))
RC ( (A(
(18)
(1 Y. (X
and the worst case distribution is exactly the training data except for the indices between io + and i, (included), corresponding some points that used to fall into the conformal prediction set and that are push at the boundary. The formula in (18) seems difficult to interpret in general but we nevertheless see some interesting similarities in its formulation with the Waudby-Smith Ramdas bound (Waudby-Snin -mith and Ramdas, 2024; Bates et al., 2021) despite the completely different approaches their construction.
Remark 11. In order include variations of X while keeping a closed form, it would be possible to consider a linear model fx(X)= (0. X) and apply the same reasoning as above. Nevertheless, the derivations become mpidly cumbersome. and the result is not much more informative.
11

---

Published in Transactions on Machine Learning Research (08/2025)
3.4.3 Regularization
Providing explicit expressions for the the solution of (15) as in the previous section is out of reach for many situations of interest, a common limitation of WDRO. Nevertheless, entropic regularization have of WDRO have been proposed and studied recently, offering promising performances (Azizian et al., 2023a;b; Wang et al., 2023). These approaches can be directly used in our context; by adding entropy regularization term to the transportation problem, the dual problem (15) becomes
(x(')) - - X1I2 7 1
D (A) = inf 7p² + +(,,),,, [og_(x.y)~0000.) exp 720
8
(19)
where 0 is the regularization strength, and Q((X, is user-defined conditional distribution. Typical choices are == p and Q((X,)) Y),
The regularized dual (19) enables to find an approximate solution of (15) by relying on gradient- methods (since the supremum in (15) is replaced by a differentiable function). Nevertheless, one still has to sample the points (X',"" in order to numerically approximate the inner expectation. Compared to the special case of the previous section or to classical conformal bounds, the numerical cost of solving this problem clearly much higher (due to the sampling and optimization in 7) but when jointly training and performing conformal prediction training methods. We refer to the papers above for more details and to Vincent et al. (2024) for practical implementation.
model, the computation times
par with classical adversarial
3.5 Simultaneous learning and conformal prediction
A crucial point in our analysis is that the statistical guarantees for WDRO are uniform in the loss functions; in Lemma 5, the result is uniform in x. This means that if the prediction depends on some parameter 6, we can learn 0 while constructing our WDRO conformal prediction sets, using the same dataset. This feature is seldom found in the conformal prediction literature where the model and conformal prediction set usually have to be learned using separate datasets (a recent exception (Braun et al., 2025)).
To make this more precise, consider the setup of Example 2.1 where Tx(x) Kye E y: ly fo(x) < A} and L(y, (x)))) (TT((=) \ Then, we can use
sup ()) (EP
E (xr)-ol-u-s(o)e
Then, choosing Ån. On so that Rg 0m)) S a, we have the same guarantees as Theorem 9. Thus, we can actually optimize the model jointly with the risk-controlled prediction set estimation, on the same dataset. Note here that the obtained model would be different from the one obtained by empirical risk minimization of even WDRO as it is learned to optimize the conformal loss. Numerically, a good solution is to rely on the regularized formulation mentioned in Section 3.4.3.
4 Numerical illustrations
In this section, we illustrate the results of the paper and the applicability of WDRO-based Risk Controlling Prediction Sets. Our goal thus to show how the WDRO-based bounds behave compared to classical ap- proaches for RCPS and how they can encompass distribution shifts and simultaneous training and conformal prediction, rather than performing a complete performance evaluation. The code used for these experiments is available at https::/ gi thub. ctw. iutzelee/rrrssswrrro
We place ourselves in the regression setting of Example 2.1 and compare the following upper- souur- functions R (A) aimed at verifying (7) in Step A:
Hoeffding: (A) = Rr, + V
1 log (1) (see (Bates et al., 2021, Sec. 3.1.1))
2n
12

---

Published in Transactions on Machine Learning Research (08/2025)
Bernstein: (A) = (A) + / log m[m log (3) where o² is the standard unbiased
variance estimator for the losses (see (Bates et al., 2021, Sec. 3.1.3))
Simple WDRO: The bound of (18)
SKWDRO: The bound of (19) where the 0/1 loss is replaced by a smooth version solved using the package skudro (Vincent et al., 2024)
In all the section, unless otherwise specified, we take a 0.1 and d 0.05. For the WDRO-based approaches, we take p c/ /n with v2 for SKWDRO and c 10v2 for Simple WDRO. Finally, note that for both WDRO-based approaches, the solved objectives ((18) and (19) are dual-based.
Remark 12 (Choice of the radius p). The choice of the radius is an important and difficult question in distributionally robust oplimization both theoretically and praclicully. We can distinguish two appronches: (i) generic measure concentration (as used in Esfahani and Kuhn (2018) and Lemma 5), which gives = C/88)(cn))) and thus suffers from the curse of dimensionality (but can be used directly to obtain statistical guarantees on WDRO estimators); or (ti) specific concentration on the WDRO (dual) objective which give p = C' IV but are unfortunately elusive for the conformal objective functions considered in the paper (mainly due to the discontinuity, see Azizian et al. (20236)). In both cases, constants are not explicit.
Nevertheless, as in classical WDRO we found that p x 1/ was performing rather well all around and this was used in our experiments as mentioned above. This finding is in line with the "practical rate" for WDRO estimation as mentioned in Sec. 7.2.D of Esfahami and Kuhn (2018).
Furthermore, for a specific dataset, several approaches can be used to tune the radius adequately. Indeed, training and conformal prediction can be done simultancously over the same dataset (contrary to most conformal prediction techniques which rely on separate calibration set), a small portion of the data can be held out for cross validating the value of p to satisfy the requirements with virtually no performance loss over the whole training procedure. This approach is detailed in (Esfahani and Kuhn, 2018, Sec. 1.2.B, C) for instance and can be on alternative to the standard 1/ /n choice.
Finally, in the case of an unknown distribution shift, these techniques could be too optimistic (unless some samples from the shifted distribution are available) and the (squore of) the radius should be theoretically increased by the measure of the shift (in W2 distance). In practice, this is often too conservative as the geometry of the Wasserstein distance with Euclidean cost may not capture adequately the loss degrudation. This is general issuc in the modeling of distribution shifts that goes beyond the scope of the present paper.
4.1 Coverage and interval size
In this section, our experimental setup is the following. We use 1000 samples generated from the make-regression function of scikit-learn." 500 are used to train a linear prediction model (using scikit-learn's LinearRegression estimator), 250 are used for calibration (which is n in the notation of the paper), 250 are used for evaluation.
In Fig. 2, we display the values of the considered saper-bounds as a function of the prediction size for one realization of the experiment. The selected prediction size Xm for each upper-bound is the smallest value of X for which the upper-bound is below the target risk (represented as a dashed line), the smaller the better as this will result in tighter prediction sets for the same guarantee. We observe that WDRO- based upper- bounds are consistently smaller than Hoeffding or Bernstein meaning they will lead to tighter prediction sets in general.
In Fig. 3, we repeat our experiment 100 times and report boxplots of the coverages (i.e., one minus the risk) evaluated on the test data as well as prediction interval sizes. We observe that WDRO-based approaches lead to admissible coverages (slightly closer to the target risk) with smaller interval sizes. Thus, the constitute interesting substitutes for Hoeffding or Bernstein- approaches.
Precisely, we replace 1u50 by -Hoxp(-3bu-1 We use the options n_ features=S, n i_informatiee=,, noise=20 leading to problem dimension d 5 with a fair amount. of noise.
13

---

Published in Transactions on Machine Learning Research (08/2025)
637
WTRD SKAURRO Read
130
12
Figure 2: Behavior of the RCPS bounds function of X
Test Covenage (1 Risk)
Prediction Interval Size
100
000
0000
00
0000
000
COOD
10
0.92
1bxciling
Beristeta
WDRO
SXWDRO
Boetding
Bonustel
WDRO
SKWIRO
Figure 3: Performance of the RCPS bounds over N = 100 runs. The dashed line represents the target coverage a. The whiskers of the boxplot are set to the 8 and 8 quantiles that larget RPCS have lower whisker above the dashed line.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Coverge
Covenaa distribation shib
1.00
00
000
1.00
000000
000
0.35
0000 00
GO0000
0000000
0.96
000000
60
0.81
000000
0.90
0.8
Bodiilag Beodaas 05
Brafling Baiitei P- 2-
Budting Bacatte
Figure 4: Performance of the RCPS bounds over N 100 with and without distribution staftt. The dashed line represents the target coverage 1 1 a. The whiskers of the boxplot are set to the ō and 1 - 8 quantiles so that the target of RPCS to have lower whisker above the dashed line. The leftmost figure corresponds the classical test coverage (as in the previous section) while the center corresponds to the coverage on the shifted dalaset. For the WDRO bound, the performance for several values of p are displayed
4.2 Distribution shifts
We place ourselves in the same setup as before. To simulate a distribution shift, we create shifted dataset by copying the test dataset and adding to the points (X;) Gaussian noise of mean (0.25, 0, O, 0, and covariance 1, resulting in Wasserstein distance of 0.612 between testing and training data. In Fig. 4, we display the coverage for the test set and the shifted set. We compare different values of for the WDRO bound. We notice first that the Bernstein and Hoeffding bounds are so pessimistic that they can actually accommodate for quite large distribution shifts in practice (without theoretical grounding though). For the WDRO-based bounds, we see that, as expected, the interval sizes and coverage grow with p, which enables take into account different magnitudes of shift (we recall that following Theorem 9, a WDRO model can accommodate for shift of magnitude in Wasserstein distance whenever p + p(n, 6) where p(n, 5) is the concentration radius that can be taken 1/vn).7 In particular, in the present case, the shift is of magnitude 0.612 and the "practical concent ration radius" is P. = 2/250 = 0.894 leading to minimal radius of 1.506. Hence, the choice p == 0.5 is smaller than the distribution shift magnitude and thus does not provide coverage in the shifted situation (middle plot) and even without shift (left plot). The choice p = 1.0 is closer to the prescribed value, provides correct coverage before shift but does not match the RCPS target after shift. satisfying performance after shift is attained with p = 2.0. This validates the interest this approach for directly integrating distribution shifts.
4.3 Simultaneous training and conformal inference
In this section, we draw n = 200 samples from the following distribution: X is drawn uniformly in [-2, 2] and Y = f(X) + U where f((1) exp(t)+exp(-r) + and U is a standard Gaussian variable.
For fixed X, we perform degree-4 polynomial regression of Y from X and compare two approaches for training and conformal inference using the SKWDRO bound:8
Separate. The regression is learned using sklearn's Linear Regression on polynomial features then the conformal upper bound is computed.
Joint. The polynomial model is trained to minimize the conformal loss, i.e., the bound of (12) is also optimized in 0.
"e do not communtt correct pointwisc distance used base for the Wasserstein distance (here the Euclidean distance) may be tunod for each problem for large performance gain. Such numerical study is out-of-scope for the present paper. 8the bound of (12) with smoothed loss solved by entropic regularization
how tune with respect distribution shift and refer to Remark 12. Furthermore, the
15

---

Published in Transactions on Machine Learning Research (08/2025)
In Fig. 5, we report the bound values and the test risk (estimated from 1000 independent samples) for separate and joint training a function of A. We observe that the joint approach enables to have a smaller bound and risk than the separate one, for any value of X. Hence, simultaneous training and conformal inference will lead to a better conformal inference performance at the cost of degrading the empirical performance.
In Fig. 6, we observe the models obtained by joint learning and conformal prediction can be very different than the one obtained separately by empirical risk minimization. Indeed, if À is small, is difficult to obtain meaningful confidence guarantees and thus any model than can grab few points will be optimal. On the over way around, if À is large, it is easy to be confident due to the large error margin and simple tendency of the data is sufficient to obtain a good conformal prediction. In between, the situation is mixed as a sufficient number of points should be correctly predicted but whenever the prediction fails, it does not matter by how much and thus the conformal loss training offers some kind of regularization to the predictor.
Thus, we can see that training a model using the distributionally robust conformal loss does not lead to good test performance even though the conformal risk is improved. Nevertheless, this opens the door for the study of WDRO-based conformal risk regularization for the empirical risk minimization which could lead to good performance but with an improved conformal coverage.
Remark 13 (On the interplay between coverage and performance). There is subtlety here in the interaction between conformal prediction (ic. risk control) which is at the core of this work and prediction performance of the base model which is not our main concern. The basic conformal prediction framework aims to quantify the uncertainty of a predictor (whatever its performance). Typically, for the regression case, if the predictor has poor performance for a given crileria and that the conformal interval is bwill with respect to this crileria, then the obtained prediction sets (ie. >) will be large. Conversely, if we mindmize the prediction size while jointly training a model (as we do in this section), then the obtained predictor should be somehow performing but has no reason to be optimal."
When jointly training, the RCPS guarantees are theoretically preserved since the generalization part only depends on measure concentration. Intuitively, by doing so we should reduce the overfitting of the predictor even if we might "overfit" the prediction sel, still this is mitigoled by the distribuutonaally robust optimization of the conformal loss. In addition, conformal prediction intervals can be obtained even when the models depend on the calibration data under some assumptions (see eg. Barber et al. (2023)).
Additional experiments exploring this setting can be found in Appendix A.
5 Conclusion
In this paper, we discussed the design of risk-controlled prediction sets using Wasserstein distributionally robust optimization. We showed that, by replacing the model loss by the conformal loss, the WDRO problem exactly leads to the type of generalization guarantees needed in the considered conformal prediction approach. This demonstrates the potential links between these two separate methods which can foster future research. In addition, the obtained WDRO-based conformal prediction methods displayed promising performance in our numerical illustrations and benefit from two major advantages: i) they can scamlessly account for distribution shifts; and ii) they allow for simultaneous training and calibration on the same dataset which is advantagcous in data-poor situations.
Acknowledgments
The authors thank the action editor and the reviewers for their fruitful remarks. F. Iutzeler acknowledges the support of the AI Cluster ANITI (ANR-23-IACI-0000) and the ANR project MAD (ANR-24-CE223-522)) A.
"Note that we consider in this work the "conformal risk control" approach, which is more general framework: shown in Example 2.1, the size of the conformal interval is then generalized through the parameter X. Both frameworks aim to get given level 1 1- a) which is then guarantee in implics that the level a would not be achieved for some realizations of (X,, Y) The
guarantees Eq. (1) (ie. getting a prediction set for the output expectation. In particular, guarantecs we are studying here (Eq. (9)) aim to ensure that the proportion of such events remains coutrolled at another level 8. This can be achieved by extrapolating concentration knowledge for the considered risk (Eq. (T)): as proposed by Bates et. al. (2021).
16

---

Published in Transactions on Machine Learning Research (08/2025)
Sepurater: bound
Seguntte:
Johar. bound
10
08
0.4
0.2
1.0
15
20
25
30
3.5
1.1
4.5
5.0
Figure 5: Conformal bounds and test risk (estimated from 1000 independent samples) for separate and joint training.
Speal Apcaliioaa
24
12
(a) À 1.2
(b) A = 2.6
(c) 2 5.0
Figure Polynomial models learned for different values of with separate and joint training.
Mazoyer has been supported by the project ROMEO (ANR-21-ASTA-0001) from the ASTRID joint program of the French National Research Agency (ANR) and the French Innovation and Defence Agency (AID).
References
A. N. Angelopoulos and S. Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494-50) 2023.
A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster. Conformal risk control. In International Conference Learning Representations, 2024.
L. Aolaritei, M. I. Jordan, Y. Marzouk, Z. O. Wang, and J. Zhu. Conformal prediction under Lévy-Prokhorov distribution shifts: Robustness to local and global perturbations. arXiv preprint arXiv:2502. 14105, 2025.
W. Azizian, F. Iutzeler, and J. Malick. Regularization for Wasserstein distributionally robust optimization. ESAIM: COCV, 29:33, 2023a.
W. Azizian, F. Iutzeler, and J. Malick. Exact generalization guarantees for (regularized) wasserstein distri- butionally robust models. Advances in Neural Information. Processing Systems, 36, 2023b.
R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, (2):8816 845, 2023.
S. Bates, A. Angelopoulos, L. Lei, Malik, and M. Jordan. Distribution-free risk-controlling prediction sets. Journal of the ACM, 68(6):1-33 14,, 2021.
17

---

Published in Transactions on Machine Learning Research (08/2025)
A. Ben-Tal, D. Den Hertog, A. De Waegenacre, B. Melenberg, and G. Rennen. Robust solutions of opti- mization problems affected by uncertain probabilities. Management Science, 59(2):441-35 -571, 2013.
J. Blanchet and K. Murthy. Quantifying distributional model risk via optimal transport. Mathemalies of Operations Research, 44(2):565-60) 600, 2019.
J. Blanchet and A. Shapiro. Statistical limit theorems in distributionally robust optimization. orNiv preprint arXiv:2303.144 14867, 2023.
J. Blanchet, K. Murthy, and V. A. Nguyen. Statistical analysis of wasscrstein distributionally robust esti- mators. In Tulorials in Operalions Research: Emerging Oplimizalion Methods and Modeling Techniquess with Applications, pages 227-20-22222 224.254. INFORMS, 2021.
J. Blanchet, K. Murthy, and N. Si. Confidence regions in wasserstein distributionally robust estimation. Biometrika, (09(2): 315, 2022.
S. Braun, L. Aolaritei, M. I. Jordan, and F. Bach. Minimum volume conformal sets for multivariate regression. arxiv preprint ar Xrc:2200 19068, 2025.
M. Cauchois, A. A. Suyash Gupta, and J. C. Duchi. Robust validation: Confident predictions even when distributions shift. Journal of the American Statislioul Associalion, 119(548):3033 3044, 2024.
E. Delage and Y. Ye. Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems. Operations Research, 58:595-612, 612, 2010.
P. M. Esfahani and D. Kuhn. Data-driven distribuutonally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Progromming, 171(1):115 166, 2018.
S. Feldman, L. Ringel, S. Bates, and Y. Romano. Achieving risk control in online learning scttings. Trans- actions on Machine Learning Research, 2023. ISSN 2835-8856.
M. Fontana, G. Zeni, and S. Vantini. Conformal prodiction: unified review of theory and new challenges. Bernoulli, 29(1):1 23, 2023.
N. Fournier and A. Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability Theory and Related Fields, 162(3):7077738)) 2015.
R. Gao and A. J. Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 48(2):603 655, 2023.
A. Gendler, T.W. Weng, L. Daniel, and Y. Romano. Adversarially robust conformal prediction. In Inter- national Conference on Leurning Representations, 2022.
I. Gibbs and E. Candès. Adaptive conformal inference under distribution shift. In Advances in Neural 2021.
Information Processing Systems, volume 34, pages
Gibbs and E. Candès. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25(162):1-36 16,, 2024.
J. Goh and M. Sim. Distributionally Robust Optimization and Its Tractable Approximations. Operalions Research, 58:902-917, 2010.
C. Gupta, A. K. Kuchibhotla, and Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127, 2022.
Z. Hu and L. J. Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available al Oplimization Online, 1(2):9, 2013.
D. Kuhn, S. Shafice, and W. Wiesemann. Distributionally robust optimization, 2024. URL https://arxi/: org/abs/244 025499.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Y. Kwon, W. Kim, J.-H. Won, and M. C. Paik. Principled learning method for wasserstein distributionally robust optimization with local perturbations. In International Conference on Machine Learning, pages 5567 5576. PMLR, 2020.
Y. Laguel, J. Malick, and Z. Harchaoui. First-order optimization for superquantile-based supervised learning. International Workshop on Machine Learning for Signal Processing, pages 1-6. IEEE, 2020.
T. Le and J. Malick. Universal generalization guarantees for wasserstein distributionally robust models. arXiv preprint ar 20x2:2402. 11981, 2024.
D. Levy, Y. Carmon, J. C. Duchi, and A. Sidford. Largo-scale methods for distributionally robust optimiza- tion. Advances in Newral Information Processing Systems, 33, 2020.
J. Li, C. Chen, and A. M.C. So. Fast Epigraphical Projection-based Incremental Algorithms for Wasserstein Distributionally Robust Support Vector Machine. In Advances in Neural Information Processing Systems, volume 33, pages 4029-4039. Curran Associates, Inc., 2020.
H. Namkoong and J. C. Duchi. Stochastic gradient methods for distributionally robust optimization with Edivergences. Advances in neural information processing systems, 29, 2016.
H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman. Inductive confidence machines for regression. European Conference on Machine Learning, pages 345 356, 2002.
A. Podkopaev and A. Ramdas. Distribution-frre uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence, volume 161, pages 844-853, 2021.
S. Shafiezzadeh Abadeh, P. M. Mohajerin Esfahani, and D. Kuhn. Distributionally Robust Logistic Re- gression. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
S. Shafieeradeh-Abadeel, D. Kuhn, and P. M. Esfahani. Regularization via Mass Transportation. Journal of Machine Learning Research, 20:1-66 68, 2019.
A. Sinha, H. Namkoong, and J. Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018.
M. Staib and S. Jegelka. Distributionally robust optimization and generalization in kernel methods. Advances Neural Information Processing Systems, 32, 2019.
R. Tibshirani, R. F. Barber, E. J. Candès, and A. Ramdas. Conformal prediction under covariate shift.
In Advances in Neural Information Processing Systems, volume 32, 2019.
F. Vincent, W. Azizian, F. Iutzeler, and J. Malick. skwdro: library for wasserstein distributionally robust machine learning. Xix preprint ar 24Xiv:2410.211 2024.
V. Vovk, A. Gammerman, and G. Shafer. Algorithmic learning in random world. Springer, 2005.
J. Wang, R. Gao, and Y. Xic. Sinkhorn distributionally robust optimization. arXic preprint arXiv:2109.11926, 2023.
I. Waudby-Smith and A. Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1-21 2024.
Yan, Y. Romano, and T.W. Weng. provably robust conformal prediction with improved efficiency. International Conference on Learning Representations, 2024.
M.-. Yue, D. Kuhn, and W. Wiesemann. On linear optimization over wasserstein balls. Mathematical Programming, 115(1):11077111 2022.
19

---

Published in Transactions on Machine Learning Research (08/2025)
J.-J. Zhu, W. Jitkrittum, M. Diehl, and B. Schölkopf. Kernel distributionally robust optimization: General- ized duality theorem and stochastic approximation. In International Conference. on Artificial Intelligence and Statistics, pages 280 288. PMLR, 2021a.
J.-J. Zhu, C. Kouridi, Y. Nemmour, and B. Schölkopf. Adversarially robust kernel smoothing. arxiv preprint 2Xxv:2102.08477 2021b.
20

---

Published in Transactions on Machine Learning Research (08/2025)
A Extra numerical illustrations for joint training and conformal inference
In this section, we use the same sctup as in Section 4.3 but change some of the parameters.
A.1 Varying the number of samples
We take = 500, 1000, 2000 samples and report the bounds and test risk in Fig. 7 and the obtain models in Fig. 8. We observe that neither the models nor the prediction change dramatically, indicating that the concentration observed is stable and hence the predictions. In particular, we do not particularly see overfitting behaviors.
A.2 Changing the variance
We take a variance of a = .1, .2, and 55 (instead of 1) in the Ganssian noise U and report the bounds and test risk in Fig. 9 and the obtain models in Fig. 10. With a smaller variance, the prediction size needed lo obtain a coverage guarantee diminishes as wee see in Fig. 9. Thus for X == 1.0, we observe that the models are better adjusted than with unit variance and for the 2.6 corresponding to an approximate coverage of 80%, the model better follows the curve compared the unit variance case (which has a coverage of approximately 65%)- Finally, for X 5.0, the size is large that virtually any model will have perfect coverage hence the observed behavior.
21

---

Published in Transactions on Machine Learning Research (08/2025)
Segurrate:: boand
Joit: bound
Joiz::
0.6
0.4
0.2
(a) 500
Sepuunder: boand
Sepunaee: rilk
Joint: ras
08-
00
0.4
0.1
(b) 1000
Segarater
Segorrate:: risk
Joint: bound
1.0
Join: rak
0.8
06-
0.
0.
(c) 2000
Figure Conformal bounds and test risk for separate and joint training.
22

---

Published in Transactions on Machine Learning Research (08/2025)
(a) 500, A 1.0
(b) 500, À 2.6
(c) 500, À 5.0
(d) 1000,
1000, 2.6
(f) TL 1000, A 5.0
14
15
30
(g) n 2000, A = 1.0
(h) n 2000, A 2.6
(i) 2 2000, À 5.0
Figure 8: Polynomial models learned for different values of À with separate and joint training.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Segurrate:: boand
Joit: bound
Joiz::
08
06-
0.2
0.0
(a)
Sepuunder:
Sepuntee: rilk
10-
Joint: ras
08
0.-
0.4
0.2
(b) 0.2
Segarater
Segorrate:: risk
Joint: bound
1.0
Join: rak
0.5
0.4
0.2
5.0
(c) 0.5
Figure Conformal bounds and test risk for separate and joint training.
24

---

Published in Transactions on Machine Learning Research (08/2025)
(a) 0.1, A 1.0
(b) 0.1, A 2.6
(c) 0.1, A 5.0
(d) 0 0.2, A
(e) 0.2, 2.6
(f) 0.2, A 5.0
Ere
Fale
-10
20
-1
(g) r 0- 1.0
(h) 0 = 0.5, 2.6
(i) 0 = 0.5, 5.0
Figure 10: Polynomial models learned for different values of with separate and joint training.
25


Reviews:
[{"review_id": "AyqsHfT5w3", "reviewer": "Reviewer_uVd6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the connection between conformal prediction and DRO is a meaningful one to investigate, I unfortunately don't see much in this paper that I think is a major contribution.\n\nThe main results reason about the existence of choices of conformal parameters $\\lambda$ which satisfy risk quantiles $\\alpha$ over distributions within some distance defined by a function $\\rho$. The paper makes no attempt to reason about what these values may be, hence, it is almost trivial by simply allowing $\\alpha$ or $\\rho$ to be arbitrarily large. In particular, I believe Assumption 3(a) *immediately* implies this result for Wasserstein-2 distance by simply letting $\\rho$ be equal to the $\\ell_2$ diameter of the space. Assumption 3(b) is similarly strong. \n\nEssentially, by choosing a very weak notion of distribution shift (Wasserstein-2 distance) and never specifying any sort of limits on how conservative the predictor may need to be to satisfy conformal risk guarantees under that shift, the proven results reduce almost entirely to just the original observation: that conformal prediction can be \"reparameterized\" as DRO.\n\nFurthermore, there seems to be little mathematical analysis in a paper focused purely on math. Most of the Lemmas are literal definitions or one-line proofs, and those that are not are taken (sometimes with no additional analysis) from the references. This includes Lemma 4 from Gao and Leywegt (2023), Lemma 5 from Fournier and Guillin (2015), and Theorem 10 from Yue et al (2022) and Gao and Leywegt (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper draws a connection between conformal prediction and distributionally robust optimization (DRO) by pointing out that the \"conformal set\" over which a prediciton function is calibrated can be connected to the uncertainty set in DRO. Thus by using methods for conformalizing the risk of a function, we recover DRO w.r.t. the conformal set for free.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "See my notes above. I think this is likely a valuable connection to pursue. But merely proving the existence of choices of $\\lambda$ and $\\rho$ for conformal prediction without any sort of bound on how $\\alpha$ or $\\rho$ grow as a function of the shift does not feel sufficient, particular in cases where we assume e.g. that $\\mathcal{X}\\times\\mathcal{Y}$ is bounded.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JCdIOdyaXO", "reviewer": "Reviewer_nLLr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n\n1. The paper is well-written and clearly structured. The key concepts are easy to follow and understand.\n2. The proposed methodology is theoretically sound and rigorously developed. The authors provide a comprehensive theoretical analysis of WDRO-based risk-controlled prediction sets (RCPS). And then, the dual formulation in Section 3.4 transforms the infinite-dimensional robust risk optimization into a tractable finite-dimensional problem. In addition, the proposed simultaneous learning is also a good choice to furthe improve the efficiency of prediciton intervals.\n3. The exmperical experiemts are extensive. The authors demonstare the uperioity of their method in the examples with and without distribution shift.\n\n# Weaknesses\n\n1. The authors should clarify the role of Section 3.4 in the overall methodology. Specifically, since the duality is claimed to be *“the computational scheme of WDRO”*, does the implementation in experiments directly solve the dual problem (e.g., Theorem 6) to estimate $\\hat{R}^\\rho_n(\\lambda)$? \n\n2. Some empirical results need to be further explained. For example, in the second subfigure of Figure 4, the average coverage under $\\rho=0.5$ falls below $1-\\alpha$, so the theoretical guarantee no longer applies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper combines the conformal risk-control framework with the distributionally robust optimization problem. Specifically, the method builds a Wasserstein ball around the empirical distribution and computes the worst-case expected loss over this set, leading to prediction intervals that remain reliable even when there is a moderate distribution shift between calibration and test data. Theoretical results, such as Theorem 1, establish that as long as the test distribution lies within this Wasserstein ball, one can control the predictive risk with high confidence. The extensive experiments show the superiority of WDRO-based upper bounds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In Figure 6, the jointly trained polynomial model exhibits a worse fit but achieves a lower bound value compared to the separately trained model. Could this be due to high noise levels? The authors are encouraged to provide an additional experiment using a standard noise variance of 0.1 instead of 1 to clarify this behavior.\n\n2. Minor typos:\n\n- Page 1: “reliablity” → “reliability”\n- Page 14: Add a comma before “i.e.” in the \"joint\" section", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OoH624CE4C", "reviewer": "Reviewer_6jTx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper establishes a formal connection between conformal risk control and distributionally robust optimization, providing uniform high-probability guarantees under Wasserstein shift.\n2. The ability to simultaneously train the model and calibrate prediction intervals on the same dataset offers an improvement over classical conformal prediction.\n3. The paper presents a tractable formulation for the WDRO objective and provides implementation strategies utilizing regularized approximations.\n\n**Weaknesses**\n1. The method’s validity depends on choosing the Wasserstein radius $\\rho$, but the paper provides no strategy to select it.\n2. In Section 4.3, models trained to minimize conformal loss achieve poor point prediction performance. This trade-off is acknowledged but not analyzed, limiting the method’s utility in tasks requiring accurate predictions.\n3. Although the paper promotes joint training and calibration as a sample-efficient alternative, it does not assess the potential for overfitting when data is reused.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new framework to construct risk-controlling prediction sets using Wasserstein Distributionally Robust Optimization (WDRO). Unlike classical conformal prediction methods, risk-controlling conformal prediction intervals can be learned jointly with the predictive model, under the WDRO formalism. The method claims to offer improved statistical guarantees and tighter prediction intervals. Theoretical results support the validity of the approach under certain assumptions, and experiments illustrate the benefits and trade-offs of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is suggested to provide guidance on selecting the Wasserstein radius, including either heuristic or empirical strategies.\n2. The degradation in predictive accuracy observed in Section 4.3 should be analyzed in more detail.\n3. It is suggested to add a comparison of joint setups under varying sample sizes to examine if data reuse leads to overfitting.\n4. It is suggested to provide more discussion or information on solving the regularized problem. For example, including a time comparison to baseline methods would be useful.", "broader_impact_concerns": "There are no additional ethical concerns identified that require discussion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)