Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization
Authors: Nicola Bariletto
Nhat Ho
Venue: NeurIPS 2024 poster (2024)
Keywords: Bayesian Nonparametrics
Distributionally Robust Optimization
Dirichlet Process
Decision Theory
Ambiguity Aversion
Machine Learning
Primary area: learning_theory
TL;DR: We introduce and study a novel distributionally robust optimization procedure, combining insights from Bayesian nonparametric (Dirichlet process) theory and a popular decision-theoretic model of smooth ambiguity-averse preferences.

Abstract: Training machine learning and statistical models often involves optimizing a data-driven risk criterion. The risk is usually computed with respect to the empirical data distribution, but this may result in poor and unstable out-of-sample performance due to distributional uncertainty. In the spirit of distributionally robust optimization, we propose a novel robust criterion by combining insights from Bayesian nonparametric (i.e., Dirichlet process) theory and a recent decision-theoretic model of smooth ambiguity-averse preferences. First, we highlight novel connections with standard regularized empirical risk minimization techniques, among which Ridge and LASSO regressions. Then, we theoretically demonstrate the existence of favorable finite-sample and asymptotic statistical guarantees on the performance of the robust optimization procedure. For practical implementation, we propose and study tractable approximations of the criterion based on well-known Dirichlet process representations. We also show that the smoothness of the criterion naturally leads to standard gradient-based numerical optimization. Finally, we provide insights into the workings of our method by applying it to a variety of tasks based on simulated and real datasets.

flourishing literature on Distributionally Robust Bayesian Nonparametries Meets Data-Driven Distributionally Robust Optimization
Nicola Bariletto
Nhat Ho
Department of Statistics and Data Sciences
Department of Statistics and Data Sciences The University of Texas at Austin Austin, TX 78712
The University of Texas Austin Austin, TX 78712
nicola. berilttoeaal
minhnhat@utexas.c edu
Abstract
Training machine learning and statistical models often involves optimizing data- driven risk criterion. The risk is usually computed with respect to the empirical data distribution, but this may result in poor and unstable out-of-sample performance due to distributional uncertainty. In the spirit of distributionally robust optimiza- tion, we propose a novel robust criterion by combining insights from Bayesian nonparametric (i.c., Dirichlet process) theory and recent decision-the -hheoretc model of smooth umbiguity-avers aveerse preferences. First, we highlight novel connections with standard regularized empirical risk minimization techniques, among which Ridge and LASSO regressions. Then, we theoretically demonstrate the existence of favorable finite- sample and asymptotic statistical guarantees on the performance the robust optimization procedure. For practical implementation, we propose and study tractable approximations of the criterion based on well-known Dirichlet process representations. We also show that the smoothness the criterion natu- rally leads to standard gradient- based numerical optimization. Finally, provide insights into the workings our method by applying it to a variety of tasks based simulated and real datasets.
Introduction
In machine learning and statistics applications, several quantities of interest solve the optimization problem
min Rp. (8), lEe
where (0) : Eg-p h(0, {)] | is the expected risk associated to decision 8, under cost function h(0 E) (measurable in the argument {) and given that the distribution of the (E, is pll For instance, we are dealing with a supervised learning task where & = (z, y) E R=-1 x R, h(OE) is usually loss function ((fo(z))) y) quantifying the cost incurred in predicting y with fa()) here the decision variable 0, which parametrizes the function fo : mp--1 - RR. For the rest the paper, we assume = C Rm θ RRd and h θ E [0, K for some K <0
data
In most cases interest the true data-generating process p. is unknown, and only a sample &" (EI..... {n) from it is available. The most popular solution is to approximate p. by the empirical distribution Pe=, and optimize Ryg. (0) = n En I0(0, {.). However, especially for small sample sizes and complex data-ges generating mechanisms, this result in poor out-of- -fammle performance, leading to the need for robust alternatives.
'Given a topological space (A, T), we denote by (AA) the Borel o-algebra generated by T. "Note that, under mild regularity (e.g., continuity) conditions on h. its boundedness ensured, for instance. by the frequently assumed compactness = and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Optimization (DRO) has provided several methods in that direction [though not always with data- driven applications as the primary focus; see 33] for a recent exhaustive review of the field]. A prominent approach is the min-max DRO (mM-DRO) one, whereby worst-case criterion over an ambiguity set of plausible distributions minimized [19] _ 3 10 46 11. Recent notable results involve the study of mM-DRO problems where the ambiguity set is defined as Wasserstein ball of probability measures centered at the empirical distribution [29] [2].
Contribution. Differently from the mM-DRO paradigm, we propose distributionally robust procedure based on the minimization of the following criterion:
Ven (0) = f
o (Rp(()))Q==
(1)
where K] + R+ is a continuous, convex and strictly increasing function, and Qe is Dirichlet process posterior conditional on 5" 13
As we show below, our proposal brings together insights from two well-established strands of literature- decision theory under ambiguity and Bayesian nonparametric statistics, contributing in a novel way the field of data-driven distributionally robust optimization. As we establish throughout the article, among the key advantages of the criterion are: (i) its favorable statistical properties in terms of probabilistic finite-sample and asymptotic performance guarantees; (ii) the availability of tractable approximations that are easy to optimize using standard gradient- methods; and (iii) its ability both improve and stabilize the out-of-sample performance of standard learning methods.
The rest of the paper is organized as follows. In Section 2, we motivate the formulation in Equation (T) by providing concise overview of decision theory under ambiguity and its connections to Bayesian statistics and regularization. In Section B] we study the statistical properties of procedures based on we propose and study tractable approximations for Vg- based on the theory of
Ve.. In Section[4] 4 DP representations. In Section[3] we highlight the robustness properties of our method by applying it variety of learning tasks based on real and simulated data. Section 6 concludes the article. Proofs of theoretical results and further background are provided in Appendices A [B] respectively, while in Appendix C we discuss how the smoothness of the proposed criterion yields straightforward gradient-based optimization, and present more details on the numerical experiments. Code to replicate our experiments can be found at the folllowing link: https: / tigggtthu...... com/nbariletto//BB for_DRO
2 Decision Theory and Bayesian Statistics
Following a long-standing tradition in Bayesian statistics and decision theory [37], the distributional uncertainty on the data-generating process p. can be dealt with by defining prior Q over it. We point out that this Bayesian approach contrasts with the classical one, where a prior would typically be placed directly on the parameter 0, whose data-driven optimal value would be determined as function (e.g., the mean or mode) of the resulting posterior distribution. In this new framework, instead, the parameter 0 is treated as a variable to be optimized, while the prior is assigned to the entire data- generreeng process. This perspective allows us to incorporate several valuable Bayesian concepts, as will clarify throughout the paper, while preserving the flexibility of the original optimization-bn based learning framework. Notably, does not need to be interpreted as the parameter of a full generative model, would be required in a classical Bayesian setting. Instead, it can represent a vector of parameters associated with a generic, possibly complex loss function, such as those employed in modern deep learning architectures.
Specifically, our Bayesian approach is equivalent to modeling the observed data CT = (Ei...., as exchangeable with de Finetti measure Q:
iid P. p
i= 1-
Q.
Due the stochasticity of p. (0) is itself random variable, and sensible procedure is to maximize its posterior expectation. Let Q5- be the posterior law of conditional on the sample g
Throughout the paper, adopt the terms "ambiguity" and "uncertainty" interchangeably.

---

Then, one solves the following problem:
min EinEp-ae Qen [R) = min L h(0, (e((()))pe))eedd (dp) min Be-p(-10") 1s (1(0) E)], DE
Bea
where S denotes the space of probability measures on ( endowed with the Borel o-algebra ( ()) generated by the topology of weak convergence, while p(dEl<") := S J ped (dp) denotes the posterior predictive distribution. In sum, within this general Bayesian framework, the data-driven problem reduces to minimizing h(0,g) {) averaged w.r.t. the posterior predictive distribution, i.e, mingge Rp(-1e-) (0).
2.1 The Dirichlet Process
natural choice is to model the prior Q as a Dirichlet process (DP), and Qę" then DP posterior. First proposed by [13], the DP is the comerstone nonparametric prior over spaces of probability measures. Its specification involves a concentration parameter a > 0 and a centering probability measure po- Intuitively, the DP is characterized by the following finiteddimensional distributions: ~ DP(a, Pn) implies (p( A1):
pl Ax)) ~ Dirichlet(opn A1),... (pp)( for any finite mea- surable partition (A1,...; A&) of = key property of the DP is its almost discreteness, which
allows to write p Σソ>1 P3 (where probability weights and atom locations are independent). Moreover, the DP is conjugate with respect to exchangeable sampling. In our case, this means
d
DP(a, Po) = Qe- DD(( a n, a+ n
n
Po
+ P(")
That is, conditional the sample t", p is again DP with larger concentration parameter + and centered at the predictive distribution p(-1 == a4n + n
a4nPee" The latter a compromise between the prior guess Po and the empirical distribution pç=, and the balance between the two is determined by the relative size of a and n. The predictive distribution is also related to the celebrated Maeckwell-MaeQueenn Pólya urn scheme (or Chinese restaurant process) to draw an exchangeable sequence (Ez)i21 distributed according to p DP(a, Pu): Draw & Po and, for all > 1 and l < i, set & = & with probability (a+ + -11 else (i.e., with probability al + 1) draw ç ~ Po [4.
Given the large support of Qe", which consists of all probability measures whose support included in that of p(-15" ((") 288. the DP is reasonable and tractable option to mitigate misspecification concerns. Then, leveraging the mentioned expression for the DP predictive distribution, the problem specializes to
R a++n Ree (0) + a _o (0)}
a
min dg
Ryo
[see also 27 45 In practice, adopting the above Bayesian approach amounts to introducing a regularization term depending the prior centering distribution po- Compared to the simple empirical risk Ryg (0), this type of criterion displays lower variance (because Rp, (0) is non-random) at the cost of some additional, asymptotically-vaniss vanniiiin bias w.r.t. the theoretical criterion Ry. (0). We also note that such bias can be attenuated in finite samples long the prior guess po and the true data-generating process pa are close enough in terms of the difference RRuo (0) Rp. (0)
Connections to Regularization in Linear Regression. One of the most ubiquitous data-driven learning tasks is linear regression [38] 77. It is well-known that, in this setting, coefficient estimation (c.g., via maximum likelihood or least squares) sample average of the squared loss function. It turns out that, applying the Bayesian regularized approach (2), interesting equivalence with standard regularization techniques such Ridge 21 and LASSO [41] emerges.
be framed
minimization problem of the
Proposition 2.1. Let h(0, (v, (y AT xx) Then, denoting Aa,n = a/n, n the following equivalences hold:
'Also, [[p[A)] (A) and Vip(A)] (1 - (A)( po() for any A (E), justifyin the
names of a and po-
'n practice, the prior guess Po can be leveraged to incorporate features of the underlying process that the
researcher suspects hold (e.g.. in regression applications, sparsity). See also Section 5
3

---

1. If (O I), then 9 solving 2 implies that it solves
min
h(0,E,) + Aa, 1
1=1
2. 1fV = diag(101)))
and po = NO V), then 1 solving 2 implies that it solves
min AEB
h(0 + Aa,n110111
Proposition 2.1 is insightful because highlights novel Bayesian interpretation of Ridge and LASSO linear regression. In fact, it is well known that both methods are equivalent to maximum- a-posteriori estimation of regression coefficients when the latter are assigned either normal or Laplace prior. In our setting, instead of parametric prior on the regression coefficients, we place nonparametric one on the joint distribution of the response and the covariates. The degree of regularization, then, is naturally guided by the prior confidence parameter a and the sample size n. We also note that sparsity only one of the possible gata-genn features one might want to enforce in regularized estimation D compared to Ridge and LASSO, to incorporate such patterns by specifying the prior expectation Po of the joint response-covariate distribution.
and the nonparametric Bayesian approach offers greater flexibility,
2.2 Ambiguity Aversion
As we just showed, adopting traditional Bayesian framework, uncertainty about the model p is resolved by using the posterior Q&- to directly average out. This procedure, however, does not take into account the (partly) subjective nature of the beliefs encoded in Qęn, and the aversion to this that a statistical decision maker (DM) might have. In fact, the result of the procedure is that the DM ends up minimizing the expected risk, where the average taken according to the predictive distribution. In practice, then, the latter is put on the same footing as an objectively known probability distribution, such as the true model.
This issue has been thoroughly studied and addressed in the economic decision theory literature 18 (). In that context, the economic DM faces an analogous expected utility maximization problem maxéce Ee-p [e <)] (e.g., to allocate her capital to a portfolio of investments subject to random economic shocks {). However, she does not possess enough objective information to pick one single model of the world p. but deems larger set of models plausible. One possibility, then, is that the DM forms second-order belief (e.g., a prior Q) over such set, and resolves uncertainty by directly averaging expected utility profiles Be~plu(( E)] w.r.t. p Q.
Just like in data-driven problem, however, direct averaging does not account for ambiguity aversion. (23] proposed and axiomatized a tractable "Smooth Ambiguity Aversion" " (SmAA) model, whereby second-order averaging is preceded by deterministic transformation induc- ing uncertainty aversion via its curvature: The DM optimizes J0 oWEe~plute [u(0, (dd)), and criterion (1) simply specializes the SmAA model the data-driven case When optimization takes the form of minimization, ambiguity aversion is driven by the degree of convexity of of In particular, convexity encodes the DM's tendency pick decisions that yield less variable pected loss levels across ambiguous probability models. To see this intuitively, examine the sim- ple case when only two models, P1 and p2, are supported by decisions 0, and 02 that, under P1 and p2. yield the expected risks marked on the horizontal axis of Figure 1 While ((d)))(
_6p + f0pz. Consider two
(Bd)))((p)) R", the convexity of & implies
'For instance, one might have prior information on specific correlation patterns among covariates, which could be useful to incorporate in regression training with few data points.
exploring Bayesian ideas in the context DRO. However, the cited works (i) focus on parametric priors and (ii) rely on either ambiguity sets or statistical risk measures to induce robustness. This is in contrast with our work. which (i) resorts to more assumption free nonparametric prior and (ii) leverages the highlighted robustness properties of the simple yet powerful convex transformation Φ.
'Our approach is also related to recent literature 48 47
40
'I the economic decision theory literature, as the DM usually maximizes a criterion (utility), convexity is replaced by concavity.

---

2.5
2.0
1.5
1.0
0.5
0.0
(8g)
(0,)
R²
(2,(0,)
(2,
- the mM-DRO setup is recovered, with Figure I: Graphical display of smooth ambiguity aversion at work. Although 0, and 02 yield the same loss RD in @-expectation, the ambiguity averse criterion favors the less variable decision 01- Graphically, this is because the orange line connecting dRR,, (01)) to (()))(11) lies (point-wise) below the line connecting &f (R,, (02)) to (R2))))))
(O))))pp)))) < (R, (0)) ((dp). That is, although 0, and 02 yield the same loss in @-expectation, the ambiguity-avers criterion favors 0 because it ensures less variability across uncertain distributions P1 and p2-
Interestingly, showed that the SmAA model belongs general class of ambiguity-averse preferences, which admit common utility function representation. For SmAA preferences with ((t) = 3 exp(6-11) -3 (with 3 > 0 and under additional technical assumptions), this representation implies the equivalence of problem ① with
min max {Epp (R (0) (PL(()) BEO P.
where KL(-II-) is the Kullback-Leibe divergence and < denotes absolute continuity. The above result further clarifies the mechanism through which distributional robustness is induced: Intuitively, instead of directly averaging measure, penalizing distributions that are further away from the posterior the latter acts as reference probability measure. Moreover, in the limiting
p Qea. one computes worst-case scenario w.r.t. the mixing
ambiguity set C = {p E P 3P < Qş",P = J {ddq)) In the other limiting case /3 > oo (with the convention oc = 0), the ambiguity neutral Bayesian criterion instead recovered.
Statistical Properties
In this section, analyze the statistical properties of the criterion Vg- (8), as function of the sample size n. first issue of interest, addressed in Proposition 3.1 is to study its asymptotic point-wise behavior (ef. [17), Corollary 4.17).
Proposition 3.1. Let &" be iid according to pa and & + h(0, {) continuous for all O E θ. Then, for all O
lim Ve- (0) (R, (0))
almost surely.
This ensures that, more data are collected, the proposed criterion approaches, with probability 1, the true theoretical risk (up to the strictly increasing transformation Φ).
While point-wise wise convergence to the target ground truth is first desirable property for any sensible criterion, is not enough characterize the behavior of optimization's 's out-of-sample performance, nor the closeness of the optimal criterion value and the criterion optimizer(s) to their theoretical counterparts. In the following subsections, we study these properties both the finite-sample regime and in the asymptotic limit +
5

---

Finite-Sample Guarantees.
Denote
fn E arg min Ve.. (0), 8, E arg minRp, (0), BEA
and we assume the above sets of minimizers to be non-empty throughout the article. In finite-sample analysis, a first question of interest is whether probabilistic performance guarantees hold for the robust criterion optimizer On. In our setting, one can naturally measure performance by the narrowness of the gap between Ry, (On) and R.. (O.). As we clarify later, Lemma 3.2 is a first step towards establishing this type of guarantees.
Lemma 3.2. Let be twice continuously difserentiable on (0, K), with Ma = suP)E(0,K) d(() to0 and 2/ == SuPsE(0.K) To(t) < +00, where (t(t) == 6² (t)))/ 2 0. Then
K2
sup [Ve- - (0)) - Ms 0
sup Rpgn (0) Rp. (0)l + a aEB
K + %0l
a a+
Lemma 3.2
links the sup distance of the criterion VG- from the theoretical risk to three key objects:
1. The classical sup distance between the empirical and theoretical risk, supece Ree (0) Rp. (0)):
2. The sup distance between the theoretical risk and the risk computed w.r.t. the base probability
measure po. supgee p.. (0) - R.p (0)l. In fact, while in the formulation of Lemma 3.2 we bound such distance by K (see the second addendum) to eliminate the dependence on the unknown but fixed ps, one could equivalently replace K by supsee RRo (0) RR. (6)).
This clarifies that, if Pu is good guess for pa, i.e., if the above sup distance is small, adopting a Bayesian prior centered at po can improve finite sample bounds;
3.
The Arrow-Prult coesficient (I( of absolute ambiguity aversion. In the economic theory literature on decision-mmking under risk, this is well-known concept measuring the degree of risk aversion of decision makers, with point-wise larger values of 76 corresponding to more risk aversion. See (23) for a discussion on the straightforward adaptation of this measure to the ambiguity (rather than risk) aversion setup we work in.
Most importantly, Lemma 3.2 allows us to prove the following Theorem, which yields the performance guarantees we are after.
Theorem 3.3. For all 1
Plo(Rp (0n)) pRpp. (0.)) S
K2 2 10 ].
2 P
sup Rpg (0) R.. (0) S M
+
K
n
2M a + 72
Theorem 3.3 allows obtain inite-sample probabilistic guarantees on the excess risk @Rpp. (0,)) (R,, (0.)) via bounds on supree 'R (0) - R,, (0) The latter is a well-studied quantity, and the sought bounds follow from standard results relying on conditions on the complexity of the function class H (h(0,.) E )), as measured by its Vapnik-Chervonenke dimension, metric entropy, etc. We refer the reader to 44 43 for systematic treatment of the topic and specific useful results.
Asymptotic Guarantees. So far, we have studied the fmiie-ssaple behavior of the sutple performance of 0n. Another closely related type of results deals with the asymptotic limit of such performance, as well as with the convergence of optimum criterion values and optimizing parameters to their ground-truth counterparts. In this Subsection, attention is turned to theoretical results of this kind.
Finite- sante- guarantees on supage RRen (8) Rp. (8) are usually of the form
RRee (8) Rp. (0) { 2 Ma dee
sup

---

with En=1 Th < oc. This implies (via straightforward application of the first Borel- Cantelli Lemma) the almost sure vanishing of supaee Reg (0) R.. (0) Thus, we include this as an assumption of the next Theorem. Moreover, we introduce functional dependence of & on n, and
denote & = on accordingly.
Theorem 3.4. Retain the assumptions of Lemma 3.2 and limg-+00 supace Ryen (0) - (0) 0
almost surely. Moreover, assume that On satisfies (T) lim Vo., = 0, (2) supn21 Mrn oc, and
(3) lime suPrE[o,K)
(6n(t) t| Then the next two almost sure limits hold:
lim Ry. (0,) pp (0.): n=000
lim VE (0x) (0.). 2--000
Theorem 3.4 is crucial because it ensures that, asymptotically, the excess risk vanishes and the saite-sample optimal value converges to the optimal value under the data generating process P Remark 3.5. From a design point of view, the type of n-dependent parametrization of Φ required in Theorem 3.4 is sensible, as it is equivalent to adopting vanishing levels of ambiguity aversion (uniformly vanishing Arrow-Pratt coefficient) as the sample size grows that is, one obtains more and more precise estimate of the true distribution p.. Moreover, this assumption is in the spirit of the condition imposed on the radius of the Wasserstein ambiguity ball in [29]. which required to vanish as the sample size grows, Finally, it is easy to show that On (t) = 8,, exp(8, (3, 't 8n. for positive 3, - 00, satisfies the conditions of Theorem 3.4 and from now on we silently assume this form for Φn.
Finally, we leverage the above results to ensure the convergence of the sequence of optimizers (0x)221 to a theoretical optimizer.
Theorem 3.6. Let 0 h(0,g) be continuous for all & E and lima-soo Ry (0n) (0.) Ry, (0.)
almost surely (e.g., ensured in Theorem 3.4. Then, almost surely, lom On 0 implies Rp. (0) = (0.).
Monte Carlo Approximation
In what follows, we fix sample <" and propose simulation strategies estimate Vee (0). The latter, in fact, is analytically intractable due to the infinite dimensionality of QE.. To that end, we exploit a key representation of DPs first established by 39 If p DP(1) q). then _ ΣØ= p5Eイ- where
E q and the sequence of weights (2,)j21 1 is constructed via ttick-breaking procedure based on iid Beta(1. n7) samples (see Appendix B). Thus, for large enough integers and N, we propose the following Stick-Breaking Monte Carlo (SBMC) approximation for (8):
N
Σ Epith(e...) j=0
(O,T, =
Σ Φ
where, T' denotes the number of stick stick-breaking steps performed before truncating each Monte Carlo sample from the DP posterior Qe", while N denotes the number of such samples. Algorithm I Appendix B details the procedure, which essentially approximates the posterior DP via truncation and takes expectations accordingly.
Remark 4.1. We propose to truncate the stick- breaking procedure at some fixed step T. Another strategy would involve truncating it at a random step 7x(e) == min {t E N
Pij S E} for some small E > 0. This allows to directly control the approximation error at each Monte Carlo sample 30 D. though it leads to simulated measures with supports of different cardinalities. For the sake of theory, we opt for the fixed-stepe seeed-steprrnndom-eeee approximation, though the random- stadlommstte/fixe-errrroo one is equally viable in practice.
Remark 4.2. On top of being theory-based approximation for Vg- (0), the criterion (3) can be interpreted as implementing form of robust Bayesian bootstrap. Instead of directly averaging the risk -0),-) with respect to the empirical distribution, we first obtain N bootstrap samples of
Using an analogous line of reasoning as in the proof of Theorem 3.4 (see Appendix A).
we note that Lemma
3.2 and Theorem can be easily adapted in to obtain finite sample bounds on supsee Ven (0) - Ry (0) and
R.. (0.. Rp. (t.) depending on supte|o..x (o(t) t).
7

---

size T from the predictive (which is compromise between the empirical and the prior centering distributions), we weight observations according the breaking procedure, and finally take grand average of the d--ransformed weighted sums. This connection with the Bayesian bootstrap suggests the following alternative Maltinomiil-Diirichhee Monte Carlo (MDMC) version of Ve (0):
1
wey 10,h((0,5m (0,,(,))))))) Eis))
where (wa1..... Werr Dirichlet (T; atn
"+") and the atoms are iid according to the predic- 10 For practical computation, we recommend using the MDMC
tive (see Algorithm 2 in Appendix B
of approximation, as it tends to yield more balanced weights, compared to SBMC, even for low values
With the following results, we ensure finite-sample and asymptotic guarantees on the closeness of optimization procedures based on the SBMC approximation versus the target Ve"..
Lemma 4.3. Assume θ is a bounded subset of IRd and, for all E E, A - h00, () is L(p)-Lipschitz
continuous. Then, for all T, N 2 1 and ε > 0,
T
sup VE- (0,T, - Ve (0)] & MoK x DE
a +n n + n++1
tE
with probability at least
1 1-2 (.MMa(rdiam(H)) ( d
exp {
3N
3Ne
}
exp
400(K) [
8
40(K) + +
for some constant Cr > O.
Heuristically, the bound in Lemma 4.3 obtained by decomposing the left-hand side of the inequality into first term depending on the truncation error induced by the threshold T, and second term reflecting the Monte Carlo error related to N. Moreover, analogously to Lemma 3.2] Lemma 4.3 easily implies finite-sample bounds on the excess "robust risk" Vgg (T, N)) Ve (0n). where &n N) E arg minage Vgn. (0, T, N). Another consequence is the following asymptotic convergence Theorem, whose proof is analogous to that of Theorem 3.4
Theorem 4.4. Under the same assumptions of Lemmu 4.3 and if suPT21 Or < oo (see Appendix A for details on Cr),
lim sup VEn (O,T, - Ve- Ve- =0 T,N BEG
almost surely. Also, almost surely
lim Vg- (N), = (0n),
lim Ve- (O, = Vg- (0,). T.N
T-N
In words, Theorem 4.4 ensures that, as the truncation and MC approximation errors vanish, the optimal approximate criterion value converges the optimal exact one, and that the exact criterion value at any approximate optimizer converges to the exact optimal value. Also note that Theorems 3.4 and 4.4. when combined, provide guarantees on the convergence of Ve- (T, , N), T, .N (the empirical criterion one has optimized in practice) Ry (0.) (the theoretical optimal target) as the sample size increases and the DP approximation improves.
Finally, as a byproduct of Theorem 4.4 convergence of any approximate robust optimizer to exact one is established as follows I
Theorem 4.5. Let O + h(0,E) E) be continuous for all g E E. Moreover, assume
lim
(T, = Vg- (f..)
T,N-500
almost surely (e.g., ensured above). Then, almost surely, limu, Noc (T. = 0,, implies Ven (0m) Ve- (0.))
In the a = 0 limit and setting 7 = the well-known "Bayesian bootstrap distribution" is recovered 17
also Appendix B for further details].
In Appendix A we also present asymptotic normality result for the approximate optimizer (T,
(Proposition A.3
8

---

5 Experiments
We applied our robust optimization procedure to a host of simulated and real datasets, and we report results in this Section. Before proceeding, we notice that, given the finite approximations proposed in Section 4 and under mild regularity assumptions on the loss function h, the proposed robust criterion is amenable to standard gradient-based optimization procedures (see Appendix [f further details and an insightful interpretation of the gradient of our criterion as yielding robustly weighted stochastic gradient descent steps).
Simulation Studies. We tested our method on three different learning tasks featuring high degree of distributional uncertainty in the data generating process, and compared performance to the corresponding ambiguity neutral (i.e., simply regularized) and unregularized procedures. First, we performed a high-dimensionaa sparse linear regression simulation experiment. We simulated 200 independent samples of size correlated with each other), only the first s = 5 of which have unitary positive marginal effect on the scalar response y. Second, we performed simulation experiment estimation in the presence of outliers. We simulated 200 independent samples of 13 observations, 10 of which come from O-mean Gaussian distribution and 3 from outlier distribution, and tested the ability of the three methods to recover the true mean (i.e., 0). Third, we performed a simulation experiment on high-dimensional sparse logistic regression for binary classification. We set up data-generating mechanism similar to the linear regression experiment, where a small subset of features linearly influence the log odds-ratio.
100 from a linear model with d = 90 features (moderately
univariate Gaussian mean
Appendix C collects further details on the above experiments as well as plots summarizing the results (see Figures 2 B and 4. All three experiments reveal the ability of our robust method to improve out-of-sample performance and estimation accuracy in two ways, i.e., (i) by yielding good results on average, and especially (ii) by reducing performance variability. The latter is key robustness property that our method is designed to achieve.
Real Data Applications. We tested our method on three diverse real-world datasets. In the first study, applied our method to predict diabetes development based host of features, as collected in the popular and public Pima Indian Diabetes dataset. Because the outcome is binary, we used logistic regression implemented (i) with our robust method, (ii) with L1 regularization, and (iii) in its plain, unregularized version. We selected hyperparameters via cross-validation and tested the out-of-sample performance of the three methods applied to disjoint batches of training observations to assess the methods' performance variability. As we report in Appendix [ our robust method outperforms both alternatives average and does significantly better in redueing variability.
We performed two further studies on linear regression applied to two popular UCT Machine Learning Repository datasets: The Wine Quality dataset [8] and the Liver Disorders dataset [5] Similarly to the first study, we compared the performance of our method to OLS (unregularized) estimation and Li-penalized (LASSO) regression. After cross- coos--vlidation for parameter selection, we train the models multiple times on separate batches of data and compute out-of-sample performance on a large held-out set of observations. As the results reported in Appendix C show, also in these settings our robust DP-based method performs better than the alternatives both on average and especially in terms of lower variability. Taken together, the experimental results described in this Section corroborate empirically the robustness properties of the proposed criterion, as examined theoretically throughout the paper.
Discussion
The paper tackled the problem of optimizing data-driven criterion in the presence of distributional uncertainty about the tata-generating mechanism. To mitigate the underperformance of classical methods, we introduced a novel distributionally robust criterion, drawing insights from Bayesian nonparametries and decision-theorede model of smooth ambiguity aversion. We established connec- tions with standard regularization techniques, including Ridge and LASSO regression, and theoretical analysis revealed favorable finite-sample and asymptotic guarantees on the performance of the robust procedure. For practical implementation, we presented and examined tractable approximations of the criterion, which are amenable to gradient- based optimization. Finally, we applied our method

---

to variety of simulated and real datasets, offering insights into its practical robustness properties. Naturally, our work presents some limitations that give rise to interesting directions for future research. In particular, we note the need for a deeper examination of the model workings in terms of (i) its parameter configuration and (ii) broader application to general learning tasks (e.g., when the loss function is adapted to accommodate deep learning architectures). Moreover, our method, as many others in the distributional robustness literature, is only suited to process homogeneously generated (e.g., iid exchangeable) data, leaving room to explore extensions to more complex dependence structures, Finally, we highlight that our study offers prospects for investigating connections among such varied yet interconnected strands of literature as optimization, decision theory, and Bayesian statistics.
Acknowledgements
Both authors would like to thank Khai Nguyen for useful advice coding implementation. NB acknowledges support from the "Giorgio Mortara" scholarship by the Bank of Italy. NH acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations of Machine Learning.
References
[1] J. Arbel, P. De Blasi, and I. Prünnster. Stochastic Approximations to the Pitman- Yor Process. Bayesian Analysis, 14(4):1201 1219, 2019.
[2] A. Ben-Tal, D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341 357, 2013.
[3] D. Bertsimas, X. V. Doan, K. Natarajan, and C.-P. Teo. Models for minimax stochastie linear optimization problems with risk aversion. Mathematics of Operations Research, 35(3):580-602, 2010.
[4] D. Blackwell and J. B. MacQueen. Ferguson distributions via Pólya urn schemes. The Annals of Statisticy, 1(2):353-355, 1973.
[5] S. Cerreia-Vioglie preferences. Journal of Economic Theory, (2444::1777533300 2011.
Maccheroni, M. Marinacci, and Montrucchio. Uncertainty averse
[6] S. Cerreia- Vioglio, F. Maccheroni, M. Marinacci, and L. Montrucchio. Ambiguity and robust statisties. Journal of Economic Theory, 148(3):974-049,, 2013.
[7] R. Christensen. Plane Answers to Complex Questions: The Theory of Linear Models. Springer, 2020.
[8] Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis. Wine Quality. UCI Machine Learning Repository, 2009. DOI:
[9] P. De Blasi, S. Favaro, A. Lijoi, R. H. Mena, I. Prünster, and M. Ruggiero. Are Gibbs-type priors the most natural generalization of the Dirichlet process? IEEE transactions on pattern analysis and machine intelligence, 37(2):212-25 2015.
[10] E. Delage and Y. Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595 612, 2010.
[11] C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378-1400) 2021.
[12] B. Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569-593. Springer, 1992.
[13] T. S. Ferguson. A Bayesian analysis of some nonparametric problems. The Annals of Statistics, pages 200,,30,110 1973.
[14] T. S. Ferguson. Prior distributions on spaces of probability measures. The Annals of Statistics, 2(4):615-629, 1974.
10

---

[15] R. S. Forsyth. Liver Disorders.
UCI
Machine Learning Repository, 1990. DOI:
htpps//dbboorg/102244452//C
[16] G. Garrigos and R. M. Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.1125: 2023.
[17] S. Ghosal and A. Van der Vaart. Fundamentals of nonparametric Bayesian inference, volume 44. Cambridge University Press, 2017.
[18] I. Gilboa and M. Marinacci. Ambiguity and the Bayesian paradigm. Readings in formal epistemology: Sourcebook, pages 385 385-439, 2016.
[19] I. Gilboa and D. Schmeidler. Maxmin expected utility with non-unique prior. Journal of mathematical economics, 18(2):111-13, 1989.
[20] A. Gnedin and J. Pitman. Exchangeable Gibbs partitions and Stirling triangles. Journal of Mathematical Sciences, 138(3)5667--5655 2006.
[21] A. E. Hoerl and R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
[22] J. F. C. Kingman. Poisson processes, volume 3. Clarendon Press, 1992.
[23] P. Klibanoff, M. Marinacci, and S. Mukerji. A smooth model of decision making under ambiguity. Econometrica, 13(6):1849-1892 2005.
[24] D. Kuhn, P. M. Esfahani, V. A. Nguyen, and Shatieezadeh-Aadchh Wasserstein distribution- ally robust optimization: Theory and applications in machine learning. In Operations research management science the age of analytics, pages 130-166. INFORMS, 2019.
[25] Lafferty, H. Liu, and L. Wasserman. Concentration of measure. Technical report, Carnegie Mellon University, 2010.
[26] A. Lijoi and I. Prünster. Models beyond the Dirichlet process. In N. L. Hjort, C. Holmes, Müller, and S. G. Walker, editors, Bayesian Nonparametrics, Cambridge Series in Statistical and Probabilistic Mathematics, page 80-136. Cambridge University Press, 2010.
[27] S. Lyddon, S. Walker, and C. C. Holmes. Nonparametric learning from Bayesian models with randomized objective functions. Advances in Neural Information Processing Systems, 31, 2018.
[28] S. Majumdar. On topological support Dirichlet prior. Statistics Probability Letters, 3555)385-388 1992.
[29] P. Mohajerin Esfahani and D. Kuhn. Data-drives distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming, (71(1-2):115-16 2018
[30] Muliere and L. Tardella. Approximating distributions of random functionals of Ferguson- Dirichlet priors. Canadian Journal of Statistics, 26(2):283-297, 1998.
[31] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2885 2011.
[32] M. Perman, J. Pitman, and M. Yor. Size-biased sampling of Poisson point processes and excursions. Probability Theory and Related Fields, 92(1):21-39, 1992.
[33] J. Pitman. Exchangeable and partially exchangeable random partitions. Probability theory and related fields, 102(2):145-1558 1995.
[34] Pitman. Some developments of the Blackwell- Blackwell-Maqqueen urn scheme. In T. Ferguson,
L. Shapley, and J. B. MacQueen, editors, Statistics, probability and game theory: Papers in honor of David Blackwell, volume 30 IMS Lecture Notes - Monograph Series, pages 245-267. Institute of Mathematical Statistics, 1996.

---

[35] H. Rahimian and S. Mehrotra. Frameworks and results in distributionally robust optimization. Open Journal of Mathematical Optimization, 3:1-85, 2022.
[36] E. Regazzini, A. Lijoi, and Prünster. Distributional results for means of normalized random measures with independent increments. The Annals of Statistics, 31(2):600-585 2003.
[37] L. J. Savage. The foundations of statistics. Courier Corporation, 1972.
[38] G. A. Seber and A. J. Lee. Linear regression analysis. John Wiley & Sons, 2003.
[39] Sethuraman. A constructive definition of Dirichlet priors. Statistica sinica, pages 639 650,
1994.
[40] A. Shapiro, E. Zhou, and Y. Lin. Bayesian distributionally robust optimization. SIAM Journal on Optimization, 1372):1279-1304, 2023.
[41] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58 58(1)2677-288 1996.
[42] A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[43] R. Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge University Press, 2018.
[44] M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
[45] S. Wang, H. Wang, and J. Honorio. Distributional robustness bounds generalization errors. arXiv preprint arXiv:2212.09966, 2022.
[46] W. Wiesemann, D. Kuhn, and B. Rustem. Robust Markov decision processes. Mathematics of Operations Research, 38(1):553-183 2013.
[47] D. Wu, H. Zhu, and E. Zhou. A Bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics. SIAM Journal on Optimization, 28(2):1588 1612, 2018
[48] E. Zhou and W. Xie. Simulation optimization when facing input uncertainty. In 2015 Winter Simulation Conference (WSC), pages 3714-3724 IEEE, 2015.
12

---

Supplement to "Bayesian Nonparametries Meets Data-Driven Distributionally Robust Optimization"
This Supplement to "Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimiza- tion" is organized follows. In Appendix we collect the proofs of all results presented in the main text. In Appendix [B] we provide further background on Dirichlet process representations and related posterior simulation algorithms. In Appendix C describe in detail gradient based optimization of our criterion and the experiments presented in the paper. Finally, in Appendix D we provide details on computational infrastructure.
A Technical Proofs and Further Results
Proof of Proposition 3.1 Let ~ denote weak convergence of probability measures. By Corollary 4.17 in T77. Qen Op. almost surely. That is,
lim
((p)Qe=(dp) f(p=)
Ps
almost surely for any bounded and continuous f :P + R. Thus, we are left to prove that p (0)) is bounded and continuous for all 0 O. is bounded because 6 is continuous the compact interval [0, K], and it continuous because p + (f) is continuous (by the definition of topology of weak convergence and because + h(6, {) is bounded and continuous) and & is continuous.
Proof of Lemma 3.2 First note that, by the stated assumptions, it follows from Taylor's theorem that
(R)( (Rpp. (0)) +' (R)) Rp..+ ("(cp.0)
[R -Rp_
2
for all p e Pg and 0 ④ and for some cp.a E []. Then
Φ' (Rp (0)) / [R((0) - RR. (0)]Qe- (dpp)
sup (0) @Rp. (0)) = sup
0E
0E6
+/=
+ /
o" ('(pp.)
R (0) - Rp. (0) 100 (dp)
2
P=
< Ma sup
a
Rep (0) +
R Rpen (0) Rp. (0)
RE6
+ n
a
(R) Ry. (0)
MR sup BEB
Q (dp)
sup (t(
P2
2
tE(0,K)
sup Rpee (0) Ry. (0) + a K K2
< M
n
sup 70)
a +
eEe
2 kE(O.K)
Proof of Theorem 3.3
Notice the following decomposition:
(RR, (0.)) ((Rr. (0.))
(4)
20
(RR, (0n)) Ve (0n) Ve-
Ven (0.) (0-) (0.) o(Rr.
00
< 2up Ve- (0) - 6(Rp. (0))).
92B
13

---

Then, Lemma 3.2 implies that, for all 6 > 0,
Plo p. (0,)) @ (0.)) _ 6]
2F P
sup
VVn (0) (Rp (0)) - 6/2
155
K²
P
M4
n
sup Renn (0) Ry. (0)l +
a
K +
sup
Tol1)] 58/2
a + BEB
a+ n
2
tE(O,K)
P
sup BEB
Ree (0) - RRp (0) 1
a + n
a
K
K
sup r6(() (0(KK)
)
2M
+
2
Proof of Theorem 3.4
Since
lim sup RRgg (0) Ry, (0) = n-000 0e8
almost surely and given assumptions (1) and (2) on On. by Lemma 3.2
we obtain
lim sup [Ve (0) 6n (R. (0)) 0 n-+00 BE0
almost surely. Then, by decomposition
lim 6n (Rp. (f,)) Φn (Rp. (0.)) 0
almost surely and almost surely, As consequence,
lim (6,) - Vg. = -0
n
lim (0.)
on (Rp. (0.))
< lim
[iee (On) (0.)) (0.) (R, (0.))
lim
(en Vie (0.) + sup V (0) on (Rp. (0))
n +00
8E
almost surely. Now recall assumption (3), i.e., the sequence (øn)n2) converges uniformly to the identity map. Then, in light of the previous observations and by noticing that
RRp. (0.) - RR. (0.) < (Rp. (0x) on (Rp. (9..)) + l (Rp. (0n)) - (Rp. (0.))
+ 1r (Rp. (0.)) (0.)l
and
VVe - Ry. (0-) I Ve- (0.) On (0-)) + I0n (Ry (0.)) (0.)).
the two desired almost sure limits follow:
lim Rp. (0n) Rp. (0-),
lim (0,) Ry. (0.).
Proof of Theorem 3.6
We have
Rp. (0.) < Rp. Espp. lim h(0m. {) lim Rp. (0x. Rp. (0.)
n
almost surely, where the first equality follows from the continuity of + h(.,E) and the second one from the Dominated Convergence Theorem. Then, (p) = Rp. (0.) almost surely, proving the result.
To Prove Lemma 4.3 immediately.
we introduce two other Lemmas. After proving those, Lemma follows
14

---

Lemma A.I. For all T,N 2 1,
T
a+n
sup
-V S M,K AEB
+sup AEB
(0. (0. (0, (0.
(5)
a
Proof. We have
sup Ven (0, T. N) - Vee #Ee
S sup AE
Ve (0, N) E Ve (0,
sup 0-6
(0, Ve- (0)
(6)
Note that
Epi Pr Φ Σ p,h(0,5) 5,) poh(0, (000..0)))
BIVE- (0, 1)]
=B Qe
Σ p,h(0,Ez) poh((0,5))
00
PEijz1D55e Ren
p,h(O,Eg) + poh((0,
Σ 3=T+1
pyh(O,E;)) (1)
j=l
}
= (0) ((0))BBy,za
B2 Aem
o' (co D, Po h(0,
Pj (0 Eg)
Po
where the last equality follows from the mean value theorem applied to endpoints and Σj=1phh(0,55 Eo) Σ5 Dy-T+1pph(0555 Then the second term in 6
Σ
f=1 P,, ((,,E,) E,)
is bounded by
T
T
MoKEy 2و2 P5de, Ben IPol
MaKE
[ Bx)
-McK
a a+n
a a+
The second term on the left-hand side of Equation instead of the form
Σ ((x)) El((X))]
sup g
N
==1
where X;
Ety) are iid random variables whose distribution determined by the truncated stick-brekking procedure.
The aim of the next Lemma is to provide sufficient conditions for finite sample bounds and asymptotic convergence of the term in 77.. Specifically, we impose complexity constraints the function
class H = (8 + h00, E) E b which allow us to obtain appropriate conditions on the derived class
-=MM0)))-0oh(0..)) (p) (p) G0 + & { EpMh(,s) + poh(0, (o) : :000
j=l
ensuring the asymptotic and non-asymptotic results seek.
Lemma A.2. Assume is a bounded subset of Rdd and + ((,0 is c(f)-Lipschitz continuous for all E. Then, for all T, N.N: and 0,
VE- (0, N) - EVV (0. (0.T,1) Se
sup dEG
with probability at least
d
(2M,Crdiam(O)v@
3 (6(() (K) +8)
3N
1
1 1-2
exp
+ exp
8
400(K))
for some constant Cr 0.
15

---

Proof. By the Lipschitz continuity assumption on h(0, E), we obtain that, for all (p).5 + (E))))((0,E))+++ po)( is ()(((13)))))0)))) -Lisschizz continuous, with
T
{(p),{)))0)) = Σ ps((S))
Indeed, for all 0,, 02 E O,
T
T
Σ py p,h(01.5)) + poh(01,.50)
Σ /=1
p,h((22:5) 5j) + poh(O2.
s M Σ p,,11(0 1(02, E,) + (ol1((11.0)) k0))
82ll-
Therefore, denoting by P the law of the vector (p), { and by N1 & (P)) the associated e-bracketing number of the class $, by Lemma 7.88 in [25] we obtain
4. (diammO)))l
Nile, $, (PC) &
with CT: = ((((())))))))))) Then the result follows by Theorem 7.86 in 25 after noticing that Su [| I sup b/ S ((K).
Proof of Theorem
We have
(0n) S (0.)
Ep-Ree
lim lim ((R,((,(T,N))
IN too T
lim lim Ven (T,
N- T-000
(B.)
almost surely, where the first two equalities follow from the continuity of 0 -> h(8, E) and ø as well as from iterated application of the Dominated Convergence Theorem (recall that h(0, 3) j0, by assumption for all 0 and {). This implies Ve (0,) Ve (0,)) almost surely.
In the next result, we will assume ((()= = <b((() 8 (x)( 33. Moreover, we will emphasize, through superscripts, the dependence of mathematical objects on /3 and the DP concentration parame- ter a. Moreover, if necessary, Nn dependent on the sample size n. Proposition
make the truncation threshold T, and the number of MC samples
A.3. Assume
is Var (0) almost surely for all 0 and
open subset of Rd and
lim lo supace ( " (0,T,N)
0. Moreover, assume that
1. A + h(E, 0) differentiable at 0, arg mingee Ry. for p. pl-almost every & E E, with gradient Vo. (E);
2. For all 02 and 02 in neighborhood of 0,, there exists a measurable function & + H(E) E Cg. such that ()((0. h(O2. E)I <. HE)] 02]1:
3. 0 + Rp. (9) admits second-order Taylor expansion at 9., with non-singular symmetric Hessian matrix Ve..
Then, with probability there exist sequences (Tn)n2 (Na)21 (Bn)21 (diverging to oc) and (O,, Joz1 (converging to 0), such that
Vn(6@mm.0n Sn (Tn, N₂) 0.) ~ V),,V),
V = Egmp. (V. Vo.
provided gam.. An (Tn, N,. 3 0, 00.
16

---

Proof. The imposed assumptions match those listed in Theorem 5.23 of [42]. The only condition left to prove is that there exist sequences (T,, 1n21. (Nn)a21, (3a)221, (diverging to oc) and (an))21 (converging to 0), such that Ree (0gm on (Tn, Nn)) inface Ry (0) S (n We do so by proving that, for any fixed n 2 1, Ryg (0%-0 (T) + inface Re (0) almost surely as + and B, I" N + oc; this implies that, for all n, with probability there exist On., Sr, Tn and N,, such that Ryg (0%.... (Tn, N.)) infoge Ryen (0) S En for any O. Moreover, it is easy to see that the result is implied by supage 1 (0,T, - Ry (0) + 0, so we prove the latter. We have
supl 2a. oe
(6.T.N) - Rpen (0)
sup ( -Ve (0)1 | + sup V (0) Rpe (0))
lee
#ge
where the first term converges 0 almost surely by assumption. Using second-order Taylor expansion of Φn (R) around Ryen (0), the second term, instead, satisfies
sup aee
V (0) Ryn (0) sup ME
(s(R)(()) (dp) Ryee (0)
K22
$ sup 858
(Rpe (0))
R2 (0) (dp) Rpe (8)
+ 2
sup (t)(
tE(0,K]
K2
sup k]
(t)
sup
Re (0) +
Rp (0) Rpen (0)
sup
(5((t) 0,
8 a + 7
a+1 n
2
tE(0,K]
+0
as a + 0 and B 00.
Remark A.4. Theorems 3.6 and 4.5 ensure that (a) for all n 2 1, provided 0, (T, .N converges almost surely to some On E O, the latter is minimizer of Ve- (0); and (b) if the above sequence (0x)221 converges almost surely to some 0, E O, the latter is a minimizer of Rp. (0). Notice that the assumptions required for these results are consistent with the ones of Proposition [A.3]
so they can be used to justify the condition @e..0nn (Tn, N₂) 2 0.. For instance, if one assumes almost sure uniqueness of minimizers and almost sure convergence of the above defined sequences, On (Tn, N,.) 2 0. can be guaranteed leveraging the preceding results.
Stochastic Gradient Descent Convergence Analysis. The following results refer to material presented in Appendix C below. For ease of exposition, we fix B and denote by E, the expectation operator conditional on the realization of the random index draws m! ..... m² iid Uniform( (1,...,M)))
Proposition A.5. Assume that V is convex and that (0:) follows Equation 9 for some starting value 6" E O and B = 1. Moreover, assume that, for all 0 E θ,
M 1
(0) Em)il² 2 oz.
Then
ET ET-I(V(0")) V(00) < 1000 8 + 02 =T mE
T 2I nt
where 0" E aarg minsee V(0), 0r = Σ 0,0 and Σx RM!
Proof of Proposition
A.5
Fix = 1.... T. We have.
|0++1
0-122 = =110* mel o o'(Hm (0') Em)) - 1²11
= ||0t 0 112 + m/ (m o (Hm, Voh()) )l?
(0' - 8¹) em 0 H (0') V
17

---

Applying the law of total expectation and the fact that Ln 6' (Hm . ) is unbiased for (oV( E,((0' em! (f") Smi)l E,(E4- 100 TT) e ml (H (('))) S.
= =Ee-11(0' - -0") VoV(('))
Hence
20p
0" 1-] 10
E.ll 1-
S. + nz. E, [1 em 0 n V vo , )1
- 1 0" ||21 E, 00 to
20
Summing over t = .., and since
E-11(0' 11 = -
because V is convex, have
T o
2
Σ nr (V < 100 11 t=
Σ
m
1=0
Dividing both sides by Σ t and exploiting (i) the linearity of the expectation operator, (ii) the convexity of the weights (14) and (iii) the convexity of V, the result follows.
Proof of Proposition 2.1 As for case 1, given the assumed form of Po and the criterion represen- left to establish an expression for Eg-po [)( Ee,.e)>w(..nn(( 0 x))].. Notice that -8,2, Therefore, Eg-pp (A(0,E) = Vly T z] + ||0]], which is easily seen to complete the proof. Finally, the proof for the LASSO case is completely analogous to the Ridge one and is therfore omitted.
tation 2).
id _N 68). independenddnnly y (0, 1), that y y- N 1+ 1013).
B Further Background on the Dirichlet Process and Approximation Algorithms
Since its definition by 13 based on the family of finite dimeeniaiaii Dirichlet distributions (as sketched in Section 2 the Dirichlet process has been characterized (and thus generalized) number of useful ways. For instance, the DP can be derived as a neutral to the right process [14),
normalized completely random measure 13 22
26
36 Gibbs-type prior (20 9., Pitman-
Yor Process 33 62), and a species sampling model 34 In what follows, we review two other
constructions of the DP which were at the basis of the approximate versions of the robust criterion Ve" proposed in Section
Steck-Breaking Construction of the Dirichlet Process. process enjoys the following brick-breaking" representation
proved that Ferguson's 1973 Dirichlet
p DP(a, P) p
where
Iz idd P, j= 1,
P1 = B1,
j-1
Pj = B, II
Bi, == 2.
1=1
B, ~ Beta(11 a),
j j= 1,2,..
18

---

Algorithm SBMC Approximation
Input: Data {", model parameters, number of MC samples N, truncation step T
for = 1 to N do
Set [ (1
Bx) =
for = to 7 do
Draw Eij a-nPh + a+n PE"
Draw Bi Beta a + n)
Set Pij = B, II1=1 (1 B&)
end for
Draw Ein ~ atn 1o + n atnPe"
a
Set Pro IIk=1 (1 Bx)
end for
Return: N E o Ej=o (t)
The name of the procedure comes from the analogy with breaking a stick of length into two pieces
of length B1 and 1 B,. then the second piece into two sub-pieces of length (1 ) B and (1 - (1) B2), and so on. In Algorithm 1 then, we simulate N realizations from Qe-, truncating the stick-breaking
procedure at step j 7. The remaining portion of the stick is then allocated to one further atom drawn from the predictive distribution. Then, the intractable integral with respect to the DP posterior is approximated via Monte Carlo average of the integrals (i.e., weighted sums) with respect to the N simulated measures.
Multinomial-Drrichl. Construction of the Dirichlet Process and Monte Carlo Algorithms. Another finite-dimensional approximation of p DP(a, P) is pr E Py0zy, with xj iid P and (pi..... .u.....PT) ~ Dirichlet(T;
As T xo, pr approaches p [see Theorem 4.19 in 17. Hence, one can approximate Vee (0) as in Algorithm 2 where the concentration parameter is a 7 and the centering distribution coincides with the predictive.
Algorithm 2 Multinomial-Drrcclee Monte Carlo (MDMC) Approximation
Input: Data E", model parameters, number of MC samples N, approximation threshold T for i = 1 to N do Initialize w: E IRT," E TT
for j 1 to T do
Update w; wi) ~ Gamma | 1) Update (i) P Po a-n P end for
Normalize wi Σ/= w.())
w.
end for Return: N EN N
o0 (w. ((,EE))
When a is negligible compared to the sample size n. one can simplify posterior simulation by setting O. Thus, one obtains DP P&=)) posterior. This distribution enjoys useful representation
p E PrE., with p; Dirichlet(re; I,... 1) [see 17 Section 4.7]. Due to its similarity to the usual bootstrap procedure [2]. this distribution is known as the "Bayesian bootstrap". Algorithm [3] iimplements the Bayesian bootstrap to approximate the criterion Ve- (0). In practice, however, we do not recommend resorting to the Bayesian bootstrap approximation, since DP assigns probability
as follows: p DP(n, )
the set of distributions with strictly positive support on e.. This goes against the prescription that, as the finite sample ş" provides only partial information on the true underlying distribution, the statistical DM should be willing to consider wider set of distributions other than the ones supported at the sample realizations.
19

---

Algorithm 3 Bayesian Bootstrap Monte Carlo (BBMC) Approximation
Input: Data E", model parameters, number of MC samples N for to N do Initialize w; E IR"
j 1 to n do
Update wi( Gamma(1, 1)
end for Normalize w Σラ w.(
end for Return: N E o (w)) (0)
C Numerical Optimization and Experiment Details
In this Section, we first describe the SGD algorithm used in practice for our experiments. Then, we describe in full detail the experiments presented in the paper.
Gradient-Bard Based Optimization. Whether resort to the SBMC the MDMC approximation of Vg" (0), we are faced with the task of minimizing a criterion of the form
V(0)
Σ Pog (ogh(0
N
1=1
The smoothness and convexity of make it appealing minimize the criterion via gradient-based convex optimization techniques. Indeed, it is enough to assume that +> h(0, {) is convex and differentiable (a standard assumption met in many applications of interest) to easily yield the same properties for V.
In light of this, assuming that 0 + h(0,E) is differentiable at every Eij and denoting H.(0) Ej Psyh(O, E.)). the gradient of
M Σ en (Hm VoV N (HH)))V)HH) = M m==1
(8)
where Lm Tpm and the m-indexxng is just recoding of the indices (with slight abuse of notation and M = N.).. That is, the gradient of (0) can written the average of M terms. Thus, to minimize (()) we propose mini-hatch Stochastic Gradient Descent algorithm which, at each iteration f, updates the parameter vector as follows:
B
1
0++1
0²
1lt B
Σ
Emg D'n (Hm, Voh (m,))
(9)
for a step-size > 0 and a random subset (mini-batch) of size B from the indices (1...., M). Under standard regularity assumptions [16), in Proposition A.5 (Appendix of the algorithm at usual rates for convex problems.
A we prove convergence
Remark C.I. Expression (8) provides some insight on how, in practice, distributional robustness is enforced. Notice that H, H, Rp, (0) is the expected risk computed according to p;. an approximate realization from Qg Thus, in the computation of the overall gradient V, (O)). the gradients associated to the p. that generate higher expected risks receive more weight (being ó convex, o' is increasing). These weights, then, reflected into which gradients, in the mini-batch SGD algorithm, are given more leverage updating the parameter vector. Thus, the procedure can be thought as implementing "soft worst-case scenario" scheme, whereby distributions in the posterior support weighted (in terms of gradient influence) more the worse they do in terms of expected risk.
Mini-Batch Stochastic Gradient Descent Algorithm. For practical optimization, we apply modification to the SGD algorithm provided in Equation ( which helps to reduce the computational
20

---

burden of the procedure. Indeed, recall the formula of the gradient of the criterion V that we need to optimize:
1 N N
VOV(0)
o' (H.(O))V,H(())
O (H,
M
Clearly, then, implementing the baseline SGD algorithm requires, each iteration, the evaluation of multiple Hm (0') terms, each consisting of I evaluations of the loss function h. To avoid this, each iteration instead sub-sample one index i = ....... .N and update the parameter vector The latter is still an unbiased estimator of
according to the associated gradient of (0, V the overall gradient of V(0).. but it requires only T evaluations of h (plus those I gradients of h. similarly to the baseline algorithm). Finally, to exploit the whole data efficiently, we sub-sample without replacement and perform multiple passes over the N MC samples. Algorithm summarizes the procedure.
Algorithm 4 Modified Stochastic Gradient Descent Algorithm
Input: Approximate criterion parameters {Piij Eay) i 1...., .N. = 1...., T), step size schedule (2200 starting value 08, number of passes P, iteration tracker / = for p to do
Initialize /= (1, NI,...,N}
for j = N do
Sample uniformly i i
Update 0c+1 0¹
11- &1 Piellp Ex)) Et Voh(O'
Update == 1 {i}
Update =
end for end for Return: PPN+1
C.1 High-Dimensional Linear Regression Experiment
Setting. In this experiment, we test the performance of our robust criterion in a high-dimensional sparse linear regression task. The high-dimensional and sparse nature of the data-generating process is expected to induce distributional uncertainty, and our method is meant to address this. In this context, we use the quadratic loss function v. + 10 10-3(y 0 pTz)², where the 10 factor serves to stabilize numerical values in the optimization process. Notice that, by the form of the ambiguity-neutral criterion (2). the multiplicative factor on the loss function does not change the equivalence with Ridge.
Data-Genee Gata-Generating Process. The data for the experiment are generated iid across simulations (200) and observations (n = 100 per simulation) as follows. For each observation = I,..., n, the d-dimensional (d 90) covariate vector follows a multivariate normal distribution with mean 0 and such that (i) each covariate has unitary variance, and (ii) any pair of distinct covariates has covariance 0.3:
0.3
0.3
Zn1
0.3
1
0.3
I;
~ NO E), Σ
E IR Iddd
Fid
00.3
0.3
Then, the response has conditional distribution yel:; (1, 1,,,,,,, 0)T Rd and a 0.5. That is, out of 90 covariates, only the first 5 have a unitary positive marginal effect on y/, and additive Gaussian noise added to the resulting linear combination. Together with 100 training samples, at each simulation we generate 5000 test samples on which we compute oft-of-sample RMSE for the ambiguity-avenn averse, ambiguity- and OL.S procedures.
NaaT 2i, 02, with
21

---

3
E315
1331
Mean
1283
54, Doy
66 Sa8
030
Figure 2: Simulation results for the high-dimensional sparse linear regression experiment. Bars report the mean and standard deviation (across 200 sample simulations) of the test RMSE, L2 distance of estimated coefficient vector 0 from the data-generating one, and the L2 norm of Ó. Results are shown for the ambiguity-averse ambiguity-neutral, and OLS procedures. Note: The left (blue) axis refers to mean values, the right (orange) axis to standard deviation values.
Robust Criterion Parameters. For each simulated sample, we run our robust procedure setting the following parameter values: <(t) exp(tt3)) 8, B E {1, oo), = a/n for E {1, 5, 10), and Po (O, I). where the 8 oo setting corresponds Ridge regression with regularization parameter a (see Proposition 2.1) Finally, we run 300 Monte Carlo simulations to approximate the 50.
criterion, and truncate the Saaiioomal--Diichhiee approximation
Stochastic Gradient Descent Parameters We initialize the algorithm at = (o....,0) 0) and set the step size at 7/ = 50/(100 + vt). The number of passes over data is set after visual inspection of convergence of the criterion value. The time per SGD run is less than second on infrastructure (see Appendix
C.2 Experiment on Gaussian Location Estimation With Outliers
Setting. In this experiment, test the performance of robust criterion on the task of estimating a univariate Gaussian mean (assuming the variance is known) when the data is corrupted by a few observations coming from distant distribution. Clearly, this is situation where a considerable level distributional uncertainty is warranted. In this setting, the loss function h(5. 9) = (E 0)² is simply the negative log-like likelihood associated to the normal model. Notice that the h is convex in and, as in the previous experiment, we pre-mmn pre-multpply it by factor of 10-3 for numerical stability reasons.
Gana-Genn Process. The data for the experiment are generated iid across simulations (100) and observations (n 13 per simulation) as follows. For each simulation, 10 iid samples z; are drawn from (0, 1) distribution (the actual data-generating process we want to learn) and samples are drawn iid from a N/ 5) outlier distribution. At each simulation we also generate 5000 test samples from the data- seaaeeeeeeaiing process (0, 1), on which we compute the out-of-sample average negative log- log-likelihood for the ambiguity-averss ambiguity-neut
and Maximum Likelihood Estimation (MLE) procedures this will be our measure out of sample performance (see Figure
Robust Criterion Parameters. For each simulated sample, we run our robust procedure setting the following parameter values: ((t) 8 exxp(t/3) 3, E [1, oc),
(1,2,5, 10). and Po = N (Wn,, I), where μ4 = (10- 0 3 /10 + 3) is weighted average of the data-generating and the outlier means. By the expression of the ambiguity- criterion it is easy show that
the 3 = oo case leads to the parameter estimate
0 +
0
V/s
a
+
71
1=1
22

---

2-2 0.5 d+10
135 Bey
03500
A.W080 7.407M
Figure 3: Simulation results from the experiment on Gaussian mean estimation with outliers. Bars report the mean and standard deviation (across 100 sample simulations) of the test mean negative log-likelihood and the absolute value distance the estimated parameter from (the data-generating value). Results shown for the ambiguity-avee The left (blue) axis refers to mean values, the right (orange) axis to standard deviation values.
ambiguity-neunt neutral, and MLE procedures. Note:
with Vi x; for I,..., and y/ Mu for = I,...., + That is, the ambiguity-neutee procedure with concentration parameter a E N is equivalent to the MLE procedure when the original training sample is enlarged with additional observations equal Mu- Finally, run 300 Monte Carlo simulations to approximate the criterion, and truncate the Aaatinmmial-Drrichhe approximation T 50.
Stochastic Gradient Descent Parameters. We initialize the algorithm at 0 = O and set the step size at 1/: 20/(100- + vt) The number of passes over data is set after visual inspection convergence of the criterion value. The time per SGD run is seconds our infrastructure (see Appendix DD.
Results. In Figure present the results of the simulation study. As for the regression experiment, ambiguity-avers: criterion brings improvement, across a values and compared to the ambiguity- neutral and the simple MLE procedures, both in terms of average performance and in terms of the latter's variabiliy (see the first row of the Figure). From the second row of Figure 3 it also emerges that, average, the ambiguity-averse procedure more accurate estimating the location parameter than the two other methods. Compared to the simple MLE procedure, the variability the estimated parameter is also significantly smaller. Taken together, these results confirm the theoretical expectation that the ambiguity-ave -vverse optimization is effective at hedging against the distributional uncertainty arising in the estimation of corrupted data such the simulated ones.
C.3 High-Dimensional Logistic Regression Experiment
Setting. In this experiment, we test the performance of our robust criterion on high-dimensionall sparse classification task using the framework of logistic regression. As in the linear regression experiment, the digg--iinnsional and sparse nature of the data-genee getaaaeenerainng process is expected to induce distributional uncertainty, and our method is meant address this. In this setting, the loss function is h(g.0) = log(1 + exp(-y- x 0)). As in the previous experiment, we pre-multiply it by a factor 10 for numerical stability reasons.
GenerGenn Process. The data for the experiment are generated iid across simulations (200) and observations (n = 100 per simulation) as follows. For each observation = 1,... n,, the d d-dimensional (d 90) covariate vector follows multivariate normal distribution with mean 0 and such that (i) each covariate has unitary variance, and (ii) any pair of distinct covariates has covariance 0.3:
0.3
0.37
0.3
1
0.3
~ NO E),
==
E Iddxd
Fid
0.3
0.3
Then, the response has conditional distribution yl ~ Binary([1, -1).p2))))) with = 1/(1 + exp(-x" a)) and = 11,1,1,1,1,0, 0) E Rd. That is, out of 90 covariates, only the first have a unitary positive marginal effect on the log-odds. Together with 100 training samples, each
23

---

4
400
0301
6000
4.00
1203
12.00
1283 Moa
54, Doy
A009
45
33.00
1283
12.00
Figure 4: Simulation results for the high-dimensional sparse logistic regression experiment. Bars report the mean and standard deviation (across 200 sample simulations) of the test average loss, L2 distance of estimated coefficient vector Ô from the data-generating one, and the L2 norm of Ô. Results shown for the ambiguity-averse, Ingrregularized, and un-regularized procedures. Note: The left (blue) axis refers to mean values, the right (orange) axis to standard deviation values.
simulation we generate 5000 samples on which we compute the out-of-sample average loss for the ambiguity-averse, Le-regularized (with regularization parameter a, see below), and un-regularized procedures.
Robust Criterion Parameters. For each simulated sample, the following parameter values: (6t) = 3 cxp(t/3) - 3, 1, = a/n n for a E {1,2, 2, 5, 10), and Po Binary( (1, -1,, 0.5) x (O, I). Finally, run 200 Monte Carlo simulations approximate the criterion, and truncate the Multinomial-Drriche approximation at T 50.
robust procedure setting
a
Stochastic Gradient Descent Parameters We initialize the algorithm at = (O...., 0) and set the step size at т) = 1000/(100) (100 + vB). The number passes over data is set after visual inspection of convergence of the criterion value. The run time per SGD run is 3 seconds our infrastructure (see Appendix
Results. In Figure 3 we present the results the simulation study. As for the regression ex- periment, the ambiguity-averse criterion brings improvement, across values and compared to Læ-regularized and the unregularized procedures, both terms of average performance and in terms of the latter's variabiliy (see the first row of the Figure). From the second row of Figure B also emerges that, on average, the ambiguity-averse procedure is more accurate and less variable at estimating the true regression coefficient than the two other methods. Also, our method is able to more effectively shrink the norm the coefficient vector towards (see the third row). Taken together, these results confirm the theoretical expectation that the ambiguity-ave effective at hedging against the distributional uncertainty arising in nigh-dimensionll classification problems (in this experimental setting, tackled via logistic regression).
optimization is
C.4 Pima Indian Diabetes Dataset Experiment
In this experiment, we use logistic regression for classification on the popular Pima Indians Diabetes dataset [2] collecting data on 768 women belonging to Native American group that lives Mexico and Arizona. The data consists of binary outcome (whether the subject developed diabetes not) and 8 features related to her physical condition (these features are standardized before running the analysis).
To test our method, we randomly select 300 training observations and leave out the rest for test sample. Then, we randomly split the training data into 15 folds of size 20 and select, via k-fold cross
Made available by the National Institute of Diabetes and Digestive and Kidney Diseases and downloaded from sottts://wwue kaggle. httpg//....cccceeeaaaaaaaaaaaaaa

---

validation, the optimal DP concentration parameter a over a wide grid of values. We do the same for the L1 -penltty coefficient used implement regularized logistic regression with the Python library scikit-learn B1. Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, L.1 ---penalized logistic regression, and unregularized logistic regression each of the folds This splitting procedure allows us (i) to test and compare the performance of our method in a setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures. The run time per SGD run 19 seconds on our infrastructure (see Appendix D
Table Trpports the results from the described procedure. Unregularized logistic regression performs quite poorly compared to the other two methods. Instead, the latter yield results in the same orders of magnitude both in terms of average performance and of performance variability, though our DP robust method features almost half of the variability produced by I Leggeaulrre logistic regression.
Unregularized
L1 Regularized
DP Robust
Average
0.0142
0.0007
0.0006
Standard Deviation
0.0127
6.2253c-05
3.9742e-05
Table 1: Comparison of average and standard deviation of the out-of-sample performance (out-of- sample expected logistic loss) of the three employed methods for binary classification on the Pima Indian Diabetes dataset.
C.5 Wine Quality Dataset Experiment
In this experiment, we applied linear regression to the popular UCI Machine Learning Repository Wine Quality dataset [8] Data consists of 4898 measurements
wines' characteristics and quality score assigned each wine. The aim is to predict the latter based the former (both features and response are standardized before running the analysis). We implement linear regression using our DP-based robust method (with the squared loss function), OLS, and LASSO (the last two methods are implemented using scikit-learn 31).
To test our method, we randomly select 300 training observations and leave out the rest for test sample. Then, we randomly split the training data into 10 folds size 30 and select, via k-fold cross validation, the optimal DP concentration parameter over wide grid of values. We do the same for the L1 -i-eenalty coefficient used to implement LASSO. Once the optimal parameters are selected based out-of -sttpll risk, again randomly split the training sample into the same number of folds, and implement our roubust DP method, LASSO regression, and OLS estimation on each of the folds. This splitting procedure allows (i) to test and compare the performance of our method in setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures. The run time per SGD run is seconds infrastructure (see Appendix D).
Table 2 reports the results from the described procedure, whose interpretation is very much in line with the results of the previous experiment.
Unregularized
La Regularized
DP Robust
Average
0.0014
0,0009
0.0009
Standard Deviation
0.0004
8.0192e-05
6.0076e-05
Table 2: Comparison of average and standard deviation of the oft-of-sample performance (out-of-
sample expected squared loss) of the three employed methods for linear regression on the Wine Quality dataset.
13 All of the implementation details (e.g., parameter values), can be found in our code. This holds for the next experiments well.
25

---

C.6 Liver Disorders Dataset Experiment
In this experiment, we applied linear regression to the popular UCI Machine Learning Repository Liver Disorders dataset Data consists of 345 measurements of blood test results and the number of drinks consumed per day by each subject. The aim predict the latter based on the former (both features and response are standardized before running the analysis). We implement linear regression using our DP-based robust method (with the squared loss function), OLS, and LASSO (the last two methods are implemented using scikit-lears (31).
To test our method, we randomly select 200 training observations and leave the for test sample. Then, we randomly split the training data into 10 folds of size 20 and select, via k-fold cross validation, the optimal DP concentration parameter over wide grid of values. We do the same for the L1 Li-penalty coefficient used to implement LASSO. Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, LASSO regression, and OLS estimation on each of the folds. This splitting procedure allows us (i) to test and compare the performance of our method in setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures. The run time per SGD run is 15 seconds on our infrastructure (see Appendix D
Table reports the results from the described procedure, whose interpretation very much in line with the results of the previous two experiments.
Unregularized
Ln Regularized
DP Robust
Average
0.0012
0.0009
0.0007
Standard Deviation
0.0005
0.0001
6.6597e-05
Table 3: Comparison of average and standard deviation of the out-of-sample performance (out-of- sample expected squared loss) of the three employed methods for linear regression the Liver Disorders dataset.
D Computational Infrastructure
All experiments were performed desktop with 12th Gen Intel(R) Core(TM) 19-12900H, 2500 Mhz, 14 Core(s), 20 Logical Processor(s) and 32.0 GB RAM.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper justifies the claims made in the abstract with either theoretical
results, or experimental evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please, refer Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
Please, refer to Sections 3 for theoretical propositions and the underlying assumptions, and Appendix for proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please, refer to Appendix C for experiment details and to our supplementary material in the form of code and a README file with instructions on how to run the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Please, refer the code included in the submitted supplementary material. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please, refer to Appendix C and the submitted code in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Variability in the reported average performance metries are of direct interest for the evaluation of our method, so we report measures of variability as separate bars in plots or separate entries tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please, refer to Appendices C and D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::/nneripspp.
Answer: [Yes]
Justification:
We have read the Code of Ethics and confirm the adherence of our paper to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Due to its theoretical and optimization-focuuaa nature, the paper has no direct foresceable socictal impact beyond that of any generic machine learning paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper only employs simulated or small publicly available datasets, as well as well-established machine learning techniques.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper properly credits all data and software used.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "d7Ngqwf2bW", "reviewer": "Reviewer_DqoX", "summary": "The paper proposes a Bayesian method for optimizing stochastic objectives under a given finite sample. More precisely, The paper assumes the Dirichlet Process for data generation and then it proposes to optimize the mean of the stochastic objective over posterior distributions given the sample. The paper proves asymptotic properties of the proposed objective. The experimental results in the appendix show that the proposed method works better than the standard L2 regularization or no regularization on linear regression tasks over simulation and real data sets.", "questions": "Is it difficult to obtain finite-sample error bound like PAC or statistical learning theory? (or any related work?)", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem is quite relevant. The paper proves basic asymptotic desirable properties. The experimental results show the superiority of the proposed method to the standard L2 regularization-based methods.", "weaknesses": "One of my concerns is the proposed method's high computational cost. However, I don’t think it is critical. When the sample size is large enough, the standard ERM would work well. The proposed method, though it shows asymptotic convergence properties, would work better when the sample size is small. On the other hand, it would need more technical improvement when the sample space is high-dimensional.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent reviewing our work and for the insightful comments, which we will incorporate in the next version of our paper by emphasizing the points brought up by the reviewer.\n\nSpecifically, we would first like to address the weakness, kindly pointed out by the reviewer, related to the computational cost of the method. We agree with the observation that the method is best suited for small-to-moderate sample sizes, as is true for any Distributionally Robust Optimization (DRO) method. In fact, distributional uncertainty is likely to affect optimization only when a relatively small sample does not allow to accurately approximate the data generating process. However, as the sample size increases, distributional uncertainty becomes less of an issue and standard ERM methods suffice for accurate parameter estimation and stable predictive performance. Nevertheless, while this is an important observation for the practical usefulness of any DRO method, we believe that sample size is not per se a computational problem for our method. Indeed, looking at Equation (3) in the paper, it is clear how in practice the criterion is estimated by cheaply sampling from the DP predictive and appropriate weight distributions. Therefore, while a larger sample size might require a higher value of $T$ (truncation threshold) and/or $N$ (number of MC samples) to make the criterion more representative of the data set (e.g., ensure that each data point is sampled at least once with high probability), gradient computations are easily vectorized and lead to very efficient mini-batch SGD updates akin to plain ERM methods. The same holds true for the dimensionality of the data and/or parameter space, which affects computational cost in the same way as it does for standard ERM.\n\nSecondly, we’d like to address the reviewer’s question about the possibility of obtaining finite-sample performance guarantees. We point out that this is precisely the goal of Theorem 3.3, which relates finite sample bounds on the predictive performance parameter estimator $\\theta_n$ to the classical $\\sup$ distance between the ERM and true risk. The latter, in turn, allows us to obtain finite-sample probabilistic bounds via conditions on the complexity (VC dimension, metric entropy, etc.) of the loss function class, as done in standard learning theory. As we point out in the paragraph after Theorem 3.3, our paper does not discuss any specific result of this kind as they are (1) standard in the literature and (2) can be immediately plugged in from classical textbook formulations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V9b8HLohmR", "reviewer": "Reviewer_uUEe", "summary": "This paper proposes a novel robust optimization criterion for training machine learning and statistical models by combining Bayesian nonparametric theory and smooth ambiguity-averse preferences, addressing distributional uncertainty to improve out-of-sample performance. The authors demonstrate theoretical guarantees for their method and show its practical implementation and effectiveness through tasks using simulated and real datasets.", "questions": "NO.", "limitations": "Discussed in Section 6.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theoretical aspects of this paper are not my research area, so I am not equipped to assess the strengths and weaknesses of the theoretical contributions. However, the proposed method and its practical implications appear to be well-founded and promising.", "weaknesses": "The theoretical aspects of this paper are not my research area, so I am not equipped to assess the strengths and weaknesses of the theoretical contributions. However, the proposed method and its practical implications appear to be well-founded and promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and time spent reviewing our work. If any further questions should come up during the next phases of the reviewing process, we will be happy to engage in further discussion with the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FjVm4anUan", "reviewer": "Reviewer_7wnT", "summary": "This paper introduces a new robust risk criterion that integrates concepts from Bayesian nonparametric theory, specifically the Dirichlet process, and a recent decision-theoretic model of smooth ambiguity-averse preferences. They show the relationships between their criterion and traditional regularization methods for empirical risk minimization, including Ridge and LASSO regressions. They also provide theoretical gurantee for the robust optimization and propose tractable approximations of the criterion.", "questions": "* I have a question about the coefficient in the prior. What will happen if we adopt the empirical Bayesian idea into your algorithm, i.e. estimate the coefficient in the prior through data? Will this cause degradation of the model？", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a solid paper that makes a good and exciting theoretical contribution. The idea of incorporating Dirichlet prior into distributionally robust is novel, and well-suited for the problem. The empirical section is adequate.", "weaknesses": "*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent reviewing our work and for the insightful comment, which we will incorporate in the next version of our paper by emphasizing its main point.\n\nSpecifically, we believe that the reviewer makes a good point that, at first sight, it could be desirable to estimate the concentration parameter $\\alpha$ from the data using empirical Bayes techniques. In principle, this solution could be viable by exploiting the exchangeable partition probability function (EPPF) of the Dirichlet Process, which can be read as the likelihood of observing the data clusters seen in the samples, given the parameter $\\alpha$. Then, one could maximize the log-EPPF (available in closed form,  see Equation 2.19 in [1]) and obtain an empirical estimate of $\\alpha$. However, unless the data contain ties (i.e., the data-generating distribution is assumed to have a discrete component, which is often unlikely in practical applications), this will lead to a degenerate $\\alpha = +\\infty$ solution – this is because the data is trivially partitioned into $n$ clusters, each with one component. One could further try to solve this issue by first applying some off-the-shelf clustering method to the data set, then maximizing the resulting “approximate EPPF”. While this option is interesting, it requires one further layer of computation, which adds to the overall cost of the optimization procedure. Instead, for practical purposes, a simple yet efficient approach might be to select the parameter value based on classical out-of-sample validation strategies. While this last solution is less grounded in the theory of Dirichlet Processes, we believe that it will be more beneficial in practice, as it is more aligned with the explicit aim of optimization-based machine learning methods that aim to maximize predictive performance.\n\n\n*[1] Pitman, J. (2006). Combinatorial stochastic processes: Ecole d'eté de probabilités de Saint-Flour XXXII-2002. Springer.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rao60c9ZjT", "reviewer": "Reviewer_Vivx", "summary": "The authors use Dirichelet process and a smooth ambiguity aversion model to approximate the solution of risk minimization problems. They demonstate the consistency of their approach, along with finite sample guarantees. They additionally give a practical means of applying their procedure and apply the procedure to a number of simulated and real problems.", "questions": "In theorem 3.4 it is assume that phi_n converges uniformly to the identity function. To me this seems like assuming that curvature of phi_n tends to 0 indictating that as n goes to infinity there is no ambiguity aversion. However, uniform convergence to the identity function is much a much stonger condition than curvature going to 0. For example convergence to 2 times the identity function still has curvature going to 0. Am I wrong in my reasoning behind convergence to the identity function? If not, is it possible to update assumption (3) in theorem 3.4 to a more general curvature to 0 assumption? It seems like the without much modification it can be instead assumed that phi_n converges to some scalar multiple of the idenity function.\n\n\nHow do the assumptions on theorem 3.3,3.4 guarantee that theta_n and theta_* exist? As is isn't assume that h is continuous or Theta is compact it seems like the argmin of V and R can be empty. From the proof of theorem 3.3 it seems like if theta_* doesn't exist it can be replaced by any arbitrary theta.\n\nAnother question. The uniform boundedness of h is strongly relied on to get the generate the results. Is it possible/are there methods available to remove this assumption? Classic problems like least squares regression over an unbounded domain will have h unbounded. This often isn't all that bad as assuming compactness of the domain is fine, but the bounds given here depend strongly on the size of the compact set used to restrict the problem to.", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Background was very informative and enlightening. Despite being very technical in nature the paper was easy to follow. Based on the references given, the idea seems quite novel. Proofs related to results in section 3 were quite easy to follow (section 4 and appendix proofs not checked)", "weaknesses": "There are many small typos. \n - In Lemma 3.2 the supremum of \\gamma^* is not needed as gamma^* does not depend on t. The statement in the proof is the supremum over gamma(t). \n- For consistency with lemma 3.2, in theorem 3.3 the supremum over gamma(t) should probably be replaced by gamma^* (or gamma(t) should be added back into lemma 3.2).\n - In line 181 I don't think the (ii) should be there. \n\nThere are some minor problems.\n - The introduction of the dependency of phi on n makes a small appearence. This idea is very interesting but I don't think it is emphasised enough. Just a couple of sentences explaining why you'd want to do such a thing would be nice.\n - In line 221 a form of phi is assumed. I assume this is supposed to be phi_n and not phi. If it is supposed to be phi I'm unsure what form is being assumed, and if it is supposed to be phi_n it doesn't seem like it is assumed as section 4 seems to use phi.\n\nAnd there are some reasonably major (but easily fixable) problems. \n- On first read it is confusing how exactly the distance between the theoretic risk and the risk computed WRT to p_0 enters into lemma 3.2. Seeing the full statement I understand why the full bound isn't included in the main text but perhaps this discussion can be reworded. \n- It is unclear if theorem 4.4 is using the same assumptions as lemma 4.3, or just the form of C_T that is given in lemma 4.3.\n- I think theorem 3.3 needs to assume that M_phi < infinity as it is moved to the otherside of an equation in the proof. I think the proof needs more discussion about the case of gamma^*=infinity as on first read it looks like it moved to another side of the equation in the proof. gamma^*=infinity seems fine as thne the bound is triival. These cases are possible as taking K=1 and phi(t) = (t-1)^(1/3) has M_phi = gamma^*=infinity.  Seems like everything can be infinite in lemma 3.2 as then the bound is trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent reviewing our work and for the insightful comments, which we will incorporate in the next version of our paper by fixing the typos/mistakes pointed out by the reviewer and by emphasizing the points brought up in the review.\n\nAs for the minor problems pointed out by the reviewer, we’d like to address them as follows:\n* Better explanation of the dependence of $\\phi$ on $n$: The paper includes a relatively careful explanation of this choice and its interpretation, as collected in Remark 3.7. We will however try to emphasize it more in the next version of the paper.\n* Specific form of $\\phi$ in line 221: The reviewer spotted a typo ($\\phi$ instead of the correct version $\\phi_n$) which creates confusion. We thank them for pointing this out and we will fix it in the next version of the paper. As for our omission of the dependence on $n$ also in the subsequent sections, we have made this choice because such sections deal with approximation and gradient optimization of the criterion for a _fixed_ sample size $n$. In contrast, the dependence of $\\phi$ on $n$ is only relevant as $n$ varies to ensure proper convergence results.\n\nAs for the more serious yet fixable problems pointed out by the reviewer, we’d like to address them as follows:\n* The reviewer states that “On first read it is confusing how exactly the distance between the theoretic risk and the risk computed WRT to p_0 enters into lemma 3.2.” We agree with the comment and move footnote 8 to the main body, so to make this relation more evident.\n* The reviewer states that “It is unclear if theorem 4.4 is using the same assumptions as lemma 4.3, or just the form of C_T that is given in lemma 4.3.” We thank the reviewer for pointing this out, which gives us the opportunity to clarify the result. The exact form of $C_T$ can be found in the proof of Lemma A2 on page 23. In Theorem 4.4 we maintain the same assumptions as in Theorem 4.3, with the additional requirement that $C_T$ (defined as above) is bounded above as $T$ varies. We will make sure to clarify these points in the next version of the paper.\n* As for the boundedness of $M_\\phi$ and $\\gamma_\\phi^*$, the reviewer is right in pointing out that our treatment silently assumes both quantities to be finite (as it is the case with the exponential functional form for $\\phi$ proposed in the paper). Because in practice these assumptions are easily satisfied, we will explicitly mention them in the next version of the paper.\n\nAs for the questions asked by the reviewer, we would like to address them as follows:\n* The reviewer asks whether assuming the uniform convergence of $\\phi_n$ to the identity is necessary, or whether it can be relaxed with convergence to some multiple of the identity. We point out that, for the sake of optimization, any affine transformation of $\\phi_n$ would yield the same optimized parameter. Hence, assuming that $\\phi_n$ converges uniformly to the identity is without loss of generality (and, in one form or the other, needed for the proof of Theorem 3.4), because $\\phi_n$ can always be normalized and centered to obtain the desired convergence. Notice that this is exactly the reason why we choose $\\phi_n(t) = \\beta_n \\exp(t/\\beta_n) - \\beta_n$ as a special functional form, even though $\\phi_n(t) = \\exp(t/\\beta_n)$ would be equivalent (optimization-wise) for all $n$.\n* The reviewer asks whether the existence of theoretical and empirical minimizers is ensured by our assumptions. We thank the reviewer for pointing out this lack of clarity, which we will resolve in the next version of the paper. The existence of minimizers is not a consequence of our assumptions on $h$ or $\\Theta$, but rather an assumption itself – we assume the optimization problem to be well posed. In practice, many loss functions (e.g., convex) will easily ensure the existence of some optimizer (both theoretical and for our criterion, which is convex if the $h$ is convex).\n* The reviewer asks about the necessity of $h$ being uniformly bounded for our results. This is a good point, which we are currently tackling in follow-up research on the method we propose in this paper. Indeed, we believe that, specializing $h$ to a well-behaved loss like the quadratic loss used in linear regression, might yield finite sample and asymptotic guarantees even in the unbounded case (e.g., under appropriate tail conditions for the data-generating process $p_\\star$). Being this paper our first, foundational contribution to this topic, we kept our analysis as general as possible in terms of loss function choice, which resulted in the need for stronger assumptions in our theoretical analysis. Nevertheless, as mentioned, the really good point made by the reviewer is currently being investigated, and we hope to have new results in that direction soon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jWlXRovbtU", "reviewer": "Reviewer_p57R", "summary": "This article proposes a new method for optimisation of risk under uncertainty, using Dirichlet Processes to introduce an extra degree of robustness to modelling uncertainty on the data generating process. The method is generally applicable to many statistical learning problems through the use of loss functions, and further links are made to the economic decision-making literature. The theoretical properties of the methods are explored, and a Monte Carlo approximation is introduced to make tractable the inference for the DP model. Experiments are performed on three simulated experiments, and three real data sets.", "questions": "Line 110: “In our setting, instead of a parametric prior on the regression coefficients, we place a nonparametric one on the joint distribution of the response and covariates” What is lost by not having priors associated with parameters? This is not necessarily a good thing to move away from: specific belief distributions for specific interpretable model components is a good thing. What the implications for estimating the posterior uncertainty on the parameter level, for example in the regression case? What kind of posterior convergence do you expect on the parameters themselves?\n\nWhat are the more general implications of putting a prior on the distributions of the data instead of the model parameters? This is suggested to be the case in line 111. Is that not what the likelihood is meant to capture? Is this merely sloppy phrasing in line 111  Does the prior on the distribution of the data come into conflict with the model at some point? What is the difference between altering the prior you have assumed here and altering the statistical model?\n\nWhat is the expected scalability of the methods to larger data sets? None of those used here are really that big, but the use of SGD implies that you have big-N scalability in mind. The wall clock times on your fairly modest desktop system are quite small, so presumably you have some ability to scale to larger data sets if desired.\n\nIs the trained DP object interpretable at all? Assuming that the learned DP object implies some sort of partition of the data-generating process, then is that partition telling us something we didn’t know beforehand? How restrictive is the DP prior on the space of measures that you might expect in reality?", "limitations": "While an elegant choice for the problem at hand, the use of a Dirichlet Process is likely to be a limiting factor for future directions of research, both in terms of the computational limits and the possible allowable partitions.\n\nThe assertion of the prior in the way performed here, in contrast to over interpretable parameters, possibly limits the likelihood of this method being widely adopted.\n\nThe theoretical results are focussed on convergence to the correct expected risk (or transformations thereof), rather than telling us anything about the parameters themselves.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This article represents a novel contribution to a very general problem, with the use of a neat mathematical representation of a statistical decision maker’s uncertainty in the form of a Dirichlet Process. The relevant theory behind the method is explored thoroughly, and there are several different applications explored. The quality of the scientific writing is fairly high. The links made with the economic decision making literature are interesting, helping elucidate the underlying point about ambiguity aversion.", "weaknesses": "The idea of placing a prior on the data generating process instead of the parameters is unusual from the traditional Bayesian perspective. While this is not necessarily wrong, the authors do not do a particularly good job of communicating the implications of constructing a prior in this way, or how existing intuitions that the readership may have can be transferred to this new approach.\n\nThe results discussion from line 299 onwards has not been prioritised for space in the manuscript: the authors attempt to concentrate reporting results from six different experiments into a single page, which is not enough exposition for the main text of the article. Many fairly important details are pushed into an Appendix and then covered by asserting the “superior ability of our robust method” in the main text. This is far too simple an analysis for an empirical investigation: I don’t trust any analysis that can be summed up so concisely, let alone six combined, with three being real data sets. Appendix C is reasonably thorough in this respect, although there is a noticeable absence of any rigorous testing of the differences in performance metrics.\n\nThere are a few small issues with the English in the article: line 8: “among which Ridge…”, line 97: “pervasive” is a strange choice of vocabulary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent reviewing our work and for the insightful comments, which we will incorporate in the next version of our paper by emphasizing the points brought up in the review.\n\nFirst, we would like to address the weaknesses pointed out by the reviewer as follows (we defer discussion of the first weakness to the questions section below):\n* We agree with the reviewer that the results of our experiments are analyzed extremely quickly at the end of the paper. Our choice was dictated by the strict page limit and the predominantly methodological nature of our contribution. However, we will make sure to extend the discussion in the next version of the paper. As for lack of uncertainty quantification for performance metrics, we note how every plot and table reporting empirical results includes standard deviations (across sample realization or data folds) on top of means. This reporting style is due to the fact that variability is of direct interest to our analysis (our method stabilizes out-of-sample performance), but it can be readily used to obtain, for instance, standard normal confidence intervals for the corresponding mean values (that is, mean value $\\pm$ standard deviation * critical value).\n* We thank the reviewer for pointing out the language issues, which we will address in the next version of the paper.\n\nSecond, we would like to address the reviewer’s questions as follows:\n* The reviewer asks for clarifications on the unusual switch between placing priors on parameters versus placing priors on the data generating process (this also echoes the first weakness mentioned by the reviewer). We’d like to point out that our starting point is data-driven optimization, where parameters of interest are learnt by minimizing some empirical approximation of an expected loss function. The Bayesian component of our methodology lies not in how we infer parameters, but in how we choose to approximate the expectation around the loss function: Instead of directly taking an empirical average, we average out the generating process using a DP posterior. Adopting this perspective, then, also allows to introduce ambiguity aversion via the convex transformation $\\phi$. Hence, our method is better understood as using _Bayesian ideas_ to improve upon optimization-based procedures. This allows to gain some key robustness properties and analytical tractability, while retaining the computational scalability of optimization-based learning that traditional Bayesian methods often lack due to burdensome inference on the whole posterior. A good example of this is our Proposition 2.1 (also mentioned by the reviewer), which uncovers a new Bayesian interpretation of Ridge and Lasso. In fact, using traditional Bayesian methods, it is well known that Ridge and Lasso are equivalent to maximum-a-posteriori estimators based on Gaussian and Laplacian priors on the linear regression coefficient vector. Our procedure is instead purely based on optimization of the expected loss function, where the generating process is averaged out via a DP posterior with appropriately chosen centering measure. Of course, as for most black-box optimization procedures, one might argue that uncertainty quantification is harder than with traditional Bayesian methods, and this also holds for our method (however, see Proposition A.3 in the Appendix for an asymptotic normality result). This comes as no surprise, as it is in line with a common trade-off between purely Bayesian methods and optimization-centric ones, which enjoy higher scalability at the cost of harder uncertainty quantification. To conclude, we are grateful to the reviewer for raising these important conceptual points, which we will clarify further in the next version of the paper.\n* The reviewer asks about scalability. The method promises to be scalable to large datasets due to the criterion form seen in Equation (3), which lends itself to easy differentiation (under differentiability assumptions for $h$, which are independent of our method) and stochastic gradient optimization. Of course, as the data size increases, it is necessary to increase the parameter $T$ (truncation threshold) and/or $N$ (MC samples) to make the criterion representative of the sample (e.g., to ensure that each observation appears in the sample at least once), but linear scaling of either parameter suffices for that purpose.\n* The reviewer asks about the interpretability and restrictiveness of the criterion, e.g. in terms of data partitions. We highlight that this latter aspect is not applicable to our setting, as the data is modeled via a DP indirectly through a posterior expectation, but learning happens through standard optimization over parameters. Hence, the usual DP clustering is absent from this framework. In terms of restrictiveness, the nonparametric DP posterior choice ensures full support over the space of data-generating measures, but still preserves tractability due to its posterior and predictive characterizations. Notice that more general choices could be considered (e.g., the PY process), but tractability would be hindered and not much would be gained, for instance, in the presence of continuous data.\n\nFinally, we’d like to address the last limitation mentioned by the reviewer:\n* The reviewer indicates that our theoretical analysis focuses on the predictive risk but not on parameter estimation. Preliminarily, we would like to point out that Theorem 3.6 indeed deals with asymptotic convergence of the estimated parameter to its true counterpart. However, we thank the reviewer for the comment and agree with them that finite-sample analysis is still lacking for parameter estimation. This is due to the fact that, to obtain results in this direction, it is most likely necessary to specialize to particular cases of the loss function $h$ (e.g., quadratic loss for linear regression), which is the object of some of our ongoing research on the general method proposed in this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the challenge of optimizing machine learning models under distributional uncertainty. The authors propose a novel method that integrates Bayesian nonparametric theory, specifically the Dirichlet Process (DP), with distributionally robust optimization (DRO). Finite-sample probabilistic guarantee on the excess risk is derived. The asymptotic vanishing of the excess risk and the convergence of the finite-sample optimal value are established. The paper's theoretical contributions are considered solid, with the proofs being clear and technically sound (see the review and comments by Reviewers 7wnT and Vivx). However, the experimental results, while promising, are briefly discussed in the main text (see the review and comments by Reviewer p57R). Some other concerns include: computational expense, especially for high-dimensional data or large sample sizes (see the review and comments by Reviewer DqoX); could better explain the implications of placing a prior on the data-generating process rather than on model parameters (see the review and comments by Reviewer p57R); minor typos (see the review and comments by Reviewer Vivx).
Decision: Accept (poster)