Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization

Authors: nicola  bariletto
nhat  ho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 06:59:08

Abstract: Training machine learning and statistical models often involves optimizing a data-driven risk criterion. The risk is usually computed with respect to the empirical data distribution, but this may result in poor and unstable out-of-sample performance due to distributional uncertainty. In the spirit of distributionally robust optimization, we propose a novel robust criterion by combining insights from Bayesian nonparametric (i.e., Dirichlet process) theory and a recent decision-theoretic model of smooth ambiguity-averse preferences. First, we highlight novel connections with standard regularized empirical risk minimization techniques, among which Ridge and LASSO regressions. Then, we theoretically demonstrate the existence of favorable finite-sample and asymptotic statistical guarantees on the performance of the robust optimization procedure. For practical implementation, we propose and study tractable approximations of the criterion based on well-known Dirichlet process representations. We also show that the smoothness of the criterion naturally leads to standard gradient-based numerical optimization. Finally, we provide insights into the workings of our method by applying it to a variety of tasks based on simulated and real datasets.

Main paper: 
 introduction: 
 In machine learning and statistics applications, several quantities of interest solve the optimization problem min θ∈Θ R p * (θ),where R p (θ) := E ξ∼p [h(θ, ξ)] is the expected risk associated to decision θ, under cost function h(θ, ξ) (measurable in the argument ξ) and given that the distribution of the (Ξ, B(Ξ))-valued data ξ is p. 1 For instance, if we are dealing with a supervised learning task where ξ = (x, y) ∈ R m-1 × R, h(θ, ξ) is usually a loss function ℓ(f θ (x), y) quantifying the cost incurred in predicting y with f θ (x) -here the decision variable is θ, which parametrizes the function f θ : R m-1 → R. For the rest of the paper, we assume Ξ ⊆ R m , Θ ⊆ R d and h : Θ × Ξ → [0, K] for some K < ∞. 2In most cases of interest the true data-generating process p * is unknown, and only a sample ξ n = (ξ 1 , . . . , ξ n ) from it is available. The most popular solution is to approximate p * by the empirical distribution p ξ n , and optimize R p ξ n (θ) ≡ n -1 n i=1 h(θ, ξ i ). However, especially for small sample sizes n and complex data-generating mechanisms, this can result in poor out-of-sample performance, leading to the need for robust alternatives. A flourishing literature on Distributionally Robust Optimization (DRO) has provided several methods in that direction [though not always with datadriven applications as the primary focus; see 35, for a recent exhaustive review of the field]. A prominent approach is the min-max DRO (mM-DRO) one, whereby a worst-case criterion over an ambiguity 3 set of plausible distributions is minimized [19, 2, 3, 10, 46, 11] . Recent notable results involve the study of mM-DRO problems where the ambiguity set is defined as a Wasserstein ball of probability measures centered at the empirical distribution [29, 24] .Contribution. Differently from the mM-DRO paradigm, we propose a distributionally robust procedure based on the minimization of the following criterion:EQUATIONwhere ϕ : [0, K] → R + is a continuous, convex and strictly increasing function, and Q ξ n is a Dirichlet process posterior conditional on ξ n [13] .As we show below, our proposal brings together insights from two well-established strands of literature -decision theory under ambiguity and Bayesian nonparametric statistics, -contributing in a novel way to the field of data-driven distributionally robust optimization. As we establish throughout the article, among the key advantages of the criterion are: (i) its favorable statistical properties in terms of probabilistic finite-sample and asymptotic performance guarantees; (ii) the availability of tractable approximations that are easy to optimize using standard gradient-based methods; and (iii) its ability to both improve and stabilize the out-of-sample performance of standard learning methods.The rest of the paper is organized as follows. In Section 2, we motivate the formulation in Equation ( 1) by providing a concise overview of decision theory under ambiguity and its connections to Bayesian statistics and regularization. In Section 3, we study the statistical properties of procedures based on V ξ n . In Section 4, we propose and study tractable approximations for V ξ n based on the theory of DP representations. In Section 5, we highlight the robustness properties of our method by applying it to a variety of learning tasks based on real and simulated data. Section 6 concludes the article. Proofs of theoretical results and further background are provided in Appendices A and B, respectively, while in Appendix C we discuss how the smoothness of the proposed criterion yields straightforward gradient-based optimization, and present more details on the numerical experiments. Code to replicate our experiments can be found at the folllowing link: https://github.com/nbariletto/BNP_ for_DRO. 
decision theory and bayesian statistics: 
 Following a long-standing tradition in Bayesian statistics and decision theory [37] , the distributional uncertainty on the data-generating process p * can be dealt with by defining a prior Q over it. We point out that this Bayesian approach contrasts with the classical one, where a prior would typically be placed directly on the parameter θ, whose data-driven optimal value would be determined as a function (e.g., the mean or mode) of the resulting posterior distribution. In this new framework, instead, the parameter θ is treated as a variable to be optimized, while the prior is assigned to the entire data-generating process. This perspective allows us to incorporate several valuable Bayesian concepts, as we will clarify throughout the paper, while preserving the flexibility of the original optimization-based learning framework. Notably, θ does not need to be interpreted as the parameter of a full generative model, as would be required in a classical Bayesian setting. Instead, it can represent a vector of parameters associated with a generic, possibly complex loss function, such as those employed in modern deep learning architectures.Specifically, our Bayesian approach is equivalent to modeling the observed data ξ n = (ξ 1 , . . . , ξ n ) as exchangeable with de Finetti measure Q:ξ i | p iid ∼ p, i = 1, . . . , n, p ∼ Q.Due to the stochasticity of p, R p (θ) is itself a random variable, and a sensible procedure is to maximize its posterior expectation. Let Q ξ n be the posterior law of p conditional on the sample ξ n . Then, one solves the following problem:min θ∈Θ E p∼Q ξ n [R p (θ)] = min θ∈Θ PΞ Ξ h(θ, ξ)p(dξ)Q ξ n (dp) = min θ∈Θ E ξ∼p(•|ξ n ) [h(θ, ξ)],where P Ξ denotes the space of probability measures on B(Ξ) endowed with the Borel σ-algebra B(P Ξ ) generated by the topology of weak convergence, while p(dξ|ξ n ) := PΞ p(dξ)Q ξ n (dp) denotes the posterior predictive distribution. In sum, within this general Bayesian framework, the data-driven problem reduces to minimizing h(θ, ξ) averaged w.r.t. the posterior predictive distribution, i.e., min θ∈Θ R p(•|ξ n ) (θ).A natural choice is to model the prior Q as a Dirichlet process (DP), and Q ξ n is then a DP posterior. First proposed by [13] , the DP is the cornerstone nonparametric prior over spaces of probability measures. Its specification involves a concentration parameter α > 0 and a centering probability measure p 0 . Intuitively, the DP is characterized by the following finite-dimensional distributions: p ∼ DP(α, p 0 ) implies (p(A 1 ), . . . , p(A k )) ∼ Dirichlet(αp 0 (A 1 ), . . . , αp 0 (A k )) for any finite measurable partition {A 1 , . . . , A k } of Ξ. 4 A key property of the DP is its almost sure discreteness, which allows to write p d = j≥1 p j δ ξj (where probability weights and atom locations are independent). Moreover, the DP is conjugate with respect to exchangeable sampling. In our case, this meansp ∼ DP(α, p 0 ) ⇒ Q ξ n = DP α + n, α α + n p 0 + n α + n p ξ n .That is, conditional on the sample ξ n , p is again a DP with larger concentration parameter α + n and centered at the predictive distribution p(•|ξ n ) := α α+n p 0 + n α+n p ξ n . The latter is a compromise between the prior guess p 0 and the empirical distribution p ξ n , and the balance between the two is determined by the relative size of α and n. The predictive distribution is also related to the celebrated Blackwell-MacQueen Pólya urn scheme (or Chinese restaurant process) to draw an exchangeable sequence (ξ i ) i≥1 distributed according to p ∼ DP(α, p 0 ): Draw ξ 1 ∼ p 0 and, for all i > 1 and ℓ < i, set ξ i = ξ ℓ with probability (α + j -1) -1 , else (i.e., with probability α(α + j -1) -1 ) draw ξ i ∼ p 0 [4] .Given the large support of Q ξ n , which consists of all probability measures whose support is included in that of p(•|ξ n ) [28] , the DP is a reasonable and tractable option to mitigate misspecification concerns. Then, leveraging the mentioned expression for the DP predictive distribution, the problem specializes toEQUATION[see also 27, 45] . In practice, adopting the above Bayesian approach amounts to introducing a regularization term depending on the prior centering distribution p 0 . Compared to the simple empirical risk R p ξ n (θ), this type of criterion displays lower variance (because R p0 (θ) is non-random) at the cost of some additional, asymptotically-vanishing bias w.r.t. the theoretical criterion R p * (θ). We also note that such bias can be attenuated in finite samples as long as the prior guess p 0 and the true data-generating process p * are close enough in terms of the difference |R p0 (θ) -R p * (θ)|. 5Connections to Regularization in Linear Regression. One of the most ubiquitous data-driven learning tasks is linear regression [38, 7] . It is well-known that, in this setting, coefficient estimation (e.g., via maximum likelihood or least squares) can be framed as a minimization problem of the sample average of the squared loss function. It turns out that, applying the Bayesian regularized approach (2), an interesting equivalence with standard regularization techniques such as Ridge [21] and LASSO [41] emerges. Proposition 2.1. Let h(θ, (y, x)) = (y -θ ⊤ x) 2 . Then, denoting λ α,n := α/n, the following equivalences hold:1. If p 0 = N (0, I), then θ solving (2) implies that it solvesmin θ∈Θ 1 n n i=1 h(θ, ξ i ) + λ α,n ∥θ∥ 2 2 ; 2. If V = diag(|θ 1 | -1 , . . . , |θ d-1 | -1) and p 0 = N (0, V ), then θ solving (2) implies that it solvesmin θ∈Θ 1 n n i=1 h(θ, ξ i ) + λ α,n ∥θ∥ 1 .Proposition 2.1 is insightful because it highlights a novel Bayesian interpretation of Ridge and LASSO linear regression. In fact, it is well known that both methods are equivalent to maximuma-posteriori estimation of regression coefficients when the latter are assigned either a normal or a Laplace prior. In our setting, instead of a parametric prior on the regression coefficients, we place a nonparametric one on the joint distribution of the response and the covariates. The degree of regularization, then, is naturally guided by the prior confidence parameter α and the sample size n.We also note that sparsity is only one of the possible data-generating features one might want to enforce in regularized estimation, 6 and the nonparametric Bayesian approach offers greater flexibility, compared to Ridge and LASSO, to incorporate such patterns by specifying the prior expectation p 0 of the joint response-covariate distribution.As we just showed, adopting a traditional Bayesian framework, uncertainty about the model p is resolved by using the posterior Q ξ n to directly average p out. This procedure, however, does not take into account the (partly) subjective nature of the beliefs encoded in Q ξ n , and the aversion to this that a statistical decision maker (DM) might have. In fact, the result of the procedure is that the DM ends up minimizing the expected risk, where the average is taken according to the predictive distribution.In practice, then, the latter is put on the same footing as an objectively known probability distribution, such as the true model.This issue has been thoroughly studied and addressed in the economic decision theory literature [18, 6] . In that context, the economic DM faces an analogous expected utility maximization problem max θ∈Θ E ξ∼p [u(θ, ξ)] (e.g., to allocate her capital to a portfolio of investments subject to random economic shocks ξ). However, she does not possess enough objective information to pick one single model of the world p, but deems a larger set of models plausible. One possibility, then, is that the DM forms a second-order belief (e.g., a prior Q) over such set, and resolves uncertainty by directly averaging expected utility profilesE ξ∼p [u(θ, ξ)] w.r.t. p ∼ Q.Just like in our data-driven problem, however, direct averaging does not account for ambiguity aversion. [23] proposed and axiomatized a tractable "Smooth Ambiguity Aversion" (SmAA) model, whereby second-order averaging is preceded by a deterministic transformation ϕ inducing uncertainty aversion via its curvature: The DM optimizes PΞ ϕ(E ξ∼p [u(θ, ξ)])Q(dp), and criterion (1) simply specializes the SmAA model to the data-driven case. 7 When optimization takes the form of minimization, ambiguity aversion is driven by the degree of convexity of ϕ. 8In particular, convexity encodes the DM's tendency to pick decisions that yield less variable expected loss levels across ambiguous probability models. To see this intuitively, examine the simple case when only two models, p 1 and p 2 , are supported by Q = 1 2 δ p1 + 1 2 δ p2 . Consider two decisions θ 1 and θ 2 that, under p 1 and p 2 , yield the expected risks marked on the horizontal axis of Figure 1 Graphically, this is because the orange line connecting ϕ(R p1 (θ 1 )) to ϕ(R p2 (θ 1 )) lies (point-wise) below the line connectingEQUATIONϕ(R p1 (θ 2 )) to ϕ(R p2 (θ 2 )). ϕ(R p (θ 1 ))Q(dp) < ϕ(R p (θ 2 ))Q(dp).That is, although θ 1 and θ 2 yield the same loss in Q-expectation, the ambiguity-averse criterion favors θ 1 because it ensures less variability across uncertain distributions p 1 and p 2 .Interestingly, [5] showed that the SmAA model belongs to a general class of ambiguity-averse preferences, which admit a common utility function representation. For SmAA preferences with ϕ(t) = β exp(β -1 t)-β (with β > 0 and under additional technical assumptions), this representation implies the equivalence of problem (1) with min θ∈Θ maxP :P ≪Q ξ n E p∼P [R p (θ)] -βKL(P ∥Q ξ n ) ,where KL(•∥•) is the Kullback-Leibler divergence and ≪ denotes absolute continuity. The above result further clarifies the mechanism through which distributional robustness is induced: Intuitively, instead of directly averaging over p ∼ Q ξ n , one computes a worst-case scenario w.r.t. the mixing measure, penalizing distributions that are further away from the posterior -the latter acts as a reference probability measure. Moreover, in the limiting case β → 0, the mM-DRO setup is recovered, with ambiguity set C = {p ∈ P Ξ : ∃P ≪ Q ξ n , p = PΞ qP (dq)}. In the other limiting case β → ∞ (with the convention 0 • ∞ = 0), the ambiguity neutral Bayesian criterion (2) is instead recovered. 
statistical properties: 
 In this section, we analyze the statistical properties of the criterion V ξ n (θ), as a function of the sample size n. A first issue of interest, addressed in Proposition 3.1, is to study its asymptotic point-wise behavior (cf. [17] , Corollary 4.17). Proposition 3.1. Let ξ n be iid according to p * and ξ → h(θ, ξ) continuous for all θ ∈ Θ. Then, for all θ ∈ Θ, limn→∞ V ξ n (θ) = ϕ(R p * (θ))almost surely.This ensures that, as more data are collected, the proposed criterion approaches, with probability 1, the true theoretical risk (up to the strictly increasing transformation ϕ).While point-wise convergence to the target ground truth is a first desirable property for any sensible criterion, it is not enough to characterize the behavior of the optimization's out-of-sample performance, nor the closeness of the optimal criterion value and the criterion optimizer(s) to their theoretical counterparts. In the following subsections, we study these properties both in the finite-sample regime and in the asymptotic limit n → ∞.Finite-Sample Guarantees. Denoteθ n ∈ arg min θ∈Θ V ξ n (θ), θ * ∈ arg min θ∈Θ R p * (θ),and we assume the above sets of minimizers to be non-empty throughout the article. In finite-sample analysis, a first question of interest is whether probabilistic performance guarantees hold for the robust criterion optimizer θ n . In our setting, one can naturally measure performance by the narrowness of the gap between R p * (θ n ) and R p * (θ * ). As we clarify later, Lemma 3.2 is a first step towards establishing this type of guarantees. Lemma 3.2. Let ϕ be twice continuously differentiable on (0, K), with M ϕ := sup t∈(0,K) ϕ ′ (t) < +∞ and γ * ϕ := sup t∈(0,K) γ ϕ (t) < +∞, where γ ϕ (t) := ϕ ′′ (t)/ϕ ′ (t) ≥ 0. Thensup θ∈Θ |V ξ n (θ) -ϕ(R p * (θ))| ≤ M ϕ n α + n sup θ∈Θ |R p ξ n (θ) -R p * (θ)| + α α + n K + K 2 2 γ * ϕ .Lemma 3.2 links the sup distance of the criterion V ξ n from the theoretical risk to three key objects: 3. The Arrow-Pratt coefficient γ ϕ (t) of absolute ambiguity aversion. In the economic theory literature on decision-making under risk, this is a well-known concept measuring the degree of risk aversion of decision makers, with point-wise larger values of γ ϕ corresponding to more risk aversion. See [23] for a discussion on the straightforward adaptation of this measure to the ambiguity (rather than risk) aversion setup we work in.Most importantly, Lemma 3.2 allows us to prove the following Theorem, which yields the performance guarantees we are after. Theorem 3.3. For all δ > 0P[ϕ(R p * (θ n )) -ϕ(R p * (θ * )) ≤ δ] ≥ P sup θ∈Θ R p ξ n (θ) -R p * (θ) ≤ α + n n δ 2M ϕ - α α + n K - K 2 2 γ * ϕ .Theorem 3.3 allows to obtain finite-sample probabilistic guarantees on the excess riskϕ(R p * (θ n )) - ϕ(R p * (θ * )) via bounds on sup θ∈Θ R p ξ n (θ) -R p * (θ). The latter is a well-studied quantity, and the sought bounds follow from standard results relying on conditions on the complexity of the function class H := {h(θ, •) : θ ∈ Θ}, as measured by its Vapnik-Chervonenkis dimension, metric entropy, etc. We refer the reader to [44, 43] for a systematic treatment of the topic and specific useful results.Asymptotic Guarantees. So far, we have studied the finite-sample behavior of the out-of-sample performance of θ n . Another closely related type of results deals with the asymptotic limit of such performance, as well as with the convergence of optimum criterion values and optimizing parameters to their ground-truth counterparts. In this Subsection, attention is turned to theoretical results of this kind.Finite-sample guarantees on sup θ∈Θ R p ξ n (θ) -R p * (θ) are usually of the formP sup θ∈Θ R p ξ n (θ) -R p * (θ) ≤ δ ≥ 1 -η n , with ∞ n=1 η n < ∞.This implies (via a straightforward application of the first Borel-Cantelli Lemma) the almost sure vanishing of sup θ∈Θ R p ξ n (θ) -R p * (θ) . Thus, we include this as an assumption of the next Theorem. Moreover, we introduce a functional dependence of ϕ on n, and denote ϕ ≡ ϕ n accordingly. Theorem 3.4. Retain the assumptions of Lemma 3.2 and lim n→∞ sup θ∈Θ R p ξ n (θ) -R p * (θ) = 0 almost surely. Moreover, assume that ϕ n satisfies (1) lim n→∞ γ * ϕn = 0, (2) sup n≥1 M ϕn < ∞, and (3) lim n→∞ sup t∈[0,K] |ϕ n (t) -t| = 0. Then the next two almost sure limits hold:lim n→∞ R p * (θ n ) = R p * (θ * ), lim n→∞ V ξ n (θ n ) = R p * (θ * ).Theorem 3.4 is crucial because it ensures that, asymptotically, the excess risk vanishes and the finite-sample optimal value converges to the optimal value under the data generating process. 9Remark 3.5. From a design point of view, the type of n-dependent parametrization of ϕ required in Theorem 3.4 is sensible, as it is equivalent to adopting vanishing levels of ambiguity aversion (uniformly vanishing Arrow-Pratt coefficient) as the sample size grows -that is, as one obtains a more and more precise estimate of the true distribution p * . Moreover, this assumption is in the spirit of the condition imposed on the radius of the Wasserstein ambiguity ball in [29] , which is required to vanish as the sample size grows. Finally, it is easy to show that ϕ n (t) = β n exp(β -1 n t) -β n , for positive β n → ∞, satisfies the conditions of Theorem 3.4, and from now on we silently assume this form for ϕ n .Finally, we leverage the above results to ensure the convergence of the sequence of optimizers (θ n ) n≥1 to a theoretical optimizer. Theorem 3.6. Let θ → h(θ, ξ) be continuous for all ξ ∈ Ξ and lim n→∞ R p * (θ n ) = R p * (θ * ) almost surely (e.g., as ensured in Theorem 3.4). Then, almost surely,lim n→∞ θ n = θ implies R p * ( θ) = R p * (θ * ).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: Please, refer to the code included in the submitted supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
monte carlo approximation: 
 In what follows, we fix a sample ξ n and propose simulation strategies to estimate V ξ n (θ). The latter, in fact, is analytically intractable due to the infinite dimensionality of Q ξ n . To that end, we exploit a key representation of DPs first established by [39] : If p ∼ DP(η, q), then p d = ∞ j=1 p j ξ j , where ξ j iid ∼ q and the sequence of weights (p j ) j≥1 is constructed via a stick-breaking procedure based on iid Beta(1, η) samples (see Appendix B). Thus, for large enough integers T and N , we propose the following Stick-Breaking Monte Carlo (SBMC) approximation for V ξ n (θ):EQUATIONwhere, T denotes the number of stick-breaking steps performed before truncating each Monte Carlo sample from the DP posterior Q ξ n , while N denotes the number of such samples. Algorithm 1 in Appendix B details the procedure, which essentially approximates the posterior DP via truncation and takes expectations accordingly. Remark 4.1. We propose to truncate the stick-breaking procedure at some fixed step T . Another strategy would involve truncating it at a random step T i (ε) := min t ∈ N : t j=1 p ij ≤ ε for some small ε > 0. This allows to directly control the approximation error at each Monte Carlo sample [30, 1] , though it leads to simulated measures with supports of different cardinalities. For the sake of theory, we opt for the fixed-step/random-error approximation, though the random-step/fixed-error one is equally viable in practice. Remark 4.2. On top of being a theory-based approximation for V ξ n (θ), the criterion (3) can be interpreted as implementing a form of robust Bayesian bootstrap. Instead of directly averaging the risk h(θ, •) with respect to the empirical distribution, we first obtain N bootstrap samples of size T from the predictive (which is a compromise between the empirical and the prior centering distributions), we weight observations according to the stick-breaking procedure, and finally take a grand average of the ϕ-transformed weighted sums. This connection with the Bayesian bootstrap suggests the following alternative Multinomial-Dirichlet Monte Carlo (MDMC) version of V ξ n (θ):1 N N i=1 ϕ T j=1 w ij h(θ, ξ ij ) ,where (w i1 , . . . , w iT ) iid ∼ Dirichlet T ; α+n T , . . . , α+n T and the atoms are iid according to the predictive (see Algorithm 2 in Appendix B). 10 For practical computation, we recommend using the MDMC approximation, as it tends to yield more balanced weights, compared to SBMC, even for low values of T .With the following results, we ensure finite-sample and asymptotic guarantees on the closeness of optimization procedures based on the SBMC approximation versus the targetV ξ n . Lemma 4.3. Assume Θ is a bounded subset of R d and, for all ξ ∈ Ξ, θ → h(θ, ξ) is c(ξ)-Lipschitz continuous. Then, for all T, N ≥ 1 and ε > 0, sup θ∈Θ Vξ n (θ, T, N ) -V ξ n (θ)] ≤ M ϕ K × α + n α + n + 1 T + ε with probability at least 1 -2 32M ϕ C T diam(Θ) √ d ε d × exp - 3N ε 2 4ϕ(K)(6ϕ(K) + ε) + exp - 3N ε 40ϕ(K) for some constant C T > 0.Heuristically, the bound in Lemma 4.3 is obtained by decomposing the left-hand side of the inequality into a first term depending on the truncation error induced by the threshold T , and a second term reflecting the Monte Carlo error related to N . Moreover, analogously to Lemma 3.2, Lemma 4.3 easily implies finite-sample bounds on the excess "robust risk" V ξ n ( θn (T, N )) -V ξ n (θ n ), where θn (T, N ) ∈ arg min θ∈Θ Vξ n (θ, T, N ). Another consequence is the following asymptotic convergence Theorem, whose proof is analogous to that of Theorem 3.4. Theorem 4.4. Under the same assumptions of Lemma 4.3, and if sup T ≥1 C T < ∞ (see Appendix A for details on C T ), limT,N →∞ sup θ∈Θ Vξ n (θ, T, N ) -V ξ n (θ)] = 0 almost surely. Also, almost surely lim T,N →∞ Vξ n ( θn (T, N ), T, N ) = V ξ n (θ n ), lim T,N →∞ V ξ n ( θn (T, N )) = V ξ n (θ n ).In words, Theorem 4.4 ensures that, as the truncation and MC approximation errors vanish, the optimal approximate criterion value converges to the optimal exact one, and that the exact criterion value at any approximate optimizer converges to the exact optimal value. Also note that Theorems 3.4 and 4.4, when combined, provide guarantees on the convergence of Vξ n ( θn (T, N ), T, N ) (the empirical criterion one has optimized in practice) to R p * (θ * ) (the theoretical optimal target) as the sample size increases and the DP approximation improves.Finally, as a byproduct of Theorem 4.4, convergence of any approximate robust optimizer to an exact one is established as follows. 11Theorem 4.5. Let θ → h(θ, ξ) be continuous for all ξ ∈ Ξ. Moreover, assume limT,N →∞ V ξ n ( θn (T, N )) = V ξ n (θ n )almost surely (e.g., as ensured above). Then, almost surely, lim T,N →∞ θn (T, N ) = θn impliesV ξ n ( θn ) = V ξ n (θ n ). 
experiments: 
 We applied our robust optimization procedure to a host of simulated and real datasets, and we report results in this Section. Before proceeding, we notice that, given the finite approximations proposed in Section 4 and under mild regularity assumptions on the loss function h, the proposed robust criterion is amenable to standard gradient-based optimization procedures (see Appendix C for further details and an insightful interpretation of the gradient of our criterion as yielding robustly weighted stochastic gradient descent steps).Simulation Studies. We tested our method on three different learning tasks featuring a high degree of distributional uncertainty in the data generating process, and compared performance to the corresponding ambiguity neutral (i.e., simply regularized) and unregularized procedures. First, we performed a high-dimensional sparse linear regression simulation experiment. We simulated 200 independent samples of size n = 100 from a linear model with d = 90 features (moderately correlated with each other), only the first s = 5 of which have unitary positive marginal effect on the scalar response y. Second, we performed a simulation experiment on univariate Gaussian mean estimation in the presence of outliers. We simulated 200 independent samples of 13 observations, 10 of which come from a 0-mean Gaussian distribution and 3 from an outlier distribution, and tested the ability of the three methods to recover the true mean (i.e., 0). Third, we performed a simulation experiment on high-dimensional sparse logistic regression for binary classification. We set up a data-generating mechanism similar to the linear regression experiment, where a small subset of features linearly influence the log odds-ratio.Appendix C collects further details on the above experiments as well as plots summarizing the results (see Figures 2, 3 , and 4). All three experiments reveal the ability of our robust method to improve out-of-sample performance and estimation accuracy in two ways, i.e., (i) by yielding good results on average, and especially (ii) by reducing performance variability. The latter is a key robustness property that our method is designed to achieve.Real Data Applications. We tested our method on three diverse real-world datasets. In the first study, we applied our method to predict diabetes development based on a host of features, as collected in the popular and public Pima Indian Diabetes dataset. Because the outcome is binary, we used logistic regression as implemented (i) with our robust method, (ii) with L 1 regularization, and (iii) in its plain, unregularized version. We selected hyperparameters via cross-validation and tested the out-of-sample performance of the three methods applied to disjoint batches of training observations to assess the methods' performance variability. As we report in Appendix C, our robust method outperforms both alternatives on average and does significantly better in reducing variability.We performed two further studies on linear regression applied to two popular UCI Machine Learning Repository datasets: The Wine Quality dataset [8] and the Liver Disorders dataset [15] . Similarly to the first study, we compared the performance of our method to OLS (unregularized) estimation and L 1 -penalized (LASSO) regression. After cross-validation for parameter selection, we train the models multiple times on separate batches of data and compute out-of-sample performance on a large held-out set of observations. As the results reported in Appendix C show, also in these settings our robust DP-based method performs better than the alternatives both on average and especially in terms of lower variability. Taken together, the experimental results described in this Section corroborate empirically the robustness properties of the proposed criterion, as examined theoretically throughout the paper. 
discussion: 
 The paper tackled the problem of optimizing a data-driven criterion in the presence of distributional uncertainty about the data-generating mechanism. To mitigate the underperformance of classical methods, we introduced a novel distributionally robust criterion, drawing insights from Bayesian nonparametrics and a decision-theoretic model of smooth ambiguity aversion. We established connections with standard regularization techniques, including Ridge and LASSO regression, and theoretical analysis revealed favorable finite-sample and asymptotic guarantees on the performance of the robust procedure. For practical implementation, we presented and examined tractable approximations of the criterion, which are amenable to gradient-based optimization. Finally, we applied our method to a variety of simulated and real datasets, offering insights into its practical robustness properties. Naturally, our work presents some limitations that give rise to interesting directions for future research.In particular, we note the need for a deeper examination of the model workings in terms of (i) its parameter configuration and (ii) its broader application to general learning tasks (e.g., when the loss function is adapted to accommodate deep learning architectures). Moreover, our method, as many others in the distributional robustness literature, is only suited to process homogeneously generated (e.g., iid or exchangeable) data, leaving room to explore extensions to more complex dependence structures. Finally, we highlight that our study offers prospects for investigating connections among such varied yet interconnected strands of literature as optimization, decision theory, and Bayesian statistics.This Supplement to "Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization" is organized as follows. In Appendix A, we collect the proofs of all results presented in the main text. In Appendix B, we provide further background on Dirichlet process representations and related posterior simulation algorithms. In Appendix C, we describe in detail gradient based optimization of our criterion and the experiments presented in the paper. Finally, in Appendix D we provide details on our computational infrastructure.Proof of Proposition 3.1. Let ⇝ denote weak convergence of probability measures. By Corollary 4.17 in [17] ,Q ξ n ⇝ δ p * almost surely. That is, lim n→∞ PΞ f (p)Q ξ n (dp) = f (p * )almost surely for any bounded and continuous f : P Ξ → R. Thus, we are left to prove that p → ϕ(R p (θ)) is bounded and continuous for all θ ∈ Θ. It is bounded because ϕ is continuous on the compact interval [0, K], and it is continuous because p → R p (θ) is continuous (by the definition of topology of weak convergence and because ξ → h(θ, ξ) is bounded and continuous) and ϕ is continuous.Proof of Lemma 3.2. First note that, by the stated assumptions, it follows from Taylor's theorem thatϕ(R p (θ)) = ϕ(R p * (θ)) + ϕ ′ (R p * (θ))[R p -R p * ] + ϕ ′′ (c p,θ ) 2 [R p -R p * ] 2for all p ∈ P Ξ and θ ∈ Θ and for some c p,θ ∈ [0, K]. Thensup θ∈Θ |V ξ n (θ) -ϕ(R p * (θ))| = sup θ∈Θ ϕ ′ (R p * (θ)) PΞ [R p (θ) -R p * (θ)]Q ξ n (dp) + PΞ ϕ ′′ (c p,θ ) 2 [R p (θ) -R p * (θ)] 2 Q ξ n (dp) ≤ M ϕ sup θ∈Θ α α + n R p0 (θ) + n α + n R p ξ n (θ) -R p * (θ) + M ϕ sup θ∈Θ PΞ [R p (θ) -R p * (θ)] 2 2 Q ξ n (dp) sup t∈(0,K) γ ϕ (t) ≤ M ϕ n α + n sup θ∈Θ |R p ξ n (θ) -R p * (θ)| + α α + n K + K 2 2 sup t∈(0,K) γ ϕ (t) .Proof of Theorem 3.3. Notice the following decomposition:ϕ(R p * (θ n )) -ϕ(R p * (θ * )) ≥0 (4) = ϕ(R p * (θ n )) -V ξ n (θ n ) + V ξ n (θ n ) -V ξ n (θ * ) ≤0 +V ξ n (θ * ) -ϕ(R p * (θ * )) ≤ 2 sup θ∈Θ |V ξ n (θ) -ϕ(R p * (θ))|.Then, Lemma 3.2 implies that, for all δ > 0,P[ϕ(R p * (θ n )) -ϕ(R p * (θ * )) ≤ δ] ≥ P sup θ∈Θ |V ξ n (θ) -ϕ(R p * (θ))| ≤ δ/2 ≥ P M ϕ n α + n sup θ∈Θ |R p ξ n (θ) -R p * (θ)| + α α + n K + K 2 2 sup t∈(0,K) γ ϕ (t) ≤ δ/2 = P sup θ∈Θ R p ξ n (θ) -R p * (θ) ≤ α + n n δ 2M ϕ - α α + n K - K 2 2 sup t∈(0,K) γ ϕ (t) .Proof of Theorem 3.4. Sincelim n→∞ sup θ∈Θ |R p ξ n (θ) -R p * (θ)| = 0almost surely and given assumptions ( 1) and ( 2) on ϕ n , by Lemma 3.2 we obtainlim n→∞ sup θ∈Θ |V ξ n (θ) -ϕ n (R p * (θ))| = 0almost surely. Then, by decomposition (4),lim n→∞ ϕ n (R p * (θ n )) -ϕ n (R p * (θ * )) = 0 almost surely and lim n→∞ V ξ n (θ n ) -V ξ n (θ * ) = 0 almost surely. As a consequence, lim n→∞ |V ξ n (θ n ) -ϕ n (R p * (θ * ))| ≤ lim n→∞ [|V ξ n (θ n ) -V ξ n (θ * )| + |V ξ n (θ * ) -ϕ n (R p * (θ * ))|] ≤ lim n→∞ |V ξ n (θ n ) -V ξ n (θ * )| + sup θ∈Θ |V ξ n (θ) -ϕ n (R p * (θ))| = 0almost surely. Now recall assumption (3), i.e., the sequence (ϕ n ) n≥1 converges uniformly to the identity map. Then, in light of the previous observations and by noticing that|R p * (θ n ) -R p * (θ * )| ≤ |R p * (θ n ) -ϕ n (R p * (θ n ))| + |ϕ n (R p * (θ n )) -ϕ n (R p * (θ * ))| + |ϕ n (R p * (θ * )) -R p * (θ * )| and |V ξ n (θ n ) -R p * (θ * )| ≤ |V ξ n (θ n ) -ϕ n (R p * (θ * ))| + |ϕ n (R p * (θ * )) -R p * (θ * )|,the two desired almost sure limits follow:lim n→∞ R p * (θ n ) = R p * (θ * ), lim n→∞ V ξ n (θ n ) = R p * (θ * ).Proof of Theorem 3.6. We haveR p * (θ * ) ≤ R p * ( θ) = E ξ∼p * lim n ′ →∞ h(θ n , ξ) = lim n→∞ R p * (θ n ) = R p * (θ * )almost surely, where the first equality follows from the continuity of θ → h(θ, ξ) and the second one from the Dominated Convergence Theorem. Then, R p * ( θ) = R p * (θ * ) almost surely, proving the result.To Prove Lemma 4.3, we introduce two other Lemmas. After proving those, Lemma 4.3 follows immediately.Lemma A.1. For all T, N ≥ 1,sup θ∈Θ | Vξ n (θ, T, N )-V ξ n (θ)| ≤ M ϕ K α + n α + n + 1 T + sup θ∈Θ Vξ n (θ, T, N )-E[ Vξ n (θ, T, 1)] . (5)Proof. We haveEQUATIONNote thatE[ Vξ n (θ, T, 1)] = E p1,ξ1,...,p T ,ξ T ϕ T j=1 p j h(θ, ξ j ) + p 0 h(θ, ξ 0 ) = E j≥1 pj δ ξ j ∼Q ξ n ϕ T j=1 p j h(θ, ξ j ) + p 0 h(θ, ξ 0 ) = E j≥1 pj δ ξ j ∼Q ξ n ϕ ∞ j=1 p j h(θ, ξ j ) + p 0 h(θ, ξ 0 ) - ∞ j=T +1 p j h(θ, ξ j ) = V ξ n (θ) + E j≥1 pj δ ξ j ∼Q ξ n ϕ ′ c θ, j≥1 pj δ ξ j p 0 h(θ, ξ 0 ) - ∞ j=T +1 p j p 0 h(θ, ξ j ) ,where the last equality follows from the mean value theorem applied to endpoints∞ j=1 p j h(θ, ξ j ) and ∞ j=1 p j h(θ, ξ j ) + p 0 h(θ, ξ 0 ) - ∞ j=T +1 p j h(θ, ξ j ).Then the second term in ( 6) is bounded byM ϕ KE j≥1 pj δ ξ j ∼Q ξ n [p 0 ] = M ϕ KE T k=1 (1 -B k ) = M ϕ K α + n α + n + 1 T .The second term on the left-hand side of Equation( 5) is instead of the formEQUATIONwhere X i = T j=0 p ij h(θ, ξ ij ) are iid random variables whose distribution is determined by the truncated stick-breaking procedure.The aim of the next Lemma is to provide sufficient conditions for finite sample bounds and asymptotic convergence to 0 of the term in (7) . Specifically, we impose complexity constraints on the function class H := {ξ → h(θ, ξ) : θ ∈ Θ} which allow us to obtain appropriate conditions on the derived classF := (p j , ξ j ) T j=0 → ϕ T j=1 p j h(θ, ξ j ) + p 0 h(θ, ξ 0 ) : θ ∈ Θ ,ensuring the asymptotic and non-asymptotic results we seek. Lemma A.2. Assume Θ is a bounded subset of R d and θ → h(θ, ξ) is c(ξ)-Lipschitz continuous for all ξ ∈ Ξ. Then, for all T, N ≥ 1 and ε > 0,sup θ∈Θ Vξ n (θ, T, N ) -E[ Vξ n (θ, T, 1)] ≤ ε with probability at least 1 -2 32M ϕ C T diam(Θ) √ d ε d exp - 3N ε 2 4ϕ(K)(6ϕ(K) + ε) + exp - 3N ε 40ϕ(K).for some constant C T > 0.Proof. By the Lipschitz continuity assumption on h(θ, ξ), we obtain that, for all (p j , ξ j ) T j=0 , θ → ϕT j=1 p j h(θ, ξ j ) + p 0 h(θ, ξ 0 ) is M ϕ c (p j , ξ j ) T j=0 -Lipschitz continuous, with c (p j , ξ j ) T j=0 := T j=0 p j c(ξ j ).Indeed, for all θ 1 , θ 2 ∈ Θ,ϕ T j=1 p j h(θ 1 , ξ j ) + p 0 h(θ 1 , ξ 0 ) -ϕ T j=1 p j h(θ 2 , ξ j ) + p 0 h(θ 2 , ξ 0 ) ≤ M ϕ T j=1 p j |h(θ 1 , ξ j ) -h(θ 2 , ξ j )| + p 0 |h(θ 1 , ξ 0 ) -h(θ 2 , ξ 0 )| ≤ M ϕ c (p j , ξ j ) T j=0 ∥θ 1 -θ 2 ∥.Therefore, denoting by P the law of the vector (p j , ξ j ) T j=0 and by N [] (ε, F , L 1 (P )) the associated ε-bracketing number of the class F , by Lemma 7.88 in [25] we obtainN [] (ε, F , L 1 (P )) ≤ 4M ϕ Cdiam(Θ) √ d ε d , with C T := c (p j , ξ j ) T j=0 dP .Then the result follows by Theorem 7.86 in [25] after noticing thatsup f ∈F ∥f ∥ L 1 (P ) ≤ sup f ∈F ∥f ∥ ∞ ≤ ϕ(K).Proof of Theorem 4.5. We haveV ξ n (θ n ) ≤ V ξ n ( θn ) = E p∼Q ξ n lim N →∞ lim T →∞ ϕ(R p ( θn (T, N ))) = lim N →∞ lim T →∞ V ξ n ( θn (T, N )) = V ξ n (θ n )almost surely, where the first two equalities follow from the continuity of θ → h(θ, ξ) and ϕ as well as from an iterated application of the Dominated Convergence Theorem (recall that h(θ, ξ) ∈ [0, K] by assumption for all θ and ξ). This implies V ξ n (θ n ) = V ξ n ( θn ) almost surely.In the next result, we will assume ϕ(t) ≡ ϕ β (t) = β exp(t/β) -β. Moreover, we will emphasize, through superscripts, the dependence of mathematical objects on β and the DP concentration parameter α. Moreover, if necessary, we make the truncation threshold T n and the number of MC samples N n dependent on the sample size n.Proposition A.3. Assume Θ is an open subset of R d and lim N →∞ lim T →∞ sup θ∈Θ | V α,β ξ n (θ, T, N ) -V α,β ξ n (θ)| = 0 almost surely for all α > 0 and β > 0. Moreover, assume that 1. θ → h(ξ, θ) is differentiable at θ * ∈ arg min θ∈Θ R p * (θ) for p * -almost every ξ ∈ Ξ, with gradient ∇ θ * (ξ);2. For all θ 1 and θ 2 in a neighborhood of θ * , there exists a measurable functionξ → H(ξ) ∈ L 2 p * such that |h(θ 1 , ξ) -h(θ 2 , ξ)| ≤ H(ξ)∥θ 1 -θ 2 ∥;3. θ → R p * (θ) admits a second-order Taylor expansion at θ * , with non-singular symmetric Hessian matrix V θ * .Then, with probability 1, there exist sequences(T n ) n≥1 , (N n ) n≥1 , (β n ) n≥1 , (diverging to ∞) and (α n ) n≥1 (converging to 0), such that √ n θαn,βn n (T n , N n ) -θ * ⇝ N (0, V ), V := V -1 θ * E ξ∼p * ∇ θ * (ξ)∇ θ * (ξ) ⊤ V -1 θ * , provided θαn,βn n (T n , N n ) p → θ * as n → ∞.Proof. The imposed assumptions match those listed in Theorem 5.23 of [42] . The only condition left to prove is that there exist sequences(T n ) n≥1 , (N n ) n≥1 , (β n ) n≥1 , (diverging to ∞) and (α n ) n≥1 (converging to 0), such that R p ξ n ( θαn,βn n (T n , N n )) -inf θ∈Θ R p ξ n (θ) ≤ o p (n -1). We do so by proving that, for any fixed n ≥ 1, R p ξ n ( θα,β n (T, N )) → inf θ∈Θ R p ξ n (θ) almost surely as α → 0 and β, T, N → ∞; this implies that, for all n, with probability 1 there exist α n , β n , T n and N n such that R p ξ n ( θαn,βnn (T n , N n )) -inf θ∈Θ R p ξ n (θ) ≤ ε n for any ε n > 0.Moreover, it is easy to see that the result is implied by sup θ∈Θ | V α,β ξ n (θ, T, N ) -R p ξ n (θ)| → 0, so we prove the latter. We havesup θ∈Θ | V α,β ξ n (θ, T, N ) -R p ξ n (θ)| ≤ sup θ∈Θ | V α,β ξ n (θ, T, N ) -V α,β ξ n (θ)| + sup θ∈Θ |V α,β ξ n (θ) -R p ξ n (θ)|,where the first term converges to 0 almost surely by assumption. Using a second-order Taylor expansion of ϕ β (R p (θ)) around R p ξ n (θ), the second term, instead, satisfiessup θ∈Θ |V α,β ξ n (θ) -R p ξ n (θ)| = sup θ∈Θ PΞ ϕ β (R p (θ))Q α ξ n (dp) -R p ξ n (θ) ≤ sup θ∈Θ ϕ ′ β (R p ξ n (θ)) PΞ R p (θ)Q α ξ n (dp) -R p ξ n (θ) + K 2 2 sup t∈[0,K] ϕ ′′ β (t) ≤ sup t∈[0,K] ϕ ′ β (t) →1 sup θ∈Θ n α + n R p ξ n (θ) + α α + n R p0 (θ) -R p ξ n (θ) →0 + K 2 2 sup t∈[0,K] ϕ ′′ β (t) →0 → 0,as α → 0 and β → ∞.Remark A.4. Theorems 3.6 and 4.5 ensure that (a) for all n ≥ 1, provided θn (T, N ) converges almost surely to some θ n ∈ Θ, the latter is a minimizer of V ξ n (θ); and (b) if the above sequence (θ n ) n≥1 converges almost surely to some θ * ∈ Θ, the latter is a minimizer of R p * (θ). Notice that the assumptions required for these results are consistent with the ones of Proposition A.3, so they can be used to justify the condition θαn,βn Stochastic Gradient Descent Convergence Analysis. The following results refer to material presented in Appendix C below. For ease of exposition, we fix B = 1 and denote by E t the expectation operator conditional on the realization of the random index draws m 1 , . . . , m t iid ∼ Uniform({1, . . . , M }). Proposition A.5. Assume that V is convex and that (θ t ) t≥1 follows Equation (9)) for some starting value θ 0 ∈ Θ and B = 1. Moreover, assume that, for all θ ∈ Θ,M -1 M m=1 ∥ℓ m ϕ ′ (H m (θ))∇ θ h(θ, ξ m )∥ 2 ≤ σ 2 ν .ThenE T -1 [V ( θT )] -V (θ * ) ≤ ∥θ 0 -θ * ∥ 2 + σ 2 ν T t=0 η 2 t 2 T t=0 η t ,where θ * ∈ arg min θ∈Θ V (θ), θT := T t=0 ν t θ t , andν t := ηt T t ′ =0 η t ′ .Proof of Proposition A.5. Fix t = 1, . . . , T . We have,∥θ t+1 -θ * ∥ 2 = ∥θ t -η t ℓ m t ϕ ′ (H m t (θ t ))∇ θ h(θ t , ξ m t ) -θ * ∥ 2 = ∥θ t -θ * ∥ 2 + η 2 t ∥ℓ m t ϕ ′ (H m t (θ t ))∇ θ h(θ t , ξ m t )∥ 2 -2η t (θ t -θ * ) ⊤ ℓ m t ϕ ′ (H m t (θ t ))∇ θ h(θ t , ξ m t ).Applying the law of total expectation and the fact thatℓ m t ϕ ′ (H m t (θ t ))∇ θ h(θ t , ξ m t ) is unbiased for ∇ θ V (θ t ), E t [(θ t -θ * ) ⊤ ℓ m t ϕ ′ (H m t (θ t ))∇ θ h(θ t , ξ m t )] = E t [E t-1 [(θ t -θ * ) ⊤ ℓ m t ϕ ′ (H m t (θ t ))∇ θ h(θ t , ξ m t )]] = E t-1 [(θ t -θ * ) ⊤ ∇ θ V (θ t )].Hence2η t E t-1 [(θ t -θ * ) ⊤ ∇ θ V (θ t )] = E t-1 [∥θ t -θ * ∥ 2 ] -E t [∥θ t+1 -θ * ∥ 2 ] + η 2 t E t [∥ℓ m t ϕ ′ (H m t (θ t ))∇ θ h(θ t , ξ m t )∥ 2 ] ≤ E t-1 [∥θ t -θ * ∥ 2 ] -E t [∥θ t+1 -θ * ∥ 2 ] ≥0 +σ 2 ν .Summing over t = 0, . . . , T and sinceE t-1 [(θ t -θ * ) ⊤ ∇ θ V (θ t )] ≥ E t-1 [V (θ t ) -V (θ * )] because V is convex, we have 2 T t=0 η t E t-1 [V (θ t ) -V (θ * )] ≤ ∥θ 0 -θ * ∥ 2 + σ 2 ν T t=0 η 2 t .Dividing both sides by T t=0 η t and exploiting (i) the linearity of the expectation operator, (ii) the convexity of the weights (ν t ) T t=0 , and (iii) the convexity of V , the result follows.Proof of Proposition 2.1. As for case 1, given the assumed form of p 0 and the criterion representation (2), we are left to establish an expression forE ξ∼p0 [h(θ, ξ)] = E (y,x)∼N (0,I) [(y -θ ⊤ x) 2 ].Notice that -θ j x j id ∼ N (0, θ 2 j ), independendently of y ∼ N (0, 1), so that y-θ ⊤ x ∼ N (0, 1+∥θ∥ 2 2 ). Therefore, E ξ∼p0 [h(θ, ξ)] = V[y -θ ⊤ x] = 1 + ∥θ∥ 2 2, which is easily seen to complete the proof. Finally, the proof for the LASSO case is completely analogous to the Ridge one and is therfore omitted.Since its definition by [13] based on the family of finite-dimensional Dirichlet distributions (as sketched in Section 2), the Dirichlet process has been characterized (and thus generalized) in a number of useful ways. For instance, the DP can be derived as a neutral to the right process [14] , a normalized completely random measure [13, 22, 26, 36] , a Gibbs-type prior [20, 9] , a Pitman-Yor Process [33, 32] , and a species sampling model [34] . In what follows, we review two other constructions of the DP which were at the basis of the approximate versions of the robust criterion V ξ n proposed in Section 4. Input: Data ξ n , model parameters, number of MC samples N , truncation stepT for i = 1 to N do Set 0 k=1 (1 -B k ) ≡ 1 for j = 1 to T do Draw ξ ij ∼ α α+n p 0 + n α+n p ξ n Draw B ij ∼ Beta(1, α + n) Set p ij = B j j-1 k=1 (1 -B k ) end for Draw ξ i0 ∼ α α+n p 0 + n α+n p ξ n Set p i0 = T k=1 (1 -B k ) end for Return: N -1 N i=1 ϕ T j=0 p ij h(θ, ξ ij )The name of the procedure comes from the analogy with breaking a stick of length 1 into two pieces of length B 1 and 1 -B 1 , then the second piece into two sub-pieces of length (1 -B 1 )B 2 and(1 -B 1 )(1 -B 2 ), and so on. In Algorithm 1, then, we simulate N realizations from Q ξ n , truncating the stick-breaking procedure at step j = T . The remaining portion of the stick is then allocated to one further atom drawn from the predictive distribution. Then, the intractable integral with respect to the DP posterior is approximated via a Monte Carlo average of the integrals (i.e., weighted sums) with respect to the N simulated measures. Input: Data ξ n , model parameters, number of MC samples N , approximation thresholdT for i = 1 to N do Initialize w i ∈ R T , ξ i ∈ Ξ T for j = 1 to T do Update w i (j) ∼ Gamma α+n T , 1 Update ξ i (j) ∼ α α+n p 0 + n α+n p ξ n end for Normalize w i = wi n j=1 wi(j) end for Return: N -1 N i=1 ϕ w ⊤ i h(θ, ξ i )When α is negligible compared to the sample size n, one can simplify posterior simulation by setting α = 0. Thus, one obtains a DP(n, p ξ n ) posterior. This distribution enjoys a useful representation as follows: p ∼ DP(n, p ξ n ) =⇒ p d = n i=1 p i ξ i , with p i ∼ Dirichlet(n; 1, . . . , 1) [see 17, Section 4.7]. Due to its similarity to the usual bootstrap procedure [12] , this distribution is known as the "Bayesian bootstrap". Algorithm 3 implements the Bayesian bootstrap to approximate the criterion V ξ n (θ). In practice, however, we do not recommend resorting to the Bayesian bootstrap approximation, since DP(n, p ξ n ) assigns probability 1 to the set of distributions with strictly positive support on ξ n . This goes against the prescription that, as the finite sample ξ n provides only partial information on the true underlying distribution, the statistical DM should be willing to consider a wider set of distributions other than the ones supported at the sample realizations.Input: Data ξ n , model parameters, number of MC samples N for i = 1 to N do Initialize w i ∈ R n for j = 1 to n do Update w i (j) ∼ Gamma(1, 1) end for Normalize w i = wi n j=1 wi(j) end for Return:N -1 N i=1 ϕ w ⊤ i h(θ, ξ n )In this Section, we first describe the SGD algorithm used in practice for our experiments. Then, we describe in full detail the experiments presented in the paper.Gradient-Based Optimization. Whether we resort to the SBMC or the MDMC approximation of V ξ n (θ), we are faced with the task of minimizing a criterion of the formV (θ) = 1 N N i=1 ϕ   T j=1 p ij h(θ, ξ ij )   .The smoothness and convexity of ϕ make it appealing to minimize the criterion via gradient-based convex optimization techniques. Indeed, it is enough to assume that θ → h(θ, ξ) is convex and differentiable (a standard assumption met in many applications of interest) to easily yield the same properties for V .In light of this, assuming that θ → h(θ, ξ) is differentiable at every ξ ij and denoting H i (θ) := T j=1 p ij h(θ, ξ ij ), the gradient of V isEQUATIONwhere ℓ m ≡ T p m and the m-indexing is just a recoding of the indices (with a slight abuse of notation and M ≡ N • T ). That is, the gradient of V (θ) can be written as the average of M terms. Thus, to minimize V (θ) we propose a mini-batch Stochastic Gradient Descent algorithm which, at each iteration t, updates the parameter vector as follows:EQUATIONfor a step-size η t > 0 and a random subset (mini-batch) of size B from the indices {1, . . . , M }. Under standard regularity assumptions [16] , in Proposition A.5 (Appendix A) we prove convergence of the algorithm at usual rates for convex problems.Remark C.1. Expression (8) provides some insight on how, in practice, distributional robustness is enforced. Notice that H i (θ) = R pi (θ) is the expected risk computed according to p i , an approximate realization from Q ξ n . Thus, in the computation of the overall gradient ∇ θ V (θ), the gradients associated to the p i 's that generate higher expected risks receive more weight (being ϕ convex, ϕ ′ is increasing). These weights, then, are reflected into which gradients, in the mini-batch SGD algorithm, are given more leverage in updating the parameter vector. Thus, the procedure can be thought of as implementing a "soft worst-case scenario" scheme, whereby distributions in the posterior support are weighted (in terms of gradient influence) more the worse they do in terms of expected risk.Mini-Batch Stochastic Gradient Descent Algorithm. For practical optimization, we apply a modification to the SGD algorithm provided in Equation (9) , which helps to reduce the computational burden of the procedure. Indeed, recall the formula of the gradient of the criterion V that we need to optimize:∇ θ V (θ) = 1 N N i=1 ϕ ′ (H i (θ))∇ θ H i (θ) ≡ 1 M M m=1 ℓ m ϕ ′ (H m (θ))∇ θ h(θ, ξ m ).Clearly, then, implementing the baseline SGD algorithm requires, at each iteration, the evaluation of multiple H m (θ t ) terms, each consisting of T evaluations of the loss function h. To avoid this, at each iteration we instead sub-sample one index i = 1, . . . , N and update the parameter vector according to the associated gradient ϕ ′ (H i (θ t ))∇ θ H i (θ t ). The latter is still an unbiased estimator of the overall gradient of V (θ t ), but it requires only T evaluations of h (plus those of T gradients of h, similarly to the baseline algorithm). Finally, to exploit the whole data efficiently, we sub-sample without replacement and perform multiple passes over the N MC samples. Algorithm 4 summarizes the procedure.Input: Approximate criterion parameters {(p ij , ξ ij ) : i = 1, . . . , N, j = 1, . . . , T }, step size schedule (η t ) t≥0 , starting value θ 0 , number of passes P , iteration tracker t = 0 for p = 1 to P doInitialize I = {1, . . . , N } for j = 1 to N do Sample uniformly i ∈ I Update θ t+1 = θ t -η t • ϕ ′ T ℓ=1 p iℓ h(θ t , ξ iℓ ) • T ℓ=1 p iℓ ∇ θ h(θ t , ξ iℓ ) Update I = I \ {i} Update t = t + 1 end for end for Return: θ P N +1Setting. In this experiment, we test the performance of our robust criterion in a high-dimensional sparse linear regression task. The high-dimensional and sparse nature of the data-generating process is expected to induce distributional uncertainty, and our method is meant to address this. In this context, we use the quadratic loss function (θ, y, x) → 10 -3 (y -θ ⊤ x) 2 , where the 10 -3 factor serves to stabilize numerical values in the optimization process. Notice that, by the form of the ambiguity-neutral criterion (2), the multiplicative factor on the loss function does not change the equivalence with Ridge.Data-Generating Process. The data for the experiment are generated iid across simulations (200) and observations (n = 100 per simulation) as follows. For each observation i = 1, . . . , n, the d-dimensional (d = 90) covariate vector follows a multivariate normal distribution with mean 0 and such that (i) each covariate has unitary variance, and (ii) any pair of distinct covariates has covariance 0.3:x i =    x i1. . .x id    ∼ N (0, Σ), Σ =     1 0.3 • • • 0.3 0.3 1 • • • 0.3 . . . . . . . . . . . . 0.3 0.3 • • • 1     ∈ R d×d .Then, the response has conditional distributiony i |x i ∼ N (a ⊤ x i , σ 2 ), with a = (1, 1, 1, 1, 1, 0, • • • , 0) ⊤ ∈ R d and σ = 0.5.That is, out of 90 covariates, only the first 5 have a unitary positive marginal effect on y i , and additive Gaussian noise is added to the resulting linear combination. Together with 100 training samples, at each simulation we generate 5000 test samples on which we compute out-of-sample RMSE for the ambiguity-averse, ambiguity-neutral, and OLS procedures. Stochastic Gradient Descent Parameters We initialize the algorithm at θ = (0, . . . , 0) and set the step size at η t = 50/(100 + √ t). The number of passes over data is set after visual inspection of convergence of the criterion value. The run time per SGD run is less than 1 second on our infrastructure (see Appendix D).Setting. In this experiment, we test the performance of our robust criterion on the task of estimating a univariate Gaussian mean (assuming the variance is known) when the data is corrupted by a few observations coming from a distant distribution. Clearly, this is a situation where a considerable level of distributional uncertainty is warranted. In this setting, the loss function h(ξ, θ) = (ξ -θ) 2 is simply the negative log-likelihood associated to the normal model. Notice that the h is convex in θ and, as in the previous experiment, we pre-multiply it by a factor of 10 -3 for numerical stability reasons.Data-Generating Process. The data for the experiment are generated iid across simulations (100) and observations (n = 13 per simulation) as follows. For each simulation, 10 iid samples x i are drawn from a N (0, 1) distribution (the actual data-generating process we want to learn) and 3 samples are drawn iid from a N (0, 5) outlier distribution. At each simulation we also generate 5000 test samples from the data-generating process N (0, 1), on which we compute the out-of-sample average negative log-likelihood for the ambiguity-averse, ambiguity-neutral, and Maximum Likelihood Estimation (MLE) procedures -this will be our measure of out of sample performance (see Figure 3 ).For each simulated sample, we run our robust procedure setting the following parameter values: ϕ(t) = β exp(t/β) -β, β ∈ {1, ∞}, α ∈ {1, 2, 5, 10}, and p 0 = N (µ 0 , I), where µ 0 = (10 • 0 + 3 • 5)/(10 + 3) is a weighted average of the data-generating and the outlier means. By the expression of the ambiguity-neutral criterion (2) , it is easy to show that the β = ∞ case leads to the parameter estimate with y i = x i for i = 1, . . . , n and y i = µ 0 for i = n + 1, . . . , n + α. That is, the ambiguity-neutral procedure with concentration parameter α ∈ N is equivalent to the MLE procedure when the original training sample is enlarged with α additional observations equal to µ 0 . Finally, we run 300 Monte Carlo simulations to approximate the criterion, and truncate the Multinomial-Dirichlet approximation at T = 50.θξ n = 1 α + nStochastic Gradient Descent Parameters. We initialize the algorithm at θ = 0 and set the step size at η t = 20/(100 + √ t). The number of passes over data is set after visual inspection of convergence of the criterion value. The run time per SGD run is 2 seconds on our infrastructure (see Appendix D).Results. In Figure 3 , we present the results of the simulation study. As for the regression experiment, the ambiguity-averse criterion brings improvement, across α values and compared to the ambiguityneutral and the simple MLE procedures, both in terms of average performance and in terms of the latter's variabiliy (see the first row of the Figure). From the second row of Figure 3 , it also emerges that, on average, the ambiguity-averse procedure is more accurate at estimating the location parameter than the two other methods. Compared to the simple MLE procedure, the variability of the estimated parameter is also significantly smaller. Taken together, these results confirm the theoretical expectation that the ambiguity-averse optimization is effective at hedging against the distributional uncertainty arising in the estimation of corrupted data such as the simulated ones.Setting. In this experiment, we test the performance of our robust criterion on a high-dimensional sparse classification task using the framework of logistic regression. As in the linear regression experiment, the high-dimensional and sparse nature of the data-generating process is expected to induce distributional uncertainty, and our method is meant to address this. In this setting, the loss function is h(ξ, θ) = log(1 + exp(-y • x ⊤ θ)). As in the previous experiment, we pre-multiply it by a factor of 10 -3 for numerical stability reasons.Data-Generating Process. The data for the experiment are generated iid across simulations (200) and observations (n = 100 per simulation) as follows. For each observation i = 1, . . . , n, the d-dimensional (d = 90) covariate vector follows a multivariate normal distribution with mean 0 and such that (i) each covariate has unitary variance, and (ii) any pair of distinct covariates has covariance 0.3:x i =    x i1. . .x id    ∼ N (0, Σ), Σ =     1 0.3 • • • 0.3 0.3 1 • • • 0.3 . . . . . . . . . . . . 0.3 0.3 • • • 1     ∈ R d×d .Then, the response has conditional distribution simulation we generate 5000 test samples on which we compute the out-of-sample average loss for the ambiguity-averse, L 2 -regularized (with regularization parameter α, see below), and un-regularized procedures.y i |x i ∼ Binary({1, -1}, p x ), with p x = 1/(1 + exp(-x ⊤ a)) and a = (1, 1, 1, 1, 1, 0, • • • , 0) ⊤ ∈ R d .Robust Criterion Parameters. For each simulated sample, we run our robust procedure setting the following parameter values:ϕ(t) = β exp(t/β) -β, β = 1, α = a/n for α ∈ {1, 2, 5, 10}, and p 0 = Binary({1, -1}, 0.5) × N (0, I). Finally, we run 200 Monte Carlo simulations to approximate the criterion, and truncate the Multinomial-Dirichlet approximation at T = 50.We initialize the algorithm at θ = (0, . . . , 0) and set the step size at η t = 1000/(100 + √ t). The number of passes over data is set after visual inspection of convergence of the criterion value. The run time per SGD run is 3 seconds on our infrastructure (see Appendix D).Results. In Figure 3 , we present the results of the simulation study. As for the regression experiment, the ambiguity-averse criterion brings improvement, across α values and compared to the L 2 -regularized and the unregularized procedures, both in terms of average performance and in terms of the latter's variabiliy (see the first row of the Figure). From the second row of Figure 3 , it also emerges that, on average, the ambiguity-averse procedure is more accurate and less variable at estimating the true regression coefficient than the two other methods. Also, our method is able to more effectively shrink the norm of the coefficient vector towards 0 (see the third row). Taken together, these results confirm the theoretical expectation that the ambiguity-averse optimization is effective at hedging against the distributional uncertainty arising in high-dimensional classification problems (in this experimental setting, tackled via logistic regression).In this experiment, we use logistic regression for classification on the popular Pima Indians Diabetes dataset, 12 collecting data on 768 women belonging to a Native American group that lives in Mexico and Arizona. The data consists of a binary outcome (whether the subject developed diabetes or not) and 8 features related to her physical condition (these features are standardized before running the analysis).To test our method, we randomly select 300 training observations and leave out the rest for as a test sample. Then, we randomly split the training data into 15 folds of size 20 and select, via k-fold cross validation, the optimal DP concentration parameter α over a wide grid of values. We do the same for the L 1 -penalty coefficient used to implement regularized logistic regression with the Python library scikit-learn [31] . Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, L1-penalized logistic regression, and unregularized logistic regression on each of the folds. 13 This splitting procedure allows us (i) to test and compare the performance of our method in a setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures. The run time per SGD run is 19 seconds on our infrastructure (see Appendix D).Table 1 reports the results from the described procedure. Unregularized logistic regression performs quite poorly compared to the other two methods. Instead, the latter yield results in the same orders of magnitude both in terms of average performance and of performance variability, though our DP robust method features almost half of the variability produced by L 1 -regularized logistic regression.Unregularized In this experiment, we applied linear regression to the popular UCI Machine Learning Repository Wine Quality dataset [8] . Data consists of 4898 measurements of 11 wines' characteristics and a quality score assigned to each wine. The aim is to predict the latter based on the former (both features and response are standardized before running the analysis). We implement linear regression using our DP-based robust method (with the squared loss function), OLS, and LASSO (the last two methods are implemented using scikit-learn [31] ).To test our method, we randomly select 300 training observations and leave out the rest for as a test sample. Then, we randomly split the training data into 10 folds of size 30 and select, via k-fold cross validation, the optimal DP concentration parameter α over a wide grid of values. We do the same for the L 1 -penalty coefficient used to implement LASSO. Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, LASSO regression, and OLS estimation on each of the folds. This splitting procedure allows us (i) to test and compare the performance of our method in a setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures. The run time per SGD run is 5 seconds on our infrastructure (see Appendix D). In this experiment, we applied linear regression to the popular UCI Machine Learning Repository Liver Disorders dataset [15] . Data consists of 345 measurements of 5 blood test results and the number of drinks consumed per day by each subject. The aim is to predict the latter based on the former (both features and response are standardized before running the analysis). We implement linear regression using our DP-based robust method (with the squared loss function), OLS, and LASSO (the last two methods are implemented using scikit-learn [31] ).To test our method, we randomly select 200 training observations and leave out the rest for as a test sample. Then, we randomly split the training data into 10 folds of size 20 and select, via k-fold cross validation, the optimal DP concentration parameter α over a wide grid of values. We do the same for the L 1 -penalty coefficient used to implement LASSO. Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, LASSO regression, and OLS estimation on each of the folds. This splitting procedure allows us (i) to test and compare the performance of our method in a setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures.The run time per SGD run is 15 seconds on our infrastructure (see Appendix D). All experiments were performed on a desktop with 12th Gen Intel(R) Core(TM) i9-12900H, 2500 Mhz, 14 Core(s), 20 Logical Processor(s) and 32.0 GB RAM.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Please, refer to Appendix C and the submitted code in the supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Variability in the reported average performance metrics are of direct interest for the evaluation of our method, so we report measures of variability as separate bars in plots or separate entries in tables.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have read the Code of Ethics and confirm the adherence of our paper to it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Due to its theoretical and optimization-focused nature, the paper has no direct foreseeable societal impact beyond that of any generic machine learning paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper only employs simulated or small publicly available datasets, as well as well-established machine learning techniques.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper properly credits all data and software used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release any new asset.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Given a topological space (A, T ), we denote by B(A) the Borel σ-algebra generated by T .Note that, under mild regularity (e.g., continuity) conditions on h, its boundedness is ensured, for instance, by the frequently assumed compactness of Ξ and Θ.Throughout the paper, we adopt the terms "ambiguity" and "uncertainty" interchangeably.Also, E[p(A)] = p0(A) and V[p(A)] = (1 + α) -1 p0(A)(1 -p0(A)) for any A ∈ B(Ξ), justifying the names of α and p0.In practice, the prior guess p0 can be leveraged to incorporate features of the underlying process that the researcher suspects to hold (e.g., in regression applications, sparsity). See also Section 5.For instance, one might have prior information on specific correlation patterns among covariates, which could be useful to incorporate in regression training with few data points.Our approach is also related to recent literature[48,47,40] exploring Bayesian ideas in the context of DRO. However, the cited works (i) focus on parametric priors and (ii) rely on either ambiguity sets or statistical risk measures to induce robustness. This is in contrast with our work, which (i) resorts to a more assumption-free nonparametric prior and (ii) leverages the highlighted robustness properties of the simple yet powerful convex transformation ϕ.In the economic decision theory literature, as the DM usually maximizes a criterion (utility), convexity is replaced by concavity.Using an analogous line of reasoning as in the proof of Theorem 3.4 (see Appendix A), we note that Lemma 3.2 and Theorem 3.3 can be easily adapted in to obtain finite sample bounds on sup θ∈Θ |V ξ n (θ) -Rp * (θ)| and Rp * (θn) -Rp * (θ * ) depending on sup t∈[0,K] |ϕ(t) -t|.In the α = 0 limit and setting T = n, the well-known "Bayesian bootstrap distribution" is recovered [17, see also Appendix B for further details].In Appendix A, we also present an asymptotic normality result for the approximate optimizer θn(T, N ) (Proposition A.3).Made available by the National Institute of Diabetes and Digestive and Kidney Diseases and downloaded from https://www.kaggle.com/datasets/kandij/diabetes-dataset?resource=download.All of the implementation details (e.g., parameter values), can be found in our code. This holds for the next two experiments as well. 
 
 Appendix:  


Decision: The paper addresses the challenge of optimizing machine learning models under distributional uncertainty. The authors propose a novel method that integrates Bayesian nonparametric theory, specifically the Dirichlet Process (DP), with distributionally robust optimization (DRO). Finite-sample probabilistic guarantee on the excess risk is derived. The asymptotic vanishing of the excess risk and the convergence of the finite-sample optimal value are established. The paper's theoretical contributions are considered solid, with the proofs being clear and technically sound (see the review and comments by Reviewers 7wnT and Vivx). However, the experimental results, while promising, are briefly discussed in the main text (see the review and comments by Reviewer p57R). Some other concerns include: computational expense, especially for high-dimensional data or large sample sizes (see the review and comments by Reviewer DqoX); could better explain the implications of placing a prior on the data-generating process rather than on model parameters (see the review and comments by Reviewer p57R); minor typos (see the review and comments by Reviewer Vivx).

Reviews: review: paper_summary: paper_summary: The paper proposes a Bayesian method for optimizing stochastic objectives under a given finite sample. More precisely, The paper assumes the Dirichlet Process for data generation and then it proposes to optimize the mean of the stochastic objective over posterior distributions given the sample. The paper proves asymptotic properties of the proposed objective. The experimental results in the appendix show that the proposed method works better than the standard L2 regularization or no regularization on linear regression tasks over simulation and real data sets.
strength_weakness: strength_weakness: One of my concerns is the proposed method's high computational cost. However, I don’t think it is critical. When the sample size is large enough, the standard ERM would work well. The proposed method, though it shows asymptotic convergence properties, would work better when the sample size is small. On the other hand, it would need more technical improvement when the sample space is high-dimensional.
questions: questions: Is it difficult to obtain finite-sample error bound like PAC or statistical learning theory? (or any related work?)
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a novel robust optimization criterion for training machine learning and statistical models by combining Bayesian nonparametric theory and smooth ambiguity-averse preferences, addressing distributional uncertainty to improve out-of-sample performance. The authors demonstrate theoretical guarantees for their method and show its practical implementation and effectiveness through tasks using simulated and real datasets.
strength_weakness: strength_weakness: The theoretical aspects of this paper are not my research area, so I am not equipped to assess the strengths and weaknesses of the theoretical contributions. However, the proposed method and its practical implications appear to be well-founded and promising.
questions: questions: NO.
limitations: limitations: Discussed in Section 6.
paper_summary: paper_summary: This paper introduces a new robust risk criterion that integrates concepts from Bayesian nonparametric theory, specifically the Dirichlet process, and a recent decision-theoretic model of smooth ambiguity-averse preferences. They show the relationships between their criterion and traditional regularization methods for empirical risk minimization, including Ridge and LASSO regressions. They also provide theoretical gurantee for the robust optimization and propose tractable approximations of the criterion.
strength_weakness: strength_weakness: *
questions: questions: * I have a question about the coefficient in the prior. What will happen if we adopt the empirical Bayesian idea into your algorithm, i.e. estimate the coefficient in the prior through data? Will this cause degradation of the model？
limitations: limitations: The authors adequately addressed the limitations of their work.
paper_summary: paper_summary: The authors use Dirichelet process and a smooth ambiguity aversion model to approximate the solution of risk minimization problems. They demonstate the consistency of their approach, along with finite sample guarantees. They additionally give a practical means of applying their procedure and apply the procedure to a number of simulated and real problems.
strength_weakness: strength_weakness: There are many small typos. 
 - In Lemma 3.2 the supremum of \gamma^* is not needed as gamma^* does not depend on t. The statement in the proof is the supremum over gamma(t). 
- For consistency with lemma 3.2, in theorem 3.3 the supremum over gamma(t) should probably be replaced by gamma^* (or gamma(t) should be added back into lemma 3.2).
 - In line 181 I don't think the (ii) should be there. 

There are some minor problems.
 - The introduction of the dependency of phi on n makes a small appearence. This idea is very interesting but I don't think it is emphasised enough. Just a couple of sentences explaining why you'd want to do such a thing would be nice.
 - In line 221 a form of phi is assumed. I assume this is supposed to be phi_n and not phi. If it is supposed to be phi I'm unsure what form is being assumed, and if it is supposed to be phi_n it doesn't seem like it is assumed as section 4 seems to use phi.

And there are some reasonably major (but easily fixable) problems. 
- On first read it is confusing how exactly the distance between the theoretic risk and the risk computed WRT to p_0 enters into lemma 3.2. Seeing the full statement I understand why the full bound isn't included in the main text but perhaps this discussion can be reworded. 
- It is unclear if theorem 4.4 is using the same assumptions as lemma 4.3, or just the form of C_T that is given in lemma 4.3.
- I think theorem 3.3 needs to assume that M_phi < infinity as it is moved to the otherside of an equation in the proof. I think the proof needs more discussion about the case of gamma^*=infinity as on first read it looks like it moved to another side of the equation in the proof. gamma^*=infinity seems fine as thne the bound is triival. These cases are possible as taking K=1 and phi(t) = (t-1)^(1/3) has M_phi = gamma^*=infinity.  Seems like everything can be infinite in lemma 3.2 as then the bound is trivial.
questions: questions: In theorem 3.4 it is assume that phi_n converges uniformly to the identity function. To me this seems like assuming that curvature of phi_n tends to 0 indictating that as n goes to infinity there is no ambiguity aversion. However, uniform convergence to the identity function is much a much stonger condition than curvature going to 0. For example convergence to 2 times the identity function still has curvature going to 0. Am I wrong in my reasoning behind convergence to the identity function? If not, is it possible to update assumption (3) in theorem 3.4 to a more general curvature to 0 assumption? It seems like the without much modification it can be instead assumed that phi_n converges to some scalar multiple of the idenity function.


How do the assumptions on theorem 3.3,3.4 guarantee that theta_n and theta_* exist? As is isn't assume that h is continuous or Theta is compact it seems like the argmin of V and R can be empty. From the proof of theorem 3.3 it seems like if theta_* doesn't exist it can be replaced by any arbitrary theta.

Another question. The uniform boundedness of h is strongly relied on to get the generate the results. Is it possible/are there methods available to remove this assumption? Classic problems like least squares regression over an unbounded domain will have h unbounded. This often isn't all that bad as assuming compactness of the domain is fine, but the bounds given here depend strongly on the size of the compact set used to restrict the problem to.
limitations: limitations: -
paper_summary: paper_summary: This article proposes a new method for optimisation of risk under uncertainty, using Dirichlet Processes to introduce an extra degree of robustness to modelling uncertainty on the data generating process. The method is generally applicable to many statistical learning problems through the use of loss functions, and further links are made to the economic decision-making literature. The theoretical properties of the methods are explored, and a Monte Carlo approximation is introduced to make tractable the inference for the DP model. Experiments are performed on three simulated experiments, and three real data sets.
strength_weakness: strength_weakness: The idea of placing a prior on the data generating process instead of the parameters is unusual from the traditional Bayesian perspective. While this is not necessarily wrong, the authors do not do a particularly good job of communicating the implications of constructing a prior in this way, or how existing intuitions that the readership may have can be transferred to this new approach.

The results discussion from line 299 onwards has not been prioritised for space in the manuscript: the authors attempt to concentrate reporting results from six different experiments into a single page, which is not enough exposition for the main text of the article. Many fairly important details are pushed into an Appendix and then covered by asserting the “superior ability of our robust method” in the main text. This is far too simple an analysis for an empirical investigation: I don’t trust any analysis that can be summed up so concisely, let alone six combined, with three being real data sets. Appendix C is reasonably thorough in this respect, although there is a noticeable absence of any rigorous testing of the differences in performance metrics.

There are a few small issues with the English in the article: line 8: “among which Ridge…”, line 97: “pervasive” is a strange choice of vocabulary.
questions: questions: Line 110: “In our setting, instead of a parametric prior on the regression coefficients, we place a nonparametric one on the joint distribution of the response and covariates” What is lost by not having priors associated with parameters? This is not necessarily a good thing to move away from: specific belief distributions for specific interpretable model components is a good thing. What the implications for estimating the posterior uncertainty on the parameter level, for example in the regression case? What kind of posterior convergence do you expect on the parameters themselves?

What are the more general implications of putting a prior on the distributions of the data instead of the model parameters? This is suggested to be the case in line 111. Is that not what the likelihood is meant to capture? Is this merely sloppy phrasing in line 111  Does the prior on the distribution of the data come into conflict with the model at some point? What is the difference between altering the prior you have assumed here and altering the statistical model?

What is the expected scalability of the methods to larger data sets? None of those used here are really that big, but the use of SGD implies that you have big-N scalability in mind. The wall clock times on your fairly modest desktop system are quite small, so presumably you have some ability to scale to larger data sets if desired.

Is the trained DP object interpretable at all? Assuming that the learned DP object implies some sort of partition of the data-generating process, then is that partition telling us something we didn’t know beforehand? How restrictive is the DP prior on the space of measures that you might expect in reality?
limitations: limitations: While an elegant choice for the problem at hand, the use of a Dirichlet Process is likely to be a limiting factor for future directions of research, both in terms of the computational limits and the possible allowable partitions.

The assertion of the prior in the way performed here, in contrast to over interpretable parameters, possibly limits the likelihood of this method being widely adopted.

The theoretical results are focussed on convergence to the correct expected risk (or transformations thereof), rather than telling us anything about the parameters themselves.

Comments: title: thank you very much
some comments
discussion
discussion
discussion
comment: I appreciate your kind response.
The authors have engaged with my concerns concerning the article, primarily about the unconventional use of Bayesian reasoning in a new context, and more pragmatic questions of scalability and the presentation of experiments. I appreciate the effort put in.

I am happy to keep my score where it is.
Dear Reviewer,

We thank you again for the time spent reading and commenting our work, and remain at your disposal should any further question arise.

Best regards,

The Authors
Dear Reviewer,

We thank you again for the time spent reading and commenting our work, and remain at your disposal should any further question arise.

Best regards,

The Authors
Dear Area Chair,

Thank you for your insightful comments and careful reading of our submission. We’d like to address your questions as follows:

1. We agree with your observation regarding the need for a discussion on the original sources for results similar to Propositions 2.1 and 3.1, and we will incorporate this in the next version of the paper. Specifically, for Proposition 3.1, we will move the citation of [1], currently in the appendix, to the main body as a standard reference on DP weak convergence to the true generating distribution. Regarding Proposition 2.1, we acknowledge that pinpointing the exact origin of the classical interpretation of Ridge regression as Bayesian linear regression with a *parametric* standard normal prior on the coefficients is challenging. However, as noted after Proposition 2.1, our result offers a novel Bayesian interpretation: by taking an optimization-centric view and placing a *nonparametric* prior directly on the data-generating distributions (instead of on the coefficients, which we optimize), we also arrive at Ridge regularization. We believe this result highlights the fundamental connections between optimization, decision theory, and Bayesian inference, and we will ensure this is clearly explained in the revised paper.

2. We also agree that our paper currently lacks an explicit discussion of our reasoning for relating $\hat V$ to $V_{\boldsymbol\xi^n}$ instead of directly to $\phi(\mathcal R_{p_\star}(\cdot))$. We will address this in the next version of the paper with the following clarifications. As you noted, one's ultimate goal may be to ensure the convergence of $\hat V$ to $\phi(\mathcal R_{p_\star}(\cdot))$, which involves three layers of approximation: from the finite sample size $n$, from the random measure truncation threshold $T$, and from the number of MC samples $N$. In Section 3, we address the first layer by studying the convergence of $V_{\boldsymbol\xi^n}$ to $\phi(\mathcal R_{p_\star}(\cdot))$. In Section 4, we focus on the latter two layers, *given a fixed sample size approximation determined* by $n$, by studying the convergence of $\hat V$ to $V_{\boldsymbol\xi^n}$. Thus, within this logical chain, $V_{\boldsymbol\xi^n}$ serves as a bridging quantity between $\hat V$ and $\phi(\mathcal R_{p_\star}(\cdot))$. The results from Sections 3 and 4 can then be combined to directly establish the convergence of $\hat V$ to $\phi(\mathcal R_{p_\star}(\cdot))$, which involves choosing $T$ and $N$ as functions of $n$ to ensure that the right-hand side of the first equation in the statement of Lemma 4.3 converges to zero. We will emphasize this crucial point in our next revision.

Thank you again for your valuable feedback,

The Authors


*[1] S. Ghosal and A. Van der Vaart. Fundamentals of nonparametric Bayesian inference, volume 44. 378 Cambridge University Press, 2017.*
Thank you for your rebuttal. I remain positive about the paper.
Dear Author(s),

I would like to invite you to clarify some points which I am uncertain about. Thank you very much!

1. It appears that Propositions 2.1 and 3.1 are well-established, though finding a convenient source for citation might be challenging. I wonder if it would be appropriate to mention this in the revised paper.

2. In comparison with the definition in (1), I wonder if it might be more straightforward to directly link V-hat (as defined in (3)) to ϕ(Rp∗(θ))—the quantity on the right-hand side of the equation in Proposition 3.1—for the analysis. Is there any particular challenge or specific reason for using (1)?
Yes, the response sufficiently addresses my concerns.
Dear Reviewer p57R,

We appreciate you submitting your review. The authors have provided replies to your comments. Could you kindly let us know if they have adequately addressed your points?

Best regards, AC
Dear Reviewer Vivx,

Thank you for completing your review. The authors have responded to your feedback. Could you please let us know if their responses sufficiently address your concerns?

Best regards, AC
Dear Reviewer DqoX,

Thank you very much for submitting your review report. The author(s) have posted responses to your review. Could you kindly provide comments on whether your concerns have been adequately addressed?

Best regards, AC