Title: Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bayesian Nonparametrics

Distributionally Robust Optimization

Dirichlet Process

Decision Theory

Ambiguity Aversion

Machine Learning
Abstract: Training machine learning and statistical models often involves optimizing a data-driven risk criterion. The risk is usually computed with respect to the empirical data distribution, but this may result in poor and unstable out-of-sample performance due to distributional uncertainty. In the spirit of distributionally robust optimization, we propose a novel robust criterion by combining insights from Bayesian nonparametric (i.e., Dirichlet process) theory and a recent decision-theoretic model of smooth ambiguity-averse preferences. First, we highlight novel connections with standard regularized empirical risk minimization techniques, among which Ridge and LASSO regressions. Then, we theoretically demonstrate the existence of favorable finite-sample and asymptotic statistical guarantees on the performance of the robust optimization procedure. For practical implementation, we propose and study tractable approximations of the criterion based on well-known Dirichlet process representations. We also show that the smoothness of the criterion naturally leads to standard gradient-based numerical optimization. Finally, we provide insights into the workings of our method by applying it to a variety of tasks based on simulated and real datasets.

Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization

Abstract

Training machine learning and statistical models often involves optimizing a datadriven risk criterion. The risk is usually computed with respect to the empirical data distribution, but this may result in poor and unstable out-of-sample performance due to distributional uncertainty. In the spirit of distributionally robust optimization, we propose a novel robust criterion by combining insights from Bayesian nonparametric (i.e., Dirichlet process) theory and a recent decision-theoretic model of smooth ambiguity-averse preferences. First, we highlight novel connections with standard regularized empirical risk minimization techniques, among which Ridge and LASSO regressions. Then, we theoretically demonstrate the existence of favorable finite-sample and asymptotic statistical guarantees on the performance of the robust optimization procedure. For practical implementation, we propose and study tractable approximations of the criterion based on well-known Dirichlet process representations. We also show that the smoothness of the criterion naturally leads to standard gradient-based numerical optimization. Finally, we provide insights into the workings of our method by applying it to a variety of tasks based on simulated and real datasets.

1 Introduction

In machine learning and statistics applications, several quantities of interest solve the optimization problem

where R p ( θ ) := E ξ ∼ p [ h ( θ, ξ )] is the expected risk associated to decision θ , under cost function h ( θ, ξ ) (measurable in the argument ξ ) and given that the distribution of the (Ξ , B (Ξ)) -valued data ξ is p . 1 For instance, if we are dealing with a supervised learning task where ξ = ( x, y ) ∈ R m -1 × R , h ( θ, ξ ) is usually a loss function ℓ ( f θ ( x ) , y ) quantifying the cost incurred in predicting y with f θ ( x ) - here the decision variable is θ , which parametrizes the function f θ : R m -1 → R . For the rest of the paper, we assume Ξ ⊆ R m , Θ ⊆ R d and h : Θ × Ξ → [0 , K ] for some K < ∞ . 2

In most cases of interest the true data-generating process p ∗ is unknown, and only a sample ξ n = ( ξ 1 , . . . , ξ n ) from it is available. The most popular solution is to approximate p ∗ by the empirical distribution p ξ n , and optimize R p ξ n ( θ ) ≡ n -1 ∑ n i =1 h ( θ, ξ i ) . However, especially for small sample sizes n and complex data-generating mechanisms, this can result in poor out-of-sample performance, leading to the need for robust alternatives. A flourishing literature on Distributionally Robust

Nhat Ho

Optimization (DRO) has provided several methods in that direction [though not always with datadriven applications as the primary focus; see 35, for a recent exhaustive review of the field]. A prominent approach is the min-max DRO (mM-DRO) one, whereby a worst-case criterion over an ambiguity 3 set of plausible distributions is minimized [19, 2, 3, 10, 46, 11]. Recent notable results involve the study of mM-DRO problems where the ambiguity set is defined as a Wasserstein ball of probability measures centered at the empirical distribution [29, 24].

Contribution. Differently from the mM-DRO paradigm, we propose a distributionally robust procedure based on the minimization of the following criterion:

where ϕ : [0 , K ] → R + is a continuous , convex and strictly increasing function, and Q ξ n is a Dirichlet process posterior conditional on ξ n [13].

As we show below, our proposal brings together insights from two well-established strands of literature - decision theory under ambiguity and Bayesian nonparametric statistics, - contributing in a novel way to the field of data-driven distributionally robust optimization. As we establish throughout the article, among the key advantages of the criterion are: (i) its favorable statistical properties in terms of probabilistic finite-sample and asymptotic performance guarantees; (ii) the availability of tractable approximations that are easy to optimize using standard gradient-based methods; and (iii) its ability to both improve and stabilize the out-of-sample performance of standard learning methods.

The rest of the paper is organized as follows. In Section 2, we motivate the formulation in Equation (1) by providing a concise overview of decision theory under ambiguity and its connections to Bayesian statistics and regularization. In Section 3, we study the statistical properties of procedures based on V ξ n . In Section 4, we propose and study tractable approximations for V ξ n based on the theory of DP representations. In Section 5, we highlight the robustness properties of our method by applying it to a variety of learning tasks based on real and simulated data. Section 6 concludes the article. Proofs of theoretical results and further background are provided in Appendices A and B, respectively, while in Appendix C we discuss how the smoothness of the proposed criterion yields straightforward gradient-based optimization, and present more details on the numerical experiments. Code to replicate our experiments can be found at the folllowing link: https://github.com/nbariletto/BNP_ for_DRO .

2 Decision Theory and Bayesian Statistics

Following a long-standing tradition in Bayesian statistics and decision theory [37], the distributional uncertainty on the data-generating process p ∗ can be dealt with by defining a prior Q over it. We point out that this Bayesian approach contrasts with the classical one, where a prior would typically be placed directly on the parameter θ , whose data-driven optimal value would be determined as a function (e.g., the mean or mode) of the resulting posterior distribution. In this new framework, instead, the parameter θ is treated as a variable to be optimized, while the prior is assigned to the entire data-generating process. This perspective allows us to incorporate several valuable Bayesian concepts, as we will clarify throughout the paper, while preserving the flexibility of the original optimization-based learning framework. Notably, θ does not need to be interpreted as the parameter of a full generative model, as would be required in a classical Bayesian setting. Instead, it can represent a vector of parameters associated with a generic, possibly complex loss function, such as those employed in modern deep learning architectures.

Specifically, our Bayesian approach is equivalent to modeling the observed data ξ n = ( ξ 1 , . . . , ξ n ) as exchangeable with de Finetti measure Q :

Due to the stochasticity of p , R p ( θ ) is itself a random variable, and a sensible procedure is to maximize its posterior expectation. Let Q ξ n be the posterior law of p conditional on the sample ξ n .

Then, one solves the following problem:

where P Ξ denotes the space of probability measures on B (Ξ) endowed with the Borel σ -algebra B ( P Ξ ) generated by the topology of weak convergence, while p (d ξ | ξ n ) := ∫ P Ξ p (d ξ ) Q ξ n (d p ) denotes the posterior predictive distribution. In sum, within this general Bayesian framework, the data-driven problem reduces to minimizing h ( θ, ξ ) averaged w.r.t. the posterior predictive distribution, i.e., min θ ∈ Θ R p ( ·| ξ n ) ( θ ) .

2.1 The Dirichlet Process

A natural choice is to model the prior Q as a Dirichlet process (DP), and Q ξ n is then a DP posterior. First proposed by [13], the DP is the cornerstone nonparametric prior over spaces of probability measures. Its specification involves a concentration parameter α > 0 and a centering probability measure p 0 . Intuitively, the DP is characterized by the following finite-dimensional distributions: p ∼ DP ( α, p 0 ) implies ( p ( A 1 ) , . . . , p ( A k )) ∼ Dirichlet ( αp 0 ( A 1 ) , . . . , αp 0 ( A k )) for any finite measurable partition { A 1 , . . . , A k } of Ξ . 4 A key property of the DP is its almost sure discreteness, which allows to write p d = ∑ j ≥ 1 p j δ ξ j (where probability weights and atom locations are independent). Moreover, the DP is conjugate with respect to exchangeable sampling. In our case, this means

That is, conditional on the sample ξ n , p is again a DP with larger concentration parameter α + n and centered at the predictive distribution p ( ·| ξ n ) := α α + n p 0 + n α + n p ξ n . The latter is a compromise between the prior guess p 0 and the empirical distribution p ξ n , and the balance between the two is determined by the relative size of α and n . The predictive distribution is also related to the celebrated Blackwell-MacQueen Pólya urn scheme (or Chinese restaurant process) to draw an exchangeable sequence ( ξ i ) i ≥ 1 distributed according to p ∼ DP ( α, p 0 ) : Draw ξ 1 ∼ p 0 and, for all i > 1 and ℓ < i , set ξ i = ξ ℓ with probability ( α + j -1) -1 , else (i.e., with probability α ( α + j -1) -1 ) draw ξ i ∼ p 0 [4].

Given the large support of Q ξ n , which consists of all probability measures whose support is included in that of p ( ·| ξ n ) [28], the DP is a reasonable and tractable option to mitigate misspecification concerns. Then, leveraging the mentioned expression for the DP predictive distribution, the problem specializes to

[see also 27, 45]. In practice, adopting the above Bayesian approach amounts to introducing a regularization term depending on the prior centering distribution p 0 . Compared to the simple empirical risk R p ξ n ( θ ) , this type of criterion displays lower variance (because R p 0 ( θ ) is non-random) at the cost of some additional, asymptotically-vanishing bias w.r.t. the theoretical criterion R p ∗ ( θ ) . We also note that such bias can be attenuated in finite samples as long as the prior guess p 0 and the true data-generating process p ∗ are close enough in terms of the difference |R p 0 ( θ ) -R p ∗ ( θ ) | . 5

Connections to Regularization in Linear Regression. One of the most ubiquitous data-driven learning tasks is linear regression [38, 7]. It is well-known that, in this setting, coefficient estimation (e.g., via maximum likelihood or least squares) can be framed as a minimization problem of the sample average of the squared loss function. It turns out that, applying the Bayesian regularized approach (2), an interesting equivalence with standard regularization techniques such as Ridge [21] and LASSO [41] emerges.

Proposition 2.1. Let h ( θ, ( y, x )) = ( y -θ ⊤ x ) 2 . Then, denoting λ α,n := α/n , the following equivalences hold:

1. If p 0 = N (0 , I ) , then ˆ θ solving (2) implies that it solves

2. If V = diag ( | θ 1 | -1 , . . . , | θ d -1 | -1 ) and p 0 = N (0 , V ) , then ˆ θ solving (2) implies that it solves

Proposition 2.1 is insightful because it highlights a novel Bayesian interpretation of Ridge and LASSO linear regression. In fact, it is well known that both methods are equivalent to maximuma-posteriori estimation of regression coefficients when the latter are assigned either a normal or a Laplace prior. In our setting, instead of a parametric prior on the regression coefficients, we place a nonparametric one on the joint distribution of the response and the covariates. The degree of regularization, then, is naturally guided by the prior confidence parameter α and the sample size n . We also note that sparsity is only one of the possible data-generating features one might want to enforce in regularized estimation, 6 and the nonparametric Bayesian approach offers greater flexibility, compared to Ridge and LASSO, to incorporate such patterns by specifying the prior expectation p 0 of the joint response-covariate distribution.

2.2 Ambiguity Aversion

As we just showed, adopting a traditional Bayesian framework, uncertainty about the model p is resolved by using the posterior Q ξ n to directly average p out. This procedure, however, does not take into account the (partly) subjective nature of the beliefs encoded in Q ξ n , and the aversion to this that a statistical decision maker (DM) might have. In fact, the result of the procedure is that the DM ends up minimizing the expected risk, where the average is taken according to the predictive distribution. In practice, then, the latter is put on the same footing as an objectively known probability distribution, such as the true model.

This issue has been thoroughly studied and addressed in the economic decision theory literature [18, 6]. In that context, the economic DM faces an analogous expected utility maximization problem max θ ∈ Θ E ξ ∼ p [ u ( θ, ξ )] (e.g., to allocate her capital to a portfolio of investments subject to random economic shocks ξ ). However, she does not possess enough objective information to pick one single model of the world p , but deems a larger set of models plausible. One possibility, then, is that the DMforms a second-order belief (e.g., a prior Q ) over such set, and resolves uncertainty by directly averaging expected utility profiles E ξ ∼ p [ u ( θ, ξ )] w.r.t. p ∼ Q .

Just like in our data-driven problem, however, direct averaging does not account for ambiguity aversion. [23] proposed and axiomatized a tractable 'Smooth Ambiguity Aversion" (SmAA) model, whereby second-order averaging is preceded by a deterministic transformation ϕ inducing uncertainty aversion via its curvature: The DM optimizes ∫ P Ξ ϕ ( E ξ ∼ p [ u ( θ, ξ )]) Q (d p ) , and criterion (1) simply specializes the SmAA model to the data-driven case. 7 When optimization takes the form of minimization, ambiguity aversion is driven by the degree of convexity of ϕ . 8 In particular, convexity encodes the DM's tendency to pick decisions that yield less variable expected loss levels across ambiguous probability models. To see this intuitively, examine the simple case when only two models, p 1 and p 2 , are supported by Q = 1 2 δ p 1 + 1 2 δ p 2 . Consider two decisions θ 1 and θ 2 that, under p 1 and p 2 , yield the expected risks marked on the horizontal axis of Figure 1. While ∫ R p ( θ 1 ) Q (d p ) = ∫ R p ( θ 2 ) Q (d p ) = R ∗ , the convexity of ϕ implies

almost surely.

This ensures that, as more data are collected, the proposed criterion approaches, with probability 1, the true theoretical risk (up to the strictly increasing transformation ϕ ).

While point-wise convergence to the target ground truth is a first desirable property for any sensible criterion, it is not enough to characterize the behavior of the optimization's out-of-sample performance, nor the closeness of the optimal criterion value and the criterion optimizer(s) to their theoretical counterparts. In the following subsections, we study these properties both in the finite-sample regime and in the asymptotic limit n →∞ .

p

1

(

2)

p

1

(

1)

*

p

2

(

1)

p

2

(

2)

0.0

0.5

1.0

1.5

2.0

2.5

∫ ϕ ( R p ( θ 1 )) Q (d p ) < ∫ ϕ ( R p ( θ 2 )) Q (d p ) . That is, although θ 1 and θ 2 yield the same loss in Q -expectation, the ambiguity-averse criterion favors θ 1 because it ensures less variability across uncertain distributions p 1 and p 2 .

Interestingly, [5] showed that the SmAA model belongs to a general class of ambiguity-averse preferences, which admit a common utility function representation. For SmAA preferences with ϕ ( t ) = β exp( β -1 t ) -β (with β > 0 and under additional technical assumptions), this representation implies the equivalence of problem (1) with

where KL ( ·∥· ) is the Kullback-Leibler divergence and ≪ denotes absolute continuity. The above result further clarifies the mechanism through which distributional robustness is induced: Intuitively, instead of directly averaging over p ∼ Q ξ n , one computes a worst-case scenario w.r.t. the mixing measure, penalizing distributions that are further away from the posterior - the latter acts as a reference probability measure. Moreover, in the limiting case β → 0 , the mM-DRO setup is recovered, with ambiguity set C = { p ∈ P Ξ : ∃ P ≪ Q ξ n , p = ∫ P Ξ qP (d q ) } . In the other limiting case β →∞ (with the convention 0 · ∞ = 0 ), the ambiguity neutral Bayesian criterion (2) is instead recovered.

3 Statistical Properties

In this section, we analyze the statistical properties of the criterion V ξ n ( θ ) , as a function of the sample size n . A first issue of interest, addressed in Proposition 3.1, is to study its asymptotic point-wise behavior (cf. [17], Corollary 4.17).

Proposition 3.1. Let ξ n be iid according to p ∗ and ξ ↦→ h ( θ, ξ ) continuous for all θ ∈ Θ . Then, for all θ ∈ Θ ,

Finite-Sample Guarantees. Denote

and we assume the above sets of minimizers to be non-empty throughout the article. In finite-sample analysis, a first question of interest is whether probabilistic performance guarantees hold for the robust criterion optimizer θ n . In our setting, one can naturally measure performance by the narrowness of the gap between R p ∗ ( θ n ) and R p ∗ ( θ ∗ ) . As we clarify later, Lemma 3.2 is a first step towards establishing this type of guarantees.

Lemma 3.2. Let ϕ be twice continuously differentiable on (0 , K ) , with M ϕ := sup t ∈ (0 ,K ) ϕ ′ ( t ) < + ∞ and γ ∗ ϕ := sup t ∈ (0 ,K ) γ ϕ ( t ) < + ∞ , where γ ϕ ( t ) := ϕ ′′ ( t ) /ϕ ′ ( t ) ≥ 0 . Then

Lemma 3.2 links the sup distance of the criterion V ξ n from the theoretical risk to three key objects:

The classical sup distance between the empirical and theoretical risk, sup θ ∈ Θ |R p ξ n ( θ ) -R p ∗ ( θ ) | ;

The sup distance between the theoretical risk and the risk computed w.r.t. the base probability measure p 0 , sup θ ∈ Θ |R p 0 ( θ ) -R p ∗ ( θ ) | . In fact, while in the formulation of Lemma 3.2 we bound such distance by K (see the second addendum) to eliminate the dependence on the unknown but fixed p ∗ , one could equivalently replace K by sup θ ∈ Θ |R p 0 ( θ ) -R p ∗ ( θ ) | . This clarifies that, if p 0 is a good guess for p ∗ , i.e., if the above sup distance is small, adopting a Bayesian prior centered at p 0 can improve finite sample bounds;

The Arrow-Pratt coefficient γ ϕ ( t ) of absolute ambiguity aversion. In the economic theory literature on decision-making under risk, this is a well-known concept measuring the degree of risk aversion of decision makers, with point-wise larger values of γ ϕ corresponding to more risk aversion. See [23] for a discussion on the straightforward adaptation of this measure to the ambiguity (rather than risk) aversion setup we work in.

Most importantly, Lemma 3.2 allows us to prove the following Theorem, which yields the performance guarantees we are after.

Theorem 3.3. For all δ > 0

Theorem 3.3 allows to obtain finite-sample probabilistic guarantees on the excess risk ϕ ( R p ∗ ( θ n )) -ϕ ( R p ∗ ( θ ∗ )) via bounds on sup θ ∈ Θ ∣ ∣ R p ξ n ( θ ) -R p ∗ ( θ ) ∣ ∣ . The latter is a well-studied quantity, and the sought bounds follow from standard results relying on conditions on the complexity of the function class H := { h ( θ, · ) : θ ∈ Θ } , as measured by its Vapnik-Chervonenkis dimension, metric entropy, etc. We refer the reader to [44, 43] for a systematic treatment of the topic and specific useful results.

Asymptotic Guarantees. So far, we have studied the finite-sample behavior of the out-of-sample performance of θ n . Another closely related type of results deals with the asymptotic limit of such performance, as well as with the convergence of optimum criterion values and optimizing parameters to their ground-truth counterparts. In this Subsection, attention is turned to theoretical results of this kind.

Finite-sample guarantees on sup θ ∈ Θ ∣ ∣ R p ξ n ( θ ) -R p ∗ ( θ ) ∣ ∣ are usually of the form

with ∑ ∞ n =1 η n < ∞ . This implies (via a straightforward application of the first Borel-Cantelli Lemma) the almost sure vanishing of sup θ ∈ Θ ∣ ∣ R p ξ n ( θ ) -R p ∗ ( θ ) ∣ ∣ . Thus, we include this as an assumption of the next Theorem. Moreover, we introduce a functional dependence of ϕ on n , and denote ϕ ≡ ϕ n accordingly.

Theorem 3.4. Retain the assumptions of Lemma 3.2 and lim n →∞ sup θ ∈ Θ ∣ ∣ R p ξ n ( θ ) -R p ∗ ( θ ) ∣ ∣ = 0 almost surely. Moreover, assume that ϕ n satisfies (1) lim n →∞ γ ∗ ϕ n = 0 , (2) sup n ≥ 1 M ϕ n < ∞ , and (3) lim n →∞ sup t ∈ [0 ,K ] | ϕ n ( t ) -t | = 0 . Then the next two almost sure limits hold:

Theorem 3.4 is crucial because it ensures that, asymptotically, the excess risk vanishes and the finite-sample optimal value converges to the optimal value under the data generating process. 9

Remark 3.5 . From a design point of view, the type of n -dependent parametrization of ϕ required in Theorem 3.4 is sensible, as it is equivalent to adopting vanishing levels of ambiguity aversion (uniformly vanishing Arrow-Pratt coefficient) as the sample size grows - that is, as one obtains a more and more precise estimate of the true distribution p ∗ . Moreover, this assumption is in the spirit of the condition imposed on the radius of the Wasserstein ambiguity ball in [29], which is required to vanish as the sample size grows. Finally, it is easy to show that ϕ n ( t ) = β n exp( β -1 n t ) -β n , for positive β n →∞ , satisfies the conditions of Theorem 3.4, and from now on we silently assume this form for ϕ n .

Finally, we leverage the above results to ensure the convergence of the sequence of optimizers ( θ n ) n ≥ 1 to a theoretical optimizer.

Theorem 3.6. Let θ ↦→ h ( θ, ξ ) be continuous for all ξ ∈ Ξ and lim n →∞ R p ∗ ( θ n ) = R p ∗ ( θ ∗ ) almost surely (e.g., as ensured in Theorem 3.4). Then, almost surely, lim n →∞ θ n = ¯ θ implies R p ∗ ( ¯ θ ) = R p ∗ ( θ ∗ ) .

4 Monte Carlo Approximation

In what follows, we fix a sample ξ n and propose simulation strategies to estimate V ξ n ( θ ) . The latter, in fact, is analytically intractable due to the infinite dimensionality of Q ξ n . To that end, we exploit a key representation of DPs first established by [39]: If p ∼ DP ( η, q ) , then p d = ∑ ∞ j =1 p j ξ j , where ξ j iid ∼ q and the sequence of weights ( p j ) j ≥ 1 is constructed via a stick-breaking procedure based on iid Beta (1 , η ) samples (see Appendix B). Thus, for large enough integers T and N , we propose the following Stick-Breaking Monte Carlo (SBMC) approximation for V ξ n ( θ ) :

where, T denotes the number of stick-breaking steps performed before truncating each Monte Carlo sample from the DP posterior Q ξ n , while N denotes the number of such samples. Algorithm 1 in Appendix B details the procedure, which essentially approximates the posterior DP via truncation and takes expectations accordingly.

Remark 4.1 . We propose to truncate the stick-breaking procedure at some fixed step T . Another strategy would involve truncating it at a random step T i ( ε ) := min { t ∈ N : ∏ t j =1 p ij ≤ ε } for some small ε > 0 . This allows to directly control the approximation error at each Monte Carlo sample [30, 1], though it leads to simulated measures with supports of different cardinalities. For the sake of theory, we opt for the fixed-step/random-error approximation, though the random-step/fixed-error one is equally viable in practice.

Remark 4.2 . On top of being a theory-based approximation for V ξ n ( θ ) , the criterion (3) can be interpreted as implementing a form of robust Bayesian bootstrap . Instead of directly averaging the risk h ( θ, · ) with respect to the empirical distribution, we first obtain N bootstrap samples of

size T from the predictive (which is a compromise between the empirical and the prior centering distributions), we weight observations according to the stick-breaking procedure, and finally take a grand average of the ϕ -transformed weighted sums. This connection with the Bayesian bootstrap suggests the following alternative Multinomial-Dirichlet Monte Carlo (MDMC) version of V ξ n ( θ ) :

where ( w i 1 , . . . , w iT ) iid ∼ Dirichlet ( T ; α + n T , . . . , α + n T ) and the atoms are iid according to the predictive (see Algorithm 2 in Appendix B). 10 For practical computation, we recommend using the MDMC approximation, as it tends to yield more balanced weights, compared to SBMC, even for low values of T .

With the following results, we ensure finite-sample and asymptotic guarantees on the closeness of optimization procedures based on the SBMC approximation versus the target V ξ n .

Lemma 4.3. Assume Θ is a bounded subset of R d and, for all ξ ∈ Ξ , θ ↦→ h ( θ, ξ ) is c ( ξ ) -Lipschitz continuous. Then, for all T, N ≥ 1 and ε > 0 ,

with probability at least

for some constant C T > 0 .

Heuristically, the bound in Lemma 4.3 is obtained by decomposing the left-hand side of the inequality into a first term depending on the truncation error induced by the threshold T , and a second term reflecting the Monte Carlo error related to N . Moreover, analogously to Lemma 3.2, Lemma 4.3 easily implies finite-sample bounds on the excess 'robust risk" V ξ n ( ˆ θ n ( T, N )) -V ξ n ( θ n ) , where ˆ θ n ( T, N ) ∈ arg min θ ∈ Θ ˆ V ξ n ( θ, T, N ) . Another consequence is the following asymptotic convergence Theorem, whose proof is analogous to that of Theorem 3.4.

Theorem 4.4. Under the same assumptions of Lemma 4.3, and if sup T ≥ 1 C T < ∞ (see Appendix A for details on C T ),

almost surely. Also, almost surely

In words, Theorem 4.4 ensures that, as the truncation and MC approximation errors vanish, the optimal approximate criterion value converges to the optimal exact one, and that the exact criterion value at any approximate optimizer converges to the exact optimal value. Also note that Theorems 3.4 and 4.4, when combined, provide guarantees on the convergence of ˆ V ξ n ( ˆ θ n ( T, N ) , T, N ) (the empirical criterion one has optimized in practice) to R p ∗ ( θ ∗ ) (the theoretical optimal target) as the sample size increases and the DP approximation improves.

Finally, as a byproduct of Theorem 4.4, convergence of any approximate robust optimizer to an exact one is established as follows. 11

Theorem 4.5. Let θ ↦→ h ( θ, ξ ) be continuous for all ξ ∈ Ξ . Moreover, assume

almost surely (e.g., as ensured above). Then, almost surely, lim T,N →∞ ˆ θ n ( T, N ) = ¯ θ n implies V ξ n ( ¯ θ n ) = V ξ n ( θ n ) .

5 Experiments

We applied our robust optimization procedure to a host of simulated and real datasets, and we report results in this Section. Before proceeding, we notice that, given the finite approximations proposed in Section 4 and under mild regularity assumptions on the loss function h , the proposed robust criterion is amenable to standard gradient-based optimization procedures (see Appendix C for further details and an insightful interpretation of the gradient of our criterion as yielding robustly weighted stochastic gradient descent steps ).

Simulation Studies. We tested our method on three different learning tasks featuring a high degree of distributional uncertainty in the data generating process, and compared performance to the corresponding ambiguity neutral (i.e., simply regularized) and unregularized procedures. First, we performed a high-dimensional sparse linear regression simulation experiment. We simulated 200 independent samples of size n = 100 from a linear model with d = 90 features (moderately correlated with each other), only the first s = 5 of which have unitary positive marginal effect on the scalar response y . Second, we performed a simulation experiment on univariate Gaussian mean estimation in the presence of outliers. We simulated 200 independent samples of 13 observations, 10 of which come from a 0-mean Gaussian distribution and 3 from an outlier distribution, and tested the ability of the three methods to recover the true mean (i.e., 0). Third, we performed a simulation experiment on high-dimensional sparse logistic regression for binary classification. We set up a data-generating mechanism similar to the linear regression experiment, where a small subset of features linearly influence the log odds-ratio.

Appendix C collects further details on the above experiments as well as plots summarizing the results (see Figures 2, 3, and 4). All three experiments reveal the ability of our robust method to improve out-of-sample performance and estimation accuracy in two ways, i.e., (i) by yielding good results on average, and especially (ii) by reducing performance variability. The latter is a key robustness property that our method is designed to achieve.

Real Data Applications. We tested our method on three diverse real-world datasets. In the first study, we applied our method to predict diabetes development based on a host of features, as collected in the popular and public Pima Indian Diabetes dataset. Because the outcome is binary, we used logistic regression as implemented (i) with our robust method, (ii) with L 1 regularization, and (iii) in its plain, unregularized version. We selected hyperparameters via cross-validation and tested the out-of-sample performance of the three methods applied to disjoint batches of training observations to assess the methods' performance variability. As we report in Appendix C, our robust method outperforms both alternatives on average and does significantly better in reducing variability.

We performed two further studies on linear regression applied to two popular UCI Machine Learning Repository datasets: The Wine Quality dataset [8] and the Liver Disorders dataset [15]. Similarly to the first study, we compared the performance of our method to OLS (unregularized) estimation and L 1 -penalized (LASSO) regression. After cross-validation for parameter selection, we train the models multiple times on separate batches of data and compute out-of-sample performance on a large held-out set of observations. As the results reported in Appendix C show, also in these settings our robust DP-based method performs better than the alternatives both on average and especially in terms of lower variability. Taken together, the experimental results described in this Section corroborate empirically the robustness properties of the proposed criterion, as examined theoretically throughout the paper.

6 Discussion

The paper tackled the problem of optimizing a data-driven criterion in the presence of distributional uncertainty about the data-generating mechanism. To mitigate the underperformance of classical methods, we introduced a novel distributionally robust criterion, drawing insights from Bayesian nonparametrics and a decision-theoretic model of smooth ambiguity aversion. We established connections with standard regularization techniques, including Ridge and LASSO regression, and theoretical analysis revealed favorable finite-sample and asymptotic guarantees on the performance of the robust procedure. For practical implementation, we presented and examined tractable approximations of the criterion, which are amenable to gradient-based optimization. Finally, we applied our method

to a variety of simulated and real datasets, offering insights into its practical robustness properties. Naturally, our work presents some limitations that give rise to interesting directions for future research. In particular, we note the need for a deeper examination of the model workings in terms of (i) its parameter configuration and (ii) its broader application to general learning tasks (e.g., when the loss function is adapted to accommodate deep learning architectures). Moreover, our method, as many others in the distributional robustness literature, is only suited to process homogeneously generated (e.g., iid or exchangeable) data, leaving room to explore extensions to more complex dependence structures. Finally, we highlight that our study offers prospects for investigating connections among such varied yet interconnected strands of literature as optimization, decision theory, and Bayesian statistics.

References

J. Arbel, P. De Blasi, and I. Prünster. Stochastic Approximations to the Pitman-Yor Process. Bayesian Analysis , 14(4):1201 - 1219, 2019.

A. Ben-Tal, D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science , 59(2):341357, 2013.

D. Bertsimas, X. V. Doan, K. Natarajan, and C.-P. Teo. Models for minimax stochastic linear optimization problems with risk aversion. Mathematics of Operations Research , 35(3):580-602, 2010.

D. Blackwell and J. B. MacQueen. Ferguson distributions via Pólya urn schemes. The Annals of Statistics , 1(2):353-355, 1973.

S. Cerreia-Vioglio, F. Maccheroni, M. Marinacci, and L. Montrucchio. Uncertainty averse preferences. Journal of Economic Theory , 146(4):1275-1330, 2011.

S. Cerreia-Vioglio, F. Maccheroni, M. Marinacci, and L. Montrucchio. Ambiguity and robust statistics. Journal of Economic Theory , 148(3):974-1049, 2013.

R. Christensen. Plane Answers to Complex Questions: The Theory of Linear Models . Springer, 2020.

P. Cortez, A. Cerdeira, F. Almeida, T. Matos, , and J. Reis. Wine Quality. UCI Machine Learning Repository, 2009. DOI: https://doi.org/10.24432/C56S3T.

P. De Blasi, S. Favaro, A. Lijoi, R. H. Mena, I. Prünster, and M. Ruggiero. Are Gibbs-type priors the most natural generalization of the Dirichlet process? IEEE transactions on pattern analysis and machine intelligence , 37(2):212-229, 2015.

E. Delage and Y. Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research , 58(3):595-612, 2010.

J. C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics , 49(3):1378-1406, 2021.

B. Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution , pages 569-593. Springer, 1992.

T. S. Ferguson. A Bayesian analysis of some nonparametric problems. The Annals of Statistics , pages 209-230, 1973.

T. S. Ferguson. Prior distributions on spaces of probability measures. The Annals of Statistics , 2(4):615-629, 1974.

R. S. Forsyth. Liver Disorders. UCI Machine Learning Repository, 1990. DOI: https://doi.org/10.24432/C54G67.

G. Garrigos and R. M. Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235 , 2023.

S. Ghosal and A. Van der Vaart. Fundamentals of nonparametric Bayesian inference , volume 44. Cambridge University Press, 2017.

I. Gilboa and M. Marinacci. Ambiguity and the Bayesian paradigm. Readings in formal epistemology: Sourcebook , pages 385-439, 2016.

I. Gilboa and D. Schmeidler. Maxmin expected utility with non-unique prior. Journal of mathematical economics , 18(2):141-153, 1989.

A. Gnedin and J. Pitman. Exchangeable Gibbs partitions and Stirling triangles. Journal of Mathematical Sciences , 138(3):5674-5685, 2006.

A. E. Hoerl and R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics , 12(1):55-67, 1970.

J. F. C. Kingman. Poisson processes , volume 3. Clarendon Press, 1992.

P. Klibanoff, M. Marinacci, and S. Mukerji. A smooth model of decision making under ambiguity. Econometrica , 73(6):1849-1892, 2005.

D. Kuhn, P. M. Esfahani, V. A. Nguyen, and S. Shafieezadeh-Abadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research &management science in the age of analytics , pages 130-166. INFORMS, 2019.

J. Lafferty, H. Liu, and L. Wasserman. Concentration of measure. Technical report, Carnegie Mellon University, 2010.

A. Lijoi and I. Prünster. Models beyond the Dirichlet process. In N. L. Hjort, C. Holmes, P. Müller, and S. G. Walker, editors, Bayesian Nonparametrics , Cambridge Series in Statistical and Probabilistic Mathematics, page 80-136. Cambridge University Press, 2010.

S. Lyddon, S. Walker, and C. C. Holmes. Nonparametric learning from Bayesian models with randomized objective functions. Advances in Neural Information Processing Systems , 31, 2018.

S. Majumdar. On topological support of Dirichlet prior. Statistics & Probability Letters , 15(5):385-388, 1992.

P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming , 171(1-2):115-166, 2018.

P. Muliere and L. Tardella. Approximating distributions of random functionals of FergusonDirichlet priors. Canadian Journal of Statistics , 26(2):283-297, 1998.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

M. Perman, J. Pitman, and M. Yor. Size-biased sampling of Poisson point processes and excursions. Probability Theory and Related Fields , 92(1):21-39, 1992.

J. Pitman. Exchangeable and partially exchangeable random partitions. Probability theory and related fields , 102(2):145-158, 1995.

J. Pitman. Some developments of the Blackwell-Macqueen urn scheme. In T. S. Ferguson, L. S. Shapley, and J. B. MacQueen, editors, Statistics, probability and game theory: Papers in honor of David Blackwell , volume 30 of IMS Lecture Notes - Monograph Series , pages 245-267. Institute of Mathematical Statistics, 1996.

H. Rahimian and S. Mehrotra. Frameworks and results in distributionally robust optimization. Open Journal of Mathematical Optimization , 3:1-85, 2022.

E. Regazzini, A. Lijoi, and I. Prünster. Distributional results for means of normalized random measures with independent increments. The Annals of Statistics , 31(2):560-585, 2003.

L. J. Savage. The foundations of statistics . Courier Corporation, 1972.

G. A. Seber and A. J. Lee. Linear regression analysis . John Wiley & Sons, 2003.

J. Sethuraman. A constructive definition of Dirichlet priors. Statistica sinica , pages 639-650, 1994.

A. Shapiro, E. Zhou, and Y. Lin. Bayesian distributionally robust optimization. SIAM Journal on Optimization , 33(2):1279-1304, 2023.

R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology , 58(1):267-288, 1996.

A. W. Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge University Press, 2018.

M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

S. Wang, H. Wang, and J. Honorio. Distributional robustness bounds generalization errors. arXiv preprint arXiv:2212.09962 , 2022.

W. Wiesemann, D. Kuhn, and B. Rustem. Robust Markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

D. Wu, H. Zhu, and E. Zhou. A Bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics. SIAM Journal on Optimization , 28(2):1588-1612, 2018.

E. Zhou and W. Xie. Simulation optimization when facing input uncertainty. In 2015 Winter Simulation Conference (WSC) , pages 3714-3724. IEEE, 2015.

Supplement to 'Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization"

This Supplement to 'Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization" is organized as follows. In Appendix A, we collect the proofs of all results presented in the main text. In Appendix B, we provide further background on Dirichlet process representations and related posterior simulation algorithms. In Appendix C, we describe in detail gradient based optimization of our criterion and the experiments presented in the paper. Finally, in Appendix D we provide details on our computational infrastructure.

A Technical Proofs and Further Results

Proof of Proposition 3.1. Let ⇝ denote weak convergence of probability measures. By Corollary 4.17 in [17], Q ξ n ⇝ δ p ∗ almost surely. That is,

almost surely for any bounded and continuous f : P Ξ → R . Thus, we are left to prove that p ↦→ ϕ ( R p ( θ )) is bounded and continuous for all θ ∈ Θ . It is bounded because ϕ is continuous on the compact interval [0 , K ] , and it is continuous because p ↦→R p ( θ ) is continuous (by the definition of topology of weak convergence and because ξ ↦→ h ( θ, ξ ) is bounded and continuous) and ϕ is continuous.

Proof of Lemma 3.2. First note that, by the stated assumptions, it follows from Taylor's theorem that

for all p ∈ P Ξ and θ ∈ Θ and for some c p,θ ∈ [0 , K ] . Then

Proof of Theorem 3.3. Notice the following decomposition:

Then, Lemma 3.2 implies that, for all δ > 0 ,

Proof of Theorem 3.4. Since

almost surely and given assumptions (1) and (2) on ϕ n , by Lemma 3.2 we obtain

almost surely. Then, by decomposition (4),

almost surely and

almost surely. As a consequence,

almost surely. Now recall assumption (3), i.e., the sequence ( ϕ n ) n ≥ 1 converges uniformly to the identity map. Then, in light of the previous observations and by noticing that

and

the two desired almost sure limits follow:

Proof of Theorem 3.6. We have

almost surely, where the first equality follows from the continuity of θ ↦→ h ( θ, ξ ) and the second one from the Dominated Convergence Theorem. Then, R p ∗ ( ¯ θ ) = R p ∗ ( θ ∗ ) almost surely, proving the result.

To Prove Lemma 4.3, we introduce two other Lemmas. After proving those, Lemma 4.3 follows immediately.

Lemma A.1. For all T, N ≥ 1 ,

Proof. We have

Note that

where the last equality follows from the mean value theorem applied to endpoints ∑ ∞ j =1 p j h ( θ, ξ j ) and ∑ ∞ j =1 p j h ( θ, ξ j ) + p 0 h ( θ, ξ 0 ) -∑ ∞ j = T +1 p j h ( θ, ξ j ) . Then the second term in (6) is bounded by

The second term on the left-hand side of Equation(5) is instead of the form

where X i = ∑ T j =0 p ij h ( θ, ξ ij ) are iid random variables whose distribution is determined by the truncated stick-breaking procedure.

The aim of the next Lemma is to provide sufficient conditions for finite sample bounds and asymptotic convergence to 0 of the term in (7). Specifically, we impose complexity constraints on the function class H := { ξ ↦→ h ( θ, ξ ) : θ ∈ Θ } which allow us to obtain appropriate conditions on the derived class

ensuring the asymptotic and non-asymptotic results we seek.

Lemma A.2. Assume Θ is a bounded subset of R d and θ ↦→ h ( θ, ξ ) is c ( ξ ) -Lipschitz continuous for all ξ ∈ Ξ . Then, for all T, N ≥ 1 and ε > 0 ,

with probability at least

for some constant C T > 0

Proof. By the Lipschitz continuity assumption on h ( θ, ξ ) , we obtain that, for all ( p j , ξ j ) T j =0 , θ ↦→ ϕ ( ∑ T j =1 p j h ( θ, ξ j ) + p 0 h ( θ, ξ 0 ) ) is M ϕ ˜ c ( ( p j , ξ j ) T j =0 ) -Lipschitz continuous, with

Indeed, for all θ 1 , θ 2 ∈ Θ ,

Therefore, denoting by P the law of the vector ( p j , ξ j ) T j =0 and by N [] ( ε, F , L 1 ( P )) the associated ε -bracketing number of the class F , by Lemma 7.88 in [25] we obtain

with C T := ∫ ˜ c ( ( p j , ξ j ) T j =0 ) d P . Then the result follows by Theorem 7.86 in [25] after noticing that sup f ∈ F ∥ f ∥ L 1 ( P ) ≤ sup f ∈ F ∥ f ∥ ∞ ≤ ϕ ( K ) .

Proof of Theorem 4.5. We have

almost surely, where the first two equalities follow from the continuity of θ ↦→ h ( θ, ξ ) and ϕ as well as from an iterated application of the Dominated Convergence Theorem (recall that h ( θ, ξ ) ∈ [0 , K ] by assumption for all θ and ξ ). This implies V ξ n ( θ n ) = V ξ n ( ¯ θ n ) almost surely.

In the next result, we will assume ϕ ( t ) ≡ ϕ β ( t ) = β exp( t/β ) -β . Moreover, we will emphasize, through superscripts, the dependence of mathematical objects on β and the DP concentration parameter α . Moreover, if necessary, we make the truncation threshold T n and the number of MC samples N n dependent on the sample size n .

Proposition A.3. Assume Θ is an open subset of R d and lim N →∞ lim T →∞ sup θ ∈ Θ | ˆ V α,β ξ n ( θ, T, N ) -V α,β ξ n ( θ ) | = 0 almost surely for all α > 0 and β > 0 . Moreover, assume that

θ ↦→ h ( ξ, θ ) is differentiable at θ ∗ ∈ arg min θ ∈ Θ R p ∗ ( θ ) for p ∗ -almost every ξ ∈ Ξ , with gradient ∇ θ ∗ ( ξ ) ;

For all θ 1 and θ 2 in a neighborhood of θ ∗ , there exists a measurable function ξ ↦→ H ( ξ ) ∈ L 2 p ∗ such that | h ( θ 1 , ξ ) -h ( θ 2 , ξ ) | ≤ H ( ξ ) ∥ θ 1 -θ 2 ∥ ;

θ ↦→R p ∗ ( θ ) admits a second-order Taylor expansion at θ ∗ , with non-singular symmetric Hessian matrix V θ ∗ .

Then, with probability 1, there exist sequences ( T n ) n ≥ 1 , ( N n ) n ≥ 1 , ( β n ) n ≥ 1 , (diverging to ∞ ) and ( α n ) n ≥ 1 (converging to 0), such that

provided ˆ θ α n ,β n n ( T n , N n ) p → θ ∗ as n →∞ .

Proof. The imposed assumptions match those listed in Theorem 5.23 of [42]. The only condition left to prove is that there exist sequences ( T n ) n ≥ 1 , ( N n ) n ≥ 1 , ( β n ) n ≥ 1 , (diverging to ∞ ) and ( α n ) n ≥ 1 (converging to 0), such that R p ξ n ( ˆ θ α n ,β n n ( T n , N n )) -inf θ ∈ Θ R p ξ n ( θ ) ≤ o p ( n -1 ) . We do so by proving that, for any fixed n ≥ 1 , R p ξ n ( ˆ θ α,β n ( T, N )) → inf θ ∈ Θ R p ξ n ( θ ) almost surely as α → 0 and β, T, N →∞ ; this implies that, for all n , with probability 1 there exist α n , β n , T n and N n such that R p ξ n ( ˆ θ α n ,β n n ( T n , N n )) -inf θ ∈ Θ R p ξ n ( θ ) ≤ ε n for any ε n > 0 . Moreover, it is easy to see that the result is implied by sup θ ∈ Θ | ˆ V α,β ξ n ( θ, T, N ) -R p ξ n ( θ ) |→ 0 , so we prove the latter. We have

where the first term converges to 0 almost surely by assumption. Using a second-order Taylor expansion of ϕ β ( R p ( θ )) around R p ξ n ( θ ) , the second term, instead, satisfies

as α → 0 and β →∞ .

Remark A.4 . Theorems 3.6 and 4.5 ensure that (a) for all n ≥ 1 , provided ˆ θ n ( T, N ) converges almost surely to some θ n ∈ Θ , the latter is a minimizer of V ξ n ( θ ) ; and (b) if the above sequence ( θ n ) n ≥ 1 converges almost surely to some θ ∗ ∈ Θ , the latter is a minimizer of R p ∗ ( θ ) . Notice that the assumptions required for these results are consistent with the ones of Proposition A.3, so they can be used to justify the condition ˆ θ α n ,β n n ( T n , N n ) p → θ ∗ . For instance, if one assumes almost sure uniqueness of minimizers and almost sure convergence of the above defined sequences, ˆ θ α n ,β n n ( T n , N n ) p → θ ∗ can be guaranteed leveraging the preceding results.

Stochastic Gradient Descent Convergence Analysis. The following results refer to material presented in Appendix C below. For ease of exposition, we fix B = 1 and denote by E t the expectation operator conditional on the realization of the random index draws m 1 , . . . , m t iid ∼ Uniform ( { 1 , . . . , M } ) .

Proposition A.5. Assume that V is convex and that ( θ t ) t ≥ 1 follows Equation (9) ) for some starting value θ 0 ∈ Θ and B = 1 . Moreover, assume that, for all θ ∈ Θ ,

Then where θ ∗ ∈ arg min θ ∈ Θ V ( θ ) , ˜ θ T := ∑ T t =0 ν t θ t , and ν t := η t ∑ T t ′ =0 η t ′ .

Proof of Proposition A.5. Fix t = 1 , . . . , T . We have,

Applying the law of total expectation and the fact that ℓ m t ϕ ′ ( H m t ( θ t )) ∇ θ h ( θ t , ξ m t ) is unbiased for ∇ θ V ( θ t ) ,

Hence

Summing over t = 0 , . . . , T and since

because V is convex, we have

Dividing both sides by ∑ T t =0 η t and exploiting (i) the linearity of the expectation operator, (ii) the convexity of the weights ( ν t ) T t =0 , and (iii) the convexity of V , the result follows.

Proof of Proposition 2.1. As for case 1, given the assumed form of p 0 and the criterion representation (2), we are left to establish an expression for E ξ ∼ p 0 [ h ( θ, ξ )] = E ( y,x ) ∼N (0 ,I ) [( y -θ ⊤ x ) 2 ] . Notice that -θ j x j id ∼ N (0 , θ 2 j ) , independendently of y ∼ N (0 , 1) , so that y -θ ⊤ x ∼ N (0 , 1+ ∥ θ ∥ 2 2 ) . Therefore, E ξ ∼ p 0 [ h ( θ, ξ )] = V [ y -θ ⊤ x ] = 1 + ∥ θ ∥ 2 2 , which is easily seen to complete the proof. Finally, the proof for the LASSO case is completely analogous to the Ridge one and is therfore omitted.

B Further Background on the Dirichlet Process and Approximation Algorithms

Since its definition by [13] based on the family of finite-dimensional Dirichlet distributions (as sketched in Section 2), the Dirichlet process has been characterized (and thus generalized) in a number of useful ways. For instance, the DP can be derived as a neutral to the right process [14], a normalized completely random measure [13, 22, 26, 36], a Gibbs-type prior [20, 9], a PitmanYor Process [33, 32], and a species sampling model [34]. In what follows, we review two other constructions of the DP which were at the basis of the approximate versions of the robust criterion V ξ n proposed in Section 4.

Stick-Breaking Construction of the Dirichlet Process. [39] proved that Ferguson's 1973 Dirichlet process enjoys the following 'stick-breaking" representation

where

Algorithm 1 SBMC Approximation

Input: Data ξ , model parameters, number of MC samples N , truncation step T

The name of the procedure comes from the analogy with breaking a stick of length 1 into two pieces of length B 1 and 1 -B 1 , then the second piece into two sub-pieces of length (1 -B 1 ) B 2 and (1 -B 1 )(1 -B 2 ) , and so on. In Algorithm 1, then, we simulate N realizations from Q ξ n , truncating the stick-breaking procedure at step j = T . The remaining portion of the stick is then allocated to one further atom drawn from the predictive distribution. Then, the intractable integral with respect to the DP posterior is approximated via a Monte Carlo average of the integrals (i.e., weighted sums) with respect to the N simulated measures.

Multinomial-Dirichlet Construction of the Dirichlet Process and Monte Carlo Algorithms.

Another finite-dimensional approximation of p ∼ DP ( α, P ) is p T = ∑ T j =1 p j δ x j , with x j iid ∼ P and ( p 1 , . . . , p T ) ∼ Dirichlet ( T ; α/T,.. . , α/T ) . As T → ∞ , p T approaches p [see Theorem 4.19 in 17]. Hence, one can approximate V ξ n ( θ ) as in Algorithm 2, where the concentration parameter is α + n and the centering distribution coincides with the predictive.

Algorithm 2 Multinomial-Dirichlet Monte Carlo (MDMC) Approximation

When α is negligible compared to the sample size n , one can simplify posterior simulation by setting α = 0 . Thus, one obtains a DP ( n, p ξ n ) posterior. This distribution enjoys a useful representation as follows: p ∼ DP ( n, p ξ n ) = ⇒ p d = ∑ n i =1 p i ξ i , with p i ∼ Dirichlet ( n ; 1 , . . . , 1) [see 17, Section 4.7]. Due to its similarity to the usual bootstrap procedure [12], this distribution is known as the 'Bayesian bootstrap". Algorithm 3 implements the Bayesian bootstrap to approximate the criterion V ξ n ( θ ) . In practice, however, we do not recommend resorting to the Bayesian bootstrap approximation, since DP ( n, p ξ n ) assigns probability 1 to the set of distributions with strictly positive support on ξ n . This goes against the prescription that, as the finite sample ξ n provides only partial information on the true underlying distribution, the statistical DM should be willing to consider a wider set of distributions other than the ones supported at the sample realizations.

C Numerical Optimization and Experiment Details

In this Section, we first describe the SGD algorithm used in practice for our experiments. Then, we describe in full detail the experiments presented in the paper.

Gradient-Based Optimization. Whether we resort to the SBMC or the MDMC approximation of V ξ n ( θ ) , we are faced with the task of minimizing a criterion of the form

The smoothness and convexity of ϕ make it appealing to minimize the criterion via gradient-based convex optimization techniques. Indeed, it is enough to assume that θ ↦→ h ( θ, ξ ) is convex and differentiable (a standard assumption met in many applications of interest) to easily yield the same properties for V .

In light of this, assuming that θ ↦→ h ( θ, ξ ) is differentiable at every ξ ij and denoting H i ( θ ) := ∑ T j =1 p ij h ( θ, ξ ij ) , the gradient of V is

where ℓ m ≡ Tp m and the m -indexing is just a recoding of the indices (with a slight abuse of notation and M ≡ N · T ). That is, the gradient of V ( θ ) can be written as the average of M terms. Thus, to minimize V ( θ ) we propose a mini-batch Stochastic Gradient Descent algorithm which, at each iteration t , updates the parameter vector as follows:

for a step-size η t > 0 and a random subset (mini-batch) of size B from the indices { 1 , . . . , M } . Under standard regularity assumptions [16], in Proposition A.5 (Appendix A) we prove convergence of the algorithm at usual rates for convex problems.

Remark C.1 . Expression (8) provides some insight on how, in practice, distributional robustness is enforced. Notice that H i ( θ ) = R p i ( θ ) is the expected risk computed according to p i , an approximate realization from Q ξ n . Thus, in the computation of the overall gradient ∇ θ V ( θ ) , the gradients associated to the p i 's that generate higher expected risks receive more weight (being ϕ convex, ϕ ′ is increasing). These weights, then, are reflected into which gradients, in the mini-batch SGD algorithm, are given more leverage in updating the parameter vector. Thus, the procedure can be thought of as implementing a 'soft worst-case scenario" scheme, whereby distributions in the posterior support are weighted (in terms of gradient influence) more the worse they do in terms of expected risk.

Mini-Batch Stochastic Gradient Descent Algorithm. For practical optimization, we apply a modification to the SGD algorithm provided in Equation (9), which helps to reduce the computational

burden of the procedure. Indeed, recall the formula of the gradient of the criterion V that we need to optimize:

Clearly, then, implementing the baseline SGD algorithm requires, at each iteration, the evaluation of multiple H m ( θ t ) terms, each consisting of T evaluations of the loss function h . To avoid this, at each iteration we instead sub-sample one index i = 1 , . . . , N and update the parameter vector according to the associated gradient ϕ ′ ( H i ( θ t )) ∇ θ H i ( θ t ) . The latter is still an unbiased estimator of the overall gradient of V ( θ t ) , but it requires only T evaluations of h (plus those of T gradients of h , similarly to the baseline algorithm). Finally, to exploit the whole data efficiently, we sub-sample without replacement and perform multiple passes over the N MCsamples. Algorithm 4 summarizes the procedure.

Algorithm 4 Modified Stochastic Gradient Descent Algorithm

C.1 High-Dimensional Linear Regression Experiment

Setting. In this experiment, we test the performance of our robust criterion in a high-dimensional sparse linear regression task. The high-dimensional and sparse nature of the data-generating process is expected to induce distributional uncertainty, and our method is meant to address this. In this context, we use the quadratic loss function ( θ, y, x ) ↦→ 10 -3 ( y -θ ⊤ x ) 2 , where the 10 -3 factor serves to stabilize numerical values in the optimization process. Notice that, by the form of the ambiguity-neutral criterion (2), the multiplicative factor on the loss function does not change the equivalence with Ridge.

Data-Generating Process. The data for the experiment are generated iid across simulations (200) and observations ( n = 100 per simulation) as follows. For each observation i = 1 , . . . , n , the d -dimensional ( d = 90 ) covariate vector follows a multivariate normal distribution with mean 0 and such that (i) each covariate has unitary variance, and (ii) any pair of distinct covariates has covariance 0.3:

Then, the response has conditional distribution y i | x i ∼ N ( a ⊤ x i , σ 2 ) , with a = (1 , 1 , 1 , 1 , 1 , 0 , · · · , 0) ⊤ ∈ R d and σ = 0 . 5 . That is, out of 90 covariates, only the first 5 have a unitary positive marginal effect on y i , and additive Gaussian noise is added to the resulting linear combination. Together with 100 training samples, at each simulation we generate 5000 test samples on which we compute out-of-sample RMSE for the ambiguity-averse, ambiguity-neutral, and OLS procedures.

0.00

0.01

0.02

0.03

0.04

0.05

Test RMSE

a

=1

a

=2

a

=5

a

=10

0.00

0.50

1.00

1.50

)

A. Neutral

OLS

0.00

1.00

2.00

3.00

L2 Norm (

)

A. Neutral

OLS

A. Neutral

OLS

A. Neutral

OLS

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

0.0000

0.1000

0.2000

0.3000

0.4000

0.5000

0.0000

0.1000

0.2000

0.3000

Mean

St. Dev

Robust Criterion Parameters. For each simulated sample, we run our robust procedure setting the following parameter values: ϕ ( t ) = β exp( t/β ) -β , β ∈ { 1 , ∞} , α = a/n for a ∈ { 1 , 2 , 5 , 10 } , and p 0 = N (0 , I ) , where the β = ∞ setting corresponds to Ridge regression with regularization parameter α (see Proposition 2.1). Finally, we run 300 Monte Carlo simulations to approximate the criterion, and truncate the Multinomial-Dirichlet approximation at T = 50 .

Stochastic Gradient Descent Parameters We initialize the algorithm at θ = (0 , . . . , 0) and set the step size at η t = 50 / (100 + √ t ) . The number of passes over data is set after visual inspection of convergence of the criterion value. The run time per SGD run is less than 1 second on our infrastructure (see Appendix D).

C.2 Experiment on Gaussian Location Estimation With Outliers

Setting. In this experiment, we test the performance of our robust criterion on the task of estimating a univariate Gaussian mean (assuming the variance is known) when the data is corrupted by a few observations coming from a distant distribution. Clearly, this is a situation where a considerable level of distributional uncertainty is warranted. In this setting, the loss function h ( ξ, θ ) = ( ξ -θ ) 2 is simply the negative log-likelihood associated to the normal model. Notice that the h is convex in θ and, as in the previous experiment, we pre-multiply it by a factor of 10 -3 for numerical stability reasons.

Data-Generating Process. The data for the experiment are generated iid across simulations (100) and observations ( n = 13 per simulation) as follows. For each simulation, 10 iid samples x i are drawn from a N (0 , 1) distribution (the actual data-generating process we want to learn) and 3 samples are drawn iid from a N (0 , 5) outlier distribution. At each simulation we also generate 5000 test samples from the data-generating process N (0 , 1) , on which we compute the out-of-sample average negative log-likelihood for the ambiguity-averse, ambiguity-neutral, and Maximum Likelihood Estimation (MLE) procedures - this will be our measure of out of sample performance (see Figure 3).

Robust Criterion Parameters. For each simulated sample, we run our robust procedure setting the following parameter values: ϕ ( t ) = β exp( t/β ) -β , β ∈ { 1 , ∞} , α ∈ { 1 , 2 , 5 , 10 } , and p 0 = N ( µ 0 , I ) , where µ 0 = (10 · 0 + 3 · 5) / (10 + 3) is a weighted average of the data-generating and the outlier means. By the expression of the ambiguity-neutral criterion (2), it is easy to show that the β = ∞ case leads to the parameter estimate

0.0000

0.0005

0.0010

0.0015

0.0020

=1

=2

=5

=10

A. Neutral

MLE

0.0000

0.2500

0.5000

0.7500

1.0000

Dist. from Truth (

)

A. Neutral

MLE

A. Neutral

MLE

A. Neutral

MLE

0.0000

0.0001

0.0002

0.0003

0.0004

0.0005

0.0000

0.0500

0.1000

0.1500

0.2000

0.2500

Mean

St. Dev

with y i = x i for i = 1 , . . . , n and y i = µ 0 for i = n +1 , . . . , n + α . That is, the ambiguity-neutral procedure with concentration parameter α ∈ N is equivalent to the MLE procedure when the original training sample is enlarged with α additional observations equal to µ 0 . Finally, we run 300 Monte Carlo simulations to approximate the criterion, and truncate the Multinomial-Dirichlet approximation at T = 50 .

Stochastic Gradient Descent Parameters. We initialize the algorithm at θ = 0 and set the step size at η t = 20 / (100 + √ t ) . The number of passes over data is set after visual inspection of convergence of the criterion value. The run time per SGD run is 2 seconds on our infrastructure (see Appendix D).

Results. In Figure 3, we present the results of the simulation study. As for the regression experiment, the ambiguity-averse criterion brings improvement, across α values and compared to the ambiguityneutral and the simple MLE procedures, both in terms of average performance and in terms of the latter's variabiliy (see the first row of the Figure). From the second row of Figure 3, it also emerges that, on average, the ambiguity-averse procedure is more accurate at estimating the location parameter than the two other methods. Compared to the simple MLE procedure, the variability of the estimated parameter is also significantly smaller. Taken together, these results confirm the theoretical expectation that the ambiguity-averse optimization is effective at hedging against the distributional uncertainty arising in the estimation of corrupted data such as the simulated ones.

C.3 High-Dimensional Logistic Regression Experiment

Setting. In this experiment, we test the performance of our robust criterion on a high-dimensional sparse classification task using the framework of logistic regression. As in the linear regression experiment, the high-dimensional and sparse nature of the data-generating process is expected to induce distributional uncertainty, and our method is meant to address this. In this setting, the loss function is h ( ξ, θ ) = log(1 + exp( -y · x ⊤ θ )) . As in the previous experiment, we pre-multiply it by a factor of 10 -3 for numerical stability reasons.

Data-Generating Process. The data for the experiment are generated iid across simulations (200) and observations ( n = 100 per simulation) as follows. For each observation i = 1 , . . . , n , the d -dimensional ( d = 90 ) covariate vector follows a multivariate normal distribution with mean 0 and such that (i) each covariate has unitary variance, and (ii) any pair of distinct covariates has covariance 0.3:

Then, the response has conditional distribution y i | x i ∼ Binary ( { 1 , -1 } , p x ) , with p x = 1 / (1 + exp( -x ⊤ a )) and a = (1 , 1 , 1 , 1 , 1 , 0 , · · · , 0) ⊤ ∈ R d . That is, out of 90 covariates, only the first 5 have a unitary positive marginal effect on the log-odds. Together with 100 training samples, at each

0.000

0.001

0.002

0.003

Test loss

a

=1

a

=2

a

=5

a

=10

0.000

5.000

10.000

15.000

)

L2 penalty

No penalty

0.000

5.000

10.000

15.000

20.000

L2 Norm (

)

L2 penalty

No penalty

L2 penalty

No penalty

L2 penalty

No penalty

0.0000

0.0002

0.0004

0.0006

0.0000

1.0000

2.0000

3.0000

0.0000

1.0000

2.0000

3.0000

Mean

St. Dev

simulation we generate 5000 test samples on which we compute the out-of-sample average loss for the ambiguity-averse, L 2 -regularized (with regularization parameter α , see below), and un-regularized procedures.

Robust Criterion Parameters. For each simulated sample, we run our robust procedure setting the following parameter values: ϕ ( t ) = β exp( t/β ) -β , β = 1 , α = a/n for α ∈ { 1 , 2 , 5 , 10 } , and p 0 = Binary ( { 1 , -1 } , 0 . 5) ×N (0 , I ) . Finally, we run 200 Monte Carlo simulations to approximate the criterion, and truncate the Multinomial-Dirichlet approximation at T = 50 .

Stochastic Gradient Descent Parameters We initialize the algorithm at θ = (0 , . . . , 0) and set the step size at η t = 1000 / (100 + √ t ) . The number of passes over data is set after visual inspection of convergence of the criterion value. The run time per SGD run is 3 seconds on our infrastructure (see Appendix D).

Results. In Figure 3, we present the results of the simulation study. As for the regression experiment, the ambiguity-averse criterion brings improvement, across α values and compared to the L 2 -regularized and the unregularized procedures, both in terms of average performance and in terms of the latter's variabiliy (see the first row of the Figure). From the second row of Figure 3, it also emerges that, on average, the ambiguity-averse procedure is more accurate and less variable at estimating the true regression coefficient than the two other methods. Also, our method is able to more effectively shrink the norm of the coefficient vector towards 0 (see the third row). Taken together, these results confirm the theoretical expectation that the ambiguity-averse optimization is effective at hedging against the distributional uncertainty arising in high-dimensional classification problems (in this experimental setting, tackled via logistic regression).

C.4 Pima Indian Diabetes Dataset Experiment

In this experiment, we use logistic regression for classification on the popular Pima Indians Diabetes dataset, 12 collecting data on 768 women belonging to a Native American group that lives in Mexico and Arizona. The data consists of a binary outcome (whether the subject developed diabetes or not) and 8 features related to her physical condition (these features are standardized before running the analysis).

To test our method, we randomly select 300 training observations and leave out the rest for as a test sample. Then, we randomly split the training data into 15 folds of size 20 and select, via k -fold cross

validation, the optimal DP concentration parameter α over a wide grid of values. We do the same for the L 1 -penalty coefficient used to implement regularized logistic regression with the Python library scikit-learn [31]. Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, L1-penalized logistic regression, and unregularized logistic regression on each of the folds. 13 This splitting procedure allows us (i) to test and compare the performance of our method in a setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures. The run time per SGD run is 19 seconds on our infrastructure (see Appendix D).

Table 1 reports the results from the described procedure. Unregularized logistic regression performs quite poorly compared to the other two methods. Instead, the latter yield results in the same orders of magnitude both in terms of average performance and of performance variability, though our DP robust method features almost half of the variability produced by L 1 -regularized logistic regression.

C.5 Wine Quality Dataset Experiment

In this experiment, we applied linear regression to the popular UCI Machine Learning Repository Wine Quality dataset [8]. Data consists of 4898 measurements of 11 wines' characteristics and a quality score assigned to each wine. The aim is to predict the latter based on the former (both features and response are standardized before running the analysis). We implement linear regression using our DP-based robust method (with the squared loss function), OLS, and LASSO (the last two methods are implemented using scikit-learn [31]).

To test our method, we randomly select 300 training observations and leave out the rest for as a test sample. Then, we randomly split the training data into 10 folds of size 30 and select, via k -fold cross validation, the optimal DP concentration parameter α over a wide grid of values. We do the same for the L 1 -penalty coefficient used to implement LASSO. Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, LASSO regression, and OLS estimation on each of the folds. This splitting procedure allows us (i) to test and compare the performance of our method in a setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures. The run time per SGD run is 5 seconds on our infrastructure (see Appendix D).

Table 2 reports the results from the described procedure, whose interpretation is very much in line with the results of the previous experiment.

Table 2: Comparison of average and standard deviation of the out-of-sample performance (out-ofsample expected squared loss) of the three employed methods for linear regression on the Wine Quality dataset.

C.6 Liver Disorders Dataset Experiment

In this experiment, we applied linear regression to the popular UCI Machine Learning Repository Liver Disorders dataset[15]. Data consists of 345 measurements of 5 blood test results and the number of drinks consumed per day by each subject. The aim is to predict the latter based on the former (both features and response are standardized before running the analysis). We implement linear regression using our DP-based robust method (with the squared loss function), OLS, and LASSO (the last two methods are implemented using scikit-learn [31]).

To test our method, we randomly select 200 training observations and leave out the rest for as a test sample. Then, we randomly split the training data into 10 folds of size 20 and select, via k -fold cross validation, the optimal DP concentration parameter α over a wide grid of values. We do the same for the L 1 -penalty coefficient used to implement LASSO. Once the optimal parameters are selected based on out-of-sample risk, we again randomly split the training sample into the same number of folds, and implement our roubust DP method, LASSO regression, and OLS estimation on each of the folds. This splitting procedure allows us (i) to test and compare the performance of our method in a setting with scarce data, where distributional uncertainty is most likely present, and (ii) to asses the sampling variability of the implemented procedures.The run time per SGD run is 15 seconds on our infrastructure (see Appendix D).

Table 3 reports the results from the described procedure, whose interpretation is very much in line with the results of the previous two experiments.

D Computational Infrastructure

All experiments were performed on a desktop with 12th Gen Intel(R) Core(TM) i9-12900H, 2500 Mhz, 14 Core(s), 20 Logical Processor(s) and 32.0 GB RAM.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper justifies the claims made in the abstract with either theoretical results, or experimental evidence.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please, refer to Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please, refer to Sections 3 for theoretical propositions and the underlying assumptions, and Appendix A for proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please, refer to Appendix C for experiment details and to our supplementary material in the form of code and a README file with instructions on how to run the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please, refer to the code included in the submitted supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please, refer to Appendix C and the submitted code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Variability in the reported average performance metrics are of direct interest for the evaluation of our method, so we report measures of variability as separate bars in plots or separate entries in tables.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please, refer to Appendices C and D

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the Code of Ethics and confirm the adherence of our paper to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Due to its theoretical and optimization-focused nature, the paper has no direct foreseeable societal impact beyond that of any generic machine learning paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper only employs simulated or small publicly available datasets, as well as well-established machine learning techniques.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly credits all data and software used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release any new asset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a new method for optimisation of risk under uncertainty, using Dirichlet Processes to introduce an extra degree of robustness to modelling uncertainty on the data generating process. The method is generally applicable to many statistical learning problems through the use of loss functions, and further links are made to the economic decision-making literature. The theoretical properties of the methods are explored, and a Monte Carlo approximation is introduced to make tractable the inference for the DP model. Experiments are performed on three simulated experiments, and three real data sets.\n\nSTRENGTHS:\nThis article represents a novel contribution to a very general problem, with the use of a neat mathematical representation of a statistical decision maker’s uncertainty in the form of a Dirichlet Process. The relevant theory behind the method is explored thoroughly, and there are several different applications explored. The quality of the scientific writing is fairly high. The links made with the economic decision making literature are interesting, helping elucidate the underlying point about ambiguity aversion.\n\nWEAKNESSES:\nThe idea of placing a prior on the data generating process instead of the parameters is unusual from the traditional Bayesian perspective. While this is not necessarily wrong, the authors do not do a particularly good job of communicating the implications of constructing a prior in this way, or how existing intuitions that the readership may have can be transferred to this new approach.\n\nThe results discussion from line 299 onwards has not been prioritised for space in the manuscript: the authors attempt to concentrate reporting results from six different experiments into a single page, which is not enough exposition for the main text of the article. Many fairly important details are pushed into an Appendix and then covered by asserting the “superior ability of our robust method” in the main text. This is far too simple an analysis for an empirical investigation: I don’t trust any analysis that can be summed up so concisely, let alone six combined, with three being real data sets. Appendix C is reasonably thorough in this respect, although there is a noticeable absence of any rigorous testing of the differences in performance metrics.\n\nThere are a few small issues with the English in the article: line 8: “among which Ridge…”, line 97: “pervasive” is a strange choice of vocabulary.\n\nQUESTIONS:\nLine 110: “In our setting, instead of a parametric prior on the regression coefficients, we place a nonparametric one on the joint distribution of the response and covariates” What is lost by not having priors associated with parameters? This is not necessarily a good thing to move away from: specific belief distributions for specific interpretable model components is a good thing. What the implications for estimating the posterior uncertainty on the parameter level, for example in the regression case? What kind of posterior convergence do you expect on the parameters themselves?\n\nWhat are the more general implications of putting a prior on the distributions of the data instead of the model parameters? This is suggested to be the case in line 111. Is that not what the likelihood is meant to capture? Is this merely sloppy phrasing in line 111  Does the prior on the distribution of the data come into conflict with the model at some point? What is the difference between altering the prior you have assumed here and altering the statistical model?\n\nWhat is the expected scalability of the methods to larger data sets? None of those used here are really that big, but the use of SGD implies that you have big-N scalability in mind. The wall clock times on your fairly modest desktop system are quite small, so presumably you have some ability to scale to larger data sets if desired.\n\nIs the trained DP object interpretable at all? Assuming that the learned DP object implies some sort of partition of the data-generating process, then is that partition telling us something we didn’t know beforehand? How restrictive is the DP prior on the space of measures that you might expect in reality?\n\nLIMITATIONS:\nWhile an elegant choice for the problem at hand, the use of a Dirichlet Process is likely to be a limiting factor for future directions of research, both in terms of the computational limits and the possible allowable partitions.\n\nThe assertion of the prior in the way performed here, in contrast to over interpretable parameters, possibly limits the likelihood of this method being widely adopted.\n\nThe theoretical results are focussed on convergence to the correct expected risk (or transformations thereof), rather than telling us anything about the parameters themselves.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors use Dirichelet process and a smooth ambiguity aversion model to approximate the solution of risk minimization problems. They demonstate the consistency of their approach, along with finite sample guarantees. They additionally give a practical means of applying their procedure and apply the procedure to a number of simulated and real problems.\n\nSTRENGTHS:\nBackground was very informative and enlightening. Despite being very technical in nature the paper was easy to follow. Based on the references given, the idea seems quite novel. Proofs related to results in section 3 were quite easy to follow (section 4 and appendix proofs not checked)\n\nWEAKNESSES:\nThere are many small typos. \n - In Lemma 3.2 the supremum of \\gamma^* is not needed as gamma^* does not depend on t. The statement in the proof is the supremum over gamma(t). \n- For consistency with lemma 3.2, in theorem 3.3 the supremum over gamma(t) should probably be replaced by gamma^* (or gamma(t) should be added back into lemma 3.2).\n - In line 181 I don't think the (ii) should be there. \n\nThere are some minor problems.\n - The introduction of the dependency of phi on n makes a small appearence. This idea is very interesting but I don't think it is emphasised enough. Just a couple of sentences explaining why you'd want to do such a thing would be nice.\n - In line 221 a form of phi is assumed. I assume this is supposed to be phi_n and not phi. If it is supposed to be phi I'm unsure what form is being assumed, and if it is supposed to be phi_n it doesn't seem like it is assumed as section 4 seems to use phi.\n\nAnd there are some reasonably major (but easily fixable) problems. \n- On first read it is confusing how exactly the distance between the theoretic risk and the risk computed WRT to p_0 enters into lemma 3.2. Seeing the full statement I understand why the full bound isn't included in the main text but perhaps this discussion can be reworded. \n- It is unclear if theorem 4.4 is using the same assumptions as lemma 4.3, or just the form of C_T that is given in lemma 4.3.\n- I think theorem 3.3 needs to assume that M_phi < infinity as it is moved to the otherside of an equation in the proof. I think the proof needs more discussion about the case of gamma^*=infinity as on first read it looks like it moved to another side of the equation in the proof. gamma^*=infinity seems fine as thne the bound is triival. These cases are possible as taking K=1 and phi(t) = (t-1)^(1/3) has M_phi = gamma^*=infinity.  Seems like everything can be infinite in lemma 3.2 as then the bound is trivial.\n\nQUESTIONS:\nIn theorem 3.4 it is assume that phi_n converges uniformly to the identity function. To me this seems like assuming that curvature of phi_n tends to 0 indictating that as n goes to infinity there is no ambiguity aversion. However, uniform convergence to the identity function is much a much stonger condition than curvature going to 0. For example convergence to 2 times the identity function still has curvature going to 0. Am I wrong in my reasoning behind convergence to the identity function? If not, is it possible to update assumption (3) in theorem 3.4 to a more general curvature to 0 assumption? It seems like the without much modification it can be instead assumed that phi_n converges to some scalar multiple of the idenity function.\n\n\nHow do the assumptions on theorem 3.3,3.4 guarantee that theta_n and theta_* exist? As is isn't assume that h is continuous or Theta is compact it seems like the argmin of V and R can be empty. From the proof of theorem 3.3 it seems like if theta_* doesn't exist it can be replaced by any arbitrary theta.\n\nAnother question. The uniform boundedness of h is strongly relied on to get the generate the results. Is it possible/are there methods available to remove this assumption? Classic problems like least squares regression over an unbounded domain will have h unbounded. This often isn't all that bad as assuming compactness of the domain is fine, but the bounds given here depend strongly on the size of the compact set used to restrict the problem to.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper introduces a new robust risk criterion that integrates concepts from Bayesian nonparametric theory, specifically the Dirichlet process, and a recent decision-theoretic model of smooth ambiguity-averse preferences. They show the relationships between their criterion and traditional regularization methods for empirical risk minimization, including Ridge and LASSO regressions. They also provide theoretical gurantee for the robust optimization and propose tractable approximations of the criterion.

STRENGTHS:
This is a solid paper that makes a good and exciting theoretical contribution. The idea of incorporating Dirichlet prior into distributionally robust is novel, and well-suited for the problem. The empirical section is adequate.

WEAKNESSES:
*

QUESTIONS:
* I have a question about the coefficient in the prior. What will happen if we adopt the empirical Bayesian idea into your algorithm, i.e. estimate the coefficient in the prior through data? Will this cause degradation of the model？

LIMITATIONS:
The authors adequately addressed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper proposes a novel robust optimization criterion for training machine learning and statistical models by combining Bayesian nonparametric theory and smooth ambiguity-averse preferences, addressing distributional uncertainty to improve out-of-sample performance. The authors demonstrate theoretical guarantees for their method and show its practical implementation and effectiveness through tasks using simulated and real datasets.

STRENGTHS:
The theoretical aspects of this paper are not my research area, so I am not equipped to assess the strengths and weaknesses of the theoretical contributions. However, the proposed method and its practical implications appear to be well-founded and promising.

WEAKNESSES:
The theoretical aspects of this paper are not my research area, so I am not equipped to assess the strengths and weaknesses of the theoretical contributions. However, the proposed method and its practical implications appear to be well-founded and promising.

QUESTIONS:
NO.

LIMITATIONS:
Discussed in Section 6.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper proposes a Bayesian method for optimizing stochastic objectives under a given finite sample. More precisely, The paper assumes the Dirichlet Process for data generation and then it proposes to optimize the mean of the stochastic objective over posterior distributions given the sample. The paper proves asymptotic properties of the proposed objective. The experimental results in the appendix show that the proposed method works better than the standard L2 regularization or no regularization on linear regression tasks over simulation and real data sets.

STRENGTHS:
The problem is quite relevant. The paper proves basic asymptotic desirable properties. The experimental results show the superiority of the proposed method to the standard L2 regularization-based methods.

WEAKNESSES:
One of my concerns is the proposed method's high computational cost. However, I don’t think it is critical. When the sample size is large enough, the standard ERM would work well. The proposed method, though it shows asymptotic convergence properties, would work better when the sample size is small. On the other hand, it would need more technical improvement when the sample space is high-dimensional.

QUESTIONS:
Is it difficult to obtain finite-sample error bound like PAC or statistical learning theory? (or any related work?)

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-23