Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport
Authors: Zifan Wang
Yi Shen
Michael M. Zavlanos
Karl Henrik Johansson
Venue: NeurIPS 2024 poster (2024)
Keywords: Distributionally robust optimization; unbalanced optimal transport; outlier
Primary area: optimization

Abstract: Distributionally Robust Optimization (DRO) accounts for uncertainty in data distributions by optimizing the model performance against the worst possible distribution within an ambiguity set. In this paper, we propose a DRO framework that relies on a new distance inspired by Unbalanced Optimal Transport (UOT). The proposed UOT distance employs a soft penalization term instead of hard constraints, enabling the construction of an ambiguity set that is more resilient to outliers.  Under smoothness conditions, we establish strong duality of the proposed DRO problem. Moreover, we introduce a computationally efficient Lagrangian penalty formulation for which we show that strong duality also holds. Finally, we provide empirical results that demonstrate that our method offers improved robustness to outliers and is computationally less demanding for regression and classification tasks.

Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport
Zifan Wang
Yi Shen Duke University yi hhen4880duke edu
KTH Royal Institute of Technology zifanwOkth.se se
Michael M. Zavlanos
Karl H. Johansson
Duke University
KTH Royal Institute of Technology kallejekth.se
michael.
Abstract
Distributionally Robust Optimization (DRO) accounts for uncertainty in data distri- butions by optimizing the model performance against the worst possible distribution In this paper, we propose DRO framework that relies on a new distance inspired by Unbalanced Optimal Transport (UOT). The pro- posed UOT distance employs soft penalization term instead of hard constraints, enabling the construction of an ambiguity set that is more resilient to outliers. Under smoothness conditions, problem. Moreover, we introduce a computationally efficient Lagrangian penalty formulation for which we show that strong duality also holds. Finally, we provide empirical results that demonstrate that our method offers improved robustness to outliers and computationally less demanding for regression and classification tasks.
within ambiguity
establish strong duality of the proposed DRO
Introduction
Consider a stochastic optimization problem that aims to find decision variable O that minimizes the expected loss function Eg-p. ((S 0)] (1, where the random samples E are drawn from fixed distribution Px and θ C R is compact and convex set. In many applications, the distribution of interest P" is not precisely known; yet datasets that contain finite samples independently drawn from the distribution J are often available, which yield an empirical distribution estimate P. For instance, in image classification problems, IP" can represent the joint distribution of the features and the labels, and IP represent the empirical distribution of a dataset that is annotated by humans []; in portfolio optimization problems, IP captures the uncertainty of set of financial products in the future months and contains historical prices of all the products [3]; in healthcare applications, T can represent the feature distribution of population of interest, and P represent the available electronic health record data
The above stochastic optimization problem is often solved using empirical risk minimization (ERM) (5) which searches for the best decision variable 0 that minimizes the expected loss function with respect to the empirical distribution, i.c., PERM = arg mingee Eg.ee((E. 0)). However, ERM is not guaranteed to provide good oft-of-sample performance, i.c., the value of B (ERM)] is high when the empirical estimate P close the true distribution P* For example, in the presence of distribution shifts or contaminated datasets, the out-of-sampll performance cannot be improved by increasing the sample size. To enhance the out-of samaee performance of the decision variable. researchers have developed methods that use distributionally robust optimization (DRO) to address
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the mismatch between IP and I 6 8 9 10 outcomes against the worst-case scenario over a set of distributions that are close to the empirical distribution according to pre-specified distance metric. This set of distributions is called the ambiguity set and is typically assumed to contain the distribution of interest, e.g., the true distribution P.. The construction of the ambiguity set is critical in the development of various DRO methods, as it ideally encompasses all prior knowledge about the true distribution. For example, [6] defines the ambiguity set using moment constraints. Moment constraints assume that the true distribution has similar moments compared to the empirical distribution, e.g., means and variances. The ambiguity set can also include distributions that are similar to P in infinite-order moments; this construction is explored in [3] and is termed Kernel DRO. When the distribution of interest is categorical, Kullback-Leible divergence (KL) is often adopted to define the ambiguity set as the categories can be permuted and KI does not consider the geometry of a distribution's 's support. DRO using KL has been extensively studied in the literature due to its computational benefits [14] In contrast, the Wasserstein distance between two distributions captures the geometry of the distributions. The Wasserstein distance can be useful, e.g., in loan approval prediction problems where the distribution represents the incomes of a population and prior knowledge may specify that two clients are similar if their incomes are close. DRO using the Wasserstein distance (WDRÓ) has been applied to many machine learning problems, such as linear regression [66 and logistic regression [17 problems.
[2]. In contrast to ERM, DRO aims to optimize
In this paper, we apply DRO to handle distribution mismatches using the Wasserstein distance as it can better capture the geometry of the distributions. However, WDRO is not suitable for contaminated datasets, e.g., datasets that contain outliers that are geometrically far from the clean distribution but cannot be easily removed. The reason is that the Wasserstein distance between the contaminated empirical distribution and the clean (true) distribution is very sensitive to the outliers; to include the clean distribution in the ambiguity set, the Wasserstein distance would need to be selected very large and inevitably the ambiguity set would include distributions that will never happen in practice.
To apply DRO to contaminated datasets, the authors in [18 19 20 take outliers into consideration and propose various methods to minimize their impact on the learning performance. Specifically, [8] proposes to minimize the most optimistic DRO risk avoid the hardest instances that are likely be outliers. In [1], a weight clipping method is proposed to truncate the excessive impact of outliers on the results. However, both [18] and [19] use KL divergence to define the ambiguity set, which fails to capture geometric uncertainty. Notably, the authors in [20] introduce an outlier-robust WDRO framework capable of addressing both geometric uncertainty and non-geometric contamination, allowing an ε fraction of data to be arbitrarily corrupted. They design an ambiguity set based on prior knowledge and demonstrate the strong duality of the resulting DRO problem, which is related to the Conditional Value at Risk (CVaR) [2I]. Nevertheless, the performance of the formulated DRO problem relies heavily on the accuracy of the estimate of E. Besides, (20) uses off-the-shelf solvers to solve the resulting DRO problem. which is not computationally efficient when applied to large datasets since the number of constraints in the proposed method grows linearly with the sample size.
The main contribution of this paper is to introduce a new framework for outlier-robust WDRO based on Unbalanced Optimal Transport (UOT), which is known for its inherent robustness to outliers and missing data [22 23 24, 25 26). Our approach involves the design of a UOT distance that substitutes hard constraints by soft penalization term. This construction, combined with some prior knowledge on the learning task, enables the design of a new ambiguity set that includes distributions of interest and penalizes distributions that contain outliers. For the DRO problem with this new ambiguity set, we establish strong duality results under specific smoothness assumptions. However, the solution of the dual problem poses significant computational challenges. Motivated by (27]. we explore Lagrangian penalty variation of the problem, which can be viewed as a Lagrangian function of the original formulation with a fixed dual variable. We show that strong duality holds for this Lagrangian penalty problem under fewer assumptions compared to the original DRO problem. The reformulated problem employs an exponential function to reweight data points, effectively diminishing the influence of outlier data on the optimization process. We solve this problem by proposing a provable stochastic (sub)-gradient algorithm, which is computationally efficient. We provide empirical results that demonstrate that our approach not only enhances robustness to outliers but also improves computational efficiency for large-scale problems.
Our work is related to data reweighted optimization problems (28 29 30), which adjust the weights of individual data to manage outliers. However, these studies construct the reweighted optimiza- tion problems heuristically and cannot handle distribution uncertainty. In contrast, our problem
2

---

formulation is able to handle distribution uncertainty via DRO, and our findings demonstrate that the reweighted problem constitutes the dual of specific class of DRO problems, with the ambiguity set defined by the UOT distance.
The rest of the paper is organized follows. Section 2 defines the UOT distance and the associated DRO problem. Section presents the strong duality results for both the original DRO problem and its Lagrangian penalty variant. The main algorithm along with its convergence analysis is presented in Section 4 Section 3 demonstrates the effectiveness of our proposed framework in addressing outliers in both regression and classification tasks. Finally, we conclude the paper in Section 6 In the Appendix, we provide all proofs as well as some additional experiments.
2 Problem definition and preliminaries
The (1-) Wasserstei distance 31
between two distributions IP and IP is defined as
W)P, inf F(E.C)m/o(E,.)). PET(P,2)
(1)
where [PP, IP) denotes the set of joint distributions such that m P and = P. where " and 72 denote the first and the second marginal distribution of 7, respectively. The function c(E, c) E + [0, (00) is lower semi-continuous and represents the cost of moving point from to S. where the support set E C IR is assumed to be compact. The joint distribution y specifies transport plan for moving the distribution from IP to P.. The Wasserstein distance captures the underlying geometry in distributions through the cost function c(.s)) +), making it popular choice in DRO problems; the WDRO problem is defined by
min sup Eg,e(1(0, ME (f)
where (f( := (IP M(E) W(P, P)) S p} denotes the ambiguity set that contains all distributions within p-distance from I and p 2 0 is user-specified parameter. Here, we denote by M(E) the space of all probability distributions supported on E.
The Wasserstein distance enforces equality constraints on the marginal distributions of y to match and IP, thereby restricting the choice of y. If the empirical distribution I contains outliers, it becomes necessary to select large radius p for the ambiguity set ensure the inclusion of the true distribution 1' as the outliers tend to increase the Wasserstein distance between IP and F However, a large can lead to DRO solution that is conservative since the larger the radius the larger the value of the inner supremum problem in 2 Though a large radius allows the inclusion of the distribution of interest, i.e., IP", it also results in the inclusion of many distributions that are not of interests, e.g., distributions that are unlikely occur in practice. As result, the Wasserstein distance is not a desired distance choice when the empirical distribution contains outliers.
In Unbalanced Optimal Transport (UOT) problems 22) the joint distribution y is not required to have fixed marginals in contrast to ① Specifically, for any two arbitrary positive measures IP and PP, the UOT distance is defined by
UW(P, inf ((s.s) DD 720
where 71 and 72 marginals of 7, and Dap and Dip2 are Csiszár divergences that measure discrepancies between positive measures, based on functions ş0 and 422, respectively. The key distinction between UOT distance and Wasserstein (OT) distance lies in the constraints of the marginals. While the OT distance in (I) strictly enforces marginal distributions, i.c., = P and 72 = IP, the UOT in 3 relaxes the equality constraints by adding mismatch penalty functions D,p in addition to the transport cost induced by c(. This relaxation makes UOT robust to outliers since 722 can represent a distribution that is different from the contaminated distribution I..
Inspired by the definition of UOT, we consider the following unbalanced Wasserstein distance (divergence)
UW(P = inf
(EIE,S),plc(E.)) (((, (PD))
(4)
(oer(e.F)
3

---

where / is a tuning parameter and IP is an intermediate optimization variable that links P and P. Note that (4) is special case of (3) by selecting 9i = [(} (i.e., (pi = and otherwise) and (22(z) log x x ++1 (i.e., D. eg denotes the KL divergence).
addition, we restrict the marginals of the joint positive measure in 33 to be probability measures because the general positive measures are not of interest in the DRO problems we consider. The particular choice of (p functions for technical reasons related to the duality analysis, as we will show in Section
The optimal unbalanced Wasserstein transport under UP)P can be interpreted as a two-step procedure: 1) P can be transported to any P by paying small price measured by its distance to IP in the KL sense (a non-geonneee non-geometric transport), and
then transported to P by minimizing the geometry- aware transport cost. The non-geometric KL divergence is the key to allow the uncontaminated clean distribution IP* to be close to the contaminated empirical distribution IP as the KL divergence does not distinguish the geometric locations the distribution supports.
We define the ambiguity set using the unbalanced Wasserstein distance by U,((P) = (P E M()): UW S p}. Then, the DRO problem using the UOT distance (4) can be formulated as
min OE
sup PEMME)) (E) P)
Eg-P!((0,E))
(5)
The ambiguity set defined by the unbalanced Wasserstein distance enables to include the clean distribution pr, but it cannot remove distributions that are close to the contaminated empirical distri- bution P as these distributions are close to both geometrically (Wasserstein) and non -engeometrrially (KL). As result, these distributions, denoted by set P, could also make the learned model conserva- tive. To remove such distributions, is impossible to design a model selection strategy using contaminated datasets without any prior knowledge. In Section 5] also show that constructing the ambiguity set without prior knowledge may result in worse performance compared to the traditional DRO; see T
need some prior/domain knowledge. As discussed in [18, 20),
We assume that any distribution PP E & incurs significantly large costs when evaluated using a given function & compared to uncontaminated distributions, i.e., [g_=((((E)) >> [g<=[[(()], V P and
P E {F: : (IP))))) S p} The function can be related to the moment constraint. For example, if we know the expectation estimate Eu of the clean distribution, we can design h({) -|5- Ea and the function value of h(E) will be high if g is an outlier. Similar choices are considered in 20
Taking into account the prior knowledge, we consider minimizing the following (primal) function
(P)
sup PEMEE): (W())
Be-F| {)] Bg-P(h(E))
sup PEM(E): CW(F)
[c,P(L(0,E))
(6)
with L(9,5) ((0 T((E) and we assume that L(O, <) continuous in Ç. For any distribution P that contains outliers, (EE) large and thus IP is less likely to attain the supremum in (6)
In summary, we first employ the unbalanced Wasserstein distance to incorporate the distributions of interest. Then, we leverage prior knowledge to systematically exclude distributions that may contain outliers. This two-step approach ensures more thorough consideration of potential distributions while effectively mitigating the impact of outliers on the learning problem.
3 Unbalanced distributionally robust optimization
Note that the primal problem in (6) is an infinite- -imeesioddimensionall optimization as the optimization variable represents probability distributions. In this section, we derive its dual problem and variant of the dual problem that is computationally tractable.
3.1 The DRO problem
Assumption 1. Every joint distribution 7 on E x E with the second marginal distribution PP has regular conditional distribution "Iş given the value of the second marginal equals Ç.

---

This assumption ensures that the joint distribution can be written as dy(5, dnc(5)dP((): see [3] for more details on regular conditional distributions. By the law of total expectation, we have Be-r [T(00, {)] = B (E) T.(0,)))
We define fx () == su {L(0, {) - Xc(E, ()). We first derive the dual problem and show the weak duality in the following lemma.
hold and suppose that (0x 5)1 < co for P-almost every S. Define the dual problem (D) == infx20 Ap + 2,3 log B eep
Lemma 1. Let Assumption
(x(0.2) A
}}
Then, we have
fA(O.C) C
[] (D).
(P)
sup (E
B [L(0
inf X20
Ap+ A3 log END exp
XB
The proof is provided in Appendix
7.1 To show the strong duality, we need two additional assump-
tions to guarantee that the function 0x () well-behaved.
Assumption The function [(0,E) be difserentiable and concave in & with respect to the norm ||-Il-
Assumption 3. The function e = + [0, 00) is differentiable and cl:, 1-strongly convex for each &o E E.
Theorem Let Assumptions 3 hold and assume that [,f(0, S)I < oo for P-almost every Ç. Suppose that the optimal dual variable x is strictly positive. Then, the strong duality holds, i.e.,
sup (E
E [0,05
inf { A3log Be exp (fx(0, (f2(0,))((AB)) ()/(())()))) X20
(7)
The proof can be found in Appendix 7.2 For technical reasons, we show the strong duality when the optimal dual variable X" is strictly positive. most applications, the optimal dual variable is in fact positive. When X 0, the inequality constraint becomes inactive and the radius of the ambiguity set becomes very large, a situation that is out of scope for most practical applications.
Although strong duality holds and the dual problem is a finite dimensional minimization prob- lem, solving problem (7) is computationally challenging due to the complex dependency of the objective on À and, in particular, the implicit dependency of fx on X. Recall that fx(0. () = supşez (L(0,E) Ac(E, })}; in general, we cannot obtain an analytical form of fx, i.c., closed-form solution of the problem supçe= (L( - Ac(E, <)}. Additionally, the dual problem (7) is not convex in X in general. The task becomes more involved when attempting to minimize over À and 0 jointly. To address these challenges, we shift focus from the constrained p-robustness problem (7) and instead consider the corresponding Lagrangian penalty problem, which allows for computational tractable formulation.
3.2 The Lagrangian penalty problem
Motivated by (27 we consider the Lagrangian penalty reformulation of 7 as follows
[e,r((((, - AUI(P))))
sup
(8)
PEM(E)
In what follows, we show that the strong duality also holds for the Lagrangian penalty problem (
Theorem Let Assumption holdd and assume that (fx <) < oo for P-almost every Ç. Then, we have that
E - XUW(PII0))
X,3 log E E exp (Fx(0,)))((1))
sup PEM(E)
The proof is provided in Appendix 7.3 Note that the strong duality of the Lagrangian penalty problem only requires Assumption to hold in contrast to Theorem [ In particular, it does not require the loss function [(0,E) <) to be differentiable, making it applicable to broader class of problems. Besides, solving problem (8) computationally tractable as the dual problem has a closed-form solution as in (9). In particular, À is parameter in (9) while it is optimization (dual) variable in (
5

---

Since the logarithm function is monotonically increasing, minimization of P in 0 is equivalent to the following form
min E c-p [exp (SUPEEz (T00,)) - h(() - (c(E))))((AA)) BEB
(10)
Comparison with standard WDRO. has been shown in [27 that the Lagrangian penalty problem using the Wasserstein metric Wo has the following strong duality result
sup (1(0,5) (10,,5) 2o(E, [EE
min GEG
sup Eg-=(((0,E]] AWo(P, PEM(E)
min DEB
E
(1I)
where W. is the Wasserstein distance metric associated with the cost function c(s.). .). When À in (T] is selected to be large, this formulation is close to the ERM problem, i.e., minimization of
E c (T(, Clearly, distribution P containing outliers would significantly deteriorate the learning performance of minimizing [,. c)l.
In contrast, when À is large, the problem TT is close to the minimization of w(()), where w(()
Be-alexp () ((C))((3)) exp (AB) acts as a re-weighting function. Recall that h(E) a function designed based on prior knowledge and assumes high values outlier points. At an outlier point, say &. M(E) would be significantly larger than at non-outlier points, resulting in a very small weight w(E). Consequently, outlier data with small weights exert less influence the resulting model. Therefore, the formulation (10) is more robust to outliers compared to standard WDRO.
E lexp
Comparison with outlier-robust WDRO 20]. 20 considers the Huber contamination model in which an E fraction of data can be arbitrarily corrupted. [20], prior knowledge on the mean of the clean distribution is needed. Specifically, the function h({) that can identify outliers in (20 is
selected h(g) A1 ll Eull?, where represents the estimated mean of the clean distribution and A > 0 is tuning parameter. It can be verified that the corresponding Lagrangian formulation is
min CVaR sup ((00 X l|s &oll Xc(3, 85
(12)
fEE
where CVaR, denotes the conditional value at risk (CVaR), interpreted as the expected value of the rightmost (1 1-- E) percentile outcomed]
Similarly, when À is very large, the formulation in 12 closely approximates the minimization of CVaR
A ||{ Eo [I] At an outlier point, say ç, the value 1(0, ((,,)) - - &oll is small due to the large distance || - Eoll- Consequently, such an outlier point generates function value that falls the left tail of the loss distribution and is thus effectively excluded by the CVaR operation. This method can also be viewed as a re-weighting method by assigning weight to outlier points and weighting other points equally. However, this approach is highly sensitive the choice of e, which is usually difficult to obtain a priori. In comparison, our method is smoothed version of this CVaR method and is less sensitive the prior knowledge of e. Besides, calculating the value of CVaR is computationally more demanding when the number of samples is large as we will demonstrate in Sections 4 and
4 Algorithm design
In this section, we focus on solving the dual of the Lagrangian penalty problem. We assume that the empirical distribution TP is constructed from samples and written as TP = Σi &g with & the i-th sample. According to Theorem 2 the Lagrangian penalty problem is equivalent to the minimization
of the following objective function
F(O) = Σ cxp Suppes
- (1606))-x(6.)))/3b3))
(13)
'The CVaR of random vector Z that has the distribution with risk level E (0, 1) is defined as CVaRi-su [2] infa T-EEz-uIZ a)+.

---

Algorithm 1: Distributionally robust optimization with outliers
Require: Sampling distribution P., constraint sets & and Z, step size sequence {ar > 0)
1: for = .,..., ... -
2: Sample Č uniformly from P
3: Find an e-approximate maximizer ž of arg maxgez L(0,, () Ac(E,
4: Construct gradient estimate gr. exp (L((), Ae(a, (t))/(A8) x ((r,
5: 0z+1 = Proje (0,- .99r))
6: end for
To solve the problem (3). we employ the stochastic sub-gradient method, detailed in Algorithm 1 Specifically, at each time step i, we sample & from the empirical distribution uniformly. We then find the c-approximate maximizer of the function [(O,..) f) Ac(S, &.) We require that the solution satisfies (zs X²) S c, where X² denotes the set of maximizers. This solvable many applications and can be achieved through, e.g., the projected sub-gradient update. The convergence analysis of Algorithm iss presented the following theorem. The proof can be found in Appendix 7.4
method [27]. Then, we perform the
Theorem 3. Suppose that θ is a convex and compact set with bounded diameter R. Assume that the loss L(0, E) is convex in O for each & E. E. For all data points Ç from the empirical distribution
P. assume that L(0.-) Ac(-, (e) is concave, L(B,E) - Xc(E, ()) I B and c(E, (.) c( 4i <
Le l1s Elll. for all 0 O, { E. E E. Assume that (0.L(0.E) -Og E')ll S Loe k E'll
(0,E s B2. [[((,E) L(0,E') LE lle - Elll. Select 17 VT. Then, Algorithu 7 satisfies
1 T Σ (BFF(.))) 0 0 / +e),
(14)
where F* represents the minimum value of the objective function
13
The accuracy parameter e in the inner maximization problem has a fixed effect on the final opti- mization accuracy, independent of T. The assumptions in Theorem are common in the analysis of optimization problems involving the exponential function in the objective function, see, e.g..
[1 28 29
Algorithmic comparison with outlier-robust WDRO (20). Our proposed formulation can be solved by the stochastic bub-gradient method, which is computationally more efficient when the sample size is large. In contrast, the approach in [20) utilizes CVaR to account for outliers and employs off-the-shelf solvers, e.g., GUROBI [33], to obtain a solution. A notable limitation of the presence of CVaR in DRO problems is that the number of constraints scales linearly with the sample size. Consequently, for large datasets, this scaling significantly increases the number of constraints, leading computational inefficiency, shown in Section 5 Designing stochastic gradient method for the CVaR- based formulation in (20) is possible, although challenging since the estimate of the CVaR gradient is usually biased. Moreover, our methodology does not rely on off-the-shelf commercial solvers and can be incorporated in machine learning packages seamlessly, which allows us to handle a wide range of loss functions via the stochastic sub-gradient method.
5 Experiments
We conduct experiments on linear regression, linear classification, and logistic regression problems. We use the stochastic sub-gradient method Algorithm to solve the Lagrangian penalty problem and the GUROBI 33 solver to solve all other DRO problems use as benchmarks. All experiments were conducted on an Intel Core i7-1 185G7 CPU (3.00GHz) using Python 3.8. Our method referred to as UOT-DRO. A discussion paramcter selection is provided at the end of this section.
7

---

booos fox
30
Slandard OK ORO
Standard DRD G8 wERo UOT 200
UOT-ORG
(a) Excess risk with various samples.
(b) Excess with varied dimensions.
Figure 1:
Excess risk of standard DRO, OR-WDRO, and UOT-DRO with varied sample size and dimension for linear regression. The error bar denotes + standard deviation.
5.1 Linear regression
We consider linear regression problem with the loss lz y) = |@TT: yl. where (z. R x R represents generic data point and 0 I& is the model we aim to train. We define 0 10T, -1]T,
< = [xT, y/T. Then, the loss function can be written as (g(5) [0Tg).
We generate clean data distribution I'm with n samples, which is uniform over {x,,, OIX; - where XI,..., are i..dd from X (N( La). Drawing uniform random subset C [n] of size [en], the corrupted data distribution P, is defined be uniform over {(CHiES)) X1, (-c²) l(iEs)
( (OIX;+ + +)) where C 0 is a corruption scaling coefficient and > O is shift coefficient. The empirical probability distribution IP, can be written as Po 1 with & the i-th sample.
We consider prior knowledge on the mean of the clean distribution, denoted as 5, and design the El]- function h(E) Ao || Ell- The value of & is determined by cheap preprocessing step, same as in We consider the Lagrangian penalty problem which, based on Theorem 2 is given by
sup | - A - <ll - X2 ||s - - )((0))
min E c~P.
exp
(15)
Define x(0) supfll=l|, : (2) xo}
where ||-lI, denotes the dual norm.
When
A2 + x()), we can show that the problem is equivalent to
min
exp
(10T
X21lE - GII)(199)
(16)
The detailed proof is provided in Appendix 7.5
We the parameters as follows:
10, 6, A2 5. All the results are averaged over 10 independent runs. We fix 0.1 and C 8. We compare the performance of our unbalanced DRO model against the standard DRO and outlier-robust WDRO provided 200) We evaluate these methods in terms of the excess risk, which we define as the difference between the loss incurred by the learned model and the minimum achievable loss. The simulation results presented in Fig. demonstrate the performance of our method under varying conditions: In Fig. (a), we set d 10 and compare these methods for various samples sizes n E {20, 40, 60, 80, 100): ; In Fig I (b), we fix
100 and explore the impact of various dimensions d (5, 10, 20, 30, 40). We observe that our method not only achieves superior robustness to outliers but is also less sensitive to data dimensions.
Besides, we compare the computational efficiency of these methods across varying sample sizes. As shown in Table [ the OR- OR-WDRO method is not applicable to large-scale datasets, with its running time exceeding 200 minutes for a sample size of n 20000. In contrast, our method only requires 20 seconds to process the same dataset while maintaining superior learning performance. Therefore, method not only achieves better robustness but also maintains computational efficiency.

---

Table 1: Comparison of running time and excess risk of different methods for linear regression. The
symbol ** indicates that running time over 12000 seconds.
Sample Size n
Standard DRO
OR-WDRO
20
UOT-DRO
Time
Excess risk
Time
Excess risk
Time
Excess risk
0.1
3.230
0.4
0.103
2.7
0.060
200
0.2
2.298
1.3
0.064
3.4
0.040
2000
3.3
0,441
29.8
0.050
4.7
0.038
5000
9.2
0.371
259.5
0.040
7.7
0.034
10000
28.9
0.352
1438.7
0.033
11.9
0.033
20000
110.8
0.380
22.2
0.031
Table 2: Excess risk with various contamination for linear classification.
Contamin. C
Standard DRO
OR-WDRO 20
UOT-DRO
Excess risk
Accuracy
Excess risk
Accuracy
Excess risk
Accuracy
0.628
73%
0.627
72%
0.298
92%
10
0.722
66%
0.560
79%
0.295
93%
0.637
68%
0.341
90%
0,241
96%
100
0.872
56%
0.191
97%
0.240
95%
5.2 Linear classification
We consider linear classification problem with the loss lo(x,y) max y(fTx)) with 0 Rd We consider the same outliers as that in 120]. Specifically, we first generate clean distribution P. by {X,, sign((n(( X,)). where XI,..., Xn are drawn i.i.d. from NN Ia). For uniform random subset S C [nx] of size [en]. we consider the corrupted distribution i.,, which is uniform over {((-C) X; G1, sign(n! ()[x,))) i=1 where C > is the contamination factor, e1 )
is the distribution shift. We select h(E) A2 ||s Ell and c(g, () = ||5 <1l in linear classification as in 20
We consider the formulation described Theorem and solve it using Algorithm We fix & = 0.1, d = 10, 100 and select À = 10, X2 1, B 2. The experimental results averaged over 10 trials. We evaluate the model performance for different choices of the contamination factor C' by analyzing the excess risk and accuracy. Excess risk defined as the difference between the returned loss and the best achievable loss. Accuracy refers to the rate of successful classification. As shown in Table the performance of standard DRO decreases when C gets large. In contrast, the OR-WDRO performs well only when C' is very large. This is because OR-W WDRO uses CVaR to filter out outliers, which requires the outliers be significantly distant from the regular data for CVaR to be effective. Nevertheless, our method performs well across the entire range of C.
5.3 Logistic regression
We consider logistic regression with the loss function (0(x, y) log(1 exp(-y(0TX)) where 0 E IR" As in [17], we assume that the feature vector X follows a multivariate standard normal distribution, and the conditional distribution of the label y E {-1, 1} is given by Prob(y|z>) = xxp(=)(f[z))]
where 8_ = (10. .......) .). This setup uniquely determines the true distribution P. We draw n samples from this distribution create the empirical distribution P,, = 1 6(x..))) Outliers are considered to occur only in the feature space. The perturbed distribution is represented as {(-))) X; + ped ) where S C [m]] is a uniform random subset of size [en]. We fix 0.1, p = 0.1, e O..., 0), and select X 10, X 1, B 2. In Fig. 2 we compare the excess risk and accuracy of standard DRO as outlined in [17] and our proposed method, described by Theorem and implemented by Algorithm [ The results are averaged over 10 runs for different sample sizes n E {20, 40, 60, 80, 100}. Besides, we conduct comparative analysis of the excess risk and accuracy between the standard DRO and the proposed method across various contamination levels. As shown in Table 3 standard DRO is susceptible to significant outliers. Instead, the proposed method demonstrates robustness throughout the entire range of contamination factor C, consistently maintaining accuracy of approximately 92%.

---

1.4
3.2
Acoracy
2.5
0.5
0.-
The parameter À2 represents the credibility level assigned to the function h. A larger value of X2 should be selected when the confidence in the reliability of highly likely to become large outlier points. Conversely, if the prior knowledge provided by is considered unreliable, the value of X2 should be reduced. If there is no reliable prior knowledge at achieving good performance impossible, shown in (a) Excess risk with various samples.
(b) Accuracy with various samples.
Figure 2: Excess risk and accuracy of standard DRO and UOT-DRO with varied sample sizes for logistic regression. The error bar denotes + standard deviation.
Table 3: Loss and accuracy with various contamination for logistic regression.
Contamination
Standard DRO 17
UOT-DRO
Loss
Accuracy
Loss
Accuracy
0.976
73%
0.390
93%
1.249
64%
0.469
93%
16
1.502
56%
0.481
92%
30
1.056
62%
0.474
92%
5.4 Parameter selection
We provide some guidelines for selecting the parameters X, Ag, 3 in Algorithm The parameter is commonly used in the DRO literature [27] penalty coefficient. If the value of À is large, then the DRO problem approaches the empirical risk minimization problem, resulting in model that performs well on the empirical distribution but is less robust to Wasserstein perturbations.
high, meaning that h(() is
all, in which case should select related literature in robust statistics [18].
The parameter 3 penalizes the mismatch between marginal distributions. A larger value of places more emphasis on minimizing this mismatch, possibly at the expense of increasing the transportation cost, thereby making the unbalanced optimal transport distance close to the balanced one. Conversely, a small value of 3 allows for larger mismatches between the marginal distributions, which can enhance the model's robustness outliers. However, when 3 is very small, possible distribution mismatches incur little penalty, and the computed distance may fail to accurately represent the true distance between the distributions. In this case, the resulting DRO problem will incur many unlikely distributions in the ambiguity set, leading to a very conservative model.
Conclusion
In this work, we introduced a novel DRO framework that employs a new distance derived from UOT. By incorporating soft penalization term in the design of the ambiguity set, our method exhibits increased resiliency outliers. We provided strong duality results for the original DRO problem and the Lagrangian penalty problem, with the latter allowing for more efficient computation via stochastic sub-gradient methods. Finally, empirical results validate our method's enhanced robustness to outliers and reduced computational demands for regression and classification tasks.
10

---

Acknowledgement
This work was supported in part by Swedish Research Council Distinguished Professor Grant 2017- 01078, Knut and Alice Wallenberg Foundation, Wallenberg Scholar Grant, the Swedish Strategic Research Foundation SUCCESS Grant, and AFOSR under award #FA9550-19-1-0166
References
[1] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezynski. Lectures on Stochastic Programming: Modeling and Theory. SIAM, 2021.
[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages
2009.
[3] Georg Ch Pflug and Mathias Pohl. A review on ambiguity in stochastic portfolio optimization. Set- set-Valudd and Variational Analysis, 26:733-757, 2018.
[4] Euan A Ashley. Towards precision medicine. Nature Reviews Genetics, 17(9):507-522, 2016.
[5] Vladimir Naumovich Vapnik, Vlamimir Vapnik, et al. Statistical Learning Theory. Wiley New York, 1998.
[6] Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations Research, 58(3):595-612, 2010.
[7] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Mathemati- cal Programming, 171(1)1115-66,, 15-166, 2018.
[8] Wolfram Wiesemann, Daniel Kuhn, and Melvyn Sim. Distributionally robust convex optimiza- tion. Operations Research, 62(6): (1000:1358-1377 1376, 2014.
(9] Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance. Mathematics of Operations Research, 48(2):603-655, 2023.
[10] Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565-600 2019.
[11] Jie Wang, Rui Gao, and Yao Xie. Sinkhorn distributionally robust optimization, arXiv preprint arXiv:2109.1 1926,, 2021.
[12] Waiss Azizian, Franck lutzeler, and Jérôme Malick. Regularization for wasserstein distribu- tionally robust optimization. ESAIM: Control, Optimisation and Calculus of Variations, 29:33, 2023.
[13] Jia-Jie Zhu, Wittawat Jitkrittum, Moritz Diehl, and Bernhard Schölkopf. Kernel distributionally robust optimization: Generalized duality theorem and stochastic approximation, In International Conference Artificial Intelligence and Statistics, pages 280-288. PMLR, 2021.
[14] Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online, 1(2):9, 2013.
[15] John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 1388)1138- 2021.
[16] Ruidi Chen and loannis Ch Paschalidis. A robust learning approach for regression models based on distributionally robust optimization. Journal Machine Learning Research, 19(13):1-43 2018.
[17] Soroosh Shafieezadeh Abadeh, Peyman M Mohajerin Esfahani, and Daniel Kuhn. Distribu- tionally robust logistic regression. Advances Neural Information Processing Systems, 28, 2015.

---

[18] Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: robust optimization. In International Conference on Machine Learning, pages 12345-12355. PMLR, 2021.
Distributional and outlier
[19] Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj. and Arun Sai Suggala. Stochas- tic re- re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.092: 2023.
[20] Sloan Nietert, Ziv Goldfeld, and Soroosh Shafiee. Outlier-robust Wasserstein dro. Advances in Neural Information Processing Systems, 36, 2024.
[21] R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2:21-42, 2000.
[22] Thibault Séjourné, Gabriel Peyré, and François-Xavier Vialard. Unbalanced optimal transport, from theory to numerics. Handbook of Numerical Analysis, 24:407-471, 2023.
[23] Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xan Xavier Vialard. Unbalanced optimal transport: Dynamic and Kantorovich formulations. Journal of Functional Analysis, 274(1 30:0000333322 123,, 2018.
[24] Quang Huy Tran, Hicham Janati, Nicolas Courty, Rémi Flamary, levgen Redko, Pinar Demetci, and Ritambhara Singh. Unbalanced co-optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10006-10016, 2023.
[25] Matthias Liero, Alexander Mielke, and Giuseppe Savaré. Optimal autropy-transpon problems and new Hellinger- Kantorovich distance between positive measures. Inventiones mathemati- cae, 211(3):969-)))) 1117, 2018.
Jellinger-Kantrrovich
[26] Khiem Pham, Khang Le, Nhat Ho, Tung Pham, and Hung Bui. On unbalanced optimal transport: An analysis of Sinkhorn algorithm. In International Conference on Machine Learning, pages 7673-7682. PMLR, 2020.
[27] Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some dis- tributional robustness with principled adversarial training. arXiv preprint arXiv: |7Xiv:17100.0571,, 2017.
[28] Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning: Theory and applications. Journal of Machine Learning Research, 24(142):1-79, 2023.
[29] Qi Qi, Yi Xu, Rong Jin, Wotao Yin, and Tianbao Yang. Attentional-biasee stochastic gradient descent. arXiv preprint arXiv:2012.06951 2020.
[30] Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K Warmuth. Exponentiated gradient reweighting for robust training under label noise and beyond. arXiv:2104.01493, 2021.
[31] Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6)3555-60 2019.
[32] Olav Kallenberg. Foundations of Modern Probability, volume 2. Springer, 1997.
[33] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
[34] MD Donsker and SRS Varadhan. Asymptotic evaluation certain Markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-212, 1983.
[35] Paul Milgrom and Ilya Segal. Envelope theorems for arbitrary choice sets. Econometrica. 70(2):583 60010, 2002.
[36] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3), 1997.
[37] Dimitri Bertsekas. Convex Optimization Theory, volume Athena Scientific. 2009.
12

---

7 Appendix
7.1 Proof of Lemma
By plugging in the definition of the UW distance in (4. we have that
sup (P)
(BeeeL((0 B(E.C)~ lot,, O) + (DkI(PI S spp
F. M.
BENPREN [[(0,E)] B (c(5, + Bew log dF(()
sup (.
dP(()
S
BeNiBEN [[(),E]] Bx- c(s,5) + log dP(())
sup (.))
aP(()) S
inf B SeppBenn [[(0,E] + ( (BEEwBBEnu c(5,5) + log
dP(()
sup (.)) A20
aP(()
aP(()
$ inf Ap+ + sup A20 P.TEM((E)
BeWpBENy
L(0,E) - >c(E:5) - AB log
aP(c)
di((c)
S inf Ap + sup
BE-P PEM(E)
sup (L(0,)) - Ae(s, {)) log EEE
(17)
A20
dTP(s)
where the second last inequality follows from the max-min incquality and the last inequality follows since Be-xx [L( Xe(E, c)l
sup L(, Ac(E, (). Recalling the definition fx(0,)) EEE
sup (L(0.E) -xc(, {)}, we obtain that eEE
dP(() (P(())
inf Ap + X33 log B exp ()(0,<) () X20
inf Ap+ + sup Ec-B fa(0,5) A3 ABg A20
A
PEM(E)
where the last equality follows from Donsker and Varadhan's variational formula [34] and the fact that (x, is integrable with respect to P. Note that the condition [fx <)l < oo for P.
almost every ( ensures cxp(f.(0.()(AAA) df(() is well-defined. The supremum is achieved by (...lexp(A(0:))//AA)))
exp(Ax(0.C)/AA8) dP(()) [c.elesp((xx (C). The proof complete.
7.2 Proof of Theorem
Given Assumption
L(O,E) - Xc(E, () is differentiable with respect to &. According to the
Envelope theorem 35 we have fx(C) differentiable in X. Given L(0, {) Ac(E, () is strongly
concave in &, we have (xfx(() c(hx(()))))))))) (), where hx(() == arg max [(, {) )c(E, EE=
A(()) ).
Define u(A) Ap 1o++Alop ABlogh B
exp
We have v(A) is differentiable and satisfies
[x(() X3
]
E SAPP
exp
[x( Aff
FAAACAA-AАCC)
Vxu(A) p Bog E
c-
exp
(18)
fA(C)
E
exp
X2
Since X > 0, due to first-order optimality condition, we must have Vx(v(A))/a=xx i.c.
fa- (5) fx) X+
]
E B
exp
(C)(C) A
( 1(hx- (())
p + Blog
exp
(19)
A=
exp
(1-(C) X-3
We define the joint distribution 7" such that dy (E,C) drg(S)dTP* (5), where dl*
expl/>-(9)/A133)) dP(), and 72(E*) 1, if 5" 1x- (S)
arg max Xte(E,()
13

---

and (E(E) 0 otherwise. Let I be the first marginal distribution of y", i..., it satisfies that y" TP" PP). In what follows, we show that IP" lies in the ambiguity set. Specifically, we have
UW(P"11))
inf P.ee((P .P)
dP(()
inf 7.er((= .P)
c(E.5) + log
dF(())
dlP (5)
$ <Eeeco
C) + -llog dTP(c)
dF*(() dP(())
= -- Rems (c($.C) + B3
log
(A-(() X+p
exp
E= [e(hx- ()), + GI
log
(x+
CoWB
exp
X+
c(1x- ((), + fx-(5)
fAx(()) X-3
]
= -
Bloog B
Bewp-
exp
X
U1xx- (())
(A-(())
= Ee-
X
33 log E E
exp
XBB
(11x- (C))
di (c)
fA. (A)(C)
- B log BeNBr
exp
A"
dP(c)
Ecw
exp
(2=(() X-3
1(hx-(())
fx)(()
3 log Log.Ea BNNP
exp
(a-(C)
X+
XmRR
<ND
exp
X-9
P:
(20)
where the inequality holds since 7 is feasible solution in I(P* P). The following equalities derived by substituting the expressions of and P-_ The last equality follows due to the first-order optimality condition (19.. Therefore, the distribution P is feasible for the primal problem. Then, the primal optimal value is lower bounded by the value
(P) > Ee-p- [L(OE)] = E [))
= BC Ben,; [L(0,E) {)]
Rewe [(hxx (())
dp dP ((hx-(<))
dip(c)
(x- X
exp
= EeWE
1(1x-(()
E
exp
(A+(C)
(x-(C)
-xp+
BE
C
exp
(D).
(21)
X+
Combining (T) with the weak duality, we have (P) (D). Thus, the strong duality holds when X" > O. The proof is complete.
14

---

7.3 Proof of Theorem 2
Based on the definition of UW in we have
sup PEM((E
B [() AUW(PIIP))
sup PEM(E)
E [(B,E)] A inf P.EE((P
{ (E(E,)))lC((EE))+ +,05555))533333. 3D (
dl((s)
sup BEM(()))ETR.))
Be [L(, (E(E,CCnlo((ESC) -
log dP((c)
L(O,E) Ac(5,5) - AB log cP
sup E.x)
Be
aP((C)
dP(())
sup PEM(E)
R CPP sup (T(0,E) - {c(E, - A,
dP(())
(22)
The last equality in (22 achieved by selecting >²(5²) = for a E arg max (X((,)))
SE=
and 7 (E) E 0 otherwise. Here, there always exists such a g since L(O,E) - Ac(E, () is continuous
and E is compact set. Recalling the definition of fx(C), we have
EE-P [)) - AUW(PIIP)
sup PEM(E)
[x(S) A,3log dIP(C)
sup PEM(E)
R
dPp(C)
[A((S
AB log Eett
exp
(23)
XB
where the last equality follows from
14 The proof ends.
7.4 Proof of Theorem
Since L(B,E) - Ac(E, (c) is convex in 8, and E is a compact set, from Danskin's 's theorem [3)] we have f0, (.) is convex in 0 and
()). = Co (00L(00,")) -(01,2")-(c
(24)
where Co denotes the convex hull of point set. Given that fx(0, &.) is convex, it is easy to verify that F(8) convex. Since the exponential function is convex and non-decreasing, for any 2f arg max; L(0., E) Ac(E, Gc), we have
G. = exp( 2 Ac(2 ()))/(A1)) 1 (0., c 0 (xp((f((0..))) A3
Note that ||+||| exp( ( Co- Since obtain e-approximate maximizer 2; of arg max L(O,,)) Ac(E, G.), the sub-gradient gi is inexact and we denote by the error e; 9t Gr.
15

---

In what follows, we amalyze the bound of sub-gradient error e₁. From its definition, we have
|eall =
A (e, 21) c(((0r=7)-Aastt (01/00)))))
0o L(t,
A
e (LL(0 [ul00ha))--ddta222000000001001,0
001(0r,
X3
+ 0, L(,, zū² (e (1(00 (1(0))))) -atoll)))dddaaaaa
1
0gL(00: =i)) (0()
A3
- _|u. -
1
=7
X
xD Log
||ze -
1
e.0. - (3D))A))
((L(0)
+
X3
A
3D Log
B2
2(0r,2)) -1 110,, Ao(zt
1|
zill +
B
2
2/3
2
2
e A0 Long
B
B2e (Tig
+ Tee)
+
X2 || 22
A3
== C1 1| - 111,,
(25)
where the first inequality follows from triangle inequality, the second inequality holds since L(0,
Ac(E, (+) < B and [ ze) - ((e, 23 )l < Log || - 2 || The third inequality hols since
lesh e01 S eB x - y| when H, y < B. The last inequality follows from our assumptions. Since (25)
holds for any zi. we have ||e/|| S Cie.
Suppose that 0 minimizes P(0). Based the update rule, have
10++1 - 11 S ||0: - 19t- - 11 = =1, r/G,- T 00'1"
= =100- 0" 1| + 1 |G. 1| + 2/ ||:.1 2(0. -00 2(0 n0,n))) (),, (26)
Taking conditional expectation both sides of (26) with respect to Ć., we have
E |10 P++1 1²
s 10 - 0-| 112 + 1 || + + |e. 200 - 8", ,,,.), - 2(0: - 0", .nE)) + ei)
|00 0-1 (7(FF(()) F*) +.. + 7
1e 2(0: - ,0,nBee) + i
< 10 8- 2 2n(EF(()) F*)+ + + + / 1| 2(0. 1001--0,n1B0 ,(eee) + Co lleell
Co
< 1 8" 1| - 2n(EF((L) - -F*) +n' + +r'c
Re + Cocc,
(27)
where the second inequality holds since F(G.] C (F.))), the third inequality follows from the fact that G, < Co and the last inequality follow since ||eal| < Cjc. Taking expectation on both sides of
27 and summing up over t I,..., T. it gives
E 100+ - 0-12
T
< E |0,
0- 2
Σ
T 220 2c + C1ce). (28)
-
Rearranging 28
we have
R²
nC
Cf2
(EF - F*) <
+
+
+ +CRc+
(29)
T
Tn
2
2
Substituting
into
29
completes the proof.
16

---

7.5 Additional details of linear regression experiment
In this section, we show the derivation of equation (T6).
Recall that 0 = [ - S = [x_, y/ and (5) = []. As la is proper. convex, and lower semicontinuous, it coincides with its bi-conjugate function 1**, see e.g. M.. Thus, we can write (E) = sup (2. 5) (=). where 2 = (= E IR: (z) < co is the effective domain of the
conjugate function. Define x(f)- = sup[1=/]. : = E 2). It is shown by [6] that x(0) = ||0||_ With
the definition of the dual norm, we have
sup (l((E) ll - <lI 22 ||s Ell SER-
= sup sup ÇER =E2 { (2. - [2) - 1115 -l1 - ->2 1211--EEl -El Ell
sup sup ŞER =E2 SA, ||=a||. Sd2 { (2,E) - (1) (21,5 - (G) (22, -
inf
=
The classical minimax theorem 337 allows us interchange the maximization and minimization and then we obtain
{
(o((E) - Alls - <lI A2 || -5-11 }
sup
ŞER"
{ (2 21 + 22.5) - 15()) - (21, (22
sup
inf
sup
*E2 || SA. [ GER
2 || SA, 11 1elll.511 Sd2 {210)(2+21 21 + 22) (2) - (21, Gi) - (22, } }
= sup
inf
sup { - [](=) (21, ()) - (22, }
inf Sx ||alll. *+81 +0 xE2
sup ||aall Sda. 8+8 + x62
inf
{ (](=) + (2,6)) + (22 22 - }
inf
{b(()) + (22, & - 2)
||aall. Sda 8+ +2 x62
{l(())) >2 ||: - =
inf Sx,, || 2+ +20 262
The second equality holds since SUPEER (2, function and characteristic function of the set Ω respectively. The third equality follows again from the classical minimax theorem (37). The fourth equality is derived by substituting 2 -2 2 into the objective function. The fifth equality holds since lo 1 Since À > A2 k(0), there exists 22
03d(2) = Xy where on and Xn denote the support
that satisfies 22 = -1 - and other constraints. Finally, we obtain
(E()) A lIE - <lI - X2 ||s - El| } lo(G)) - >>2 -NalIss - =
sup EERR
The proof is complete.
7.6 Additional experiments on linear regression
We evaluate the sensitivity of the model performance to the tuning of designed parameters. We first fix d = 5 and explore the impact of varying X and ,3. As shown in Table the model's performance demonstrates minimal sensitivity to changes in these parameters. Then, we fix 3 = 6. and assess
the sensitivity to changes in A2- The simulation results are presented in Table 5 When X2 =
meaning we abandon the use of the prior knowledge term A2 E - & our method performs worse
than the traditional DRO method. This is because the unbalanced optimal transport distance itself includes distributions that contain worse outliers, thereby making model overly conservative. As X2 increases, there is a noticeable improvement in learning performance, which we attribute to the enhanced role of prior knowledge in outlier defection.
We also explore how the contamination factor, denoted as C, affects model performance. The simulation result is presented in T While standard DRO is highly sensitive to the contamination
17

---

Table 4: Excess risk with various parameters for linear regression.
Excess Risk
À == 6
À =
À = 12
8
0,064
0.059
0,060
3
0.057
0.056
0.058
= 10
0.059
0.055
0.060
Table Excess risk with various X2 for linear regression.
X2
Standard DRO
UOT-DRO (A 8)
UOT-DRO (A
A2
3.008
4.914
5.381
A2 1
3.008
1.340
0.882
A2 =
3.008
0.061
0.057
À2 4
3,008
0.059
0,057
X2 = 3.008
0.063
0.063
factor C, both OR-WDRO and our method are not sensitive to contamination factor C' in linear regression tasks.
Table 6: Excess risk with various contamination for linear regression.
Contamination factor C
Standard DRO
OR-WDRO 20
UOT-DRO
0.141
0.103
0.063
2.807
0.119
0.058
12
6.371
0.125
0.058
We use the same preprocessing step as in 20 to obtain a robust estimate of the expected value of the clean distribution, which involves a parameter é. This parameter can be viewed an estimate the proportion of data corruption. We evaluate the sensitivity of the model's 's performance to variations in é and the result presented Table We observe that our method achieves better robustness and less sensitive to changes in compared to OR-WDRO in 20). This is because OR-WDRO relies on é during both the preprocessing and the solver optimization steps while our method only requires during the preprocessing step.
Table 7: Excess risk with various preprocessing parameter é for linear regression (E 0.1).
standard DRO
OR-WDRO 20
UOT-DRO
0.05
2.661
0.088
0,049
0.1
2.661
0.066
0.049
0.2
2.661
0.064
0,050
7.7 Additional experiments on linear classification
We conducted additional experiments on linear classification, supplementing those presented in the main body of the paper. For these experments, select d 10. First, evaluate the model performance across various sample sizes. As illustrated in Fig. 3 our method outperforms others in terms of both excess risk and accuracy. Next, assess the model performance for different dimension d, as shown in Fig. We observe that our method achieves superior robustness and maintains the accuracy around 94%.
In addition, we compare the computational efficiency these methods linear classification. As detailed in Table 8 OR- WDRO exhibits low computational time when the sample size is small, but faces scalability issues as the dimension increases. In contrast, our method demonstrates stable computational times and superior performance across various sample sizes.
18

---

60-1
4x
3x30
70
2x
60
50
70
90
100
20
50
70
100
Samples
(a) Excess risk with varied samples.
(b) Accuracy with varied samples.
Figure 3: Excess risk and accuracy standard DRO, OR-WDRO, and UOT-DRO with varied sample sizes for linear classification. The error bar denotes + standard deviation.
100
80
4xx10-1
Standard DRD
Standard WDRD
OR OR-WDRO UCT-CRO
60-
10
30
35
1.0
15
20
25
30
35
40
Dimenssoo on
Dimension
Excess risk with varied samples.
(b) Accuracy with varied samples.
Figure 4: Excess risk and accuracy of standard DRO, dimensions for linear classification. The error bar denotes + standard deviation.
and UOT-DRO with varied
Table 8: Comparison of computation time, loss values, and accuracy for different methods for linear classification. 'ER' stands for excess risk.
Sample Size
Standard DRO
OR-WDRO
20
UOT-DRO
Time
ER
Accuracy
Time
ER
Accuracy
Time
ER
Accuracy
80
0.1
0.686
65%
0.5
0,605
80%
12.8
0.227
96%
200
0.2
0.710
70%
0.8
0.536
88%
13.6
0.265
95%
2000
3.1
1,013
43%
28.8
0.546
96%
13.5
0.299
98%
5000
8.2
1.009
41%
270.5
0.536
96%
14.2
0.292
98%
10000
19.4
1.016
30%
1478.4
0.544
97%
14.4
0.301
98%
20000
64.2
1.021
35%
*
*
14.9
0.295
99%
19

---

As in the linear regression task, we evaluate the model's sensitivity to the preprocessing step, specifically the parameter The parameter is again the estimate of the proportion data corruption. As illustrated in T our method is less sensitive to the accuracy of the estimate é. The reason mirrors that the linear regression task.
Table Excess risk with various for linear classification.
Standard DRO
OR-W WDRO
20
UOT-DRO
Parameter
Excess Risk
Accuracy
Excess Risk
Accuracy
Excess Risk
Accuracy
0.05
0.713
67%
0.447
81%
0.288
93%
0.1
0.713
67%
0.505
85%
0,283
93%
0.2
0.713
67%
0.484
86%
0.282
93%
Finally, evaluate the sensitivity of the model performance the tuning parameters À and B. We fix C' 10. Table shows that our method achieves the accuracy above 90% across the range of values from 0.5 20 and À values from 5 20. This performance is better than that of standard DRO with accuracy 66%, and OR-WDRO with accuracy 79%. These simulation results indicate that method is not sensitive these parameters in linear classification tasks.
Table 10: Accuracy of various parameters for linear classification when C = 10. Standard DRO: 66%; OR-WOR-WWW RR-W 79%.
Parameter 8
X
À
10
20
X
40
À
80
0.1
65%
71%
77%
88%
94%
94%
0.5
81%
90%
95%
95%
95%
95%
2
91%
94%
94%
94%
93%
93%
93%
93%
92%
92%
90%
85%
20
92%
92%
91%
90%
82%
68%
40
91%
89%
85%
75%
63%
57%
100
93%
86%
74%
61%
54%
47%
7.8 Additional experiments logistic regression
We evaluate the sensitivity of the model performance to the tuning parameters in logistic regression. As shown in Tables TT and 12] our method achieves the accuracy above 85% across the considered ranges of À from to 15 and 3 from 0.4 to 10. Based on all of these simulation results, we find that method's performance remains stable across wide range of parameters.
Table 11: Loss and accuracy of various 3 for logistic regression.
À
5 Accuracy
10
X
15
Parameter
Loss
Loss
Accuracy
Loss
Accuracy
0.4
0.530
85%
0.466
90%
0.449
92%
0.443
91%
0.484
92%
0.510
92%
10
0.563
91% %
0.564
91% %
0.586
91%
Table 12: Loss and accuracy of various Å2 for logistic regression.
X
5
A
10
15
Parameter X2
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
0.1
0.269
93%
0.360
89%
0.431
87%
0.4
0.388
92%
0.451
91 %
0.476
91 %
0.440
94%
0.478
93%
0.509
94%
4
0.613
90%
0.601
91 %
0.600
91%
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sections 3, 4, 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed in Section that the problem derived from Theorem is hard to solve.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
We provide all the required assumptions and complete proofs in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the required information for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [No]
Justification:
The main contribution of this paper is theoretical.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ceiiioooPolyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwtc//ggui////eeeeceoocc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Sections and 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Sections 5 and 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttps://euur:: http:r/:///uips..eeeeee cwterris.ccppuuuee httpr//ss..cppbb.ic//ttttttttooo
Answer: [Yes]
Justification: This paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthcee....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
25

---

Answer:
[NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "FDOobVvIV8", "reviewer": "Reviewer_8Mev", "summary": "The authors leverage unbalanced optimal transport (UOT) to build a new DRO model. Strong dual reformulations together with efficient algorithm design have been proposed. Numerical studies on convex loss function demonstrate the superior performance of this framework.", "questions": "2. In Theorem 1, the assumption seems too restrictive, including boundedness of $f_{\\lambda}(\\theta,\\zeta)$, differentiability and concavity of $L(\\theta,\\xi)$, differentiability and concavity of $c$, and strict positivity of dual variable $\\lambda^*$. In existing DRO literature such as [JGX2023], only the finiteness of cost function, lighted tailed condition, measurability of loss, and law of total probability is required. In standard WDRO, the technical assumption for strong duality holds is mild as well. \n\n3. I also believe the assumption that dual variable is strictly positive is not necessary: when you take sufficiently large radius, the constraint is unbinding and the dual variable $\\lambda^*=0$. Please follow existing DRO literature to explore how to build strong duality in this case. \n\n4. In line 220, the authors implicitly used the assumption that $c(\\zeta,\\zeta)=0$. I don’t remember whether it has been assumed in the main context before. \n\n5. I have concerns regarding Theorem 3. The algorithm design is actually inspired from [Sinha et. Al 2017]. The authors in that reference studied optimization for non convex loss function, such as neural networks, and perform biased SGD update to achieve near-stationary points. However, the authors in this paper only considered convex optimization. I strongly believe their results can be extended for non-convex smooth optimization. Please consider the exploration in this direction. Some useful references include \n\n[HCH2021] Hu Y, Chen X, He N (2021) On the bias-variance-cost tradeoff of stochastic optimization. Advances in Neural Information Processing Systems. \n\n[ HZCH2020] Hu Y, Zhang S, Chen X, He N (2020) Biased stochastic first-order methods for conditional stochas- tic optimization and applications in meta learning. Advances in Neural Information Processing Systems, volume 33, 2759–2770. \n\n\n6. For the numerical study part, the exploration of outlier DRO with neural network loss functions is also quite promising.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this is a good paper. The idea for using UOT is innovative and useful. Theoretical contributions are solid and I have checked their proofs and to the best of my knowledge, I don’t find mistakes. I leave some comments in next section, which may further help the authors improve the quality of this paper in terms of literature review and theoretical soundness.", "weaknesses": "1. I think the authors may miss an important citation and comparison: \n    [JGX2023] Wang J, Gao R, Xie Y (2023) Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926.\n\nSpecifically, the authors considered UOT-based DRO, where UOT is a variant of entropic regularized OT distance. In contrast, [JGX23] also considered another variant of entropic regularized OT distance (which they call it as Sinkhorn distance) to construct the ambiguity set:\n\n$$S(\\mathbb{P}, \\widehat{\\mathbb{P}}) = \\inf_{\\gamma \\in \\Gamma(\\mathbb{P}, \\widehat{\\mathbb{P}})}~\\Big\\{\n\\mathbb{E}_{(\\xi, \\zeta)\\sim \\gamma}[c(\\xi,\\zeta)] + \\beta \\mathbb{E}_{(\\xi, \\zeta)\\sim \\gamma}\\left[ \n\\log\\frac{\\mathrm{d}\\gamma(\\xi,\\zeta)}{\\mathrm{d}\\nu(\\xi)\\mathrm{d}\\widehat{\\mathbb{P}}(\\zeta)}\n\\right]\n\\Big\\}$$\n\nwhere  $\\nu(\\cdot)$ can be any reference measure (such as Lebesgue measure) on $\\Xi$. The difference is that the authors in this paper considered relative entropy regularization between  $\\bar{\\mathbb{P}}$ and $\\widehat{\\mathbb{P}}$, and [JGX23] considered relative entropy regularization between conditional conditional distribution $\\gamma_{\\zeta}(\\cdot)$  and measure  $\\nu(\\cdot)$. Also, please compare the dual reformulation in this paper and that in [JGX23]. The expressions share some connections.\n\n\nIn addition, the usage of entropic regularization for Wasserstein DRO has been largely explored in other literature. I hope the authors also add the comparisons among them.\n\n    [JY2022] J. Wang and Y. Xie, “A data-driven approach to robust hypothesis testing using sinkhorn uncertainty sets,” in 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022, pp. 3315–3320. \n\n     [WFJ2023] \tW.Azizian, F.Iutzeler, and J.Malick “Regularization for wasserstein distributionally robust optimization,” ESAIM:Control, Optimisation and Calculus of Variations, vol. 29, p. 33, 2023. \n\n     [JRY2022] J. Wang, R. Moore, Y. Xie, and R. Kamaleswaran, “Improving sepsis prediction model generalization with optimal transport,” in Machine Learning for Health. PMLR, 2022, pp. 474–488 \n\n     [SZ2023] \tS.-B. Yang and Z. Li, “Distributionally robust chance-constrained optimization with sinkhorn ambiguity set,” AIChE Journal, vol. 69, no. 10, p. e18177, 2023. \n\n     [CFAB2023] C. Dapogny, F. Iutzeler, A. Meda, and B. Thibert, “Entropy-regularized wasserstein distributionally robust shape and topology optimization,” Structural and Multidisciplinary Optimization, vol. 66, no. 3, p. 42, 2023. \n\n     [JGX2024] Wang J, Gao R, Xie Y (2024) Non-Convex Robust Hypothesis Testing using Sinkhorn Uncertainty sets. in 2024 IEEE International Symposium on Information Theory (ISIT). IEEE, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful feedback. Please see the detailed response below.\n\n---\n> **W1(Q1): Reference [JGX2023]**\n\nWe thank the reviewer for bringing our attention to the reference [JGX2023] and other related references about regularized optimal transport distance. We will add these references and additional comparative analyses into our revised manuscript.\n\nThe fundamental difference between regularized and unbalanced optimal transport distance lies in whether the marginal distributions of the couplings are allowed to be different from the empirical distribution. Take Sinkhorn Wasserstein DRO (WDRO) in [JGX23] as an example. Both Sinkhorn WDRO and unbalanced WDRO use the KL divergence, albeit in fundamentally different ways. In Sinkhorn WDRO, the KL divergence acts as a regularization term that facilitates computational efficiency at the expense of computation error. Importantly, the coupling $\\gamma \\in \\Gamma(\\hat{\\mathbb{P}},\\mathbb{P})$ in Sinkhorn WDRO must have at least one marginal distribution that is identical to the empirical distribution $\\hat{\\mathbb{P}}$. In contrast, unbalanced WDRO employs KL divergence to penalize discrepancies between marginal distributions, allowing for the coupling $\\gamma\\in \\Gamma(\\bar{\\mathbb{P}},\\mathbb{P})$ to have a different marginal distribution $\\bar{\\mathbb{P}}$. One of the resulting technical differences, as the reviewer pointed out, is the relative entropy regularization between two couples of distributions.\n\nThe strong duality of Lagrangian penalty formulation of several related problems is as follows:\n\nTraditional WDRO: $\\mathbb{E}_{\\zeta \\sim \\hat{\\mathbb{P}}} \\Big[ \\sup _ {\\xi}  \\\\{  L(\\xi)-\\lambda c(\\xi,\\zeta) \\\\} \\Big],$\n\nSinkhorn WDRO: $\\lambda \\beta \\mathbb{E} _ {\\zeta\\sim \\hat{\\mathbb{P}}} \\Big[ \\log  \\mathbb{E} _ {\\xi\\sim \\nu } \\Big[{\\rm{exp}} \\left(\\frac{L(\\xi)-\\lambda c(\\xi,\\zeta)}{\\lambda\\beta} \\right) \\Big]\\Big],$\n\nUnbalanced WDRO: $\\lambda \\beta \\log \\mathbb{E} _ {\\zeta\\sim \\hat{\\mathbb{P}}} \\Big[ {\\rm{exp}}\\left( \\frac{\\sup _ {\\xi} \\\\{L(\\xi)-\\lambda c(\\xi,\\zeta)\\\\}} {\\lambda\\beta}\\right)\\Big].$\n\nFrom these dual formulations, Sinkhorn WDRO modifies traditional WDRO by substituting the $\\sup$ with a log-sum-exp smoothing function, thereby enhance the computational efficiency. In contrast, unbalanced WDRO employs the log-sum-exp function differently. Here, it is used to re-weight samples from $\\hat{\\mathbb{P}}$ rather than replacing the $\\sup$ operation. As a result, outliers exert a reduced impact on the decision-making process. Computationally, unbalanced is more challenging as it involves an inner-maximization problem, similar to traditional WDRO.\n\n---\n> **Q2: Restrictive assumptions in Theorem 1**\n\nCompared to Sinkhorn DRO in [JGX2023], the additional assumption about the differentiability and concavity of $L(\\theta,\\zeta)$ is due to the existence of an inner sup in the dual formulation, which is smoothed out in Sinkhorn DRO. Another assumption about $\\lambda^*=0$ is discussed in Q3. Relaxing these assumptions is left as our future work.\n\n---\n> **Q3: Strictly positive dual variable is not necessary**\n \nIt may happen that $\\lambda^*=0$ when we select a sufficiently large radius. However, this scenario is uncommon in practice because choosing a very large radius typically results in overly conservative outcomes that are generally less appealing. We agree that removing this assumption is promising under additional conditions. We plan to address this in our future work as it is not straightforward by applying techniques from current DRO literature, e.g., [JGX2023].\n\n---\n> **Q4: Implicit assumption $c(\\zeta,\\zeta)=0$**\n\nThank you for raising this point. We have added it in the updated version.\n\n---\n> **Q5: Extension to non-convex smooth optimization**\n\nThank you very much for providing these two related references. We will incorporated them in our revised draft. Thank you for proposing the non-convex smooth optimization problem. This is definitely interesting and left as our future work.\n\n---\n> **Q6: Neural network implementation**\n\nThank you for affirming the potential of our method for use with neural networks (NNs). We believe our method can be applied to NN loss functions, and we plan to explore this in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l2KKYqPblA", "reviewer": "Reviewer_KUtP", "summary": "This paper introduces a novel outlier-robust Wasserstein Distributionally Robust Optimization (WDRO) framework based on unbalanced optimal transport (UOT). For the UOT-DRO, the authors establish strong duality results under specific smoothness assumptions. To enhance computational efficiency, they propose a Lagrangian penalty variant of the problem and prove the strong duality of this variant as well. The authors then employ stochastic gradient descent (SGD) to solve the Lagrangian penalty variant, providing a theoretical convergence guarantee for their method.", "questions": "1. In line 238-239, the authors mentioned that outlier-robust WDRO is highly sensitive to $\\varepsilon$, the contamination level in the outlier-robust optimal transport. However, proposition 12 in [A] provides a robustness guarantee when the chosen $\\hat \\varepsilon$ does not match the true $\\varepsilon$. Also, the numerical experiments in [17] show that OR-WDRO can perform well when the the chosen $\\hat \\varepsilon$ does not match the true $\\varepsilon$. Could you further explain why outlier-robust WDRO is sensitive to $\\varepsilon$ in line 238-239?\n2. It is encouraged to show an empirical evaluation on the convergence of the proposed Algorithm 1. This would help demonstrate the practical applicability and efficiency of the algorithm.\n3. For the numerical experiments, could the authors further explain how the values of the hyperparameters were determined for linear classification and logistic regression? Additionally, the choice of hyperparameters for the baseline methods is also missing and should be included for a comprehensive comparison.\n\n[A] Nietert, Sloan, Ziv Goldfeld, and Rachel Cummings. \"Outlier-robust optimal transport: Duality, structure, and statistical analysis.\" *International Conference on Artificial Intelligence and Statistics*. PMLR, 2022.", "limitations": "The authors discuss the limitations of their work in Section 3. Notably, this work does not have a negative societal impact.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow.\n\n2. The paper introduces a novel formulation of DRO based on UOT to address outliers in the empirical distribution. Rather than adopting UOT directly, the authors consider a variant of UOT (introduced in equation (4)) that restricts the positive measure to be a probability measure, which aligns with the focus of DRO on probability measures. Additionally, the authors introduce a function $h$ to measure the distance to the uncontaminated distribution, effectively ruling out distributions close to the empirical distribution but containing more outliers.\n\n3. The authors propose a Lagrangian penalty variant of the new outlier-robust DRO problem and employ stochastic gradient descent (SGD) to solve this variant with theoretical convergence guarantee for their method.\n\n4. In experiments conducted on linear regression, logistic regression, and linear classification, the proposed algorithm demonstrates superior performance compared to the baseline methods WDRO and OR-DRO. This validates the efficiency and effectiveness of the proposed approach.", "weaknesses": "1. The current draft does not include a theoretical analysis of the excess risk associated with the proposed UOT-DRO. Including such an analysis would strengthen the paper by providing a deeper understanding of the performance and limitations of the method.\n2. The discussion on parameter selection is currently limited. The UOT-DRO relies on several hyperparameters, including $\\lambda$, $\\lambda_2$, and $\\beta$, which may not be entirely independent. Although Section 7.6 addresses hyperparameter selection for linear regression, the approach for selecting these parameters in other tasks remains unclear. A more comprehensive discussion on tuning these parameters for general cases would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address the concerns below.\n\n---\n> **W1: Excess risk**\n\nThank you for raising this point. Analyzing the excess risk of UOT-DRO is quite challenging due to the existence of unbalanced optimal transport. Moreover, while [17] gives the analysis of excess risk, the distance used in the excess risk analysis is different from the distance used to obtain the DRO tractable reformulation. Analyzing excess risk is of course interesting and left as our future work. This work mainly focuses on computational efficiency compared to previous methods.\n\n---\n> **W2: Parameter selection**\n\nWe agree that this warrants further discussion. We have added thorough discussions on the selection of all the parameters $\\lambda,\\beta,\\lambda_2$. Please see **Common Responses C2 and C3**.\n\n---\n> **Q1: Sensitivity to $\\varepsilon$**\n\nThank you for providing the additional interesting reference [A]. Upon careful review, we find that the analysis in [A] cannot be applied to assess the sensitivity to $\\varepsilon$ in [17], since the distance that the reference [17] uses to define the DRO problem is different from that analyzed in [A]. The numerical experiments in [17] show that the performance of OR-WDRO deteriorates when $\\hat{\\varepsilon}$ is much smaller than the true $\\varepsilon$, as illustrated in Figure 2 in [17]. This finding aligns with the simulation results presented in Tables 6 and 8 of our paper.\n\nThe mechanism behind OR-WDRO is that it cuts off the leftmost $\\hat{\\varepsilon}$ percentile of the random value $\\sup _ {\\xi \\in \\Xi} \\left\\\\{ l(\\theta,\\xi) - \\lambda_1 \\left\\| \\xi - \\xi_0 \\right\\|^2- \\lambda c(\\xi,\\zeta)\\right\\\\}$ in Eq (12), which is induced by outliers, and then computes the average of the remaining $(1-\\hat{\\varepsilon})$ percentile. If $\\hat{\\varepsilon}$ is much smaller than the true $\\varepsilon$, it results in inefficient outlier exclusion and thus poor performance of OR-WDRO. Therefore, OR-WDRO depends on a relatively accurate estimate of $\\varepsilon$. In contrast, our unbalanced method does not rely on cutting off outliers but instead re-weights samples, offering a smoother approach. As a result, our method does not require an accurate estimate of $\\varepsilon$. For more details about how our method deals with outliers, please see **Common Response C1**. \n\n---\n> **Q2: Convergence of Algorithm 1**\n\nWe run Algorithm 1 in linear regression and the convergence result is shown in the new Figure 1. Since $F^*$ is hard to compute, we plot the evolution of $F(\\theta_t)$.\n\n---\n> **Q3: Parameter tuning for the proposed method and the baseline method**\n\nThank you for raising this significant point. The guidelines for parameter selections can be found in **Common Response C2**. Following the guidelines, we tune the parameters and we find that the performance is not sensitive to parameter selections in three tasks. For more details about sensitivity analysis, please see **Common Response C3** and the attached PDF file, where we conducted additional experiments on linear classification and logistic regression.\n\nThe baseline method proposed in [17] requires tuning the parameters $\\lambda,\\sigma,\\epsilon$. We adopted the parameter selections from the original implementation of [17], which we find is nearly optimally tuned. Specifically, it sets $\\sigma = \\sqrt{d}$, see discussions in [17]; $\\epsilon=0.1$, aligning with the 10\\% data corruption rate; $\\rho = 0.1$, corresponding to the size of Wasserstein perturbation. We will include these in the updated draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GO5p14Q38h", "reviewer": "Reviewer_QxeQ", "summary": "The paper proposes a new Distributionally Robust Optimization (DRO) framework based on Unbalanced Optimal Transport (UOT) distance. Under some conditions, the authors establish strong duality results for a Lagrangian penalty variation of the proposed problem. The designed algorithm is tested in linear, logistic regression and linear classification tasks where it shows an increased robustness to outliers in comparison to the considered baselines.", "questions": "1) Please explain the choice of cost function $c(\\xi,\\zeta)=|\\xi-\\zeta|$ in your experiments\n2) How do you choose parameters $\\beta$, $\\lambda$ in your experiments? Is it an heuristic choice? An ablation study of these parameters is highly anticipated\n3) Why is the performance of UOT-DRO for contamination $C=30$ higher than for $C=100$? How further increase of $C$ will influence the performance?\n4) Why you did not perform comparison with additional outlier-robust DRO approaches [1,3,4] cited in the paper?\n\n*Minor limitations*:\n- bibliography is not in alphabetical order,\n- line 286: $\\theta_*$ is not defined,\n- lines 311-312: '*only* requires *only* 20 seconds'.\n\n**References.**\n\n[1] Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj, and Arun Sai Suggala. Stochastic re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.09222, 2023.\n\n[2] Sloan Nietert, Ziv Goldfeld, and Soroosh Shafiee. Outlier-robust Wasserstein dro. Advances in Neural Information Processing Systems, 36, 2024.\n\n[3] Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning, pages 12345–12355. PMLR, 2021.\n\n[4] R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2:21–42, 2000.", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes an original approach to the DRO problem utilizing the UOT distance to define the ambiguity set.  The authors provide theoretical justifications for the dual form of the problem under consideration and convergence analysis of the constructed algorithm. They empirically demonstrate that the proposed approach allows for increasing the robustness to outliers in comparison to the chosen baselines.", "weaknesses": "**Quality.** Some of the assumptions related to the ‘outlier-robustness’ property of the proposed approach are not sufficiently explained.\n\n1) The authors claim that the ambiguity set of the proposed approach (UOT DRO) necessarily includes ‘the clean distribution $P^*$' (line 146-147) due to the soft marginal constraints in the UOT problem. However, actually, this ‘softness’ of the marginals depends on the chosen divergence $\\mathcal{D}_{\\phi_2}$ and the parameter $\\beta$ in the unbalanced divergence definition (4). Typically, increasing the parameter $\\beta$, one gets closer to the balanced OT formulation which does not have ‘outlier robustness’ property, and does not allow one to ignore the outliers in empirical dataset. This dependence on parameter $\\beta$ is not explained and verified from theoretical or practical perspectives. Clarification of this point seems to be very important since this parameter is set differently in experiments considered in the paper.\n\n2) Besides, the authors introduce the function $h$ penalizing potential distributions with outliers in an ambiguity set of the primal problem (6) of UOT-DRO. Thanks to this function, the solution (model) of the Lagrangian penalty reformulation of UOT-DRO will be less affected by outlier data as explained in lines 215-225. However, the authors provide explanations for the case of large $\\lambda$, while in the experimental section they consider $\\lambda=10$. The influence of outliers on the model in this case is still not obvious.\n\n**Clarity.** Some parts of the text should be written more clearly, e.g., the formula in line 289. The choice of cost function and parameters in experiments (Section 5.1) should be explained.\n\n**Evaluation.** The comparison with the baselines seems to be rather limited. The authors have cited several other works which propose approaches for the outlier-robust DRO problem [1,2,3,4]. However, the empirical evaluation of the proposed approach misses the comparison with some of these approaches [1,3,4].\n\n\n**In summary**, I am not sufficiently convinced that the constructed approach actually solves the problem that it was designed for, i.e., allows for solving the DRO problem while effectively ignoring the outliers in the empirical dataset. The paper does not provide any theoretical results which support this claim. Thus, it should be accurately justified at least from the empirical side. However, the experimental part misses ablation study of parameters $\\beta$, $\\lambda$, and thorough comparison with other outlier-robust DRO approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address the concerns below.\n\n---\n> **Quality 1: Chosen divergence and impact of $\\beta$**\n\nThank you for highlighting the chosen divergence in UOT. This work uses the well-studied KL divergence, which finds diverse applications in information theory, machine learning and optimal transport. Exploring other metrics or divergences is of course interesting and left as our future work.  \n\nThank you for highlighting the parameter $\\beta$. We have added a thorough discussion on the impact of $\\beta$ from both the theoretical and practical perspectives. The theoretical discussion can be found in **Common Response C2.3**. Practically, we conducted additional experiments on linear classification to assess the model's performance across a broader range of $\\beta$. Since changes in $\\beta$ affect the function values, we used accuracy as the performance measure. As shown in the new Table 3, our method achieves the accuracy above 90\\% across the range of $\\beta$ values from 0.5 to 20 and $\\lambda$ values from 5 to 20. This performance is better than that of standard DRO with accuracy 66\\%, and OR-WDRO with accuracy 79\\%. Besides, as analyzed in **Common Response C2.3**, the model's performance degrades when $\\beta$ is either too small or too large. Overall, these simulation results show that our method is relatively insensitive to the choice of $\\beta$ and easy to be tuned. \n\n---\n> **Quality 2: Explanation of our method and large $\\lambda$**\n\nIn lines 215-225, we explained our method using the case of large $\\lambda$ because it simplifies the function, making it easier to explain. We would like to clarify that our method UOT-DRO has outlier robustness regardless of the $\\lambda$ value. Please see **Common Response C1** for a general explanation. \n\nIn Section 5.1 on linear regression, we show that Eq (15) can be simplified to Eq (16) if $\\lambda\\geq \\lambda_2 + \\kappa(\\theta)$. This condition is satisfied when we select $\\lambda=10$, $\\lambda_2=5$, since the true $\\theta$ we use to generate data has a norm $1$ and $\\kappa(\\theta) = \\left\\| [\\theta,1]\\right\\|_*\\leq 2$. We apologize for any lack of clarity previously. The influence of outliers is reduced through assigning a small weight thanks to the function $h$, see lines 215-225 and **Common Response C1** for more discussions.\n\n---\n> **Clarity**\n\nThank you for your suggestion regarding clarity. We will enhance the clarity in the revised draft. The cost functions for linear regression and classification are the same as those in [17] to facilitate fair comparison. The cost function of logistic regression is taken from the seminal work [15].\n\n---\n> **Evaluation and Q4: Comparison with approaches in [1,3,4]**\n\nThrough this comparison, our goal is to highlight the advantage of our method in handling outliers by utilizing unbalanced optimal transport. We did not compare our method with the approaches in [1] and [3] because the DRO problem they address differs from ours, making a direct comparison infeasible. Specifically, references [1] and [3] use the KL divergence to construct the ambiguity set, resulting in a KL DRO problem, whereas [2] and this paper employ Wasserstein distance, leading to a Wasserstein DRO problem. The comparison between this paper and [1] or [3] would be unfair since there are two key differences: the distance used to build the ambiguity set and method to deal with outliers. Any observed performance differences could not be solely attributed to the outlier-handling methods, making the comparison inconclusive. While it might be possible to adapt our unbalanced method for outlier handling to the KL DRO framework, this would constitute a different problem altogether, requiring a complete reevaluation, which we have left as future work due to time constraints. Regarding reference [4], it is a classical paper on CVaR that does not propose any method for handling outliers. Though the dual of CVaR can be seen as a DRO problem, it cannot be applied to handle outliers directly.\n\n---\n> **Q1: Choice of cost function $c(\\xi,\\zeta)=\\left\\|\\xi-\\zeta\\right\\|_1$**\n\nWe adopt the cost function for linear regression from [17] to ensure a fair comparison between our method and theirs.\n\n---\n> **Q2: Parameter selection**\n\nThe parameters are tuned based on the parameter selection guideline as outlined in **Common Response C2**. Moreover, we conducted a thorough sensitivity analysis in three tasks. As shown in Tables 3 and 4 for linear regression, Table 9 and new Table 3 for linear classification, new Tables 1 and 2 for logistic regression, our method performs well across a wide range of parameters and is not sensitive to the choice of parameters. Please see **Common Response C3** for a detailed discussion on sensitivity analysis.\n\n---\n> **Q3: Performance of UOT-DRO for contamination $C=30$ and $C=100$**\n\nRecall that our method reduces the effect of outliers by assigning them a small (but non-zero) weight. When the contamination is high, the outlier points are significantly distant from the normal data. In this case, outlier points, despite their minimal weighting, may slightly influence model performance due to their extreme deviation from the normal data. To assess the impact of increasing $C$, we conducted additional experiments on linear classification, keeping the parameters fixed. The results indicate an accuracy of 95.8\\% for $C=30$, 95.2\\% for $C=100$, 95.0\\% for $C=300$, 94.6\\% for $C=600$, and 93.4\\% for $C=1000$. Given that $C=1000$ represents a substantially large contamination level relative to the variance of the data distribution, which is $\\sqrt{10}$, these findings suggest that our method is not sensitive to $C$ and performs well even at high contamination levels.\n\n---\n> **Minor Limitations**\n\nThank you for pointing out these typos. We have refined them in the updated draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Common Response

> **C1: How does our method deal with outliers?**  

Consider the Lagrangian penalty problem of minimizing $F(\theta) = \sum_{i=1}^n \exp\left(\frac{f _ {\lambda}(\theta,\hat{\zeta}_i)}  {\lambda \beta} \right)$ in Eq (13), where $\hat{\zeta}_i$ is the $i$-th sample that may be an outlier. The objective function $F$ comprises $n$ terms, where each term depends on the sample $\hat{\zeta}_i$, respectively. The key to achieving outlier robustness against outliers lies in reducing their contributions to the function $F$ compared to the normal data. The function $h$ helps this by ensuring that $f _ {\lambda}$ outputs a minimal value at outlier points regardless of the $\lambda$ value. Specifically, for an outlier $\hat{\zeta}_i$, the term  $\exp\left(\frac{f _ {\lambda}(\theta,\hat{\zeta}_i)}  {\lambda \beta} \right)$ would be very small and contribute less to the overall objective function $F$. 

---
> **C2: Parameter selections.**

Here, we discuss some guidelines for selecting the parameters $\lambda,\lambda_2,\beta$ in Algorithm 1.

**C2.1: $\lambda$.** The parameter $\lambda$ is commonly used in DRO literature [25] as a penalty coefficient. A larger value of $\lambda$ makes the DRO problem get close to the empirical risk minimization, resulting in a model that performs well on the empirical distribution but is less robust to Wasserstein perturbations.

**C2.2: $\lambda_2$.** The parameter $\lambda_2$ represents the credibility level assigned to the function $h$.  A larger value of $\lambda_2$ should be selected when there is high confidence in the reliability of $h$, meaning that $h(\xi)$ is very likely to yield a large value at outlier points. Conversely, if the prior knowledge provided by $h$ is considered unreliable, the value of $\lambda_2$ should be reduced. If there is no reliable prior knowledge at all, in which we should select $\lambda_2=0$, achieving good performance is impossible as discussed in the robust statistics literature [18].

**C2.3 : $\beta$.** The parameter $\beta$ quantifies the degree of penalization for the mismatch of marginal distributions. A larger value of $\beta$ places more emphasis on minimizing this mismatch, possibly at the expense of increasing the transportation cost, thereby making the unbalanced optimal transport distance close to the balanced one. Conversely, a small value of $\beta$ allows for greater flexibility in the distribution mismatch, which can enhance the model's robustness to outliers. However, when $\beta$ is extremely small, the mismatch incurs little penalty, and the computed distance may fail to accurately represent the true distance between the distributions.  In this case, the resulting DRO problem would include too many unlikely distributions in the ambiguity set, leading to an overly conservative model.

Theoretically, we discuss the impact of parameter $\beta$ from the perspective of original problem formulation (13). Following the discussion in Common Response C1, consider the objective function $F(\theta) = \sum_{i=1}^n \exp\left(\frac{f _ {\lambda}(\theta,\hat{\zeta}_i)}  {\lambda \beta} \right)$, which consists $n$ terms, each induced by a data point. A suitable value of $\beta$ would enable $\exp\left(\frac{f _ {\lambda}(\theta,\hat{\zeta}_i)}  {\lambda \beta} \right)$ to be smaller at an outlier point compared to normal data, thus reducing the impact of outliers. However, if $\beta$ is overly large, each term will equally approach 1 regardless of the value of $f _ {\lambda}$, thus failing to reduce the impact of outliers. This observation aligns with the previous analysis that a large value of $\beta$ makes the unbalanced method lose the outlier robustness as it approaches the balanced one. Conversely, an extremely small $\beta$ can also degrade performance because the objective function becomes overly focused on the sample that yields the largest value of $f _ {\lambda}$, neglecting the rest.

---
**C3: Parameter sensitivity.** When tuning the parameters, we find that our method allows a wide range of parameter selections without significant performance variations in linear regression, classification and logistic regression problems. This is verified through a comprehensive sensitivity evaluation. For linear regression, as shown in Tables 3 and 4, the model's performance shows minimal sensitivity to variations in these parameters. For linear classification, as shown in Table 9 and new Table 3, our method maintains an accuracy above 90\% across a variety of parameter settings. For logistic regression, we conducted additional experiments to evaluate the sensitivity of parameters $\lambda,\beta,\lambda_2$. As shown in the new Tables 1 and 2, our method achieves the accuracy above 90\% in most cases across the considered ranges of $\lambda$ from 5 to 15 and $\beta$ from 0.4 to 10. Based on all of these simulation results, we find that our method's performance remains stable across a wide range of parameters and it is easy to tune these parameters.

Meta-review: The authors present a novel approach by leveraging unbalanced optimal transport (UOT) to develop a new Wasserstein Distributionally Robust Optimization (WDRO) framework, accompanied by strong duality results for a Lagrangian penalty variation of the proposed problem. The framework is tested across multiple scenarios, including linear, logistic regression, and linear classification tasks, demonstrating enhanced robustness to outliers. Although there are areas for further improvement, such as deeper theoretical analysis and sensitivity to contamination levels, the paper offers valuable contributions and has sufficient merit for acceptance.
Decision: Accept (poster)