Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport

Authors: zifan  wang
yi  shen
michael m zavlanos
karl h johansson
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:06:50

Abstract: Distributionally Robust Optimization (DRO) accounts for uncertainty in data distributions by optimizing the model performance against the worst possible distribution within an ambiguity set. In this paper, we propose a DRO framework that relies on a new distance inspired by Unbalanced Optimal Transport (UOT). The proposed UOT distance employs a soft penalization term instead of hard constraints, enabling the construction of an ambiguity set that is more resilient to outliers.  Under smoothness conditions, we establish strong duality of the proposed DRO problem. Moreover, we introduce a computationally efficient Lagrangian penalty formulation for which we show that strong duality also holds. Finally, we provide empirical results that demonstrate that our method offers improved robustness to outliers and is computationally less demanding for regression and classification tasks.

Main paper: 
 introduction: 
 Consider a stochastic optimization problem that aims to find a decision variable θ ∈ Θ that minimizes the expected loss function E ξ∼P * [l(ξ, θ)] [1] , where the random samples ξ ∈ Ξ are drawn from a fixed distribution P * and Θ ⊂ R d is a compact and convex set. In many applications, the distribution of interest P * is not precisely known; yet datasets that contain finite samples independently drawn from the distribution P * are often available, which yield an empirical distribution estimate P. For instance, in image classification problems, P * can represent the joint distribution of the features and the labels, and P can represent the empirical distribution of a dataset that is annotated by humans [2] ; in portfolio optimization problems, P * captures the uncertainty of a set of financial products in the future months and P contains historical prices of all the products [3] ; in healthcare applications, P * can represent the feature distribution of a population of interest, and P represent the available electronic health record data [4] .The above stochastic optimization problem is often solved using empirical risk minimization (ERM) [5] , which searches for the best decision variable θ that minimizes the expected loss function with respect to the empirical distribution, i.e., θ ERM = arg min θ∈Θ E ξ∼ P[l(ξ, θ)]. However, ERM is not guaranteed to provide good out-of-sample performance, i.e., the value of E ξ∼P * [l(ξ, θ ERM )] is high when the empirical estimate P is not close to the true distribution P * . For example, in the presence of distribution shifts or contaminated datasets, the out-of-sample performance cannot be improved by increasing the sample size. To enhance the out-of-sample performance of the decision variable, researchers have developed methods that use distributionally robust optimization (DRO) to address the mismatch between P and P * [6, 7, 8, 9, 10, 11, 12] . In contrast to ERM, DRO aims to optimize outcomes against the worst-case scenario over a set of distributions that are close to the empirical distribution according to a pre-specified distance metric. This set of distributions is called the ambiguity set and is typically assumed to contain the distribution of interest, e.g., the true distribution P * . The construction of the ambiguity set is critical in the development of various DRO methods, as it ideally encompasses all prior knowledge about the true distribution. For example, [6] defines the ambiguity set using moment constraints. Moment constraints assume that the true distribution has similar moments compared to the empirical distribution, e.g., means and variances. The ambiguity set can also include distributions that are similar to P in infinite-order moments; this construction is explored in [13] and is termed Kernel DRO. When the distribution of interest is categorical, Kullback-Leibler divergence (KL) is often adopted to define the ambiguity set as the categories can be permuted and KL does not consider the geometry of a distribution's support. DRO using KL has been extensively studied in the literature due to its computational benefits [14, 15] . In contrast, the Wasserstein distance between two distributions captures the geometry of the distributions. The Wasserstein distance can be useful, e.g., in loan approval prediction problems where the distribution represents the incomes of a population and prior knowledge may specify that two clients are similar if their incomes are close. DRO using the Wasserstein distance (WDRO) has been applied to many machine learning problems, such as linear regression [16] and logistic regression [17] problems.In this paper, we apply DRO to handle distribution mismatches using the Wasserstein distance as it can better capture the geometry of the distributions. However, WDRO is not suitable for contaminated datasets, e.g., datasets that contain outliers that are geometrically far from the clean distribution but cannot be easily removed. The reason is that the Wasserstein distance between the contaminated empirical distribution and the clean (true) distribution is very sensitive to the outliers; to include the clean distribution in the ambiguity set, the Wasserstein distance would need to be selected very large and inevitably the ambiguity set would include distributions that will never happen in practice.To apply DRO to contaminated datasets, the authors in [18, 19, 20] take outliers into consideration and propose various methods to minimize their impact on the learning performance. Specifically, [18] proposes to minimize the most optimistic DRO risk to avoid the hardest instances that are likely to be outliers. In [19] , a weight clipping method is proposed to truncate the excessive impact of outliers on the results. However, both [18] and [19] use KL divergence to define the ambiguity set, which fails to capture geometric uncertainty. Notably, the authors in [20] introduce an outlier-robust WDRO framework capable of addressing both geometric uncertainty and non-geometric contamination, allowing an ε fraction of data to be arbitrarily corrupted. They design an ambiguity set based on prior knowledge and demonstrate the strong duality of the resulting DRO problem, which is related to the Conditional Value at Risk (CVaR) [21] . Nevertheless, the performance of the formulated DRO problem relies heavily on the accuracy of the estimate of ε. Besides, [20] uses off-the-shelf solvers to solve the resulting DRO problem, which is not computationally efficient when applied to large datasets since the number of constraints in the proposed method grows linearly with the sample size.The main contribution of this paper is to introduce a new framework for outlier-robust WDRO based on Unbalanced Optimal Transport (UOT), which is known for its inherent robustness to outliers and missing data [22, 23, 24, 25, 26] . Our approach involves the design of a UOT distance that substitutes hard constraints by a soft penalization term. This construction, combined with some prior knowledge on the learning task, enables the design of a new ambiguity set that includes distributions of interest and penalizes distributions that contain outliers. For the DRO problem with this new ambiguity set, we establish strong duality results under specific smoothness assumptions. However, the solution of the dual problem poses significant computational challenges. Motivated by [27] , we explore a Lagrangian penalty variation of the problem, which can be viewed as a Lagrangian function of the original formulation with a fixed dual variable. We show that strong duality holds for this Lagrangian penalty problem under fewer assumptions compared to the original DRO problem. The reformulated problem employs an exponential function to reweight data points, effectively diminishing the influence of outlier data on the optimization process. We solve this problem by proposing a provable stochastic (sub)-gradient algorithm, which is computationally efficient. We provide empirical results that demonstrate that our approach not only enhances robustness to outliers but also improves computational efficiency for large-scale problems.Our work is related to data reweighted optimization problems [28, 29, 30] , which adjust the weights of individual data to manage outliers. However, these studies construct the reweighted optimization problems heuristically and cannot handle distribution uncertainty. In contrast, our problem formulation is able to handle distribution uncertainty via DRO, and our findings demonstrate that the reweighted problem constitutes the dual of a specific class of DRO problems, with the ambiguity set defined by the UOT distance.The rest of the paper is organized as follows. Section 2 defines the UOT distance and the associated DRO problem. Section 3 presents the strong duality results for both the original DRO problem and its Lagrangian penalty variant. The main algorithm along with its convergence analysis is presented in Section 4. Section 5 demonstrates the effectiveness of our proposed framework in addressing outliers in both regression and classification tasks. Finally, we conclude the paper in Section 6. In the Appendix, we provide all proofs as well as some additional experiments. 
problem definition and preliminaries: 
 The (1-)Wasserstein distance [31] between two distributions P and P is defined asEQUATIONwhere Γ(P, P) denotes the set of joint distributions such that γ 1 = P and γ 2 = P, where γ 1 and γ 2 denote the first and the second marginal distribution of γ, respectively. EQUATIONwhere B ρ ( P) := {P ∈ M(Ξ) : W(P, P) ≤ ρ} denotes the ambiguity set that contains all distributions within ρ-distance from P and ρ ≥ 0 is a user-specified parameter. Here, we denote by M(Ξ) the space of all probability distributions supported on Ξ.The Wasserstein distance enforces equality constraints on the marginal distributions of γ to match P and P, thereby restricting the choice of γ. If the empirical distribution P contains outliers, it becomes necessary to select a large radius ρ for the ambiguity set to ensure the inclusion of the true distribution P * as the outliers tend to increase the Wasserstein distance between P and P * . However, a large ρ can lead to a DRO solution that is conservative since the larger the radius the larger the value of the inner supremum problem in (2) . Though a large radius allows the inclusion of the distribution of interest, i.e., P * , it also results in the inclusion of many distributions that are not of interests, e.g., distributions that are unlikely to occur in practice. As a result, the Wasserstein distance is not a desired distance choice when the empirical distribution contains outliers.In Unbalanced Optimal Transport (UOT) problems [22] , the joint distribution γ is not required to have fixed marginals in contrast to (1) . Specifically, for any two arbitrary positive measures P and P, the UOT distance is defined byEQUATIONwhere γ 1 and γ 2 are marginals of γ, and D φ1 and D φ2 are Csiszár divergences that measure discrepancies between positive measures, based on functions φ 1 and φ 2 , respectively. The key distinction between UOT distance and Wasserstein (OT) distance lies in the constraints of the marginals. While the OT distance in (1) strictly enforces marginal distributions, i.e., γ 1 = P and γ 2 = P, the UOT in (3) relaxes the equality constraints by adding mismatch penalty functions D φ in addition to the transport cost induced by c(•, •). This relaxation makes UOT robust to outliers since γ 2 can represent a distribution that is different from the contaminated distribution P.Inspired by the definition of UOT, we consider the following unbalanced Wasserstein distance (divergence) UW(P|| P) = inf P,γ∈Γ(P, P)EQUATIONwhere β is a tuning parameter and P is an intermediate optimization variable that links P and P. Note that (4) is a special case of (3) by selecting φ 1 = ι {1} (i.e., φ 1 (1) = 0 and ∞ otherwise) and φ 2 (x) = x log x -x + 1 (i.e., D φ2 denotes the KL divergence). In addition, we restrict the marginals of the joint positive measure γ in (3) to be probability measures because the general positive measures are not of interest in the DRO problems we consider. The particular choice of φ functions is for technical reasons related to the duality analysis, as we will show in Section 3.The optimal unbalanced Wasserstein transport under UW(P|| P) can be interpreted as a two-step procedure: 1) P can be transported to any P by paying a small price measured by its distance to P in the KL sense (a non-geometric transport), and 2) P is then transported to P by minimizing the geometryaware transport cost. The non-geometric KL divergence is the key to allow the uncontaminated clean distribution P * to be close to the contaminated empirical distribution P as the KL divergence does not distinguish the geometric locations of the distribution supports.We define the ambiguity set using the unbalanced Wasserstein distance by U ρ ( P) = {P ∈ M(Ξ) : UW(P|| P) ≤ ρ}. Then, the DRO problem using the UOT distance (4) can be formulated asEQUATIONThe ambiguity set defined by the unbalanced Wasserstein distance enables us to include the clean distribution P * , but it cannot remove distributions that are close to the contaminated empirical distribution P as these distributions are close to P both geometrically (Wasserstein) and non-geometrically (KL). As a result, these distributions, denoted by set P, could also make the learned model conservative. To remove such distributions, we need some prior/domain knowledge. As discussed in [18, 20] , it is impossible to design a model selection strategy using contaminated datasets without any prior knowledge. In Section 5, we also show that constructing the ambiguity set without prior knowledge may result in worse performance compared to the traditional DRO; see Table 5 .We assume that any distribution P ∈ P incurs significantly large costs when evaluated using a given function h compared to uncontaminated distributions, i.e., E ξ∼ P[h(ξ)] ≫ E ξ∼P [h(ξ)], ∀ P ∈ P and ∀ P ∈ {P : UW(P|| P) ≤ ρ}\ P. The function h can be related to the moment constraint. For example, if we know the expectation estimate ξ 0 of the clean distribution, we can design h(ξ) = ∥ξ -ξ 0 ∥ and the function value of h(ξ) will be high if ξ is an outlier. Similar choices are considered in [20] .Taking into account the prior knowledge, we consider minimizing the following (primal) functionEQUATIONwith L(θ, ξ) = l(θ, ξ) -h(ξ) and we assume that L(θ, ξ) is continuous in ξ. For any distribution P that contains outliers, E ξ∼ P[h(ξ)] is large and thus P is less likely to attain the supremum in (6) .In summary, we first employ the unbalanced Wasserstein distance to incorporate the distributions of interest. Then, we leverage prior knowledge to systematically exclude distributions that may contain outliers. This two-step approach ensures a more thorough consideration of potential distributions while effectively mitigating the impact of outliers on the learning problem. 
unbalanced distributionally robust optimization: 
 Note that the primal problem in ( 6) is an infinite-dimensional optimization as the optimization variable represents probability distributions. In this section, we derive its dual problem and a variant of the dual problem that is computationally tractable. This assumption ensures that the joint distribution γ can be written as dγ(ξ, ζ) = dγ ζ (ξ)d P(ζ); see [32] for more details on regular conditional distributions. By the law of total expectation, we haveE ξ∼P [L(θ, ξ)] = E ζ∼ PE ξ∼γ ζ (ξ) [L(θ, ξ)].We define f λ (θ, ζ) := sup ξ∈Ξ {L(θ, ξ) -λc(ξ, ζ)}. We first derive the dual problem and show the weak duality in the following lemma.Lemma EQUATIONThe proof can be found in Appendix 7.2. For technical reasons, we show the strong duality when the optimal dual variable λ * is strictly positive. In most applications, the optimal dual variable λ * is in fact positive. When λ * = 0, the inequality constraint becomes inactive and the radius of the ambiguity set becomes very large, a situation that is out of scope for most practical applications.Although strong duality holds and the dual problem is a finite dimensional minimization problem, solving problem ( 7) is computationally challenging due to the complex dependency of the objective on λ and, in particular, the implicit dependency of f λ on λ. Recall that f λ (θ, ζ) = sup ξ∈Ξ {L(θ, ξ) -λc(ξ, ζ)}; in general, we cannot obtain an analytical form of f λ (θ, ζ), i.e., a closed-form solution of the problem sup ξ∈Ξ {L(θ, ξ) -λc(ξ, ζ)}. Additionally, the dual problem (7) is not convex in λ in general. The task becomes more involved when attempting to minimize over λ and θ jointly. To address these challenges, we shift focus from the constrained ρ-robustness problem (7) and instead consider the corresponding Lagrangian penalty problem, which allows for a computational tractable formulation.Motivated by [27] , we consider the Lagrangian penalty reformulation of ( 7) as followsEQUATIONIn what follows, we show that the strong duality also holds for the Lagrangian penalty problem (8). EQUATION)The proof is provided in Appendix 7.3. Note that the strong duality of the Lagrangian penalty problem only requires Assumption 1 to hold in contrast to Theorem 1. In particular, it does not require the loss function L(θ, ξ) to be differentiable, making it applicable to a broader class of problems. Besides, solving problem ( 8) is computationally tractable as the dual problem has a closed-form solution as in (9) . In particular, λ is a parameter in (9) while it is an optimization (dual) variable in (7) .Since the logarithm function is monotonically increasing, minimization of ( 9) in θ is equivalent to the following formEQUATIONComparison with standard WDRO. It has been shown in [27] that the Lagrangian penalty problem using the Wasserstein metric W c has the following strong duality resultEQUATIONwhere W c is the Wasserstein distance metric associated with the cost function c(•, •). When λ in ( 11) is selected to be large, this formulation is close to the ERM problem, i.e., minimization ofE ζ∼ P[l(θ, ζ)].Clearly, a distribution P containing outliers would significantly deteriorate the learning performance of minimizingE ζ∼ P[l(θ, ζ)].In contrast, when λ is large, the problem ( 10) is close to the minimization ofE ζ∼ P[exp ((l(θ, ζ) -h(ζ))/(λβ))] = E ζ∼ P[exp (l(θ, ζ)/(λβ)) w(ζ)],where w(ζ) = exp (-h(ζ)/(λβ)) acts as a re-weighting function. Recall that h(ξ) is a function designed based on prior knowledge and assumes high values at outlier points. At an outlier point, say ξ, h( ξ) would be significantly larger than at non-outlier points, resulting in a very small weight w( ξ). Consequently, outlier data with small weights exert less influence on the resulting model. Therefore, the formulation ( 10) is more robust to outliers compared to standard WDRO.Comparison with outlier-robust WDRO [20] . [20] considers the Huber contamination model in which an ε fraction of data can be arbitrarily corrupted. In [20] , prior knowledge on the mean of the clean distribution is needed. Specifically, the function h(ξ) that can identify outliers in [20] is selected ash(ξ) = λ 1 ∥ξ -ξ 0 ∥2 , where ξ 0 represents the estimated mean of the clean distribution and λ 1 > 0 is a tuning parameter. It can be verified that the corresponding Lagrangian formulation isEQUATIONwhere CVaR 1-ε denotes the conditional value at risk (CVaR), interpreted as the expected value of the rightmost (1 -ε) percentile of outcomes1 .Similarly, when λ is very large, the formulation in (12) closely approximates the minimization ofCVaR 1-ε, P[l(θ, ζ) -λ 1 ∥ζ -ξ 0 ∥]. At an outlier point, say ξ, the value l(θ, ξ) -λ 1 || ξ -ξ 0 || is smalldue to the large distance || ξ -ξ 0 ||. Consequently, such an outlier point generates a function value that falls on the left tail of the loss distribution and is thus effectively excluded by the CVaR operation. This method can also be viewed as a re-weighting method by assigning weight 0 to outlier points and weighting other points equally. However, this approach is highly sensitive to the choice of ε, which is usually difficult to obtain a priori. In comparison, our method is a smoothed version of this CVaR method and is less sensitive to the prior knowledge of ε. Besides, calculating the value of CVaR is computationally more demanding when the number of samples is large as we will demonstrate in Sections 4 and 5. 
algorithm design: 
 In this section, we focus on solving the dual of the Lagrangian penalty problem. We assume that the empirical distribution P is constructed from n samples and written as P = 1 n n i=1 δ ζi with ζi the i-th sample. According to Theorem 2, the Lagrangian penalty problem is equivalent to the minimization of the following objective function EQUATIONθ t+1 = Proj Θ (θ t -ηg t ) 6: end forTo solve the problem (13), we employ the stochastic sub-gradient method, detailed in Algorithm 1. Specifically, at each time step t, we sample ζt from the empirical distribution P uniformly. We then find the ϵ-approximate maximizer z t of the function L(θ t , ξ) -λc(ξ, ζt ). We require that the solution satisfies dist(z t , X * t ) ≤ ϵ, where X * t denotes the set of maximizers. This is solvable in many applications and can be achieved through, e.g., the (sub)-gradient method [27] . Then, we perform the projected sub-gradient update. The convergence analysis of Algorithm 1 is presented in the following theorem. The proof can be found in Appendix 7.4. Theorem 3. Suppose that Θ is a convex and compact set with bounded diameter R. Assume that the loss L(θ, ξ) is convex in θ for each ξ ∈ Ξ. For all data points ζ i from the empirical distribution P, assume that L(θ,EQUATIONwhere F * represents the minimum value of the objective function (13) .The accuracy parameter ϵ in the inner maximization problem has a fixed effect on the final optimization accuracy, independent of T . The assumptions in Theorem 3 are common in the analysis of optimization problems involving the exponential function in the objective function, see, e.g., [19, 28, 29] .Algorithmic comparison with outlier-robust WDRO [20] . Our proposed formulation can be solved by the stochastic sub-gradient method, which is computationally more efficient when the sample size is large. In contrast, the approach in [20] utilizes CVaR to account for outliers and employs off-the-shelf solvers, e.g., GUROBI [33] , to obtain a solution. A notable limitation of the presence of CVaR in DRO problems is that the number of constraints scales linearly with the sample size. Consequently, for large datasets, this scaling significantly increases the number of constraints, leading to computational inefficiency, as shown in Section 5. Designing a stochastic gradient method for the CVaR-based formulation in [20] is possible, although challenging since the estimate of the CVaR gradient is usually biased. Moreover, our methodology does not rely on off-the-shelf commercial solvers and can be incorporated in machine learning packages seamlessly, which allows us to handle a wide range of loss functions via the stochastic sub-gradient method. 
experiments: 
 We conduct experiments on linear regression, linear classification, and logistic regression problems. We use the stochastic sub-gradient method in Algorithm 1 to solve the Lagrangian penalty problem and the GUROBI [33] solver to solve all other DRO problems we use as benchmarks. All experiments were conducted on an Intel Core i7-1185G7 CPU (3.00GHz) using Python 3.8. Our method is referred to as UOT-DRO. A discussion on parameter selection is provided at the end of this section. We consider a linear regression problem with the loss l θ (x, y) = |θ ⊺ x -y|, where (x, y) ∈ R d × R represents a generic data point and θ ∈ R d is the model we aim to train. We define θ = [θ ⊺ , -1] ⊺ , ξ = [x ⊺ , y] ⊺ . Then, the loss function can be written as l θ (ξ) = | θ⊺ ξ|.We generate a clean data distribution P n with n samples, which is uniform over{X i , θ ⊺ * X i } n i=1, where X 1 , . . . , X n are i.i.d. from X ∼ N (0, I d ). Drawing a uniform random subset S ⊂ [n] of size ⌊εn⌋, the corrupted data distribution Pn is defined to be uniform overC 1{i∈S} X i , -C 2 1{i∈S} (X i + ρ) n i=1, where C > 0 is a corruption scaling coefficient and ρ > 0 is a shift coefficient. The empirical probability distribution Pn can be written as Pn = 1 n n i=1 δ ζi with ζi the i-th sample. We consider prior knowledge on the mean of the clean distribution, denoted as ξ, and design the function h(ξ) = λ 2 ξ -ξ . The value of ξ is determined by a cheap preprocessing step, same as in [20] . We consider the Lagrangian penalty problem which, based on Theorem 2, is given byEQUATION)Define κ(θ) = sup{∥z∥ * : l * (z) < ∞} = θ * , where ∥•∥ * denotes the dual norm. When λ ≥ λ 2 + κ(θ), we can show that the problem ( 15) is equivalent toEQUATIONThe detailed proof is provided in Appendix 7.5.We set the parameters as follows: λ = 10, β = 6, λ 2 = 5. All the results are averaged over 10 independent runs. We fix ε = 0.1 and C = 8. We compare the performance of our unbalanced DRO model against the standard DRO and outlier-robust WDRO provided in [20] . We evaluate these methods in terms of the excess risk, which we define as the difference between the loss incurred by the learned model and the minimum achievable loss. The simulation results presented in Fig. 1 demonstrate the performance of our method under varying conditions: In Fig. 1 (a), we set d = 10 and compare these methods for various samples sizes n ∈ {20, 40, 60, 80, 100}; In Fig. 1 (b), we fix n = 100 and explore the impact of various dimensions d ∈ {5, 10, 20, 30, 40}. We observe that our method not only achieves superior robustness to outliers but is also less sensitive to data dimensions.Besides, we compare the computational efficiency of these methods across varying sample sizes. As shown in Table 1 , the OR-WDRO method is not applicable to large-scale datasets, with its running time exceeding 200 minutes for a sample size of n = 20000. In contrast, our method only requires 20 seconds to process the same dataset while maintaining superior learning performance. Therefore, our method not only achieves better robustness but also maintains computational efficiency. We consider a linear classification problem with the loss l θ (x, y) = max{0, 1 -y(θ ⊺ x)} with θ ∈ R d . We consider the same outliers as that in [20] . Specifically, we first generate a clean distribution P n by {X i , sign(θ ⊺ * X i )}, where X 1 , . . . , X n are drawn i.i.d. from N (0, I d ). For a uniform random subset S ⊂ [n] of size ⌊εn⌋, we consider the corrupted distribution Pn which is uniform over(-C) 1{i∈S} X i + e 1 , sign(θ ⊺ * X i ) n i=1, where C > 0 is the contamination factor, e 1 is the distribution shift. We select h(ξ) = λ 2 ξ -ξ 2 and c(ξ, ζ) = ∥ξ -ζ∥ 2 in linear classification as in [20] .We consider the formulation described in Theorem 2 and solve it using Algorithm 1. We fix ε = 0.1, d = 10, n = 100 and select λ = 10, λ 2 = 1, β = 2. The experimental results are averaged over 10 trials. We evaluate the model performance for different choices of the contamination factor C by analyzing the excess risk and accuracy. Excess risk is defined as the difference between the returned loss and the best achievable loss. Accuracy refers to the rate of successful classification. As shown in Table 2 , the performance of standard DRO decreases when C gets large. In contrast, the OR-WDRO performs well only when C is very large. This is because OR-WDRO uses CVaR to filter out outliers, which requires the outliers to be significantly distant from the regular data for CVaR to be effective. Nevertheless, our method performs well across the entire range of C.We consider a logistic regression with the loss function l θ (x, y) = log(1 + exp(-y(θ ⊺ X))), where θ ∈ R 10 . As in [17] , we assume that the feature vector X follows a multivariate standard normal distribution, and the conditional distribution of the label y ∈ {-1, 1} is given by Prob(y|x) = [1 + exp(-y(θ ⊺ * x))] -1, where θ * = (10, 0, . . . , 0). This setup uniquely determines the true distribution P. We draw n samples from this distribution to create the empirical distribution P n = 1 n n i=1 δ (Xi,yi) . Outliers are considered to occur only in the feature space. The perturbed distribution is represented as {(-C) 1{i∈S} X i + ρe 1 } n i=1 , where S ⊂ [n] is a uniform random subset of size ⌊εn⌋. We fix ε = 0.1, ρ = 0.1, e 1 = (1, 0, . . . , 0), and select λ = 10, λ 2 = 1, β = 2. In Fig. 2 , we compare the excess risk and accuracy of standard DRO as outlined in [17] and our proposed method, described by Theorem 2 and implemented by Algorithm 1. The results are averaged over 10 runs for different sample sizes n ∈ {20, 40, 60, 80, 100}. Besides, we conduct a comparative analysis of the excess risk and accuracy between the standard DRO and the proposed method across various contamination levels. As shown in Table 3 , standard DRO is susceptible to significant outliers. Instead, the proposed method demonstrates robustness throughout the entire range of contamination factor C, consistently maintaining an accuracy of approximately 92%. We provide some guidelines for selecting the parameters λ, λ 2 , β in Algorithm 1. The parameter λ is commonly used in the DRO literature [27] as a penalty coefficient. If the value of λ is large, then the DRO problem approaches the empirical risk minimization problem, resulting in a model that performs well on the empirical distribution but is less robust to Wasserstein perturbations.The parameter λ 2 represents the credibility level assigned to the function h. A larger value of λ 2 should be selected when the confidence in the reliability of h is high, meaning that h(ξ) is highly likely to become large at outlier points. Conversely, if the prior knowledge provided by h is considered unreliable, the value of λ 2 should be reduced. If there is no reliable prior knowledge at all, in which case we should select λ 2 = 0, achieving good performance is impossible, as shown in related literature in robust statistics [18] .The parameter β penalizes the mismatch between marginal distributions. A larger value of β places more emphasis on minimizing this mismatch, possibly at the expense of increasing the transportation cost, thereby making the unbalanced optimal transport distance close to the balanced one. Conversely, a small value of β allows for larger mismatches between the marginal distributions, which can enhance the model's robustness to outliers. However, when β is very small, possible distribution mismatches incur little penalty, and the computed distance may fail to accurately represent the true distance between the distributions. In this case, the resulting DRO problem will incur many unlikely distributions in the ambiguity set, leading to a very conservative model. 
conclusion: 
 In this work, we introduced a novel DRO framework that employs a new distance derived from UOT. By incorporating a soft penalization term in the design of the ambiguity set, our method exhibits increased resiliency to outliers. We provided strong duality results for the original DRO problem and the Lagrangian penalty problem, with the latter allowing for more efficient computation via stochastic sub-gradient methods. Finally, empirical results validate our method's enhanced robustness to outliers and reduced computational demands for regression and classification tasks. 
proof of lemma 1: 
 By plugging in the definition of the UW distance in (4), we have that (P) = sup P, P∈M(Ξ),γ∈Γ(P, P)EQUATIONwhere the second last inequality follows from the max-min inequality and the last inequality follows sinceE ξ∼γ ζ [L(θ, ξ) -λc(ξ, ζ)] ≤ sup ξ∈Ξ L(θ, ξ) -λc(ξ, ζ). Recalling the definition f λ (θ, ζ) = sup ξ∈Ξ {L(θ, ξ) -λc(ξ, ζ)}, we obtain that inf λ≥0 λρ + sup P∈M(Ξ) E ζ∼ P f λ (θ, ζ) -λβ log d P(ζ) d P(ζ) = inf λ≥0 λρ + λβ log E ζ∼ P exp f λ (θ, ζ) λβ ,where the last equality follows from Donsker and Varadhan's variational formula [34] and the fact that f λ (θ, ζ) is integrable with respect to P. Note that the condition |f λ (θ, ζ)| < ∞ for Palmost every ζ ensures exp(f λ (θ,ζ)/(λβ)) E ζ∼ P [exp(f λ (θ,ζ)/(λβ))] d P(ζ) is well-defined. The supremum is achieved by d P(ζ) = exp(f λ (θ,ζ)/(λβ))E ζ∼ P[exp(f λ (θ,ζ)/(λβ))] d P(ζ). The proof is complete.Given Assumption 2-3, L(θ, ξ) -λc(ξ, ζ) is differentiable with respect to ξ. According to the Envelope theorem [35] , we havef λ (ζ) is differentiable in λ. Given L(θ, ξ) -λc(ξ, ζ) is strongly concave in ξ, we have ∇ λ f λ (ζ) = -c(h λ (ζ), ζ), where h λ (ζ) := arg max ξ∈Ξ {L(θ, ξ) -λc(ξ, ζ)}. Define v(λ) = λρ + λβ log E ζ∼ P exp f λ (ζ)Based on the definition of UW in (4), we have EQUATION)The last equality in ( 22 EQUATIONwhere the last equality follows from [14] . The proof ends.Since L(θ, ξ) -λc(ξ, ζt ) is convex in θ, and Ξ is a compact set, from Danskin's theorem [36] , we have f λ (θ, ζt ) is convex in θ andEQUATION)where Co denotes the convex hull of a point set. Given that f λ (θ, ζt ) is convex, it is easy to verify that F (θ) is convex. Since the exponential function is convex and non-decreasing, for any z * t ∈ arg max ξ L(θ t , ξ) -λc(ξ, ζt ), we haveG t = exp((L(θ t , z * t ) -λc(z * t , ζt ))/(λβ)) 1 λβ ∂ θ L(θ t , z * t ) ⊂ ∂ θ exp(f λ (θ t , ζt )/(λβ)).Note that ∥G t ∥ ≤ exp( B λβ ) B2 λβ := C 0 . Since we obtain ϵ-approximate maximizer z t of arg max ξ L(θ t , ξ) -λc(ξ, ζt ), the sub-gradient g t is inexact and we denote by the error e t = g t -G t .In what follows, we analyze the bound of sub-gradient error e t . From its definition, we haveEQUATION)where the first inequality follows from triangle inequality, the second inequality holds sinceL(θ, ξ) - λc(ξ, ζt ) ≤ B and ∥∂ θ L(θ t , z t ) -∂ θ L(θ t , z * t )∥ ≤ L θξ ∥z t -z * t ∥.The third inequality hols since |e x -e y | ≤ e B |x -y| when x, y < B. The last inequality follows from our assumptions. Since (25) holds for any z * t , we have ∥e t ∥ ≤ C 1 ϵ. Suppose that θ * minimizes F (θ). Based on the update rule, we haveEQUATION)Taking conditional expectation on both sides of ( 26) with respect to ζt , we haveEQUATION)where the second inequality holds since E[G t ] ⊂ ∂F (θ t ), the third inequality follows from the fact that G t ≤ C 0 and the last inequality follow since ∥e t ∥ ≤ C 1 ϵ. Taking expectation on both sides of ( 27) and summing up over t = 1, . . . , T , it givesEQUATIONRearranging (28), we haveEQUATIONSubstituting η = 1 √ T into (29) completes the proof.In this section, we show the derivation of equation (16) .Recall that θ = [θ ⊺ , -1] ⊺ , ζ = [x ⊺ , y] ⊺ and l θ (ξ) = | θ⊺ ξ|.As l θ is proper, convex, and lower semicontinuous, it coincides with its bi-conjugate function l * * , see e.g. [7] . Thus, we can write l θ (ξ) = sup z∈Z ⟨z, ξ⟩ -l * θ (z), where Z = {z ∈ R d : l * (z) < ∞ is the effective domain of the conjugate function. Define κ(θ) = sup{∥z∥ * : z ∈ Z}. It is shown by [16] that κ(θ) = θ * . With the definition of the dual norm, we havesup ξ∈R d l θ (ξ) -λ ∥ξ -ζ∥ -λ 2 ξ -ξ = sup ξ∈R d sup z∈Z ⟨z, ξ⟩ -l * θ (z) -λ ∥ξ -ζ∥ -λ 2 ξ -ξ = sup ξ∈R d sup z∈Z inf ∥z1∥ * ≤λ, ∥z2∥ * ≤λ2 ⟨z, ξ⟩ -l * θ (z) -⟨z 1 , ξ -ζi ⟩ -⟨z 2 , ξ -ξ⟩ .The classical minimax theorem [37] allows us to interchange the maximization and minimization and then we obtainsup ξ∈R d l θ (ξ) -λ ∥ξ -ζ∥ -λ 2 ξ -ξ = sup z∈Z inf ∥z1∥ * ≤λ, ∥z2∥ * ≤λ2 sup ξ∈R d ⟨z + z 1 + z 2 , ξ⟩ -l * θ (z) -⟨z 1 , ζi ⟩ -⟨z 2 , ξ⟩ = sup z∈Z inf ∥z1∥ * ≤λ, ∥z2∥ * ≤λ2 X {0} (z + z 1 + z 2 ) -l * θ (z) -⟨z 1 , ζi ⟩ -⟨z 2 , ξ⟩ = inf ∥z1∥ * ≤λ, ∥z2∥ * ≤λ2, z+z1+z2=0 sup z∈Z -l * θ (z) -⟨z 1 , ζi ⟩ -⟨z 2 , ξ⟩ = inf ∥z1∥ * ≤λ, ∥z2∥ * ≤λ2, z+z1+z2=0 sup z∈Z -l * θ (z) + ⟨z, ζi ⟩ + ⟨z 2 , ζi -ξ⟩ = inf ∥z1∥ * ≤λ, ∥z2∥ * ≤λ2, z+z1+z2=0, z∈Z l θ ( ζi ) + ⟨z 2 , ζi -ξ⟩ = inf ∥z1∥ * ≤λ, ∥z2∥ * ≤λ2, z+z1+z2=0, z∈Z l θ ( ζi ) -λ 2 ζi -ξ * .The second equality holds since sup ξ∈R d ⟨z, ξ⟩ = σ R d (z) = X 0 , where σ Ω and X Ω denote the support function and characteristic function of the set Ω, respectively. The third equality follows again from the classical minimax theorem [37] . The fourth equality is derived by substituting z 1 = -z 2 -z into the objective function. The fifth equality holds since l θ = l * * θ . Since λ > λ 2 + κ(θ), there exists z 2 that satisfies z 2 = -z 1 -z and other constraints. Finally, we obtainsup ξ∈R d l θ (ξ) -λ ∥ξ -ζ∥ -λ 2 ξ -ξ = l θ ( ζi ) -λ 2 ζi -ξ * .The proof is complete.We evaluate the sensitivity of the model performance to the tuning of designed parameters. We first fix λ 2 = 5 and explore the impact of varying λ and β. As shown in Table 4 , the model's performance demonstrates minimal sensitivity to changes in these parameters. Then, we fix β = 6, and assess the sensitivity to changes in λ 2 . The simulation results are presented in Table 5 . When λ 2 = 0, meaning we abandon the use of the prior knowledge term λ 2 ξζi , our method performs worse than the traditional DRO method. This is because the unbalanced optimal transport distance itself includes distributions that contain worse outliers, thereby making our model overly conservative. As λ 2 increases, there is a noticeable improvement in learning performance, which we attribute to the enhanced role of prior knowledge in outlier detection.We also explore how the contamination factor, denoted as C, affects model performance. The simulation result is presented in Table 6 . While standard DRO is highly sensitive to the contamination We use the same preprocessing step as in [20] to obtain a robust estimate of the expected value of the clean distribution, which involves a parameter ε. This parameter can be viewed as an estimate of the proportion of data corruption. We evaluate the sensitivity of the model's performance to variations in ε and the result is presented in Table 7 . We observe that our method achieves better robustness and is less sensitive to changes in ε compared to OR-WDRO in [20] . This is because OR-WDRO relies on ε during both the preprocessing and the solver optimization steps while our method only requires ε during the preprocessing step. We conducted additional experiments on linear classification, supplementing those presented in the main body of the paper. For these experments, we select d = 10. First, we evaluate the model performance across various sample sizes. As illustrated in Fig. 3 , our method outperforms others in terms of both excess risk and accuracy. Next, we assess the model performance for different dimension d, as shown in Fig. 4 . We observe that our method achieves superior robustness and maintains the accuracy around 94%.In addition, we compare the computational efficiency of these methods in linear classification. As detailed in Table 8 , OR-WDRO exhibits low computational time when the sample size is small, but it faces scalability issues as the dimension increases. In contrast, our method demonstrates stable computational times and superior performance across various sample sizes. As in the linear regression task, we evaluate the model's sensitivity to the preprocessing step, specifically to the parameter ε. The parameter ε is again the estimate of the proportion of data corruption. As illustrated in Table 9 , our method is less sensitive to the accuracy of the estimate ε.The reason mirrors that in the linear regression task. Finally, we evaluate the sensitivity of the model performance to the tuning parameters λ and β. We fix C = 10. Table 10 shows that our method achieves the accuracy above 90% across the range of β values from 0.5 to 20 and λ values from 5 to 20. This performance is better than that of standard DRO with accuracy 66%, and OR-WDRO with accuracy 79%. These simulation results indicate that our method is not sensitive to these parameters in linear classification tasks. We evaluate the sensitivity of the model performance to the tuning parameters in logistic regression.As shown in Tables 11 and 12 , our method achieves the accuracy above 85% across the considered ranges of λ from 5 to 15 and β from 0.4 to 10. Based on all of these simulation results, we find that our method's performance remains stable across a wide range of parameters. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: See Section 5. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: . We have v(λ) is differentiable and satisfiesEQUATIONSince λ * > 0, due to first-order optimality condition, we must have ∇ λ (v(λ))| λ=λ * = 0, i.e.,EQUATIONWe EQUATIONwhere the inequality holds since γ * is a feasible solution in Γ(P * , P). The following equalities are derived by substituting the expressions of γ * and P * . The last equality follows due to the first-order optimality condition (19) . Therefore, the distribution P * is feasible for the primal problem. Then, the primal optimal value is lower bounded by the valueEQUATION)Combining (21) with the weak duality, we have (P) = (D). Thus, the strong duality holds when λ * > 0. The proof is complete.38th Conference on Neural Information Processing Systems (NeurIPS 2024).The CVaR of a random vector Z that has the distribution µ with risk level ε ∈ (0, 1) is defined asCVaR1-ε,µ[Z] = infα α + 1 1-ε EZ∼µ[Z -α]+. 


Decision: The authors present a novel approach by leveraging unbalanced optimal transport (UOT) to develop a new Wasserstein Distributionally Robust Optimization (WDRO) framework, accompanied by strong duality results for a Lagrangian penalty variation of the proposed problem. The framework is tested across multiple scenarios, including linear, logistic regression, and linear classification tasks, demonstrating enhanced robustness to outliers. Although there are areas for further improvement, such as deeper theoretical analysis and sensitivity to contamination levels, the paper offers valuable contributions and has sufficient merit for acceptance.

Reviews: review: paper_summary: paper_summary: The authors leverage unbalanced optimal transport (UOT) to build a new DRO model. Strong dual reformulations together with efficient algorithm design have been proposed. Numerical studies on convex loss function demonstrate the superior performance of this framework.
strength_weakness: strength_weakness: 1. I think the authors may miss an important citation and comparison: 
    [JGX2023] Wang J, Gao R, Xie Y (2023) Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926.

Specifically, the authors considered UOT-based DRO, where UOT is a variant of entropic regularized OT distance. In contrast, [JGX23] also considered another variant of entropic regularized OT distance (which they call it as Sinkhorn distance) to construct the ambiguity set:

$$S(\mathbb{P}, \widehat{\mathbb{P}}) = \inf_{\gamma \in \Gamma(\mathbb{P}, \widehat{\mathbb{P}})}~\Big\{
\mathbb{E}_{(\xi, \zeta)\sim \gamma}[c(\xi,\zeta)] + \beta \mathbb{E}_{(\xi, \zeta)\sim \gamma}\left[ 
\log\frac{\mathrm{d}\gamma(\xi,\zeta)}{\mathrm{d}\nu(\xi)\mathrm{d}\widehat{\mathbb{P}}(\zeta)}
\right]
\Big\}$$

where  $\nu(\cdot)$ can be any reference measure (such as Lebesgue measure) on $\Xi$. The difference is that the authors in this paper considered relative entropy regularization between  $\bar{\mathbb{P}}$ and $\widehat{\mathbb{P}}$, and [JGX23] considered relative entropy regularization between conditional conditional distribution $\gamma_{\zeta}(\cdot)$  and measure  $\nu(\cdot)$. Also, please compare the dual reformulation in this paper and that in [JGX23]. The expressions share some connections.


In addition, the usage of entropic regularization for Wasserstein DRO has been largely explored in other literature. I hope the authors also add the comparisons among them.

    [JY2022] J. Wang and Y. Xie, “A data-driven approach to robust hypothesis testing using sinkhorn uncertainty sets,” in 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022, pp. 3315–3320. 

     [WFJ2023] 	W.Azizian, F.Iutzeler, and J.Malick “Regularization for wasserstein distributionally robust optimization,” ESAIM:Control, Optimisation and Calculus of Variations, vol. 29, p. 33, 2023. 

     [JRY2022] J. Wang, R. Moore, Y. Xie, and R. Kamaleswaran, “Improving sepsis prediction model generalization with optimal transport,” in Machine Learning for Health. PMLR, 2022, pp. 474–488 

     [SZ2023] 	S.-B. Yang and Z. Li, “Distributionally robust chance-constrained optimization with sinkhorn ambiguity set,” AIChE Journal, vol. 69, no. 10, p. e18177, 2023. 

     [CFAB2023] C. Dapogny, F. Iutzeler, A. Meda, and B. Thibert, “Entropy-regularized wasserstein distributionally robust shape and topology optimization,” Structural and Multidisciplinary Optimization, vol. 66, no. 3, p. 42, 2023. 

     [JGX2024] Wang J, Gao R, Xie Y (2024) Non-Convex Robust Hypothesis Testing using Sinkhorn Uncertainty sets. in 2024 IEEE International Symposium on Information Theory (ISIT). IEEE, 2024
questions: questions: 2. In Theorem 1, the assumption seems too restrictive, including boundedness of $f_{\lambda}(\theta,\zeta)$, differentiability and concavity of $L(\theta,\xi)$, differentiability and concavity of $c$, and strict positivity of dual variable $\lambda^*$. In existing DRO literature such as [JGX2023], only the finiteness of cost function, lighted tailed condition, measurability of loss, and law of total probability is required. In standard WDRO, the technical assumption for strong duality holds is mild as well. 

3. I also believe the assumption that dual variable is strictly positive is not necessary: when you take sufficiently large radius, the constraint is unbinding and the dual variable $\lambda^*=0$. Please follow existing DRO literature to explore how to build strong duality in this case. 

4. In line 220, the authors implicitly used the assumption that $c(\zeta,\zeta)=0$. I don’t remember whether it has been assumed in the main context before. 

5. I have concerns regarding Theorem 3. The algorithm design is actually inspired from [Sinha et. Al 2017]. The authors in that reference studied optimization for non convex loss function, such as neural networks, and perform biased SGD update to achieve near-stationary points. However, the authors in this paper only considered convex optimization. I strongly believe their results can be extended for non-convex smooth optimization. Please consider the exploration in this direction. Some useful references include 

[HCH2021] Hu Y, Chen X, He N (2021) On the bias-variance-cost tradeoff of stochastic optimization. Advances in Neural Information Processing Systems. 

[ HZCH2020] Hu Y, Zhang S, Chen X, He N (2020) Biased stochastic first-order methods for conditional stochas- tic optimization and applications in meta learning. Advances in Neural Information Processing Systems, volume 33, 2759–2770. 


6. For the numerical study part, the exploration of outlier DRO with neural network loss functions is also quite promising.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces a novel outlier-robust Wasserstein Distributionally Robust Optimization (WDRO) framework based on unbalanced optimal transport (UOT). For the UOT-DRO, the authors establish strong duality results under specific smoothness assumptions. To enhance computational efficiency, they propose a Lagrangian penalty variant of the problem and prove the strong duality of this variant as well. The authors then employ stochastic gradient descent (SGD) to solve the Lagrangian penalty variant, providing a theoretical convergence guarantee for their method.
strength_weakness: strength_weakness: 1. The current draft does not include a theoretical analysis of the excess risk associated with the proposed UOT-DRO. Including such an analysis would strengthen the paper by providing a deeper understanding of the performance and limitations of the method.
2. The discussion on parameter selection is currently limited. The UOT-DRO relies on several hyperparameters, including $\lambda$, $\lambda_2$, and $\beta$, which may not be entirely independent. Although Section 7.6 addresses hyperparameter selection for linear regression, the approach for selecting these parameters in other tasks remains unclear. A more comprehensive discussion on tuning these parameters for general cases would be beneficial.
questions: questions: 1. In line 238-239, the authors mentioned that outlier-robust WDRO is highly sensitive to $\varepsilon$, the contamination level in the outlier-robust optimal transport. However, proposition 12 in [A] provides a robustness guarantee when the chosen $\hat \varepsilon$ does not match the true $\varepsilon$. Also, the numerical experiments in [17] show that OR-WDRO can perform well when the the chosen $\hat \varepsilon$ does not match the true $\varepsilon$. Could you further explain why outlier-robust WDRO is sensitive to $\varepsilon$ in line 238-239?
2. It is encouraged to show an empirical evaluation on the convergence of the proposed Algorithm 1. This would help demonstrate the practical applicability and efficiency of the algorithm.
3. For the numerical experiments, could the authors further explain how the values of the hyperparameters were determined for linear classification and logistic regression? Additionally, the choice of hyperparameters for the baseline methods is also missing and should be included for a comprehensive comparison.

[A] Nietert, Sloan, Ziv Goldfeld, and Rachel Cummings. "Outlier-robust optimal transport: Duality, structure, and statistical analysis." *International Conference on Artificial Intelligence and Statistics*. PMLR, 2022.
limitations: limitations: The authors discuss the limitations of their work in Section 3. Notably, this work does not have a negative societal impact.
paper_summary: paper_summary: The paper proposes a new Distributionally Robust Optimization (DRO) framework based on Unbalanced Optimal Transport (UOT) distance. Under some conditions, the authors establish strong duality results for a Lagrangian penalty variation of the proposed problem. The designed algorithm is tested in linear, logistic regression and linear classification tasks where it shows an increased robustness to outliers in comparison to the considered baselines.
strength_weakness: strength_weakness: **Quality.** Some of the assumptions related to the ‘outlier-robustness’ property of the proposed approach are not sufficiently explained.

1) The authors claim that the ambiguity set of the proposed approach (UOT DRO) necessarily includes ‘the clean distribution $P^*$' (line 146-147) due to the soft marginal constraints in the UOT problem. However, actually, this ‘softness’ of the marginals depends on the chosen divergence $\mathcal{D}_{\phi_2}$ and the parameter $\beta$ in the unbalanced divergence definition (4). Typically, increasing the parameter $\beta$, one gets closer to the balanced OT formulation which does not have ‘outlier robustness’ property, and does not allow one to ignore the outliers in empirical dataset. This dependence on parameter $\beta$ is not explained and verified from theoretical or practical perspectives. Clarification of this point seems to be very important since this parameter is set differently in experiments considered in the paper.

2) Besides, the authors introduce the function $h$ penalizing potential distributions with outliers in an ambiguity set of the primal problem (6) of UOT-DRO. Thanks to this function, the solution (model) of the Lagrangian penalty reformulation of UOT-DRO will be less affected by outlier data as explained in lines 215-225. However, the authors provide explanations for the case of large $\lambda$, while in the experimental section they consider $\lambda=10$. The influence of outliers on the model in this case is still not obvious.

**Clarity.** Some parts of the text should be written more clearly, e.g., the formula in line 289. The choice of cost function and parameters in experiments (Section 5.1) should be explained.

**Evaluation.** The comparison with the baselines seems to be rather limited. The authors have cited several other works which propose approaches for the outlier-robust DRO problem [1,2,3,4]. However, the empirical evaluation of the proposed approach misses the comparison with some of these approaches [1,3,4].


**In summary**, I am not sufficiently convinced that the constructed approach actually solves the problem that it was designed for, i.e., allows for solving the DRO problem while effectively ignoring the outliers in the empirical dataset. The paper does not provide any theoretical results which support this claim. Thus, it should be accurately justified at least from the empirical side. However, the experimental part misses ablation study of parameters $\beta$, $\lambda$, and thorough comparison with other outlier-robust DRO approaches.
questions: questions: 1) Please explain the choice of cost function $c(\xi,\zeta)=|\xi-\zeta|$ in your experiments
2) How do you choose parameters $\beta$, $\lambda$ in your experiments? Is it an heuristic choice? An ablation study of these parameters is highly anticipated
3) Why is the performance of UOT-DRO for contamination $C=30$ higher than for $C=100$? How further increase of $C$ will influence the performance?
4) Why you did not perform comparison with additional outlier-robust DRO approaches [1,3,4] cited in the paper?

*Minor limitations*:
- bibliography is not in alphabetical order,
- line 286: $\theta_*$ is not defined,
- lines 311-312: '*only* requires *only* 20 seconds'.

**References.**

[1] Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj, and Arun Sai Suggala. Stochastic re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.09222, 2023.

[2] Sloan Nietert, Ziv Goldfeld, and Soroosh Shafiee. Outlier-robust Wasserstein dro. Advances in Neural Information Processing Systems, 36, 2024.

[3] Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning, pages 12345–12355. PMLR, 2021.

[4] R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2:21–42, 2000.
limitations: limitations: The authors adequately addressed the limitations.

Comments: comment: I thank the authors for additional clarifications and conducted experiments, and increase my score.
We thank the reviewer for the kind response!
We thank the reviewer for the kind response and for increasing the score.
I thank the authors for successfully addressing all my concerns. I will raise my score to 7
I thank the authors for answering my questions. I will keep my score.
Dear Reviewers,

We encourage you to participate in the discussion actively. Please post a comment on the authors' response, either to highlight any remaining concerns, acknowledge that some of your concerns have been addressed, or simply to confirm that you have reviewed their response.

Best wishes, AC