Title: Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport
Venue: NeurIPS 2024
Accepted: yes
Keywords: Distributionally robust optimization; unbalanced optimal transport; outlier
Abstract: Distributionally Robust Optimization (DRO) accounts for uncertainty in data distributions by optimizing the model performance against the worst possible distribution within an ambiguity set. In this paper, we propose a DRO framework that relies on a new distance inspired by Unbalanced Optimal Transport (UOT). The proposed UOT distance employs a soft penalization term instead of hard constraints, enabling the construction of an ambiguity set that is more resilient to outliers.  Under smoothness conditions, we establish strong duality of the proposed DRO problem. Moreover, we introduce a computationally efficient Lagrangian penalty formulation for which we show that strong duality also holds. Finally, we provide empirical results that demonstrate that our method offers improved robustness to outliers and is computationally less demanding for regression and classification tasks.

Outlier-Robust Distributionally Robust Optimization via Unbalanced Optimal Transport

Abstract

Distributionally Robust Optimization (DRO) accounts for uncertainty in data distributions by optimizing the model performance against the worst possible distribution within an ambiguity set. In this paper, we propose a DRO framework that relies on a new distance inspired by Unbalanced Optimal Transport (UOT). The proposed UOT distance employs a soft penalization term instead of hard constraints, enabling the construction of an ambiguity set that is more resilient to outliers. Under smoothness conditions, we establish strong duality of the proposed DRO problem. Moreover, we introduce a computationally efficient Lagrangian penalty formulation for which we show that strong duality also holds. Finally, we provide empirical results that demonstrate that our method offers improved robustness to outliers and is computationally less demanding for regression and classification tasks.

1 Introduction

Consider a stochastic optimization problem that aims to find a decision variable θ ∈ Θ that minimizes the expected loss function E ξ ∼ P ∗ [ l ( ξ, θ )] [1], where the random samples ξ ∈ Ξ are drawn from a fixed distribution P ∗ and Θ ⊂ R d is a compact and convex set. In many applications, the distribution of interest P ∗ is not precisely known; yet datasets that contain finite samples independently drawn from the distribution P ∗ are often available, which yield an empirical distribution estimate ˆ P . For instance, in image classification problems, P ∗ can represent the joint distribution of the features and the labels, and ˆ P can represent the empirical distribution of a dataset that is annotated by humans [2]; in portfolio optimization problems, P ∗ captures the uncertainty of a set of financial products in the future months and ˆ P contains historical prices of all the products [3]; in healthcare applications, P ∗ can represent the feature distribution of a population of interest, and ˆ P represent the available electronic health record data [4].

The above stochastic optimization problem is often solved using empirical risk minimization (ERM) [5], which searches for the best decision variable θ that minimizes the expected loss function with respect to the empirical distribution, i.e., θ ERM = arg min θ ∈ Θ E ξ ∼ ˆ P [ l ( ξ, θ )] . However, ERM is not guaranteed to provide good out-of-sample performance, i.e., the value of E ξ ∼ P ∗ [ l ( ξ, θ ERM )] is high when the empirical estimate ˆ P is not close to the true distribution P ∗ . For example, in the presence of distribution shifts or contaminated datasets, the out-of-sample performance cannot be improved by increasing the sample size. To enhance the out-of-sample performance of the decision variable, researchers have developed methods that use distributionally robust optimization (DRO) to address

Yi Shen

Karl H. Johansson

the mismatch between ˆ P and P ∗ [6, 7, 8, 9, 10, 11, 12]. In contrast to ERM, DRO aims to optimize outcomes against the worst-case scenario over a set of distributions that are close to the empirical distribution according to a pre-specified distance metric. This set of distributions is called the ambiguity set and is typically assumed to contain the distribution of interest, e.g., the true distribution P ∗ . The construction of the ambiguity set is critical in the development of various DRO methods, as it ideally encompasses all prior knowledge about the true distribution. For example, [6] defines the ambiguity set using moment constraints. Moment constraints assume that the true distribution has similar moments compared to the empirical distribution, e.g., means and variances. The ambiguity set can also include distributions that are similar to ˆ P in infinite-order moments; this construction is explored in [13] and is termed Kernel DRO. When the distribution of interest is categorical, Kullback-Leibler divergence (KL) is often adopted to define the ambiguity set as the categories can be permuted and KL does not consider the geometry of a distribution's support. DRO using KL has been extensively studied in the literature due to its computational benefits [14, 15]. In contrast, the Wasserstein distance between two distributions captures the geometry of the distributions. The Wasserstein distance can be useful, e.g., in loan approval prediction problems where the distribution represents the incomes of a population and prior knowledge may specify that two clients are similar if their incomes are close. DRO using the Wasserstein distance (WDRO) has been applied to many machine learning problems, such as linear regression [16] and logistic regression [17] problems.

In this paper, we apply DRO to handle distribution mismatches using the Wasserstein distance as it can better capture the geometry of the distributions. However, WDRO is not suitable for contaminated datasets, e.g., datasets that contain outliers that are geometrically far from the clean distribution but cannot be easily removed. The reason is that the Wasserstein distance between the contaminated empirical distribution and the clean (true) distribution is very sensitive to the outliers; to include the clean distribution in the ambiguity set, the Wasserstein distance would need to be selected very large and inevitably the ambiguity set would include distributions that will never happen in practice.

To apply DRO to contaminated datasets, the authors in [18, 19, 20] take outliers into consideration and propose various methods to minimize their impact on the learning performance. Specifically, [18] proposes to minimize the most optimistic DRO risk to avoid the hardest instances that are likely to be outliers. In [19], a weight clipping method is proposed to truncate the excessive impact of outliers on the results. However, both [18] and [19] use KL divergence to define the ambiguity set, which fails to capture geometric uncertainty. Notably, the authors in [20] introduce an outlier-robust WDRO framework capable of addressing both geometric uncertainty and non-geometric contamination, allowing an ε fraction of data to be arbitrarily corrupted. They design an ambiguity set based on prior knowledge and demonstrate the strong duality of the resulting DRO problem, which is related to the Conditional Value at Risk (CVaR) [21]. Nevertheless, the performance of the formulated DRO problem relies heavily on the accuracy of the estimate of ε . Besides, [20] uses off-the-shelf solvers to solve the resulting DRO problem, which is not computationally efficient when applied to large datasets since the number of constraints in the proposed method grows linearly with the sample size.

The main contribution of this paper is to introduce a new framework for outlier-robust WDRO based on Unbalanced Optimal Transport (UOT), which is known for its inherent robustness to outliers and missing data [22, 23, 24, 25, 26]. Our approach involves the design of a UOT distance that substitutes hard constraints by a soft penalization term. This construction, combined with some prior knowledge on the learning task, enables the design of a new ambiguity set that includes distributions of interest and penalizes distributions that contain outliers. For the DRO problem with this new ambiguity set, we establish strong duality results under specific smoothness assumptions. However, the solution of the dual problem poses significant computational challenges. Motivated by [27], we explore a Lagrangian penalty variation of the problem, which can be viewed as a Lagrangian function of the original formulation with a fixed dual variable. We show that strong duality holds for this Lagrangian penalty problem under fewer assumptions compared to the original DRO problem. The reformulated problem employs an exponential function to reweight data points, effectively diminishing the influence of outlier data on the optimization process. We solve this problem by proposing a provable stochastic (sub)-gradient algorithm, which is computationally efficient. We provide empirical results that demonstrate that our approach not only enhances robustness to outliers but also improves computational efficiency for large-scale problems.

Our work is related to data reweighted optimization problems [28, 29, 30], which adjust the weights of individual data to manage outliers. However, these studies construct the reweighted optimization problems heuristically and cannot handle distribution uncertainty. In contrast, our problem

formulation is able to handle distribution uncertainty via DRO, and our findings demonstrate that the reweighted problem constitutes the dual of a specific class of DRO problems, with the ambiguity set defined by the UOT distance.

The rest of the paper is organized as follows. Section 2 defines the UOT distance and the associated DRO problem. Section 3 presents the strong duality results for both the original DRO problem and its Lagrangian penalty variant. The main algorithm along with its convergence analysis is presented in Section 4. Section 5 demonstrates the effectiveness of our proposed framework in addressing outliers in both regression and classification tasks. Finally, we conclude the paper in Section 6. In the Appendix, we provide all proofs as well as some additional experiments.

2 Problem definition and preliminaries

The (1-)Wasserstein distance [31] between two distributions P and ˆ P is defined as

where Γ( P , ˆ P ) denotes the set of joint distributions such that γ 1 = P and γ 2 = ˆ P , where γ 1 and γ 2 denote the first and the second marginal distribution of γ , respectively. The function c ( ξ, ζ ) : Ξ × Ξ → [0 , + ∞ ) is lower semi-continuous and represents the cost of moving a point from ξ to ζ , where the support set Ξ ⊂ R d is assumed to be compact. The joint distribution γ specifies a transport plan for moving the distribution from P to ˆ P . The Wasserstein distance captures the underlying geometry in distributions through the cost function c ( · , · ) , making it a popular choice in DRO problems; the WDRO problem is defined by

where B ρ ( ˆ P ) := { P ∈ M (Ξ) : W( P , ˆ P ) ≤ ρ } denotes the ambiguity set that contains all distributions within ρ -distance from ˆ P and ρ ≥ 0 is a user-specified parameter. Here, we denote by M (Ξ) the space of all probability distributions supported on Ξ .

The Wasserstein distance enforces equality constraints on the marginal distributions of γ to match P and ˆ P , thereby restricting the choice of γ . If the empirical distribution ˆ P contains outliers, it becomes necessary to select a large radius ρ for the ambiguity set to ensure the inclusion of the true distribution P ∗ as the outliers tend to increase the Wasserstein distance between ˆ P and P ∗ . However, a large ρ can lead to a DRO solution that is conservative since the larger the radius the larger the value of the inner supremum problem in (2). Though a large radius allows the inclusion of the distribution of interest, i.e., P ∗ , it also results in the inclusion of many distributions that are not of interests, e.g., distributions that are unlikely to occur in practice. As a result, the Wasserstein distance is not a desired distance choice when the empirical distribution contains outliers.

In Unbalanced Optimal Transport (UOT) problems [22], the joint distribution γ is not required to have fixed marginals in contrast to (1). Specifically, for any two arbitrary positive measures P and ˆ P , the UOT distance is defined by

where γ 1 and γ 2 are marginals of γ , and D φ 1 and D φ 2 are Csiszár divergences that measure discrepancies between positive measures, based on functions φ 1 and φ 2 , respectively. The key distinction between UOT distance and Wasserstein (OT) distance lies in the constraints of the marginals. While the OT distance in (1) strictly enforces marginal distributions, i.e., γ 1 = P and γ 2 = ˆ P , the UOT in (3) relaxes the equality constraints by adding mismatch penalty functions D φ in addition to the transport cost induced by c ( · , · ) . This relaxation makes UOT robust to outliers since γ 2 can represent a distribution that is different from the contaminated distribution ˆ P .

Inspired by the definition of UOT, we consider the following unbalanced Wasserstein distance (divergence)

where β is a tuning parameter and ¯ P is an intermediate optimization variable that links P and ˆ P . Note that (4) is a special case of (3) by selecting φ 1 = ι { 1 } (i.e., φ 1 (1) = 0 and ∞ otherwise) and φ 2 ( x ) = x log x -x +1 (i.e., D φ 2 denotes the KL divergence). In addition, we restrict the marginals of the joint positive measure γ in (3) to be probability measures because the general positive measures are not of interest in the DRO problems we consider. The particular choice of φ functions is for technical reasons related to the duality analysis, as we will show in Section 3.

The optimal unbalanced Wasserstein transport under UW( P || ˆ P ) can be interpreted as a two-step procedure: 1) ˆ P can be transported to any ¯ P by paying a small price measured by its distance to ˆ P in the KL sense (a non-geometric transport), and 2) ¯ P is then transported to P by minimizing the geometryaware transport cost. The non-geometric KL divergence is the key to allow the uncontaminated clean distribution P ∗ to be close to the contaminated empirical distribution ˆ P as the KL divergence does not distinguish the geometric locations of the distribution supports.

We define the ambiguity set using the unbalanced Wasserstein distance by U ρ ( ˆ P ) = { P ∈ M (Ξ) : UW( P || ˆ P ) ≤ ρ } . Then, the DRO problem using the UOT distance (4) can be formulated as

The ambiguity set defined by the unbalanced Wasserstein distance enables us to include the clean distribution P ∗ , but it cannot remove distributions that are close to the contaminated empirical distribution ˆ P as these distributions are close to ˆ P both geometrically (Wasserstein) and non-geometrically (KL). As a result, these distributions, denoted by set ˜ P , could also make the learned model conservative. To remove such distributions, we need some prior/domain knowledge. As discussed in [18, 20], it is impossible to design a model selection strategy using contaminated datasets without any prior knowledge. In Section 5, we also show that constructing the ambiguity set without prior knowledge may result in worse performance compared to the traditional DRO; see Table 5.

We assume that any distribution ˜ P ∈ ˜ P incurs significantly large costs when evaluated using a given function h compared to uncontaminated distributions, i.e., E ξ ∼ ˜ P [ h ( ξ )] ≫ E ξ ∼ P [ h ( ξ )] , ∀ ˜ P ∈ ˜ P and ∀ P ∈ { P : UW( P || ˆ P ) ≤ ρ }\ ˜ P . The function h can be related to the moment constraint. For example, if we know the expectation estimate ξ 0 of the clean distribution, we can design h ( ξ ) = ∥ ξ -ξ 0 ∥ and the function value of h ( ξ ) will be high if ξ is an outlier. Similar choices are considered in [20].

Taking into account the prior knowledge, we consider minimizing the following (primal) function

with L ( θ, ξ ) = l ( θ, ξ ) -h ( ξ ) and we assume that L ( θ, ξ ) is continuous in ξ . For any distribution ˜ P that contains outliers, E ξ ∼ ˜ P [ h ( ξ )] is large and thus ˜ P is less likely to attain the supremum in (6).

In summary, we first employ the unbalanced Wasserstein distance to incorporate the distributions of interest. Then, we leverage prior knowledge to systematically exclude distributions that may contain outliers. This two-step approach ensures a more thorough consideration of potential distributions while effectively mitigating the impact of outliers on the learning problem.

3 Unbalanced distributionally robust optimization

Note that the primal problem in (6) is an infinite-dimensional optimization as the optimization variable represents probability distributions. In this section, we derive its dual problem and a variant of the dual problem that is computationally tractable.

3.1 The DRO problem

Assumption 1. Every joint distribution γ on Ξ × Ξ with the second marginal distribution ¯ P has a regular conditional distribution γ ζ given the value of the second marginal equals ζ .

This assumption ensures that the joint distribution γ can be written as dγ ( ξ, ζ ) = dγ ζ ( ξ ) d ¯ P ( ζ ) ; see [32] for more details on regular conditional distributions. By the law of total expectation, we have E ξ ∼ P [ L ( θ, ξ )] = E ζ ∼ ¯ P E ξ ∼ γ ζ ( ξ ) [ L ( θ, ξ )] .

We define f λ ( θ, ζ ) := sup ξ ∈ Ξ { L ( θ, ξ ) -λc ( ξ, ζ ) } . We first derive the dual problem and show the weak duality in the following lemma.

Lemma 1. Let Assumption 1 hold and suppose that | f λ ( θ, ζ ) | < ∞ for ˆ P -almost every ζ . Define the dual problem (D) := inf λ ≥ 0 { λρ + λβ log E ζ ∼ ˆ P [ exp ( f λ ( θ,ζ ) λβ )]} . Then, we have

The proof is provided in Appendix 7.1. To show the strong duality, we need two additional assumptions to guarantee that the function f λ ( θ, ζ ) is well-behaved.

.

Assumption 2. The function L ( θ, ξ ) be differentiable and concave in ξ with respect to the norm ∥·∥ Assumption 3. The function c : Ξ × Ξ → [0 , ∞ ) is differentiable and c ( · , ξ 0 ) is 1-strongly convex for each ξ 0 ∈ Ξ .

Theorem 1. Let Assumptions 1-3 hold and assume that | f λ ( θ, ζ ) | < ∞ for ˆ P -almost every ζ . Suppose that the optimal dual variable λ ∗ is strictly positive. Then, the strong duality holds, i.e.,

The proof can be found in Appendix 7.2. For technical reasons, we show the strong duality when the optimal dual variable λ ∗ is strictly positive. In most applications, the optimal dual variable λ ∗ is in fact positive. When λ ∗ = 0 , the inequality constraint becomes inactive and the radius of the ambiguity set becomes very large, a situation that is out of scope for most practical applications.

Although strong duality holds and the dual problem is a finite dimensional minimization problem, solving problem (7) is computationally challenging due to the complex dependency of the objective on λ and, in particular, the implicit dependency of f λ on λ . Recall that f λ ( θ, ζ ) = sup ξ ∈ Ξ { L ( θ, ξ ) -λc ( ξ, ζ ) } ; in general, we cannot obtain an analytical form of f λ ( θ, ζ ) , i.e., a closed-form solution of the problem sup ξ ∈ Ξ { L ( θ, ξ ) -λc ( ξ, ζ ) } . Additionally, the dual problem (7) is not convex in λ in general. The task becomes more involved when attempting to minimize over λ and θ jointly. To address these challenges, we shift focus from the constrained ρ -robustness problem (7) and instead consider the corresponding Lagrangian penalty problem, which allows for a computational tractable formulation.

3.2 The Lagrangian penalty problem

Motivated by [27], we consider the Lagrangian penalty reformulation of (7) as follows

In what follows, we show that the strong duality also holds for the Lagrangian penalty problem (8).

Theorem 2. Let Assumption 1 hold and assume that | f λ ( θ, ζ ) | < ∞ for ˆ P -almost every ζ . Then, we have that

The proof is provided in Appendix 7.3. Note that the strong duality of the Lagrangian penalty problem only requires Assumption 1 to hold in contrast to Theorem 1. In particular, it does not require the loss function L ( θ, ξ ) to be differentiable, making it applicable to a broader class of problems. Besides, solving problem (8) is computationally tractable as the dual problem has a closed-form solution as in (9). In particular, λ is a parameter in (9) while it is an optimization (dual) variable in (7).

Since the logarithm function is monotonically increasing, minimization of (9) in θ is equivalent to the following form

Comparison with standard WDRO. It has been shown in [27] that the Lagrangian penalty problem using the Wasserstein metric W c has the following strong duality result

where W c is the Wasserstein distance metric associated with the cost function c ( · , · ) . When λ in (11) is selected to be large, this formulation is close to the ERM problem, i.e., minimization of E ζ ∼ ˆ P [ l ( θ, ζ )] . Clearly, a distribution ˆ P containing outliers would significantly deteriorate the learning performance of minimizing E ζ ∼ ˆ P [ l ( θ, ζ )] .

In contrast, when λ is large, the problem (10) is close to the minimization of E ζ ∼ ˆ P [exp (( l ( θ, ζ ) -h ( ζ )) / ( λβ ))] = E ζ ∼ ˆ P [exp ( l ( θ, ζ ) / ( λβ )) w ( ζ )] , where w ( ζ ) = exp( -h ( ζ ) / ( λβ )) acts as a re-weighting function. Recall that h ( ξ ) is a function designed based on prior knowledge and assumes high values at outlier points. At an outlier point, say ˜ ξ , h ( ˜ ξ ) would be significantly larger than at non-outlier points, resulting in a very small weight w ( ˜ ξ ) . Consequently, outlier data with small weights exert less influence on the resulting model. Therefore, the formulation (10) is more robust to outliers compared to standard WDRO.

Comparison with outlier-robust WDRO [20]. [20] considers the Huber contamination model in which an ε fraction of data can be arbitrarily corrupted. In [20], prior knowledge on the mean of the clean distribution is needed. Specifically, the function h ( ξ ) that can identify outliers in [20] is selected as h ( ξ ) = λ 1 ∥ ξ -ξ 0 ∥ 2 , where ξ 0 represents the estimated mean of the clean distribution and λ 1 > 0 is a tuning parameter. It can be verified that the corresponding Lagrangian formulation is

where CVaR 1 -ε denotes the conditional value at risk ( CVaR ), interpreted as the expected value of the rightmost (1 -ε ) percentile of outcomes 1 .

Similarly, when λ is very large, the formulation in (12) closely approximates the minimization of CVaR 1 -ε, ˆ P [ l ( θ, ζ ) -λ 1 ∥ ζ -ξ 0 ∥ ] . At an outlier point, say ˜ ξ , the value l ( θ, ˜ ξ ) -λ 1 || ˜ ξ -ξ 0 || is small due to the large distance || ˜ ξ -ξ 0 || . Consequently, such an outlier point generates a function value that falls on the left tail of the loss distribution and is thus effectively excluded by the CVaR operation. This method can also be viewed as a re-weighting method by assigning weight 0 to outlier points and weighting other points equally. However, this approach is highly sensitive to the choice of ε , which is usually difficult to obtain a priori. In comparison, our method is a smoothed version of this CVaR method and is less sensitive to the prior knowledge of ε . Besides, calculating the value of CVaR is computationally more demanding when the number of samples is large as we will demonstrate in Sections 4 and 5.

4 Algorithm design

In this section, we focus on solving the dual of the Lagrangian penalty problem. We assume that the empirical distribution ˆ P is constructed from n samples and written as ˆ P = 1 n ∑ n i =1 δ ˆ ζ i with ˆ ζ i the i -th sample. According to Theorem 2, the Lagrangian penalty problem is equivalent to the minimization of the following objective function

Algorithm 1: Distributionally robust optimization with outliers

Require: Sampling distribution ˆ P , constraint sets Θ and Z , step size sequence { α t > 0 } T -

1: for t = 1 , . . . , T do

2: Sample ˆ ζ t uniformly from ˆ P

3: Find an ϵ -approximate maximizer z t of arg max ξ ∈ Ξ L ( θ t , ξ ) -λc ( ξ, ˆ ζ t )

4: Construct gradient estimate g t = exp ( ( L ( θ t , z t ) -λc ( z t , ˆ ζ t )) / ( λβ ) ) 1 λβ ∂ θ L ( θ t , z t )

5: θ t +1 = Proj Θ ( θ t -ηg t )

6: end for

To solve the problem (13), we employ the stochastic sub-gradient method, detailed in Algorithm 1. Specifically, at each time step t , we sample ˆ ζ t from the empirical distribution ˆ P uniformly. We then find the ϵ -approximate maximizer z t of the function L ( θ t , ξ ) -λc ( ξ, ˆ ζ t ) . We require that the solution satisfies dist( z t , X ∗ t ) ≤ ϵ , where X ∗ t denotes the set of maximizers. This is solvable in many applications and can be achieved through, e.g., the (sub)-gradient method [27]. Then, we perform the projected sub-gradient update. The convergence analysis of Algorithm 1 is presented in the following theorem. The proof can be found in Appendix 7.4.

Theorem 3. Suppose that Θ is a convex and compact set with bounded diameter R . Assume that the loss L ( θ, ξ ) is convex in θ for each ξ ∈ Ξ . For all data points ζ i from the empirical distribution ˆ P , assume that L ( θ, · ) -λc ( · , ζ i ) is concave, L ( θ, ξ ) -λc ( ξ, ζ i ) ≤ B and | c ( ξ, ζ i ) -c ( ξ ′ , ζ i ) | ≤ L c ∥ ξ -ξ ′ ∥ , for all θ ∈ Θ , ξ ∈ Ξ , ξ ′ ∈ Ξ . Assume that ∥ ∂ θ L ( θ, ξ ) -∂ θ L ( θ, ξ ′ ) ∥ ≤ L θξ ∥ ξ -ξ ′ ∥ , ∥ ∂ θ L ( θ, ξ ) ∥ ≤ B 2 , ∥ L ( θ, ξ ) -L ( θ, ξ ′ ) ∥ ≤ L ξ ∥ ξ -ξ ′ ∥ . Select η = 1 √ T . Then, Algorithm 1 satisfies

where F ∗ represents the minimum value of the objective function (13) .

The accuracy parameter ϵ in the inner maximization problem has a fixed effect on the final optimization accuracy, independent of T . The assumptions in Theorem 3 are common in the analysis of optimization problems involving the exponential function in the objective function, see, e.g., [19, 28, 29].

Algorithmic comparison with outlier-robust WDRO [20]. Our proposed formulation can be solved by the stochastic sub-gradient method, which is computationally more efficient when the sample size is large. In contrast, the approach in [20] utilizes CVaR to account for outliers and employs off-the-shelf solvers, e.g., GUROBI [33], to obtain a solution. A notable limitation of the presence of CVaR in DRO problems is that the number of constraints scales linearly with the sample size. Consequently, for large datasets, this scaling significantly increases the number of constraints, leading to computational inefficiency, as shown in Section 5. Designing a stochastic gradient method for the CVaR-based formulation in [20] is possible, although challenging since the estimate of the CVaR gradient is usually biased. Moreover, our methodology does not rely on off-the-shelf commercial solvers and can be incorporated in machine learning packages seamlessly, which allows us to handle a wide range of loss functions via the stochastic sub-gradient method.

5 Experiments

We conduct experiments on linear regression, linear classification, and logistic regression problems. We use the stochastic sub-gradient method in Algorithm 1 to solve the Lagrangian penalty problem and the GUROBI [33] solver to solve all other DRO problems we use as benchmarks. All experiments were conducted on an Intel Core i7-1185G7 CPU (3.00GHz) using Python 3.8. Our method is referred to as UOT-DRO. A discussion on parameter selection is provided at the end of this section.

1 t =0

20

30

40

50

60

70

80

90

100

10

1

10 0

Excess risk

Standard DRO

OR-WDRO

UOT-DRO

(a) Excess risk with various samples.

5

10

15

20

25

30

35

40

Dimension

10

1

10 0

Excess risk

Standard DRO

OR-WDRO

UOT-DRO

(b) Excess risk with varied dimensions.

5.1 Linear regression

We consider a linear regression problem with the loss l θ ( x, y ) = | θ ⊺ x -y | , where ( x, y ) ∈ R d × R represents a generic data point and θ ∈ R d is the model we aim to train. We define ¯ θ = [ θ ⊺ , -1] ⊺ , ξ = [ x ⊺ , y ] ⊺ . Then, the loss function can be written as l θ ( ξ ) = | ¯ θ ⊺ ξ | .

,

We generate a clean data distribution P n with n samples, which is uniform over { X i , θ ⊺ ∗ X i } n i =1 where X 1 , . . . , X n are i.i.d. from X ∼ N (0 , I d ) . Drawing a uniform random subset S ⊂ [ n ] of size ⌊ εn ⌋ , the corrupted data distribution ˆ P n is defined to be uniform over {( C 1 { i ∈ S } X i , ( -C 2 ) 1 { i ∈ S } ( θ ⊺ ∗ X i + ρ ) )} n i =1 , where C > 0 is a corruption scaling coefficient and ρ > 0 is a shift coefficient. The empirical probability distribution ˆ P n can be written as ˆ P n = 1 ∑ n δ ˆ with ˆ ζ i the i -th sample.

n i =1 ζ i

We consider prior knowledge on the mean of the clean distribution, denoted as ¯ ξ , and design the function h ( ξ ) = λ 2 ∥ ∥ ξ -¯ ξ ∥ ∥ . The value of ¯ ξ is determined by a cheap preprocessing step, same as in [20]. We consider the Lagrangian penalty problem which, based on Theorem 2, is given by

Define κ ( θ ) = sup {∥ z ∥ ∗ : l ∗ ( z ) < ∞} = ∥ ∥ ¯ θ ∥ ∥ ∗ , where ∥·∥ ∗ denotes the dual norm. When λ ≥ λ 2 + κ ( θ ) , we can show that the problem (15) is equivalent to

The detailed proof is provided in Appendix 7.5.

We set the parameters as follows: λ = 10 , β = 6 , λ 2 = 5 . All the results are averaged over 10 independent runs. We fix ε = 0 . 1 and C = 8 . We compare the performance of our unbalanced DRO model against the standard DRO and outlier-robust WDRO provided in [20]. We evaluate these methods in terms of the excess risk, which we define as the difference between the loss incurred by the learned model and the minimum achievable loss. The simulation results presented in Fig. 1 demonstrate the performance of our method under varying conditions: In Fig. 1 (a), we set d = 10 and compare these methods for various samples sizes n ∈ { 20 , 40 , 60 , 80 , 100 } ; In Fig. 1 (b), we fix n = 100 and explore the impact of various dimensions d ∈ { 5 , 10 , 20 , 30 , 40 } . We observe that our method not only achieves superior robustness to outliers but is also less sensitive to data dimensions.

Besides, we compare the computational efficiency of these methods across varying sample sizes. As shown in Table 1, the OR-WDRO method is not applicable to large-scale datasets, with its running time exceeding 200 minutes for a sample size of n = 20000 . In contrast, our method only requires 20 seconds to process the same dataset while maintaining superior learning performance. Therefore, our method not only achieves better robustness but also maintains computational efficiency.

5.2 Linear classification

We consider a linear classification problem with the loss l θ ( x, y ) = max { 0 , 1 -y ( θ ⊺ x ) } with θ ∈ R d . We consider the same outliers as that in [20]. Specifically, we first generate a clean distribution P n by { X i , sign( θ ⊺ ∗ X i ) } , where X 1 , . . . , X n are drawn i.i.d. from N (0 , I d ) . For a uniform random subset S ⊂ [ n ] of size ⌊ εn ⌋ , we consider the corrupted distribution ˆ P n which is uniform over {( ( -C ) 1 { i ∈ S } X i + e 1 , sign( θ ⊺ ∗ X i ) )} n i =1 , where C > 0 is the contamination factor, e 1 is the distribution shift. We select h ( ξ ) = λ 2 ∥ ∥ ξ -¯ ξ ∥ ∥ 2 and c ( ξ, ζ ) = ∥ ξ -ζ ∥ 2 in linear classification as in [20].

We consider the formulation described in Theorem 2 and solve it using Algorithm 1. We fix ε = 0 . 1 , d = 10 , n = 100 and select λ = 10 , λ 2 = 1 , β = 2 . The experimental results are averaged over 10 trials. We evaluate the model performance for different choices of the contamination factor C by analyzing the excess risk and accuracy. Excess risk is defined as the difference between the returned loss and the best achievable loss. Accuracy refers to the rate of successful classification. As shown in Table 2, the performance of standard DRO decreases when C gets large. In contrast, the OR-WDRO performs well only when C is very large. This is because OR-WDRO uses CVaR to filter out outliers, which requires the outliers to be significantly distant from the regular data for CVaR to be effective. Nevertheless, our method performs well across the entire range of C .

5.3 Logistic regression

We consider a logistic regression with the loss function l θ ( x, y ) = log(1 + exp( -y ( θ ⊺ X ))) , where θ ∈ R 10 . As in [17], we assume that the feature vector X follows a multivariate standard normal distribution, and the conditional distribution of the label y ∈ {-1 , 1 } is given by Prob( y | x ) = [1 + exp( -y ( θ ⊺ ∗ x ))] -1 , where θ ∗ = (10 , 0 , . . . , 0) . This setup uniquely determines the true distribution P . We draw n samples from this distribution to create the empirical distribution P n = 1 n ∑ n i =1 δ ( X i ,y i ) . Outliers are considered to occur only in the feature space. The perturbed distribution is represented as { ( -C ) 1 { i ∈ S } X i + ρe 1 } n i =1 , where S ⊂ [ n ] is a uniform random subset of size ⌊ εn ⌋ . We fix ε = 0 . 1 , ρ = 0 . 1 , e 1 = (1 , 0 , . . . , 0) , and select λ = 10 , λ 2 = 1 , β = 2 . In Fig. 2, we compare the excess risk and accuracy of standard DRO as outlined in [17] and our proposed method, described by Theorem 2 and implemented by Algorithm 1. The results are averaged over 10 runs for different sample sizes n ∈ { 20 , 40 , 60 , 80 , 100 } . Besides, we conduct a comparative analysis of the excess risk and accuracy between the standard DRO and the proposed method across various contamination levels. As shown in Table 3, standard DRO is susceptible to significant outliers. Instead, the proposed method demonstrates robustness throughout the entire range of contamination factor C , consistently maintaining an accuracy of approximately 92% .

20

30

40

50

60

70

80

90

100

60

70

80

90

100

Standard DRO

UOT-DRO

(a) Excess risk with various samples.

(b) Accuracy with various samples.

5.4 Parameter selection

We provide some guidelines for selecting the parameters λ, λ 2 , β in Algorithm 1. The parameter λ is commonly used in the DRO literature [27] as a penalty coefficient. If the value of λ is large, then the DRO problem approaches the empirical risk minimization problem, resulting in a model that performs well on the empirical distribution but is less robust to Wasserstein perturbations.

The parameter λ 2 represents the credibility level assigned to the function h . A larger value of λ 2 should be selected when the confidence in the reliability of h is high, meaning that h ( ξ ) is highly likely to become large at outlier points. Conversely, if the prior knowledge provided by h is considered unreliable, the value of λ 2 should be reduced. If there is no reliable prior knowledge at all, in which case we should select λ 2 = 0 , achieving good performance is impossible, as shown in related literature in robust statistics [18].

The parameter β penalizes the mismatch between marginal distributions. A larger value of β places more emphasis on minimizing this mismatch, possibly at the expense of increasing the transportation cost, thereby making the unbalanced optimal transport distance close to the balanced one. Conversely, a small value of β allows for larger mismatches between the marginal distributions, which can enhance the model's robustness to outliers. However, when β is very small, possible distribution mismatches incur little penalty, and the computed distance may fail to accurately represent the true distance between the distributions. In this case, the resulting DRO problem will incur many unlikely distributions in the ambiguity set, leading to a very conservative model.

6 Conclusion

In this work, we introduced a novel DRO framework that employs a new distance derived from UOT. By incorporating a soft penalization term in the design of the ambiguity set, our method exhibits increased resiliency to outliers. We provided strong duality results for the original DRO problem and the Lagrangian penalty problem, with the latter allowing for more efficient computation via stochastic sub-gradient methods. Finally, empirical results validate our method's enhanced robustness to outliers and reduced computational demands for regression and classification tasks.

References

Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. Lectures on Stochastic Programming: Modeling and Theory . SIAM, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Georg Ch Pflug and Mathias Pohl. A review on ambiguity in stochastic portfolio optimization. Set-Valued and Variational Analysis , 26:733-757, 2018.

Euan A Ashley. Towards precision medicine. Nature Reviews Genetics , 17(9):507-522, 2016.

Vladimir Naumovich Vapnik, Vlamimir Vapnik, et al. Statistical Learning Theory . Wiley New York, 1998.

Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations Research , 58(3):595-612, 2010.

Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming , 171(1):115-166, 2018.

Wolfram Wiesemann, Daniel Kuhn, and Melvyn Sim. Distributionally robust convex optimization. Operations Research , 62(6):1358-1376, 2014.

Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance. Mathematics of Operations Research , 48(2):603-655, 2023.

Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research , 44(2):565-600, 2019.

Jie Wang, Rui Gao, and Yao Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926 , 2021.

Waïss Azizian, Franck Iutzeler, and Jérôme Malick. Regularization for wasserstein distributionally robust optimization. ESAIM: Control, Optimisation and Calculus of Variations , 29:33, 2023.

Jia-Jie Zhu, Wittawat Jitkrittum, Moritz Diehl, and Bernhard Schölkopf. Kernel distributionally robust optimization: Generalized duality theorem and stochastic approximation. In International Conference on Artificial Intelligence and Statistics , pages 280-288. PMLR, 2021.

Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online , 1(2):9, 2013.

John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics , 49(3):1378-1406, 2021.

Ruidi Chen and Ioannis Ch Paschalidis. A robust learning approach for regression models based on distributionally robust optimization. Journal of Machine Learning Research , 19(13):1-48, 2018.

Soroosh Shafieezadeh Abadeh, Peyman M Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. Advances in Neural Information Processing Systems , 28, 2015.

Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning , pages 12345-12355. PMLR, 2021.

Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj, and Arun Sai Suggala. Stochastic re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.09222 , 2023.

Sloan Nietert, Ziv Goldfeld, and Soroosh Shafiee. Outlier-robust Wasserstein dro. Advances in Neural Information Processing Systems , 36, 2024.

R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk , 2:21-42, 2000.

Thibault Séjourné, Gabriel Peyré, and François-Xavier Vialard. Unbalanced optimal transport, from theory to numerics. Handbook of Numerical Analysis , 24:407-471, 2023.

Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Vialard. Unbalanced optimal transport: Dynamic and Kantorovich formulations. Journal of Functional Analysis , 274(11):3090-3123, 2018.

Quang Huy Tran, Hicham Janati, Nicolas Courty, Rémi Flamary, Ievgen Redko, Pinar Demetci, and Ritambhara Singh. Unbalanced co-optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10006-10016, 2023.

Matthias Liero, Alexander Mielke, and Giuseppe Savaré. Optimal entropy-transport problems and a new Hellinger-Kantorovich distance between positive measures. Inventiones mathematicae , 211(3):969-1117, 2018.

Khiem Pham, Khang Le, Nhat Ho, Tung Pham, and Hung Bui. On unbalanced optimal transport: An analysis of Sinkhorn algorithm. In International Conference on Machine Learning , pages 7673-7682. PMLR, 2020.

Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571 , 2017.

Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning: Theory and applications. Journal of Machine Learning Research , 24(142):1-79, 2023.

Qi Qi, Yi Xu, Rong Jin, Wotao Yin, and Tianbao Yang. Attentional-biased stochastic gradient descent. arXiv preprint arXiv:2012.06951 , 2020.

Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K Warmuth. Exponentiated gradient reweighting for robust training under label noise and beyond. arXiv:2104.01493 , 2021.

Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

Olav Kallenberg. Foundations of Modern Probability , volume 2. Springer, 1997.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.

MDDonsker and SRS Varadhan. Asymptotic evaluation of certain Markov process expectations for large time. iv. Communications on Pure and Applied Mathematics , 36(2):183-212, 1983.

Paul Milgrom and Ilya Segal. Envelope theorems for arbitrary choice sets. Econometrica , 70(2):583-601, 2002.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3), 1997.

Dimitri Bertsekas. Convex Optimization Theory , volume 1. Athena Scientific, 2009.

7 Appendix

7.1 Proof of Lemma 1

By plugging in the definition of the UW distance in (4), we have that

where the second last inequality follows from the max-min inequality and the last inequality follows since E ξ ∼ γ ζ [ L ( θ, ξ ) -λc ( ξ, ζ )] ≤ sup ξ ∈ Ξ L ( θ, ξ ) -λc ( ξ, ζ ) . Recalling the definition f λ ( θ, ζ ) = sup ξ ∈ Ξ { L ( θ, ξ ) -λc ( ξ, ζ ) } , we obtain that

where the last equality follows from Donsker and Varadhan's variational formula [34] and the fact that f λ ( θ, ζ ) is integrable with respect to ˆ P . Note that the condition | f λ ( θ, ζ ) | < ∞ for ˆ P -almost every ζ ensures exp( f λ ( θ,ζ ) / ( λβ )) E ζ ∼ ˆ P [exp( f λ ( θ,ζ ) / ( λβ ))] d ˆ P ( ζ ) is well-defined. The supremum is achieved by d ¯ P ( ζ ) = exp( f λ ( θ,ζ ) / ( λβ )) E ζ ∼ ˆ P [exp( f λ ( θ,ζ ) / ( λβ ))] d ˆ P ( ζ ) . The proof is complete.

7.2 Proof of Theorem 1

Given Assumption 2-3, L ( θ, ξ ) -λc ( ξ, ζ ) is differentiable with respect to ξ . According to the Envelope theorem [35], we have f λ ( ζ ) is differentiable in λ . Given L ( θ, ξ ) -λc ( ξ, ζ ) is strongly concave in ξ , we have ∇ λ f λ ( ζ ) = -c ( h λ ( ζ ) , ζ ) , where h λ ( ζ ) := arg max ξ ∈ Ξ { L ( θ, ξ ) -λc ( ξ, ζ ) } .

Define v ( λ ) = λρ + λβ log E ζ ∼ ˆ P [ exp ( f λ ( ζ ) λβ )] . We have v ( λ ) is differentiable and satisfies

Since λ ∗ > 0 , due to first-order optimality condition, we must have ∇ λ ( v ( λ )) | λ = λ ∗ = 0 , i.e.,

We define the joint distribution γ ∗ such that d γ ∗ ( ξ, ζ ) = d γ ∗ ζ ( ξ )d ¯ P ∗ ( ζ ) , where d ¯ P ∗ ( ζ ) = exp( f λ ∗ ( ζ ) / ( λ ∗ β )) E ζ ∼ ˆ P [exp( f λ ∗ ( ζ ) / ( λ ∗ β ))] d ˆ P ( ζ ) , and γ ∗ ζ ( ξ ∗ ) = 1 , if ξ ∗ = h λ ∗ ( ζ ) = arg max ξ { L ( θ, ξ ) -λ ∗ c ( ξ, ζ ) }

and γ ∗ ζ ( ξ ) = 0 otherwise. Let P ∗ be the first marginal distribution of γ ∗ , i.e., it satisfies that γ ∗ ∈ Γ( P ∗ , ¯ P ∗ ) . In what follows, we show that P ∗ lies in the ambiguity set. Specifically, we have

where the inequality holds since γ ∗ is a feasible solution in Γ( P ∗ , ¯ P ) . The following equalities are derived by substituting the expressions of γ ∗ and ¯ P ∗ . The last equality follows due to the first-order optimality condition (19). Therefore, the distribution P ∗ is feasible for the primal problem. Then, the primal optimal value is lower bounded by the value

Combining (21) with the weak duality, we have (P) = (D). Thus, the strong duality holds when λ ∗ > 0 . The proof is complete.

7.3 Proof of Theorem 2

Based on the definition of UW in (4), we have

The last equality in (22) is achieved by selecting γ ∗ ζ ( ξ ∗ ) = 1 for a ξ ∗ ∈ arg max ξ ∈ Ξ { L ( θ, ξ ) -λc ( ξ, ζ ) } and γ ∗ ζ ( ξ ) ∈ 0 otherwise. Here, there always exists such a ξ ∗ since L ( θ, ξ ) -λc ( ξ, ζ ) is continuous and Ξ is a compact set. Recalling the definition of f λ ( ζ ) , we have

where the last equality follows from [14]. The proof ends.

7.4 Proof of Theorem 3

Since L ( θ, ξ ) -λc ( ξ, ˆ ζ t ) is convex in θ , and Ξ is a compact set, from Danskin's theorem [36], we have f λ ( θ, ˆ ζ t ) is convex in θ and

where Co denotes the convex hull of a point set. Given that f λ ( θ, ˆ ζ t ) is convex, it is easy to verify that F ( θ ) is convex. Since the exponential function is convex and non-decreasing, for any z ∗ t ∈ arg max ξ L ( θ t , ξ ) -λc ( ξ, ˆ ζ t ) , we have

Note that ∥ G t ∥ ≤ exp( B λβ ) B 2 λβ := C 0 . Since we obtain ϵ -approximate maximizer z t of arg max ξ L ( θ t , ξ ) -λc ( ξ, ˆ ζ t ) , the sub-gradient g t is inexact and we denote by the error e t = g t -G t .

In what follows, we analyze the bound of sub-gradient error e t . From its definition, we have

where the first inequality follows from triangle inequality, the second inequality holds since L ( θ, ξ ) -λc ( ξ, ˆ ζ t ) ≤ B and ∥ ∂ θ L ( θ t , z t ) -∂ θ L ( θ t , z ∗ t ) ∥ ≤ L θξ ∥ z t -z ∗ t ∥ . The third inequality hols since | e x -e y | ≤ e B | x -y | when x, y < B . The last inequality follows from our assumptions. Since (25) holds for any z ∗ t , we have ∥ e t ∥ ≤ C 1 ϵ .

Suppose that θ ∗ minimizes F ( θ ) . Based on the update rule, we have

Taking conditional expectation on both sides of (26) with respect to ˆ ζ t , we have

where the second inequality holds since E [ G t ] ⊂ ∂F ( θ t ) , the third inequality follows from the fact that G t ≤ C 0 and the last inequality follow since ∥ e t ∥ ≤ C 1 ϵ . Taking expectation on both sides of (27) and summing up over t = 1 , . . . , T , it gives

Rearranging (28), we have

Substituting η = 1 √ T into (29) completes the proof.

7.5 Additional details of linear regression experiment

In this section, we show the derivation of equation (16).

Recall that ¯ θ = [ θ ⊺ , -1] ⊺ , ζ = [ x ⊺ , y ] ⊺ and l θ ( ξ ) = | ¯ θ ⊺ ξ | . As l θ is proper, convex, and lower semicontinuous, it coincides with its bi-conjugate function l ∗∗ , see e.g. [7]. Thus, we can write l θ ( ξ ) = sup z ∈Z ⟨ z, ξ ⟩ -l ∗ θ ( z ) , where Z = { z ∈ R d : l ∗ ( z ) < ∞ is the effective domain of the conjugate function. Define κ ( θ ) = sup {∥ z ∥ ∗ : z ∈ Z} . It is shown by [16] that κ ( θ ) = ∥ ∥ ¯ θ ∥ ∥ ∗ . With the definition of the dual norm, we have

The classical minimax theorem [37] allows us to interchange the maximization and minimization and then we obtain

The second equality holds since sup ξ ∈ R d ⟨ z, ξ ⟩ = σ R d ( z ) = X 0 , where σ Ω and X Ω denote the support function and characteristic function of the set Ω , respectively. The third equality follows again from the classical minimax theorem [37]. The fourth equality is derived by substituting z 1 = -z 2 -z into the objective function. The fifth equality holds since l θ = l ∗∗ θ . Since λ > λ 2 + κ ( θ ) , there exists z 2 that satisfies z 2 = -z 1 -z and other constraints. Finally, we obtain

The proof is complete.

7.6 Additional experiments on linear regression

We evaluate the sensitivity of the model performance to the tuning of designed parameters. We first fix λ 2 = 5 and explore the impact of varying λ and β . As shown in Table 4, the model's performance demonstrates minimal sensitivity to changes in these parameters. Then, we fix β = 6 , and assess the sensitivity to changes in λ 2 . The simulation results are presented in Table 5. When λ 2 = 0 , meaning we abandon the use of the prior knowledge term λ 2 ∥ ∥ ∥ ¯ ξ -ˆ ζ i ∥ ∥ ∥ , our method performs worse than the traditional DRO method. This is because the unbalanced optimal transport distance itself includes distributions that contain worse outliers, thereby making our model overly conservative. As λ 2 increases, there is a noticeable improvement in learning performance, which we attribute to the enhanced role of prior knowledge in outlier detection.

We also explore how the contamination factor, denoted as C , affects model performance. The simulation result is presented in Table 6. While standard DRO is highly sensitive to the contamination

factor C , both OR-WDRO and our method are not sensitive to contamination factor C in linear regression tasks.

We use the same preprocessing step as in [20] to obtain a robust estimate of the expected value of the clean distribution, which involves a parameter ˆ ε . This parameter can be viewed as an estimate of the proportion of data corruption. We evaluate the sensitivity of the model's performance to variations in ˆ ε and the result is presented in Table 7. We observe that our method achieves better robustness and is less sensitive to changes in ˆ ε compared to OR-WDRO in [20]. This is because OR-WDRO relies on ˆ ε during both the preprocessing and the solver optimization steps while our method only requires ˆ ε during the preprocessing step.

7.7 Additional experiments on linear classification

We conducted additional experiments on linear classification, supplementing those presented in the main body of the paper. For these experments, we select d = 10 . First, we evaluate the model performance across various sample sizes. As illustrated in Fig. 3, our method outperforms others in terms of both excess risk and accuracy. Next, we assess the model performance for different dimension d , as shown in Fig. 4. We observe that our method achieves superior robustness and maintains the accuracy around 94% .

In addition, we compare the computational efficiency of these methods in linear classification. As detailed in Table 8, OR-WDRO exhibits low computational time when the sample size is small, but it faces scalability issues as the dimension increases. In contrast, our method demonstrates stable computational times and superior performance across various sample sizes.

20

30

40

50

60

70

80

90

100

50

60

70

80

90

100

Standard DRO

OR-WDRO

UOT-DRO

100

6 ×10-1

王

Excess risk

4 × 10-1

3 ×10-1

2 ×10-1

Standard DRO

OR-WDRO

UOT-DRO

20

30

40

50

60

70

80

90

100

(a) Excess risk with varied samples.

(b) Accuracy with varied samples.

5

10

15

20

25

30

35

40

Dimension

3×10 1

4×10 1

6×10 1

Excess risk

Standard DRO

OR-WDRO

UOT-DRO

5

10

15

20

25

30

35

40

Dimension

60

70

80

90

100

Standard DRO

OR-WDRO

UOT-DRO

(a) Excess risk with varied samples.

(b) Accuracy with varied samples.

As in the linear regression task, we evaluate the model's sensitivity to the preprocessing step, specifically to the parameter ˆ ε . The parameter ˆ ε is again the estimate of the proportion of data corruption. As illustrated in Table 9, our method is less sensitive to the accuracy of the estimate ˆ ε . The reason mirrors that in the linear regression task.

Finally, we evaluate the sensitivity of the model performance to the tuning parameters λ and β . We fix C = 10 . Table 10 shows that our method achieves the accuracy above 90% across the range of β values from 0.5 to 20 and λ values from 5 to 20. This performance is better than that of standard DRO with accuracy 66%, and OR-WDRO with accuracy 79%. These simulation results indicate that our method is not sensitive to these parameters in linear classification tasks.

7.8 Additional experiments on logistic regression

We evaluate the sensitivity of the model performance to the tuning parameters in logistic regression. As shown in Tables 11 and 12, our method achieves the accuracy above 85% across the considered ranges of λ from 5 to 15 and β from 0.4 to 10. Based on all of these simulation results, we find that our method's performance remains stable across a wide range of parameters.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Sections 3, 4, 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed in Section 3 that the problem derived from Theorem 1 is hard to solve.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide all the required assumptions and complete proofs in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the required information for the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The main contribution of this paper is theoretical.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Sections 5 and 7.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Sections 5 and 7.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper has no societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new Distributionally Robust Optimization (DRO) framework based on Unbalanced Optimal Transport (UOT) distance. Under some conditions, the authors establish strong duality results for a Lagrangian penalty variation of the proposed problem. The designed algorithm is tested in linear, logistic regression and linear classification tasks where it shows an increased robustness to outliers in comparison to the considered baselines.\n\nSTRENGTHS:\nThe paper proposes an original approach to the DRO problem utilizing the UOT distance to define the ambiguity set.  The authors provide theoretical justifications for the dual form of the problem under consideration and convergence analysis of the constructed algorithm. They empirically demonstrate that the proposed approach allows for increasing the robustness to outliers in comparison to the chosen baselines.\n\nWEAKNESSES:\n**Quality.** Some of the assumptions related to the ‘outlier-robustness’ property of the proposed approach are not sufficiently explained.\n\n1) The authors claim that the ambiguity set of the proposed approach (UOT DRO) necessarily includes ‘the clean distribution $P^*$' (line 146-147) due to the soft marginal constraints in the UOT problem. However, actually, this ‘softness’ of the marginals depends on the chosen divergence $\\mathcal{D}_{\\phi_2}$ and the parameter $\\beta$ in the unbalanced divergence definition (4). Typically, increasing the parameter $\\beta$, one gets closer to the balanced OT formulation which does not have ‘outlier robustness’ property, and does not allow one to ignore the outliers in empirical dataset. This dependence on parameter $\\beta$ is not explained and verified from theoretical or practical perspectives. Clarification of this point seems to be very important since this parameter is set differently in experiments considered in the paper.\n\n2) Besides, the authors introduce the function $h$ penalizing potential distributions with outliers in an ambiguity set of the primal problem (6) of UOT-DRO. Thanks to this function, the solution (model) of the Lagrangian penalty reformulation of UOT-DRO will be less affected by outlier data as explained in lines 215-225. However, the authors provide explanations for the case of large $\\lambda$, while in the experimental section they consider $\\lambda=10$. The influence of outliers on the model in this case is still not obvious.\n\n**Clarity.** Some parts of the text should be written more clearly, e.g., the formula in line 289. The choice of cost function and parameters in experiments (Section 5.1) should be explained.\n\n**Evaluation.** The comparison with the baselines seems to be rather limited. The authors have cited several other works which propose approaches for the outlier-robust DRO problem [1,2,3,4]. However, the empirical evaluation of the proposed approach misses the comparison with some of these approaches [1,3,4].\n\n\n**In summary**, I am not sufficiently convinced that the constructed approach actually solves the problem that it was designed for, i.e., allows for solving the DRO problem while effectively ignoring the outliers in the empirical dataset. The paper does not provide any theoretical results which support this claim. Thus, it should be accurately justified at least from the empirical side. However, the experimental part misses ablation study of parameters $\\beta$, $\\lambda$, and thorough comparison with other outlier-robust DRO approaches.\n\nQUESTIONS:\n1) Please explain the choice of cost function $c(\\xi,\\zeta)=|\\xi-\\zeta|$ in your experiments\n2) How do you choose parameters $\\beta$, $\\lambda$ in your experiments? Is it an heuristic choice? An ablation study of these parameters is highly anticipated\n3) Why is the performance of UOT-DRO for contamination $C=30$ higher than for $C=100$? How further increase of $C$ will influence the performance?\n4) Why you did not perform comparison with additional outlier-robust DRO approaches [1,3,4] cited in the paper?\n\n*Minor limitations*:\n- bibliography is not in alphabetical order,\n- line 286: $\\theta_*$ is not defined,\n- lines 311-312: '*only* requires *only* 20 seconds'.\n\n**References.**\n\n[1] Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj, and Arun Sai Suggala. Stochastic re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.09222, 2023.\n\n[2] Sloan Nietert, Ziv Goldfeld, and Soroosh Shafiee. Outlier-robust Wasserstein dro. Advances in Neural Information Processing Systems, 36, 2024.\n\n[3] Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning, pages 12345–12355. PMLR, 2021.\n\n[4] R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2:21–42, 2000.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel outlier-robust Wasserstein Distributionally Robust Optimization (WDRO) framework based on unbalanced optimal transport (UOT). For the UOT-DRO, the authors establish strong duality results under specific smoothness assumptions. To enhance computational efficiency, they propose a Lagrangian penalty variant of the problem and prove the strong duality of this variant as well. The authors then employ stochastic gradient descent (SGD) to solve the Lagrangian penalty variant, providing a theoretical convergence guarantee for their method.\n\nSTRENGTHS:\n1. This paper is well-written and easy to follow.\n\n2. The paper introduces a novel formulation of DRO based on UOT to address outliers in the empirical distribution. Rather than adopting UOT directly, the authors consider a variant of UOT (introduced in equation (4)) that restricts the positive measure to be a probability measure, which aligns with the focus of DRO on probability measures. Additionally, the authors introduce a function $h$ to measure the distance to the uncontaminated distribution, effectively ruling out distributions close to the empirical distribution but containing more outliers.\n\n3. The authors propose a Lagrangian penalty variant of the new outlier-robust DRO problem and employ stochastic gradient descent (SGD) to solve this variant with theoretical convergence guarantee for their method.\n\n4. In experiments conducted on linear regression, logistic regression, and linear classification, the proposed algorithm demonstrates superior performance compared to the baseline methods WDRO and OR-DRO. This validates the efficiency and effectiveness of the proposed approach.\n\nWEAKNESSES:\n1. The current draft does not include a theoretical analysis of the excess risk associated with the proposed UOT-DRO. Including such an analysis would strengthen the paper by providing a deeper understanding of the performance and limitations of the method.\n2. The discussion on parameter selection is currently limited. The UOT-DRO relies on several hyperparameters, including $\\lambda$, $\\lambda_2$, and $\\beta$, which may not be entirely independent. Although Section 7.6 addresses hyperparameter selection for linear regression, the approach for selecting these parameters in other tasks remains unclear. A more comprehensive discussion on tuning these parameters for general cases would be beneficial.\n\nQUESTIONS:\n1. In line 238-239, the authors mentioned that outlier-robust WDRO is highly sensitive to $\\varepsilon$, the contamination level in the outlier-robust optimal transport. However, proposition 12 in [A] provides a robustness guarantee when the chosen $\\hat \\varepsilon$ does not match the true $\\varepsilon$. Also, the numerical experiments in [17] show that OR-WDRO can perform well when the the chosen $\\hat \\varepsilon$ does not match the true $\\varepsilon$. Could you further explain why outlier-robust WDRO is sensitive to $\\varepsilon$ in line 238-239?\n2. It is encouraged to show an empirical evaluation on the convergence of the proposed Algorithm 1. This would help demonstrate the practical applicability and efficiency of the algorithm.\n3. For the numerical experiments, could the authors further explain how the values of the hyperparameters were determined for linear classification and logistic regression? Additionally, the choice of hyperparameters for the baseline methods is also missing and should be included for a comprehensive comparison.\n\n[A] Nietert, Sloan, Ziv Goldfeld, and Rachel Cummings. \"Outlier-robust optimal transport: Duality, structure, and statistical analysis.\" *International Conference on Artificial Intelligence and Statistics*. PMLR, 2022.\n\nLIMITATIONS:\nThe authors discuss the limitations of their work in Section 3. Notably, this work does not have a negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors leverage unbalanced optimal transport (UOT) to build a new DRO model. Strong dual reformulations together with efficient algorithm design have been proposed. Numerical studies on convex loss function demonstrate the superior performance of this framework.\n\nSTRENGTHS:\nOverall this is a good paper. The idea for using UOT is innovative and useful. Theoretical contributions are solid and I have checked their proofs and to the best of my knowledge, I don’t find mistakes. I leave some comments in next section, which may further help the authors improve the quality of this paper in terms of literature review and theoretical soundness.\n\nWEAKNESSES:\n1. I think the authors may miss an important citation and comparison: \n    [JGX2023] Wang J, Gao R, Xie Y (2023) Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926.\n\nSpecifically, the authors considered UOT-based DRO, where UOT is a variant of entropic regularized OT distance. In contrast, [JGX23] also considered another variant of entropic regularized OT distance (which they call it as Sinkhorn distance) to construct the ambiguity set:\n\n$$S(\\mathbb{P}, \\widehat{\\mathbb{P}}) = \\inf_{\\gamma \\in \\Gamma(\\mathbb{P}, \\widehat{\\mathbb{P}})}~\\Big\\{\n\\mathbb{E}_{(\\xi, \\zeta)\\sim \\gamma}[c(\\xi,\\zeta)] + \\beta \\mathbb{E}_{(\\xi, \\zeta)\\sim \\gamma}\\left[ \n\\log\\frac{\\mathrm{d}\\gamma(\\xi,\\zeta)}{\\mathrm{d}\\nu(\\xi)\\mathrm{d}\\widehat{\\mathbb{P}}(\\zeta)}\n\\right]\n\\Big\\}$$\n\nwhere  $\\nu(\\cdot)$ can be any reference measure (such as Lebesgue measure) on $\\Xi$. The difference is that the authors in this paper considered relative entropy regularization between  $\\bar{\\mathbb{P}}$ and $\\widehat{\\mathbb{P}}$, and [JGX23] considered relative entropy regularization between conditional conditional distribution $\\gamma_{\\zeta}(\\cdot)$  and measure  $\\nu(\\cdot)$. Also, please compare the dual reformulation in this paper and that in [JGX23]. The expressions share some connections.\n\n\nIn addition, the usage of entropic regularization for Wasserstein DRO has been largely explored in other literature. I hope the authors also add the comparisons among them.\n\n    [JY2022] J. Wang and Y. Xie, “A data-driven approach to robust hypothesis testing using sinkhorn uncertainty sets,” in 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022, pp. 3315–3320. \n\n     [WFJ2023] \tW.Azizian, F.Iutzeler, and J.Malick “Regularization for wasserstein distributionally robust optimization,” ESAIM:Control, Optimisation and Calculus of Variations, vol. 29, p. 33, 2023. \n\n     [JRY2022] J. Wang, R. Moore, Y. Xie, and R. Kamaleswaran, “Improving sepsis prediction model generalization with optimal transport,” in Machine Learning for Health. PMLR, 2022, pp. 474–488 \n\n     [SZ2023] \tS.-B. Yang and Z. Li, “Distributionally robust chance-constrained optimization with sinkhorn ambiguity set,” AIChE Journal, vol. 69, no. 10, p. e18177, 2023. \n\n     [CFAB2023] C. Dapogny, F. Iutzeler, A. Meda, and B. Thibert, “Entropy-regularized wasserstein distributionally robust shape and topology optimization,” Structural and Multidisciplinary Optimization, vol. 66, no. 3, p. 42, 2023. \n\n     [JGX2024] Wang J, Gao R, Xie Y (2024) Non-Convex Robust Hypothesis Testing using Sinkhorn Uncertainty sets. in 2024 IEEE International Symposium on Information Theory (ISIT). IEEE, 2024\n\nQUESTIONS:\n2. In Theorem 1, the assumption seems too restrictive, including boundedness of $f_{\\lambda}(\\theta,\\zeta)$, differentiability and concavity of $L(\\theta,\\xi)$, differentiability and concavity of $c$, and strict positivity of dual variable $\\lambda^*$. In existing DRO literature such as [JGX2023], only the finiteness of cost function, lighted tailed condition, measurability of loss, and law of total probability is required. In standard WDRO, the technical assumption for strong duality holds is mild as well. \n\n3. I also believe the assumption that dual variable is strictly positive is not necessary: when you take sufficiently large radius, the constraint is unbinding and the dual variable $\\lambda^*=0$. Please follow existing DRO literature to explore how to build strong duality in this case. \n\n4. In line 220, the authors implicitly used the assumption that $c(\\zeta,\\zeta)=0$. I don’t remember whether it has been assumed in the main context before. \n\n5. I have concerns regarding Theorem 3. The algorithm design is actually inspired from [Sinha et. Al 2017]. The authors in that reference studied optimization for non convex loss function, such as neural networks, and perform biased SGD update to achieve near-stationary points. However, the authors in this paper only considered convex optimization. I strongly believe their results can be extended for non-convex smooth optimization. Please consider the exploration in this direction. Some useful references include \n\n[HCH2021] Hu Y, Chen X, He N (2021) On the bias-variance-cost tradeoff of stochastic optimization. Advances in Neural Information Processing Systems. \n\n[ HZCH2020] Hu Y, Zhang S, Chen X, He N (2020) Biased stochastic first-order methods for conditional stochas- tic optimization and applications in meta learning. Advances in Neural Information Processing Systems, volume 33, 2759–2770. \n\n\n6. For the numerical study part, the exploration of outlier DRO with neural network loss functions is also quite promising.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}