Title: Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications
Venue: NeurIPS 2023
Accepted: no
Keywords: Distributionally Robust Optimization

Free Energy

Over-pessimism

Calibration term
Abstract: Machine learning algorithms minimizing average risk are susceptible to distributional shifts. Distributionally Robust Optimization (DRO) addresses this issue by optimizing the worst-case risk within an uncertainty set. However, DRO suffers from over-pessimism, leading to low-confidence predictions, poor parameter estimations as well as poor generalization. In this work, we conduct a theoretical analysis of a probable root cause of over-pessimism: excessive focus on noisy samples. To alleviate the impact of noise, we incorporate data geometry into calibration terms in DRO, resulting in our novel Geometry-Calibrated DRO (GCDRO) for regression. We establish that our risk objective aligns with the Helmholtz free energy in statistical physics, and this free-energy-based risk can extend to standard DRO methods. Leveraging gradient flow in Wasserstein space, we develop an approximate minimax optimization algorithm with a bounded error ratio and standard convergence rate and elucidate how our approach mitigates noisy sample effects. Comprehensive experiments confirm GCDRO's superiority over conventional DRO methods.

Abstract

Machine learning algorithms minimizing average risk are susceptible to distributional shifts. Distributionally Robust Optimization (DRO) addresses this issue by optimizing the worst-case risk within an uncertainty set. However, DRO suffers from over-pessimism, leading to low-confidence predictions, poor parameter estimations as well as poor generalization. In this work, we conduct a theoretical analysis of a probable root cause of over-pessimism: excessive focus on noisy samples. To alleviate the impact of noise, we incorporate data geometry into calibration terms in DRO, resulting in our novel Geometry-Calibrated DRO (GCDRO) for regression. We establish that our risk objective aligns with the Helmholtz free energy in statistical physics, and this free-energy-based risk can extend to standard DRO methods. Leveraging gradient flow in Wasserstein space, we develop an approximate minimax optimization algorithm with a bounded error ratio and standard convergence rate and elucidate how our approach mitigates noisy sample effects. Comprehensive experiments confirm GCDRO's superiority over conventional DRO methods.

1 Introduction

Machine learning algorithms with empirical risk minimization (ERM) have been shown to perform poorly under distributional shifts, especially sub-population shifts where substantial data subsets are underrepresented in the average risk due to their small sample sizes. As an alternative, Distributionally Robust Optimization (DRO) (Namkoong and Duchi, 2017; Blanchet and Murthy, 2019; Blanchet et al., 2019a; Duchi and Namkoong, 2021; Zhai et al., 2021; Liu et al., 2022a; Gao and Kleywegt, 2022; Gao et al., 2022) aims to optimize against the worst-case risk distribution within a predefined uncertainty set. This uncertainty set is centered around the training distribution, and generalization performance can be guaranteed when the test distribution falls within this set.

However, DRO methods have been found to experience the over-pessimism problem in practice (Hu et al., 2018; Zhai et al., 2021) (

i.e.

, low-confidence predictions, poor parameter estimations, and

generalization), recent studies have sought to address this issue. From the uncertainty set perspective ,

Blanchet et al. (2019b); Liu et al. (2022a,b) proposed data-driven methods to learn distance metrics 28

from data. However, these approaches remain vulnerable to noisy samples, as demonstrated in Table 2. 29

Recently, Słowik and Bottou (2022); Agarwal and Zhang (2022) observed that DRO may overly focus 30

on sub-populations with higher noise levels, leading to suboptimal generalization. Consequently, 31

from the risk objective perspective , they suggest incorporating calibration terms to mitigate this issue. 32

Nevertheless, applicable calibration terms either require expert knowledge or are computationally 33

intensive, and few practical algorithms have been proposed. 34

To devise a practical calibration term for DRO, we first aim to identify the root causes of over35

pessimism, which we attribute to the excessive focus on noisy samples that frequently exhibit higher 36

prediction errors. For typical DRO methods (Namkoong and Duchi, 2017; Staib and Jegelka, 2019; 37 Duchi and Namkoong, 2021; Liu et al., 2022b), based on a simple yet insightful linear example, 38 we theoretically demonstrate that the variance of estimated parameters becomes substantially large 39 when noisy samples have higher densities, in line with the empirical findings reported in (Zhai et al., 40 2021). Furthermore, we demonstrate that existing outlier-robust regression methods are not directly 41 applicable for mitigating noisy samples in DRO scenarios where both noisy samples and distribution 42 shifts coexist, highlighting the non-trivial nature of this problem. 43

44

45

46

47

48

49

50

51

52

53

54

55

In this work, inspired by the ideas in (Słowik and Bottou, 2022; Agarwal and Zhang, 2022), we design calibration terms, i.e. , total variation and entropy regularization, to prevent DRO from excessively focusing on random noisy samples. In conjunction with the Geometric Wasserstein uncertainty set (Liu et al., 2022b) utilized in our methods, these calibration terms effectively incorporate information from the data manifold, leading to improved regulation of the worst-case distribution in DRO. Specifically, during the optimization, the total variation term penalizes the variation of weighted prediction errors along the data manifold, preventing random noisy samples from gaining excessive densities. The entropy regularization term, also used in (Liu et al., 2022b), acts as a non-linear graph Laplacian operator that enforces the smoothness of the sample weights along the manifold. These calibration terms work together to render the worst-case distribution more reasonable for DRO, leading to our Geometry-Calibrated DRO (GCDRO) approach. We validate the effectiveness of our GCDRO on both simulation and real-world data.

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

Furthermore, from a statistical physics perspective, we demonstrate that our risk objective corresponds to the Helmholtz free energy, comprising three components: interaction energy, potential energy, and entropy. The free energy formulation generalizes typical DRO methods such as KL-DRO, χ 2 -DRO (Duchi and Namkoong, 2021), MMD-DRO (Staib and Jegelka, 2019) and GDRO (Liu et al., 2022b). This physical interpretation provides a novel perspective for understanding different DRO methods by drawing parallels between the worst-case distribution and the steady state in statistical physics, offering valuable insights. From the free energy point of view, our GCDRO specifically addresses the interaction energy between samples to mitigate the effects of noisy samples . Motivated by the study of the Fokker-Planck equation (FPE, Chow et al. (2017); Esposito et al. (2021)), through gradient flow in the Geometric Wasserstein space, we derive an approximate minimax algorithm with a bounded error ratio e -CT in after T in inner-loop iterations and a convergence rate of O (1 / √ T out ) after T out outer-loop iterations. Our optimization method supports any quadratic form of interaction energy, potentially paving the way for designing more effective calibration terms for DRO in the future.

2 Preliminaries: Noisy Samples Bring Over-Pessimism in DRO

Notations. X ∈ X denotes the covariates, Y ∈ Y denotes the target, f θ ( · ) : X → Y is the predictor parameterized by θ ∈ Θ . ˆ P N denotes the empirical counterpart of distribution P ( X,Y ) with N samples, and p = ( p 1 , . . . , p N ) T ∈ R N + is the probability vector. [ N ] = { 1 , 2 , . . . , N } denotes the set of integers from 1 to N . The random variable of data points is denoted by Z = ( X,Y ) ∈ Z . The random vector of n dimension is denoted by ⃗ h n = ( h 1 , . . . , h n ) T . G N = ( V, E, W ) denotes a finite weighted graph with N nodes, where V = [ N ] is the vertex set, E is the edge set and W = { w ij } ( i,j ) ∈ E is the weight matrix of the graph. And ( x ) + = max( x, 0) .

Distributionally Robust Optimization (DRO) is formulated as:

where ℓ is the loss function (typically mean square error) and P ( P ) = { Q : Dist ( Q,P ) ≤ ρ } denotes the ρ -radius uncertainty ball around the distribution P . Different distance metrics derive different DRO methods, e.g., f -divergence DRO ( f -DRO, Namkoong and Duchi (2017); Duchi and Namkoong (2021)) with the Cressie-Read family of Rényi divergence, Wasserstein DRO (WDRO, Sinha et al. (2018); Blanchet and Murthy (2019); Blanchet et al. (2019a,b)), MMD-DRO (Staib and Jegelka, 2019) with maximum mean discrepancy, and Geometric DRO (GDRO, Liu et al. (2022b)) with Geometric Wasserstein distance. Although DRO methods are designed to resist sub-population shifts, they have been observed to have poor generalization performances (Hu et al., 2018; Frogner et al., 2019; Słowik and Bottou, 2022) in practice, which is referred to as over-pessimism.

In this section, we identify one of the root causes of the over-pessimism of DRO: the excessive focus 87 on noisy samples with typically high prediction errors . 88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

KL-DRO

X²-DRO

GDRO

GCDRO

Sample weights

Sample weights

Sample weights

Sample weights

0.73

0.64

0.64

0.68

0.30

0.29

0.23

0.22

0.03

0.06

0.07

0.10

Major

Group

Minor

Group Samples

Noisy

Major

Greup

Group Samples

Minor

Noisy

Major

Group

Minor

Group Samples

Noisy

Major

Group

Minor

Group Samples

Noisy

We showcase DRO methods' excessive focus on noisy samples in practice and reveal their probability densities are linked to high prediction errors in worst-case distributions.

Through a simple yet insightful regression example, we prove that such a phenomenon leads to high estimation variances and subsequently poor generalization performance.

We demonstrate that existing outlier-robust regression methods are not directly applicable for mitigating noisy samples in DRO scenarios, emphasizing the non-trivial nature of this problem.

Problem Setting Given the underlying clean distribution P clean = (1 -α ) P major + αP minor , 0 < α < 1 2 , the goal of DRO can be viewed as achieving good performance across all possible sub-populations P minor . Denote the observed contaminated training distribution by P train . Based on Huber's ϵ -contamination model (Huber, 1992), we formulate P train as:

where ˜ Q is an arbitrary noisy distribution (typically with larger noise scale), 0 < ϵ < 1 2 is the noise level. Note that the minor sub-population could represent any distribution with a proportion of α in P . However, we explicitly specify it here to emphasize the distinction between our setting and the traditional Huber's ϵ -contaminated setting, as the latter does not take sub-population shifts into account.

Empirical Observations. Following a typical regression setting (Duchi and Namkoong, 2021; Liu et al., 2022b), we demonstrate the worst-case distribution of KL-DRO, χ 2 -DRO, and GDRO in Figure 1, where the size of each point is proportional to its density. In this scenario, the underlying distribution P comprises a known major sub-population (95%, blue points) and a minor sub-population (5%, green points). And the noise level ϵ in P train is 2% . DRO methods are expected to upweigh samples from minor sub-population to learn a model with uniform performances w.r.t. sub-populations. However, from Figure 1, we could observe that KL-DRO, χ 2 -DRO and GDRO excessively focus on noisy samples, resulting in a noise level 10 to 15 times larger than the original. This observation helps to explain their poor performance on this task (detailed results can be found in Table 2).

Theoretical Analysis. To support our observations, we first analyze the worst distribution of KL-DRO, χ 2 -DRO and GDRO, shedding light on the underlying reasons for this phenomenon.

Proposition 2.1 (Worst-case Distribution) . Let ˆ Q ∗ N = ( q ∗ 1 , q ∗ 2 , . . . , q ∗ N ) T ∈ R N + denotes the worstcase distribution, and ℓ ( f θ ( x i ) , y i ) ( abbr. ℓ i ) denotes the prediction error of sample i ∈ [ N ] . For different choices of Dist ( · , · ) in P ( P ) = { Q : Dist ( Q,P ) ≤ ρ } , we have:

KL-DRO: q ∗ i /q ∗ j ∝ exp( ℓ i -ℓ j ) ;

GDRO's final state (gradient flow step T →∞ ): q ∗ i /q ∗ j ∝ exp( ℓ i -ℓ j ) ;

χ 2 -DRO: q ∗ i /q ∗ j = ( ℓ i -λ ) + / ( ℓ j -λ ) + , and λ ≥ 0 is the dual parameter independent of i .

Proposition 2.1 demonstrates that for KL-DRO, χ 2 -DRO, and GDRO (large gradient flow step), the 121 relative density between samples is solely determined by their prediction errors, indicating that a 122 larger prediction error results in a higher density. However, in our problem setting, samples from both 123 minor sub-population P minor and noisy sub-population ˜ Q exhibit high prediction errors. The primary 124 goal of DRO is to focus on the minor sub-population P minor , but the presence of noisy samples in 125 ˜ Q significantly interferes with this objective and hurts model learning. As shown in Figure 1, for 126 KL-DRO, χ 2 -DRO and GDRO, noisy samples attract much density. Intuitively, it is not surprising 127 that an excessive focus on noisy samples can have a detrimental impact. As KL-DRO, χ 2 -DRO, and 128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

GDRO can be viewed as optimization within a weighted empirical distribution, we use the following simple example with the weighted least square model to demonstrate how this excessive focus on noisy samples can lead to high estimation variance, ultimately causing over-pessimism.

Example (Weighted Least Square) . Consider the data generation process as Y = kX + ξ , where X,Y ∈ R and random noise ξ satisfies ξ ⊥ X , E [ ξ ] = 0 and E [ ξ 2 ] ( abbr. σ 2 ) is finite. Assume that the training dataset X D consists of clean samples { x ( i ) c , y ( i ) c } i ∈ [ N c ] and noisy samples { x ( i ) o , y ( i ) o } i ∈ [ N o ] with σ 2 c < σ 2 o . Consider the weighted least-square model f ( X ) = θX . Denote the sample weight of a clean sample ( x ( i ) c , y ( i ) c ) as w ( i ) c ∈ R + , i ∈ [ N c ] , and the sample weight of a noisy sample ( x ( i ) o , y ( i ) o ) as w ( i ) o ∈ R + , i ∈ [ N o ] with ∑ i ∈ [ N c ] w ( i ) c + ∑ i ∈ [ N o ] w ( i ) o = 1 . The variance of the estimator ˆ θ is given by:

where X D = { x ( i ) c } N c 1 ∪ { x ( i ) o } N o 1 are the sampled covariates in the dataset. Besides, the minimum variance is achieved if and only if ∀ 1 ≤ i ≤ N c , 1 ≤ j ≤ N o , w ( j ) o /w ( i ) c = σ 2 c /σ 2 o < 1 .

From the results, we make the following remarks:

If noisy samples have higher weights than clean samples (e.g., w o /w c > 1 ), the variance of the estimated parameter ˆ θ will be larger, suggesting that the learned ˆ θ could be significantly unstable.

· In conjunction with Proposition 2.1, DRO methods tend to assign high weights to noisy samples, which can lead to unstable parameter estimation. While this example is relatively simple, this phenomenon aligns with the empirical findings in Zhai et al. (2021), which demonstrate that DRO methods can be quite unstable when confronted with label noise.

Relationship with Conventional Outlier-robust Regression. We would like to explain why conventional outlier-robust regression methods cannot be directly applied to our problem. The main challenge stems from the coexistence of noisy samples and minor sub-populations, both of which typically exhibit high prediction errors, leading to a misleading worst-case distribution in DRO. Conventional outlier-robust regression methods (Diakonikolas and Kane, 2018; Klivans et al., 2018; Diakonikolas et al., 2022) primarily focus on mitigating the effects of outliers without considering sub-population shifts. For instance, the L 2 -estimation-error of outlier-robust linear regression is O ( ϵ log(1 /ϵ )) (Diakonikolas and Kane, 2018), where ϵ represents the noise level in Equation 1. However, as analyzed in Proposition 2.1 and demonstrated in Figure 1, during the optimization of DRO, the noise level ϵ significantly increases, rendering even outlier-robust estimation quite inaccurate. Moreover, Klivans et al. (2018) propose finding a pseudo distribution with minimal prediction errors to avoid outliers (see Algorithm 5.2 in (Klivans et al., 2018)). Nevertheless, this approach might inadvertently exclude minor sub-populations, which should be the focus under sub-population shifts, due to the main challenge: the coexistence of noisy samples and minor subpopulations. Zhai et al. (2021) incorporate this idea into DRO. Still, their method requires an implicit assumption that the prediction errors of noisy samples are higher than those of minor sub-populations, which does not always hold in practice. And Bennouna and Van Parys (2022) build the uncertainty set via two measures, KL-divergence and Wasserstein distance, leading to a combined approach of KL-DRO and ridge regression. Despite this, as we discussed earlier, DRO tends to increase the noise level in data, making it difficult to fix using ridge regression.

Based on the analysis above, we stress the importance of integrating more data-derived information. In pursuit of this, we propose to leverage the unique geometric properties that distinguish noisy samples from minor sub-populations to address this issue.

3 Proposed Method

In this work, with a focus on regression, we introduce our Geometry-Calibrated DRO (GCDRO). The fundamental idea is to utilize data geometry to distinguish between random noisy samples and minor sub-populations. It is motivated by the fact that prediction errors for minor sub-populations typically exhibit local smoothness along the data manifold, a property that is not shared by noisy samples.

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

Discrete Geometric Wasserstein Distance. We briefly revisit the definition of the discrete geometric Wasserstein distance. Given a weighted finite graph G N = ( V, E, W ) , the probability set P ( G N ) supported on the vertex set V is defined as P ( G N ) = { p ∈ R N | ∑ N i =1 p i = 1 , p i ≥ 0 , for i ∈ V } , and its interior is denoted as P o ( G N ) . A velocity field v = ( v ij ) i,j ∈ V ∈ R N × N on G N is defined on the edge set E satisfying that v ij = -v ji if ( i, j ) ∈ E . ξ ij ( p ) is a function interpolated with the associated nodes' densities p i , p j . The flux function pv ∈ R N × N on G N is defined as pv := ( v ij ξ ij ( p )) ( i,j ) ∈ E and its divergence is defined as div G N ( pv ) := -( ∑ j ∈ V :( i,j ) ∈ E √ w ij v ij ξ ij ( p )) N i =1 ∈ R N . Then for distributions p 0 , p 1 ∈ P o ( G N ) , the discrete geometric Wasserstein distance (Chow et al., 2017; Liu et al., 2022b) is defined as:

Equation 4 computes the shortest (geodesic) length among all potential plans, integrating the total kinetic energy of the velocity field throughout the transportation process. A key distinction from the Wasserstein distance is that it only permits density to appear at the graph nodes.

Formulation Given training dataset D tr = { ( x i , y i ) } N i =1 and a finite weighted graph G N = ( V, E, W ) representing the inherent structure of sample covariates. Denote the empirical marginal distribution as ˆ P X , the formulation of GCDRO is:

where ρ is the pre-defined radius of the uncertainty set, ℓ i is the loss on the i -th sample and w ij ∈ W denotes the edge weight between sample i and j . α and β are hyper-parameters.

Illustrations . In our formulation, for any distribution q within the uncertainty set, Calibration term I ( ∑ ( i,j ) ∈ E w ij q i q j ( ℓ i -ℓ j ) 2 ) calculates the graph total variation of prediction errors along the data manifold that is characterized by G N . Intuitively, when selecting the worst-case distribution , this term imposes a penalty on distributions that allocate high densities to random noisy samples, as this allocation significantly amplifies the overall variation in prediction errors. Conversely, this term does not penalize distributions that allocate high densities to minor sub-populations, as their errors are smooth and have a relatively small impact on the total variation along the manifold. This differing phenomenon arises from the distinct geometric properties of random noisy samples and minor sub-populations, as samples from the latter typically cluster together on the data manifold. Further, during the optimization of model parameter θ , this term acts like a variance term, resulting in a quantile-like risk objective, which helps to mitigate the effects of outliers.

Calibration term II ( ∑ N i =1 q i log q i ) represents the negative entropy of distribution q . As discussed in Section 3.2, during optimization, this term transforms into a non-linear graph Laplacian operator that encourages sample weights to be smooth along the manifold, avoiding extreme sample weights in the worst-case distribution.

3.1 Free Energy Implications on Worst-case Distribution

We first demonstrate the free energy implications of our risk objective R N ( θ, q ) . Intuitively, the change of sample weights across N samples (the inner maximization problem of R N ( θ, q ) ) can be analogously related to the dynamics of particles in a system, wherein the concentration of densities coincides with the aggregation of particle masses at N distinct locations (in the case of infinite samples, these locations converge to the data manifold). As a result, a deeper understanding of the steady state in a particle system can offer valuable insights into the worst-case distribution for DRO.

Building on this analogy, we can dive deeper into the physics of particle interactions. When particles exist within a potential energy field, they are subject to external forces. Simultaneously, there are interactions among the particles themselves, leading to a constant state of motion within the system. In statistical physics, a key point of interest is identifying when a system reaches a steady state. In a standard process like the reversible isothermal process, it is established that spontaneous reactions consistently move in the direction of decreasing Helmholtz free energy (Fu et al., 1990; Reichl, 1999;

Friston, 2010), which consists of interaction energy, potential energy and the negative entropy: 221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

By taking V = -ℓ ⃗ and K ij = α 2 w ij ( ℓ i -ℓ j ) 2 for ( i, j ) ∈ E , our risk objective is a special case of Helmholtz free energy, where the potential energy of sample i is -ℓ i q i and the interaction energy between sample i and j is α 2 w ij ( ℓ i -ℓ j ) 2 q i q j . Specifically, such mutual interactions can manifest as repulsive forces between adjacent particles , thereby preventing the concentration of mass in locations where local prediction errors are significantly high. And this explains from a physical perspective why our calibration term I could mitigate random noisy samples.

Additionally, Proposition 3.1 offers physical interpretations to comprehend the worst-case distribution of various DRO methods. We make some remarks: (1) current DRO methodologies, except MMDDRO, do not explicitly formulate the interaction term between samples in their design considerations ( χ 2 -DRO does not involve interaction between samples), despite the corresponding interaction energy between particles being a common phenomenon in physics; (2) MMD-DRO simply uses kernel gram matrix for interaction and lacks efficient optimization algorithms; (3) by considering this interaction energy , our proposed GCDRO is capable of mitigating the impacts of random noisy samples.

Proposition 3.1 (Free Energy Implications) . The dual reformulations of some typical DRO methods are equivalent to the free-energy-based minimax problem min θ ∈ Θ ,λ ≥ 0 max q ∈ P { λρ -E ( q , θ, λ ) } with different choices of P , ρ and K,V,H [ q ] in the free energy E . Details are shown in Table 1.

Through free energy, we could understand the type of energy or steady state that DRO methods strive to achieve, and design better interaction energy terms in DRO. Moreover, our optimization, as outlined in Section 3.2, could accommodate multiple quadratic forms of interaction energy.

3.2 Optimization

Then we derive an approximate minimax optimization for our GCDRO. For the inner maximization problem, we approximately deal with it via the gradient flow of -R N ( θ, Q ) w.r.t. Q in the geometric Wasserstein space ( P o ( G N ) , GW G N ) . We show that the error rate is O ( e -CT in ) after T in iterations inner loop, which gives a nice approximation. For the outer minimization w.r.t. model parameters θ , we analyze the convergence rate of O (1 / √ T out ) after T out iterations outer loop when the risk function satisfies Lipschitzian smoothness conditions.

Inner Maximization . We denote the Continuous gradient flow as q : [0 , T ] → P o ( G N ) , the probability density of sample i at time t is abbreviated as q i ( t ) , and the Time-discretized gradient flow with time step τ as ˆ q τ . For inner maximization, we utilize the τ -time-discretized gradient flow (Villani, 2021) for -R N ( θ, q ) in the geometric Wasserstein space ( P o ( G N ) , GW 2 G N ) as:

The gradient of q in Equation 7 is given as (when τ → 0 ):

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

where E is the edge set of G N , w ij is the edge weight between node i and j , N ( i ) denotes the set of neighbors of node i , ℓ i denotes the loss of sample i , and ξ ij ( · , · ) : P ( G N ) × R → R is:

which is the upwind interpolation commonly used in statistical physics and guarantees that the probability vector q keeps positive. From the gradient, we could see that the entropy regularization acts as a non-linear graph Laplacian operator to make the sample weights smooth along the manifold. In our algorithm, we fix the steps of the gradient flow to be T in and prove that the error ratio is e -CT in compared with the ground-truth worst-case risk R N ( θ, q ∗ ) constrained in an ρ ( θ, T in ) -radius ball.

Proposition 3.2 (Approximation Error Ratio) . Given the model parameter θ , denote the distribution after time T in as q T in ( θ ) , and the distance to training distribution ˆ P X as ρ ( θ, T in ) := GW 2 G N ( ˆ P X , q T in ( θ )) ( abbr. ρ ( θ ) ). Assume R N ( θ, q ) is convex w.r.t q . Then define the ground-truth worst-case distribution q ∗ ( θ ) within the ρ ( θ ) -radius ball as:

The upper bound of the error rate of the objective function R N ( θ, q T in ) satisfies:

where ˆ L is the Laplacian matrix of G N . λ sec , λ min are the second smallest and smallest eigenvalue, m,r are constants depending on R N , G N , β .

We make some remarks:

For the assumption that R N is convex w.r.t. q , the Hessian is given by ∇ 2 R N = β diag (1 /q 1 , ..., 1 /q N ) + 2 K . Since K is a sparse matrix whose nonzero elements in each row is far smaller than N , it is easily satisfied in empirical settings that the Hessian matrix ∇ 2 R is diagonally dominant and thus positive definite, making the inner maximization concave w.r.t q .

During the optimization, our algorithm finds an approximate worst-case distribution that is close to the ground-truth one within a ρ ( θ ) -radius uncertainty set. Our robustness guarantee is similar to Sinha et al. (2018) (see Equation 12 in Sinha et al. (2018)).

The error ratio is e -CT in , enabling to find a nice approximation efficiently with finite T in steps.

Outer Minimization. The convergence property relies on the risk objective R N ( θ, q ) . When R N ( θ, q ) is smooth w.r.t. θ , the following proposition guarantees convergence to a stationary point of problem 5 at a standard rate of O (1 / √ T ) .

Proposition 3.3 (Convergence) . Assume F ( θ ) := sup q : GW 2 GN ( ˆ P X , q ) ≤ ρ ( θ ) R N ( θ, q ) is L -smooth, and R N ( θ, q ) is L q -smooth w.r.t. q such that ∥∇ q R N ( θ, q ) - ∇ q R N ( θ, q ′ ) ∥ 2 ≤ L q ∥ q -q ′ ∥ 2 . ρ ( θ ) follows the definition in Proposition 3.2. Take a constant ∆ F ≥ F ( θ (0) ) -inf θ F ( θ ) and set step size as α = √ ∆ F / ( LT ) . For ∥ q T in -q ∗ ∥ 2 2 ≤ γ of the inner maximization problem, we have:

From Proposition 3.3, as T →∞ , ∇ θ F ( θ ( t ) ) exhibits a standard square-root convergence. Furthermore, the parameter γ can be effectively controlled, owing to the concavity inherent in the inner maximization problem and the rapidly diminishing error ratio as described in Proposition 3.2.

3.3 Mitigate the Effects of Random Noisy Samples

Finally, we prove that our GCDRO method effectively de-emphasizes 'noisy samples' with locally non-smooth prediction errors. Due to the challenge of assessing intermediate states in gradient flow, we focus on its final state (as T in →∞ ). Notably, in Proposition 3.2, the convergence rate of gradient flow is O ( e -CT in ) , implying that an efficient approximation of the final state is feasible.

For the worst-case distribution q ∗ , we denote the density ratio between samples as γ ( i, j ) := q ∗ i /q ∗ j . In sensitivity analysis, when only sample i is perturbed with label noises, we denote the density ratio

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

in the new worst-case distribution ˜ q ∗ as γ noisy ( i, j ) := ˜ q ∗ i / ˜ q ∗ j . The sample weight sensitivity ξ ( i, j ) is defined as ξ ( i, j ) = log γ noisy ( i, j ) -log γ ( i, j ) , which measures how much density ratio changes under perturbations on one sample. Larger ξ ( i, j ) indicates larger sensitivity to noisy samples.

Proposition 3.4. Assume ℓ noisy i -ℓ i ≥ 2( ∑ k ∈ N ( i ) q ∗ k w ik ℓ k ∑ k ∈ N ( i ) q ∗ k w ik -ℓ i ) which is locally non-smooth. For any α > 0 (in Equation 5), we have ξ GCDRO < ξ GDRO. Furthermore, there exists M > 0 such that for any α > M , we have ξ GCDRO ( i, j ) < 0 < min { ξ χ 2 -DRO ( i, j ) , ξ GDRO ( i, j )(= ξ KL-DRO ( i, j )) } , indicating that GCDRO is not sensitive to locally non-smooth noisy samples.

In practice, we do a grid search over α ∈ [0 . 1 , 10] on an independent held-out validation dataset to select the best α . The complexity of gradient flow scales linearly with sample size.

4 Experiments

In this section, we test the empirical performances of our proposed GCDRO on simulation data and real-world regression datasets with natural distributional shifts. As for the baselines, we compare with empirical risk minimization (ERM), WDRO, two typical f -DRO methods, including KL-DRO, χ 2 -DRO (Duchi and Namkoong, 2021), GDRO (Liu et al., 2022b), HRDRO (Bennouna and Van Parys, 2022) and DORO (Zhai et al., 2021), where HRDRO and DORO are designed to mitigate label noises.

4.1 Simulation Data

Data Generation. We design simulation settings with both sub-population shifts and noisy samples. The input covariates X = [ S, U, V ] T ∈ R 10 consist of stable covariates S ∈ R 5 , irrelevant ones U ∈ R 4 and the unstable covariate V ∈ R :

) ).

Analysis. From Table 2, (1) compared with ERM, all typical DRO methods, especially χ -DRO and KL-DRO, are strongly affected by label noises. (2) Although DORO is designed to mitigate outliers, it does not perform well under strong noises ( κ = 5% ), because it relies on the assumption that noisy points have the largest prediction errors, which does not always hold. (3) Our proposed GCDRO outperforms all baselines under different strengths of label noises, which demonstrates its effectiveness. (4) Compared with GDRO, we could see that our calibration terms in Equation where θ S ∈ R 5 is the coefficients of the true model, | r | > 1 is the adjustment factor for each sub-population, and Laplace ( · , · ) denotes the Laplace distribution. From the data generation, the relationship between S and Y stays invariant under different r , U ⊥ Y , while the relationship between V and Y is controlled by r , which varies across sub-populations . Intuitively, sign ( r controls whether the spurious correlation V -Y is positive or negative. And | r | controls the strength of the spurious correlation: the larger | r | is, the stronger the spurious correlation is. Furthermore, in order to conform to real data which are naturally assembled with label noises (Zhai et al., 2021), we introduce label noises by an ϵ proportion of labels as Y ′ ∼ N (0 , Std ( Y )) . ϵ controls the noise level. Settings. In training, we generate 9,500 points with r = 1 . 9 ( majority , strong positive spurious correlation V -Y ) and 500 points with r = -1 . 3 ( minority , weak negative spurious correlation V -Y In testing, we vary r ∈ { 3 . 0 , 2 . 3 , -1 . 9 , -2 . 7 } to simulate different spurious correlations V -Y . We use linear model with mean square error (MSE) and report the prediction root-mean-square errors (RMSE) for each sub-population, the mean and standard deviation of prediction errors among all testing sub-populations. Also, we report the parameter estimation errors ∥ ˆ θ -θ ∗ ∥ 2 of all methods ( θ ∗ = ( θ T S , 0 , . . . , 0) T ). The results over 10 runs are shown in Table 2. 2

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

BikeSharingDataset

HousePriceDataset

TemperatureDataset

ERM

HRDRO

WDRO

DORO

L.5

X²-DRO

GDRO

2.0

8.5

KL-DRO

GCDRO

RMSE

RMSE

RMSE

ERM

HRDRO

ERM

HRDRO

WDRO

DORO

WDRO

DORO

x²-DRO

GDRO

x²-DRO

GDRO

KL-DRO

GCDRO

1.0

KL-DRO

GCDRO

0.25

1

2

Season

3

0.5

<1920

1929-1945

Built Year

1945-1970

1970-1995

>1995

2013

2014

Year

2015

2016

2017

5 is effective to mitigate label noises. From Figure 1, the worst-case distribution of our GCDRO significantly upweighs on the minority (green points) and does not put much density on the noisy data (red points), while the others put much higher weights on the noisy samples and perform poorly.

4.2 Real-world Data

We use three real-world regression datasets with natural distributional shifts, including bike-sharing prediction, house price, and temperature prediction. For all these experiments, we use a two-layer MLP model with mean square error (MSE). We use the Adam optimizer Kingma and Ba (2015) with the default learning rate 1 e -3 . And all methods are trained for 5 e 3 epochs.

Datasets. (1) Bike-sharing dataset (Dua and Graff, 2017) contains the daily count of rental bikes in the Capital bike-sharing system with the corresponding 11 weather and seasonal covariates. The task is to predict the count of rental bikes of casual users . Note that the count of casual users is likely to be more random and noisy , which is suitable to verify the effectiveness of our method. We split the dataset according to the season for natural shifts. In the training data, the ratio of four seasons' data is 9 : 7 : 5 : 3 . We test on the rest of the data and report the prediction error of each season. (2) House Price dataset 1 contains house sales prices from King County, USA. The task is to predict the transaction price of the house via 17 predictive covariates such as the number of bedrooms, square footage of the house, etc. We divide the data into 5 sub-populations according to the built year of each house with each sub-population covering a span of 25 years. In training, we use data from the first group (built year < 1920 ) and report the prediction error for each testing group. (3) Temperature dataset (Dua and Graff, 2017) is largely composed of the LDAPS model's next day's forecast data, in-situ maximum and minimum temperatures of present-day, and geographic auxiliary variables in South Korea from 2013 to 2017. The task is to predict the next-day's maximum air temperatures based on the 22 covariates. We divide the data into 5 groups corresponding with 5 years. In the training data, the ratio of five years' data is 9 : 7 : 5 : 3 : 1 . We test on the rest of the data and report the prediction error of each year. More details could be found in Appendix.

Analysis. (1) From the results in Figure 4.1, we could see that the performances of ERM drop a lot under distributional shifts, and DRO methods have better performance as well as robustness. (2) Our proposed GCDRO outperforms all baselines under strong shifts, with the most stable performances under natural distributional shifts. (3) As for the k NN graph's fitting accuracy of the data manifold, we visualize the learned manifold in Appendix and we could see that the learned k NN graph fits the data manifold well. Besides, we show in Appendix that the performances of our GCDRO are relatively stable across different choices of k . Also, our GCDRO only needs the input graph G N to represent the data structure and any manifold learning or graph learning methods could be plugged in to give a better estimation of G N .

5 Future Directions

Our work deals with the over-pessimism in DRO via geometric calibration terms and provides free energy implications. The high-level idea could inspire future research on (1) relating free energy with DRO; (2) designing more reasonable calibration terms in DRO; (3) incorporating data geometry in general risk minimization algorithms.

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

References

Agarwal, A. and Zhang, T. (2022). Minimax regret optimization for robust machine learning under distribution shift. arXiv preprint arXiv:2202.05436 .

Bennouna, A. and Van Parys, B. (2022). Holistic robust data-driven decisions. arXiv preprint arXiv:2207.09560 .

Blanchet, J., Kang, Y., and Murthy, K. (2019a). Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability , 56(3):830-857.

Blanchet, J. and Murthy, K. (2019). Quantifying distributional model risk via optimal transport. Mathematics of Operations Research , 44(2):565-600.

Blanchet, J. H., Kang, Y., Murthy, K. R. A., and Zhang, F. (2019b). Data-driven optimal transport cost selection for distributionally robust optimization. In 2019 Winter Simulation Conference, WSC 2019, National Harbor, MD, USA, December 8-11, 2019 , pages 3740-3751. IEEE.

Chow, S.-N., Li, W., and Zhou, H. (2017). Entropy dissipation of fokker-planck equations on graphs. arXiv preprint arXiv:1701.04841 .

Diakonikolas, I. and Kane, D. M. (2018). Algorithmic high-dimensional robust statistics. Webpage http://www. iliasdiakonikolas. org/simons-tutorial-robust. html .

Diakonikolas, I., Kane, D. M., Pensia, A., and Pittas, T. (2022). Streaming algorithms for highdimensional robust statistics. In International Conference on Machine Learning , pages 5061-5117. PMLR.

Dua, D. and Graff, C. (2017). Uci machine learning repository.

Duchi, J. C. and Namkoong, H. (2021). Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics , 49(3):1378-1406.

Esposito, A., Patacchini, F. S., Schlichting, A., and Slepˇ cev, D. (2021). Nonlocal-interaction equation on graphs: gradient flow structure and continuum limit. Archive for Rational Mechanics and Analysis , 240(2):699-760.

Friston, K. (2010). The free-energy principle: a unified brain theory? Nature reviews neuroscience , 11(2):127-138.

Frogner, C., Claici, S., Chien, E., and Solomon, J. (2019). Incorporating unlabeled data into distributionally robust learning. arXiv preprint arXiv:1912.07729 .

Fu, X., Shen, W., Yao, T., and Hou, W. (1990). Physical chemistry. Higher Education, Beijing .

Gao, R., Chen, X., and Kleywegt, A. J. (2022). Wasserstein distributionally robust optimization and variation regularization. Operations Research .

Gao, R. and Kleywegt, A. (2022). Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research .

Hu, W., Niu, G., Sato, I., and Sugiyama, M. (2018). Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning , pages 2029-2037. PMLR.

Huber, P. J. (1992). Robust estimation of a location parameter. Breakthroughs in statistics: Methodology and distribution , pages 492-518.

Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings .

Klivans, A., Kothari, P. K., and Meka, R. (2018). Efficient algorithms for outlier-robust regression. In Conference On Learning Theory , pages 1420-1430. PMLR.

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

Liu, J., Shen, Z., Cui, P., Zhou, L., Kuang, K., and Li, B. (2022a). Distributionally robust learning with stable adversarial training. IEEE TKDE .

Liu, J., Wu, J., Li, B., and Cui, P. (2022b). Distributionally robust optimization with data geometry. In Advances in Neural Information Processing Systems .

Namkoong, H. and Duchi, J. C. (2017). Variance-based regularization with convex objectives. Advances in neural information processing systems , 30.

Reichl, L. E. (1999). A modern course in statistical physics.

Sinha, A., Namkoong, H., and Duchi, J. C. (2018). Certifying some distributional robustness with principled adversarial training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net.

Słowik, A. and Bottou, L. (2022). On distributionally robust optimization and data rebalancing. In International Conference on Artificial Intelligence and Statistics , pages 1283-1297. PMLR.

Staib, M. and Jegelka, S. (2019). Distributionally robust optimization and generalization in kernel methods. Advances in Neural Information Processing Systems , 32.

Villani, C. (2021). Topics in optimal transportation. 58.

Zhai, R., Dan, C., Kolter, Z., and Ravikumar, P. (2021). Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning , pages 12345-12355. PMLR.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates a root cause of the over-pessimism issue of existing distributionally robust optimization (DRO) methods: excessive focus on noisy samples. To mitigate this issue, the authors proposed a novel DRO method called Geometrically-Calibrated DRO (GCDRO). They introduce the free energy implications of their method (Section 3.1) and approximate optimization method (Section 3.2). Finally, they empirically validate their method for both simulation data (Section 4.1) and real-world data (Section 4.2).\n\nSTRENGTHS:\n- While this paper is a theoretical work (for the ML community), I believe that the problems addressed in this paper can be applied in practical situations. Specifically, existing debiasing methods [1, 2] focus on minor group samples with loss/gradient information. Similar to existing DRO methods (as the paper shows), these debiasing methods will focus excessively on noisy samples. In this case, GCDRO (or a more computationally scalable approximation version) can be applied to mitigate this issue.\n    - [1] Nam, Junhyun, et al. \"Learning from failure: De-biasing classifier from biased classifier.\" *Advances in Neural Information Processing Systems* 33 (2020): 20673-20684.\n    - [2] Ahn, Sumyeong, Seongyoon Kim, and Se-young Yun. \"Mitigating Dataset Bias by Using Per-sample Gradient.\" *arXiv preprint arXiv:2205.15704* (2022).\n- The free energy perspective of DRO using duality seems novel and interesting.\n\nWEAKNESSES:\n- The organization of this paper is somewhat puzzling and hard, although it deals with theoretical topics that are difficult for non-experts to understand. For example, it would be recommended to highlight the key difference between DRO and ERM in Section 2. As ERM does not assume graph structure G_N, readers unfamiliar with DRO would expect a detailed explanation/usage of the graph in L74.\n- It would be recommended to add a confidence interval for the results in Figure 2. Also, a vector image format provides more clear results as it does not blur when zooming in.\n\nQUESTIONS:\nCan you provide the overall computation cost in terms of wall clock time? What is the computational bottleneck in the proposed method? Can the k-NN graph construction used in Appendix A be applied to debiasing settings in [1,2]?\n\nLIMITATIONS:\nThe authors did not discuss the limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a novel approach called Geometry-Calibrated Distributionally Robust Optimization (GCDRO) to address the over-pessimism issue in traditional Distributionally Robust Optimization (DRO) methods. DRO aims to optimize worst-case risk within an uncertainty set to mitigate the effects of distributional shifts in machine learning algorithms. However, DRO often leads to low-confidence predictions, poor parameter estimations, and limited generalization.\n\nThe authors analyze a possible cause of over-pessimism in DRO, which is the excessive focus on noisy samples. To mitigate the impact of noise, they incorporate data geometry into calibration terms in DRO, resulting in GCDRO specifically designed for regression tasks. They demonstrate that their risk objective aligns with the concept of Helmholtz free energy in statistical physics, and this free-energy-based risk can be extended to standard DRO methods.\n\nTo optimize the GCDRO objective, the authors leverage gradient flow in the Wasserstein space and develop an approximate minimax optimization algorithm. This algorithm guarantees a bounded error ratio and standard convergence rate. The authors further explain how their approach alleviates the effects of noisy samples in the optimization process.\n\nTo validate the effectiveness of GCDRO, the authors conduct comprehensive experiments. These experiments demonstrate that GCDRO outperforms conventional DRO methods in terms of prediction accuracy, parameter estimation, and generalization performance.\n\nOverall, the proposed GCDRO approach addresses the over-pessimism issue in DRO by incorporating data geometry into the calibration terms. The authors provide a theoretical analysis and empirical evidence to support the superiority of GCDRO over conventional DRO methods, showing its potential for improving the robustness and performance of machine learning algorithms in the face of distributional shifts and noisy samples.\n\nSTRENGTHS:\nNovel Approach: The paper introduces a novel approach called Geometry-Calibrated Distributionally Robust Optimization (GCDRO) to address the over-pessimism issue in traditional Distributionally Robust Optimization (DRO) methods. This new approach incorporates data geometry into calibration terms, offering a unique perspective on mitigating the impact of noisy samples and improving the performance of machine learning algorithms.\n\nTheoretical Analysis: The paper provides a theoretical analysis of the over-pessimism issue in DRO and establishes a connection between the risk objective in GCDRO and the Helmholtz free energy in statistical physics. This theoretical analysis deepens the understanding of the problem and the proposed solution, providing a solid foundation for the proposed method.\n\nOptimization Algorithm: The paper leverages gradient flow in Wasserstein space to develop an approximate minimax optimization algorithm for GCDRO. This algorithm guarantees a bounded error ratio and standard convergence rate, providing a reliable and efficient optimization framework for GCDRO.\n\nEmpirical Evaluation: The paper includes comprehensive experiments to evaluate the performance of GCDRO compared to conventional DRO methods. The experimental results demonstrate the superiority of GCDRO in terms of prediction accuracy, parameter estimation, and generalization performance. The empirical evaluation strengthens the claims made in the paper and highlights the practical benefits of the proposed approach.\n\nPractical Significance: The paper addresses an important problem in machine learning—dealing with distributional shifts and noisy samples—and offers a practical solution that can improve the robustness and generalization of machine learning algorithms. The proposed GCDRO method has the potential to be applied in real-world scenarios where distributional shifts are prevalent, making it highly relevant and valuable.\n\nWEAKNESSES:\nLimited Comparison: The paper compares GCDRO only with conventional DRO methods, without exploring a broader range of state-of-the-art approaches or alternative methods for addressing the over-pessimism issue in DRO. Including a more diverse set of baselines would provide a more comprehensive evaluation and better contextualize the performance of GCDRO.\n\nLack of Real-world Applications: The paper focuses on theoretical analysis and empirical evaluations using synthetic or benchmark datasets. However, the absence of real-world applications or case studies limits the understanding of how GCDRO would perform in practical scenarios. Extending the evaluation to real-world datasets and applications would enhance the applicability and relevance of the proposed method.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the impact of noisy samples on Distributionally Robust Optimization (DRO) algorithms. Using a simple model and empirically, they show how DRO variants tend to give to much importance to those samples, which make them overly pessimistic in finding the worst case distribution shift. To solve this problem they propose a new method called Geometrically Calibrated DRO (GCDRO), which adds two new calibration terms. One of this term is looking at the relationship between samples (provided by a graph $G_N$) and penalizes shifted distributions $\\mathbf{q}$ which assign high probability mass to connected samples when those have very different prediction losses.The other entropy term favors shifted distributions $\\mathbf{q}$ with larger entropy, penalizing $\\mathbf{q}$ which overly focus on a subset of samples. The authors provide an algorithm to approximately solve the inner maximization loop which they use to apply their algorithms to several benchmarks in which GCDRO is outperforming prior methods. Moreover, the authors draw a parallel between their method and Helmholtz free energy.\n\nSTRENGTHS:\nOriginality: While the two added calibration terms are not entirely novel on their own, I find some novelty in combining them and (i) demonstrating a parallel with Helmholtz free energy, as well as (ii) showing how those can tackle the specific problem of noisy samples in DRO.\n\nClarity: Albeit dense, the paper is clear enough. I would maybe recommend clarifying early on how to get the graph $G_N$.\n\nQuality & significance: This work is well motivated and I can see it being impactful. Especially, the link between DRO methods and statistical physics could motivate interesting future directions.\n\nWEAKNESSES:\nThe method requires a graph $G_N$, which might not be easy to get. There should be a paragraph on how this graph was obtained for each dataset in the experiments section. In the supplementary, it is mentioned that $k$-NN graphs are used and the authors show that the results do not depends so much on $k$. While $k$-NN graphs might work on simple regression datasets, I doubt they would be efficient on more complex ones. For instance, DORO has been applied to CelebA, I'm not sure a $k$-NN graph would work there. Some more complex regression datasets can be found as part of WILDS [1]. I understand the goal of the paper is not manifold learning, yet better understanding the limitations associated with requiring $G_N$ seems important to grasp how useful the proposed method can be in practice.   \n\nWould proposition 3.3 still hold considering the error obtained from the inner maximization? \n\nReferences:\n\n[1] WILDS: A Benchmark of in-the-Wild Distribution Shifts (https://www-cs-faculty.stanford.edu/people/jure/pubs/wilds-icml21.pdf)\n\nQUESTIONS:\n- Would proposition 3.3 still hold considering the error obtained from the inner maximization?\n\nLIMITATIONS:\nThe limitations of the method are somewhat discussed in the supplementary. I like that the authors did experiments with several $k$-NN graphs, but I believe this analysis can be more convincing by testing the method on more complex regression datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to tackle the issue of overconservativeness in conventional DRO models. The proposed method incorporates data geometry properties into the design of the objective function and ambiguity set. The authors leverage the discrete geometric Wasserstein distance, initially presented in Chow et al. (2017), as a probability metric to construct the ambiguity set. Additionally, they enhance the model's performance by introducing the graph total variation quantity to the objective function. This modification effectively diminishes the influence of detrimental data points, thus mitigating their impact on the overall optimization process. \nThe effectiveness of the proposed approach is extensively validated through experiments conducted on synthetic and real datasets. The results demonstrate its superiority over conventional DRO models, exhibiting improved accuracy and robustness.\n\nSTRENGTHS:\nI acknowledge the significance of addressing the challenge posed by noisy examples in machine learning, and I commend the motivation behind the design presented in this paper. The proposed method is intriguing, and the introduction of new calibration terms appears to be a reasonable approach. The experimental results showcase promise, particularly on synthetic datasets, and also demonstrate competitiveness on real datasets.\n\nWEAKNESSES:\n1.The Discrete Geometric Wasserstein Distance restricts the worst-case distribution to have the same support as the training dataset. If this is the case, then the method you propose is fundamentally distinct from Wasserstein DRO. I am interested in understanding the specific scenarios in which Wasserstein DRO outperforms divergence-based methods. As far as I comprehend, GCDRO aims to enhance phi-divergence DRO methods by incorporating graph information. \n2. I kindly request verification of the assumption made in Proposition 3.3, which states that F(\\theta) is L-smooth. It appears that the convergence results were obtained based on this assumption alone.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}