Distributed Distributionally Robust Optimization with Non-Convex Objectives
Authors: Yang Jiao
Kai Yang
Dongjin Song
Venue: NeurIPS 2022 Accept (2022)

Abstract: Distributionally Robust Optimization (DRO), which aims to find an optimal decision that minimizes the worst case cost over the ambiguity set of probability distribution, has been applied in diverse applications, e.g., network behavior analysis, risk management, etc. However, existing DRO techniques face three key challenges: 1) how to deal with the asynchronous updating in a distributed environment;  2) how to leverage the prior distribution effectively; 3) how to properly adjust the degree of robustness according to difference scenarios. To this end, we propose an asynchronous distributed algorithm, named Asynchronous Single-looP alternatIve gRadient projEction (ASPIRE) algorithm with the itErative Active SEt method (EASE) to tackle the distributed distributionally robust optimization (DDRO) problem. Furthermore, a new uncertainty set, i.e., constrained $D$-norm uncertainty set, is developed to effectively leverage the prior distribution and flexibly control the degree of robustness. Finally, our theoretical analysis elucidates that the proposed algorithm is guaranteed to converge and the iteration complexity is also analyzed. Extensive empirical studies on real-world datasets demonstrate that the proposed method can not only achieve fast convergence, remain robust against data heterogeneity and malicious attacks, but also tradeoff robustness with performance.

Distributed Distributionally Robust Optimization with
Non-Convex Objectives
Yang Jiao
Kai Yang" Tongji University kaiyangStongji edu. on
Dongjin Song University of Connecticut dongj in. songduconn. edu
Tongji University
yangjiaoßtoonji edu. on
Abstract
Distributionally Robust Optimization (DRO), which aims to find an optimal de- cision that minimizes the worst case cost over the ambiguity set of probability distribution, has been widely applied in diverse applications, e.g., network behavior analysis, risk management, etc. However, existing DRO techniques face three key challenges: 1) how to deal with the asynchronous updating in distributed envi- ronment; 2) how leverage the prior distribution effectively; 3) how properly adjust the degree of robustness according to different scenarios. To this end, we propose asynchronous distributed algorithm, named Asynchronous Single-looP alternatIve gRadient projEction (ASPIRE) algorithm with the itFrative Active SEi method (EASE) tackle the distributed distributionally robust optimization (DDRO) problem. Furthermore, a new uncertainty set, i.e., constrained D-norm uncertainty set, developed effectively leverage the prior distribution and flex- ibly control the degree of robustness. Finally, our theoretical analysis elucidates that the proposed algorithm is guaranteed to converge and the iteration complexity is also analyzed. Extensive empirical studies on real-world datasets demonstrate that the proposed method can not only achieve fast convergence, and remain robust against data heterogeneity as well as malicious attacks, but also tradeoff robustness with performance.
Introduction
The past decade has witnessed the proliferation of smartphones and Internet of Things (IoT) devices, which generate a plethora of data everyday. Centralized machine learning requires gathering the data particular server to train models which incurs high communication overhead [46] and suffers privacy risks (3), As remedy, distributed machine learning methods have been proposed. Considering distributed system composed of N workers (devices), we denote the dataset of these workers as .Di,.... D. For the jth (1 و N) worker, the labeled dataset is given as Dj = {x] v where x IRd and yi [1..... c} denote the jth data sample and the corresponding label, respectively. The distributed learning tasks can be formulated as the following optimization problem,
min F(w) with F(w) = E, Jz(w): WEW
where w E R model parameter to be learned and W RP is a nonempty closed convex set, (-()) is the empirical risk over the jth worker involving only the local data:
fy(w)
E EXtED, [D,] (x),, vf: w),
(2)
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

where Cj is the local objective function over the 5th worker. Problem in Eq. (1) arises in numerous
areas, such as distributed signal processing [9]. multi-agen optimization [6T, etc. However, such problem does not consider the data heterogeneity [5 40 39
among different workers (i.e., data distribution of workers could be substantially different from each other (14).. Indeed, it has been shown that traditional federated approaches, such as FedAvg [3], built for independent and identically distributed (IID) data may perform poorly when applied to Non-IID data [27]. This issue can be mitigated via learning a robust model that aims to achieve uniformly good performance over all workers by solving the following distributionally robust optimization (DRO) problem in a distributed manner:
30
min max F(w,p) = -, P)f)(w), weW PEDCAN
(3)
where p = pp,... , PN] ER is the adversarial distribution in N workers, the jth entry in this vector, i..., P2 represents the adversarial distribution value for the jth worker. AN - {p R 1pp - 1} and $2 is subset of A N. Agnostic federated learning (AFL) [35 firstly introduces the distributionally robust (agnostic) loss in federated learning and provides the convergence rate for (strongly) convex functions. However, AFL does not discuss the setting of f.. DRFA-Prox [16] considers &2 = AN and imposes a regularizer on adversarial distribution to leverage the prior distribution. Nevertheless, three key challenges have not yet been addressed by prior works. First, whether it is possible to construct an uncertainty framework that can not only flexibly maintain the trade-off between the model robustness and performance but also effectively leverage the prior distribution? Second, how to design asynchronous algorithms with guaranteed convergence? Compared to synchronous algorithms, the master in asynchronous algorithms can update its parameters after receiving updates from only a small subset of workers [58] TO Asynchronous algorithms are particularly desirable in practice since they can relax striet data dependencies and ensure convergence even in the presence of device failures [58]. Finally. whether it is possible to flexibly adjust the degree of robustness? Moreover, is necessary to provide convergence guarantee when the objectives (i.e., f,,,), Vj) are non-convex.
To this end, we propose ASPIRE-EASE to effectively address the aforementioned challenges. Firstly, different from existing works, the prior distribution is incorporated within the constraint in our formulation, which can not only leverage the prior distribution more effectively but also achieve guaranteed feasibility for any adversarial distribution within the uncertainty set. The prior distribution can be obtained from side information or uniform distribution [41), which is necessary to construct the uncertainty (ambiguity) set and obtain a more robust model [16]. Specifically, we formulate the prior distribution informed distributionally robust optimization (PD-DRO) problem as:
min max E, P3 (wn,) (w;EW) PEP
s.t. == wj j= 1.
var. 2, W1. W2
2DN,
where z E RP is the global consensus variable, wy Rp is the local variable (local model parameter) of jthh worker and 2 CR is a nonempty closed convex set. P PCR is the uncertainty (ambiguity) set of adversarial distribution p. which set based on the prior distribution. To solve the PD-DRO problem in an asynchronous distributed manner, we first propose Asynchronous Single-looP alternative gRadient projEction (ASPIRE), which employs simple gradient projection steps for the update of primal and dual variables at every iteration, thus is computationally efficient. Next, the itErative Active SEt method (EASE) is employed to replace the traditional cutting plane method improve the computational efficiency and speed up the convergence. We further provide the convergence guarantee for the proposed algorithm. Furthermore, new uncertainty set, iee, constrained D-norm (CD-norm), is proposed in this paper and its advantages include: 1) it can flexibly control the degree of robustness; 2) the resulting subproblem is computationally simple; 3) it can effectively leverage the prior distribution and flexibly set the bounds for every Pj.
Contributions. Our contributions can be summarized as follows:
1. We formulate a PD-DRO problem with C'Dnnorm uncertainty set. PD-DRO incorporates the prior distribution constraints which can leverage prior distribution more effectively and guarantee ro- bustness. In addition, CD-norm is developed to model the ambiguity set around the prior distribution and it provides a flexible way control the trade-off between model robustness and performance.
2. We develop a single-loop asynchronous algorithm, namely ASPIRE-EASE, to optimize PD- DRO in an asynchronous distributed manner. ASPIRE employs simple gradient projection steps to
2

---

update the variables at every iteration, which is computationally efficient. And EASE is proposed to replace cutting plane method enhance the computational efficiency and speed up the convergence. We demonstrate that even if the objectives fj( (w;)) Vj are non-convex, the proposed algorithm is guaranteed to converge. We also theoretically derive the iteration complexity of ASPIRE-EASE.
3. Extensive empirical studies on four different real world datasets demonstrate the superior perfor- mance of the proposed algorithm. It is seen that ASPIRE-EASE can not only ensure the model's robustness against data heterogeneity but also mitigate malicious attacks.
2
Preliminaries
2.1 Distributionally Robust Optimization
Optimization problems often contain uncertain parameters. A small perturbation of the parameters could render the optimal solution of the original optimization problem infeasible or completely meaningless [3]. Distributionally robust optimization (DRO) [28 7 assumes that the probability distributions of uncertain parameters are unknown but remain in an ambiguity (uncertainty) set and aims to find decision that minimizes the worst case expected cost over the ambiguity set, whose general form can be expressed as,
min max Er(r(xr, xEĀ PEP
where E X represents the decision variable, P is the ambiguity set of probability distributions P of uncertain parameters ç. Existing methods for solving DRO can be broadly grouped into two widely-used categories [42]: 1) Dual methods [15 50 [8] reformulate the primal DRO problems as deterministic optimization problems through duality theory. Ben-Tal et
reformulate the robust linear optimization (RLO) problem with an ellipsoidal uncertainty set second-order cone optimization problem (SOCP). 2) Cutting plane methods [34 6 (also called adversarial approaches (1I) continuously solve approximate problem with finite number of constraints of the primal DRO problem, and subsequently check whether new constraints are needed to refine the feasible set. Recently, several new methods 41 29 23 have been developed to solve DRO, which need to solve the inner maximization problem at every iteration.
2.2 Cutting Plane Method for PD-DRO
In this section, we introduce the cutting plane method for PD-DRO in Eq. (4). We first reformulate PD-DRO by introducing additional variable h E H (HCR) is nonempty closed convex set) and protection function g( w. [55]. Introducing additional variable h is an epigraph reformulation 3 [6)). In this case, Eq. can be reformulated as the form with uncertainty in the constraints:
min #c #EZ,(w, (w,,),,W),.k NEE
h
S.L. Σ pf (w,,) + (((() h50,
2 w j=l,
var. 2, 11, W.. WN, h,
where p the nominal value of the adversarial distribution for every worker and (((w,)) max E, (P3 (1,) is the protection function. Eq. (6) is semi-infinite program (SIP) which PEP
contains infinite constraints and cannot be solved directly (42]. Denoting the set of cutting plane parameters in (++1)tth th iteration as A' I the following function is used to approximate g( (w,
g( (( max at f(w) = max aiEA!
Σ, (w;))
a.EA'
where a, (ar,,.. [ N denotes the parameters of 7th cutting plane in A' and f(w) [fi (wi),... fx (WN) E Substituting the protection function g((w,) }) with g( (w;) }), we can obtain the following approximate problem:
h # Z EWJ.EE
min
s.t.
Σ
(p ( h h50 Va E A
(8)
z = Wj j=l...,,
var. W1, W2 wN h.
3

---

3 ASPIRE
Distributed optimization is an attractive approach for large-scale learning tasks [54] g since it does not require data aggregation, which protects data privacy while also reducing bandwidth requirements 45]. When the neural network models (i.e. ff(w;) ) Vj are non-convex functions) are used, solving problem in Eq. (8) in a distributed manner facing two challenges: 1) Computing the optimal solution to a non-conves subproblem requires a large number of iterations and therefore is highly computationally intensive if not impossible. Thus, the traditional Alternating Direction Method of Multipliers (ADMM) is ineffective. 2) The communication delays of workers may differ significantly [1I] thus, asynchronous algorithms are strongly preferred.
To this end, we propose the Asynchronous Single-looP alternatIve gRadient projEEcion (ASPIRE). The advantages of the proposed algorithm include: 1) ASPIRE uses simple gradient projection steps to update variables in each iteration and therefore it is computationally more efficient than the traditional ADMM method, which seeks to find the optimal solution in non-convex (for wj Vj) and convex (for z and h) optimization subproblems every iteration, 2) the proposed asynchronous algorithm does not need strict synchronization among different workers. Therefore, ASPIRE remains resilient against communication delays and potential hardware failures from workers. Details of the algorithm are given below. Firstly, we define the node as master which is responsible for updating the global variable 2, and we define the node which is responsible for updating the local variable uy as worker j. In each iteration, the master updates its variables once it receives updates from at least S workers, i.e., active workers, satisfying S S S N. Q denotes the index subset of workers from which the master receives updates during (t + 1)th iteration. We also assume the master will receive updated variables from every worker at least once for each iterations. The augmented Lagrangian function of Eq. (8) can be written as:
w, Lp h+ E (T ((a))))))))))))))))))))))))))) -)) (=) Σ, R1 || w/l1?
where
h (A) ( A and Φ, Φ. Vj represent the dual variables of inequality and equality constraints in Eq. (8) respectively. A C R. and $ C R' are nonempty closed convex sets, constant K1 > 0 is a penalty parameter. Note that Eq. (9) does not consider the second-order penalty term for inequality constraint since it will invalidate the distributed optimization. Following (52), the regularized version of Eq. (9) is employed to update all variables as follows.
(p(( (,, h (A). (0,1) Lp E, of |Al E,
oe 2 110
(10)
where of and cf denote the regularization terms in (t + 1)th iteration. To avoid enumerating the whole dataset, the mini-batch loss could be used. A batch of instances with size m can be randomly sampled from each worker during each iteration. The loss function of these instances from jth
worker is given by f2),) = > 1 (x),, U5: wzy). It is evident that [w;)( = fywy and =11 m
(V; In (t + 1)th master iteration, the proposed algorithm proceeds as follows.
1) Active workers update the local variables w, as follows.
7 Pw(wf- - and Ww, w, Lp((w)) 1,5 No 15 M M..j) wj. Vj & Q4+1,
(II)
where T, is the last iteration during which worker j was active. is seen that Ej+ w w 7
and Φf 0"" L where T and constant The will be introduced below. Pw represents the projection onto the closed convex set W and we set W = {wyl| || |wy S a1), a1 is a positive constant. And then, the active workers (j E Q 1l transmit their local model parameters w/tt and loss fy())) to the master.
represents the step-size and let a = when T<T; and a The when t = Ti.
2) After receiving the updates from active workers, the master updates the global consensus variable z, additional variable h and dual variables \ as follows,
=P2(2" - n/a V (L Jaz H², (()))))) 1 - nhi V.IL (1 112 ,h' ((((())))))))))
(12)
(13)

---

= Lotrof 41 1,2
41 1 ((1T-19))) 1-1,,,, I4
(14)
where n v/g and #1 represent the step-sizes. Pz. PH and PA respectively represent the projection
onto the closed convex sets Z, H and A. We set 2 = {zl |z| >o S @1), H = (h) 0 كI 02) } and
A = {Ai] f X S 03). where 02 and ag are positive constants. A denotes the number of cutting
planes. Then, master broadcasts 2*+1, h++,, {25+1) } to the active workers.
3) Active workers update the local dual variables 3 as follows,
1 Po ( + +22 T ((w ΦJ. Vj & Q
net1 (, 2
(15)
where P2 represents the step-size and Pa represents the projection onto the closed convex set o and we set + { Φ | S a Ora is positive constant. And master can also obtain (
according to Eq. (15) It is seen that the projection operation in each step is computationally simple since the closed convex sets have simple structures [4.
4 Iterative Active Set Method
Cutting plane methods may give rise to numerous linear constraints and lots of extra message passing 55 Moreover, more iterations are required to obtain the c-stationary point when the size of a set containing cutting planes increases (which corresponds to larger M), which can be seen in Theorem I. To improve the computational efficiency and speed up the convergence, we consider removing the inactive cutting planes. The proposed itErative Active SEt method (EASE) can be divided into the two steps: during Ti iterations, 1) solving the cutting plane generation subproblem to generate cutting plane, and 2) removing the inactive cutting plane every k iterations, where k 0 a pre-set constant and can be controlled flexibly.
The cutting planes are generated according to the uncertainty set. For example, if we employ ellipsoid uncertainty set, the cutting plane is generated via solving a SOCP. In this paper, we propose CDmmoom uncertainty set, which can expressed as follows,
p= 9j SPj, E,
Pj 4
1-
(16)
Pj
where P can flexibly control the level of robustness, q ( IN) ETR represents the prior distribution, -P, and Pý (P, 2 0) represent the lower and upper bounds for p3 qj. respectively. The setting of q and P.. Vjj are based on the prior knowledge. D-norm is a classical uncertainty set (which is also called as budget uncertainty set) [5]. We call Eq. (66 CDDmnorm uncertainty set since non-negative and add up to exactly one, i... 1 p = 1. Due to the special structure of CD-nrom, the cutting plane generation subproblem is easy to solve and the level of robustness in terms of the outage probability, i.e., probabilistic bounds of the violations of constraints can be flexibly adjusted via a single parameter r. We claim that 4 I-normm (or twice total variation distance) uncertainty set is closely related to CD-norm uncertainty set. Nevertheless, there are two differences: 1) CD-norm uncertainty set could be regarded as a weighted I noom with additional constraints. 2) CD-nnrm uncertainty set can flexibly set the lower and upper bounds for every Pj (i.e,, 93 - Pi Sp S PPy+P;) while 0 o pj S Vj in 4 norm uncertainty set. Based on the C'Dmnorm uncertainty set, the cutting plane can be derived as follows,
is a probability vector so all the entries of this vector
1) Solve the following problem,
p 1+1 arg max P1. -PN
Σ, (p)
s.t.
P3 4j
<r, - 93 Sp. Vj. E,,, p1
(17)
Pj
var.
Pl PN.
where p'+1 = 1pi PN E R. Let ++ = =p p +1 P. where p = 7. ........ p) RN This
first step aims to obtain the distribution -' I by solving problem in Eq. 177) This problem can
be effectively solved through combining merge sort [13] (for sorting p,f, [w
N) with [t 1, ,N). Since N is relatively large in
few basic arithmetic operations (for obtaining (+1 p/,1,
5

---

Algorithm ASPIRE-EASE
Initialization: iteration variables {wi],, 20, h., {N/, {69)) and set A
repeat
for active worker do updates local w f+1 according Eq. 11
end for
active workers transmit local model parameters and loss to master;
master receives updates from active workers do
updates z ==+1, h { 1x++1
{Ø++1) in master according to Eq.
12
13
(44
15
master broadcasts ==+1 h 4+1 {X+1)
to active workers;
for active worker do
updates local " according to Eq. end for
15
if (L +1 mod k = and / < Ti then
master updates A ++1 according Eq. 19 and (0).. and broadcast parameters to all workers; end if'
+ until convergence
distributed system, the arithmetic complexity solving problem Eq. 17 is dominated by merge sort, which can be regarded as (N (og(N)).
2) Let f(w) (wi), fN((NN) R. check the feasibility of the following constraints:
a (()) )
max ai f(r)... arEA
(18)
3) If Eq. 188 violated, a1+1 will be added into A
A' A' Eq. 18 is violated,
A 2+1
(19)
A', otherwise,
X++1 when a new cutting plane is added, its corresponding dual variable AjA+++1 0 will be generated. After the cutting plane subproblem is solved, the inactive cutting plane will removed, that is:
{ C C c+ {ar), if 1 and X =,
A ++1
(20)
A c-1 otherwise,
where CA++ {at} is the complement of {ar} in A++1, and the dual variable will be removed. Then master broadcasts A+1 (X++)) to all workers. Details of algorithm are summarized Algorithm
Convergence Analysis
Definition (Stationarity gap) Following iteration is defined as:
52
53 32
the stationarity gap of our problem thh
oo (w) Pw - V, (,((w (Af), (1;1)))
1 - - V (gfh))
VG-=
1h'- -irno (1f), (of))))
(21)
PA (A),, (d)))))
-P +2 ((w (A), (+())))
where G is the simplified form of VVV ( ,,,, ($))).
Definition 2 (E-stationary point) (fw) ),² h², (X1), {0)1) is e-stationary point (e 2 0) of differentiable function Lp. if ||VG*|| < T(e) is the first iteration index such that |[VG Se, i.e. T(s)= -c)=miin min ||V SE..
Assumption (Smoothness/Gradient Lipschitz) I.p has Lipschitz, continuous gradients. assume that there exists L > 0 satisfying

---

- [ll/wact - Deat: 2 2; h; Acat - -Saati Deat - Eemlll:
where 0 E {[w, ; 2, ( and [:] represents the concatenation. Weat -eatt = 101; .... WN - wN E RRN,, Acat - Aeat = [ - ;AAi
>AAyy E BIA'I Dcat Deat
(0) 161-011-1 &N ( Φ
Assumption (Boundedness) Before obtaining the e-stationary point (i.e., 1 (T we assume variables in master satisfy that relative small constant. The change of the variables in master is upper bounded within T derations:
2 0, where > 0
|| =1² S STKy,, where 1 > is constant.
||ht - ht 111 S tk10, E,
Setting 1 (Bounded (A') | M, ie., an upper bound is set for the number of cutting planes.
Setting 2 (Setting of ci, cb) c 11-m11 pı(t+1jē 2 S and d2 -
pa(t+1)² 2 L2 are nonnegative non- increasing sequences, where & and C2 are positive constants and meet Ma + N
Theorem (Iteration complexity) Suppose Assumption 1 and hold. We set Nhu = 1// = 1/f
IA4T222 N-L2 and Iw Ltp.ML2+paN My62 eg(c))
And we set
Ltp [
1
constants P1 <inn L+24 15Tk)N12) and P2 L+203' respectively. For a given e, we have:
4(do + P (d + (T++))3)3J
T(e)~O(max 4M01² 2
4No2
2
P1
22
c6
2²
(2))
where d2, d2. % T, kdd d. ds, de and Ti constants. The detailed proof is given in Appendix
There exists wide array of works regarding the convergence analysis of various algorithms for nonconvex/convex optimization problems involved in machine learning 25 53 Our analysis, however, differs from existing works in two aspects. First, solve the non-convex PD-DRO in an asynchronous distributed manner. To our best knowledge, there are few works focusing on solving the DRO in distributed manner. Compared solving the non-convex PD-DRO in centralized manner, solving it in an asynchronous distributed manner poses significant challenges in algorithm design and convergence analysis. Secondly, we do not assume the inner problem can be solved nearly optimally for each outer iteration, which is numerically difficult to achieve in practice 4 Instead, ASPIRE-EASE is single loop and involves simple gradient projection operation at each step.
Experiment
In this section, we conduct experiments on four real-world datasets assess the performance of the proposed method. Specifically, we evaluate the robustness against data heterogeneity, robustness against malicious attacks and efficiency of the proposed method. Ablation study is also carried out to demonstrate the excellent performance of ESPIRE-EASE.
6.1 Datasets and Baseline Methods
We compare the proposed ASPIRE-EASE with baseline methods based on SHL 120
Person Ac-
tivity (26), Single Chest-Mounte Accelerometer (SM-AC) [9] and Fashion MNIST [51] datasets.
The baseline methods include Ind, (learning the model from an individual worker j). Mix
(learning the model from all workers with even weights using ASPIRE), FedAvg 33 AFL 335 and
DRFA-Prox [16]. The detailed descriptions of datasets and baselines are given Appendix C
In our empirical studies, since the downstream tasks are multi-class classification, the cross entropy loss is used on each worker (i.e., C.).), Vj). For SHL, Person Activity and SM-AC datasets, we adopt the deep multilayer perceptron [49 as the base model. And we use the same logistic regression model as in (35] [6] for Fashion MNIST dataset. The base models are trained with SGD. More
details are given in Appendix C Following related works in this direction [41 35 16), worst case performance are reported for the comparison of robustness. Specifically, we use Acc,, and Loss,, to represent the worst case test accuracy and training loss (i.e., the test accuracy and training loss on the worker with worst performance), respectively. We also report the standard deviation Std of
7

---

Table I: Performance comparisons based on Acc.' (%) t. Loss + and Std + (1 and 1. respectively denote higher scores represent better performance and lower scores represent better performance). The boldfaced digits represent the best results, "-" represents not available.
Model
SHI.
Person Activity
SC-MA
Fashion MNIST
7
Lossge
Std]
Ace,"
Loss,
Std]
Accu
Loss,
Std
Acc. 1
Loss,
Std.
19.0620.65
29.1
49.3840.08
8.32
22.56i80.78
17.5
Mixpovo
09.8723.10
0.800=0.018
4.81
56.31+0.09
1.16500017
3.00
49.8120211
L424400024
6.99
66.8060.18
0,78450.003
10.1
FedArg 33
09.663.07
0,802=0.023
5.21
56,28+0.3
L154=0019
49.5360.9
1441-0015
7.17
66,5840.99
0,781+9.002
10.2
AFI.
1
78:11=1.99 0.582+0:021
1081-00014
0.99
54.5640.99
1,17240018
3:50
77.32+0.15
0,70340.001
1.86
DRFA-Pro Prox 16
48
0.532:0.054
1.85
58
62662.01 16
1096:0.037
1.26
54.6130.76
L151:0009
4.69
77.9510.51
0,70240.007
1.34
ASPIREEEASE
79.16=1.13
0.515=00.019
1,02
59.4360044
1053500000
0.82
56,31+0.29
1.12660021
3.16
78.82+1.17
0.69%6490004
1,01
ASPIRF-FASE
78.9-1.27
0.521=0.023
1.36
59.54+0.21
1051=00016
0.79
56.71#0.16
1.11900.028
3.48
78.7540.06
0.69840.006
1.09
[Ace1, ... Acex] (the test accuracy on every worker). In the experiment, S is set as 1, that means the master will make an update once it receives message. Each experiment is repeated 10 times, both mean and standard deviations are reported. We implement our model with PyTorch and conduct all the experiments on server with two TITAN V GPUS.
6.2 Results
Robustness against Data Heterogeneity. We first assess the robustness of the proposed ASPIRE- EASE by comparing it with baseline methods when data are heterogeneously distributed across different workers. Specifically, we compare the Acc.. Loss and Std of different methods on all datasets. The performance comparison results are shown in Table In this table, we can observe that max[Indy). which represents the best performance of individual training over all workers, exhibits the worst robustness on SHI., Person Activity, and SC-MA. This is because individual training (max (nnx)Tnd))) }) only learns from the data in its local worker and cannot generalize to other workers due to different data distributions. Note that max [Ind,] } is unavailable for Fashion MNIST since each worker only contains one class of data and cross entropy loss cannot be used in this case. max{Ind [Id,]] } also does not have Loss since Indy is trained only on individual worker j. The FedAvg and Mix exhibit better performance than max[Ind,] since they consider the data from all workers. Nevertheless, FedAvg and Mixeven only assign the fixed weight for each worker. AFL is more robust than FedAvg and Mixgven since it not only utilizes the data from all workers but also considers optimizing the weight of cach worker. DRFA-Prox outperforms AFL since it also considers the prior distribution and regards it as a regularizer in the objective function. Finally, we can observe that the proposed ASPIRE-EASE shows excellent robustness, which can be attributed to two factors: 1) ASSIRE-EESSS considers data from all workers and can optimize the weight of each worker; 2) compared with DRFA-Prox which uses prior distribution as regularizer, the prior distribution is incorporated within the constraint in our formulation (Bq. 4. which can be leveraged more effectively. And it is seen that ASPIRE-EASE can perform periodic communication since ASPIRE-EASE, which represents ASPIRE-EASE with periodic communication, also has excellent performance.
Within ASPIRE-EASE, the level of robustness can be controlled by adjusting I'. Specially, when I'= O, we obtain a nominal optimization problem in which no adversarial distribution considered. The size of the uncertainty set will increase with I' (when I' S NV, which enhances the adversaiial robustness of the model. As shown in Figure the robustness of ASPIRE-EASE can be gradually enhanced when I increases. More results are available in Figure C2 of Appendix C
Robustness against Malicious Attacks. To assess the model robustness against malicious attacks, malicious workers with backdoor attacks I 48), which attempt to mislead the model training process, added to the distributed system. Following [14] we report the success attack rate of backdoor attacks for comparison. It can be calculated by checking how many instances in the backdoor dataset can be misled and categorized into the target labels. Lower success attack rates indicate more robustness against backdoor attacks. The comparison results are summarized in Table ala more detailed settings of backdoor attacks are available in Appendix C In Table we observe that AFL can be attacked easily since it could assign higher weights to malicious workers. Compared to AFL, FedAvg and Mixgven achieve relatively lower success attack rates since they assign equal weights to the malicious workers and other workers. DRFA-Prox can achieve even lower success attack rates since it can leverage the prior distribution to assign lower weights for malicious workers. The proposed ASPIRE-EASE achieves the lowest success attack rates since it can leverage the prior distribution more effectively. Specifically, it will assign lower weights to malicious workers with tight theoretical guarantees.
8

---

SORE CAe ACCHTO TM4 18 1
1ea1 Core Trannro Lie.
DRFA-IES
12
13 D L3 28,25
4000
2230
1548
30300
2000
(a) Person Activity
(b) SC-MA
(a) Person Activity
SC-MA
I control the degree of robustness (worst case performance in the problem) on (a)
Figure 1:
Figure 2: Comparison of the convergence time worst worker (a) Person Activity, (b) SC-MA datasets.
Person Activity, (b) SC-MA datasets.
Table 2: Performance comparisons about the suc- cess attack rate (%) 4. The boldfaced digits repre- sent the best results.
Marter d
Model
SHL
Person Activity
SC-MA
Fushion MNIST
Mixriss
36.21a2.23
34.32a2.18
52.14
83.18:2.07
bodAvg 35
38,15+3.02
33,25=2,49
55.3963.13
82.04=1.4
(a) Person Activity
(b) SC-MA
68.6344.24
43.662387
758164.03
9000:2252
DRFA-Prox
16
21.23:3.63
27.2763.11
30.79:3.65
63.24::247
Figure 3:
Comparison of ASPIRE-CP CP and
ASPIRE-EASE
9.17a1.65
22.36x2.35
14.51s3.21
45.10s1.64
ASPIRE-EAE EASE regarding the number of cutting
planes on (a) Person Activity, (b) SC-MA datasets.
Efficiency. In Figure 2 we compare the convergence speed of the proposed ASPIRE-EASE with AFL and DRFA-Proo by considering different communication and computation delays for each worker. The proposed ASPIRE-EASE has two variants, ASPIRE-CP (ASPIRE with cutting plane method), ASPIRE-ER ASPPRE-EASEO-) available in Figure C3 of Appendix C Based on the comparison, we can observe that the proposed ASPIRE-EASE generally converges faster than baseline methods and its variants. This is because 1) compared with AFI., DRFA-Prox, and ASPIRE-FASE()) ASPIRE-EASE is an asynchronous algorithm in which the master updates its parameters only after receiving the updates from active workers instead of all workers: 2) unlike DRFA-Prox, the master in ASPIRE-EASE only needs to communicate with active workers once per iteration; 3) compared with ASPIRE-CP, ASPIRE-EASE utilizes active set method instead of cutting plane method, which is more efficient. It is seen from Figure 2 that, the convergence speed of ASPIRE-EASE mainly benefits from the asynchronous setting.
EASE without asynchronous setting). More results
Ablation Study. For ASPIRE, compared with cutting plane method, EASE is more efficient since it considers removing the inactive cutting planes. To demonstrate the efficiency of EASE, we firstly compare ASPIRE-EASE with ASPIRE-CP concerning the number of cutting planes used during the we can observe that ASPIRE-EASE thus is more efficient. The convergence speed of ASPIRE-EASE and ASPIRE-CP in Figure also suggests that ASPIRE-EASE converges much faster than ASPIRE-CP. More results are available in Figure and C4
training. In Figure 3
uses fewer cutting planes than ASPIRE-CP,
Appendix C
Conclusion
In this paper, we present ASPIRE-EASE method to effectively solve the distributed distributionally robust optimization problem with non-convex objectives. In addition, C'D-norm uncertainty has been proposed to effectively incorporate the prior distribution into the problem formulation, which allows for flexible adjustment of the degree of robustness of DRO. Theoretical analysis has also been conducted to analyze the convergence properties and the iteration complexity of ESPIRE-EASE ASPIRE-EASE exhibits strong empirical performance on multiple real-world datasets and is effective in tackling DRO problems in a fully distributed and asynchronous manner. In the future work, more uncertainty sets could be designed for our framework and more update rule for variables ASPIRE could be considered.

---

Acknowledgments and Disclosure of Funding
The work Yang Jiao and Kai Yang was supported in part by the Fundamental Research Funds for the Central Universities of China, in part by the Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS), in part by the National Natural Science Foundation of China under Grant 61771013, and in part by the Fundamental Research Funds of Shanghai Jiading District.
References
[1] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938-2948. PMLR, 2020.
[2] A. Ben-Tal and A. Nemirovski. Robust solutions of uncertain linear programs, Operations research letters, 25(1):1-13, 1999.
[3] A. Ben-Tal L. El Ghaoui, and A. Nemirovski. Robust optimization. Princeton university press. 2009.
[4] D. P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3): 334-334, 1997.
[5] D. Bertsimas and M. Sim. The price robustness. Operations research, 52(1):35-53, 2004.
[6] D. Bertsimas, I. Dumming, and M. Lubin. Reformulation versus sutting-planes for robust optimization. Computational Management Science, 13(2):15-2217 2016.
[7] J. Blanchet and K. Murthy. Quantifying distributional model risk via optimal transport. Mathe- matics of Operations Research, 44(2):565-600, 2019.
[8] L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.
[9] Casale, O. Pujol, and P. Radeva. Personalization and user verification in wearable systems using biometric walking patterns. Personal and Ubiquitous Computing, 16(5):563-580, 2012.
[10] T.-H. Chang, M. Hong, W.-C. Liao, and X. Wang. Asynchronous distributed ADMM for large-scale optimization Part I: Algorithm and convergence analysis. IEEE Transactions on Signal Processing, 64(12):3115 11:::311121111 8-3130, 2016.
[11] Y. Chen, Y. Ning. M. Slawski, and H. Rangwala. Asynchronous online federated learning for edge devices with Non-IID data. 2020 IEEE International Conference on Big Data (Big Data), pages 15-24. IEEF, 2020.
[12] A. Cohen, A. Daniely, Y. Drori, T. Koren, and M. Schain. Asynchronous stochastic optimization robust to arbitrary delays. Advances Neural Information Processing Systems, 34:9024-9035, 2021.
[13] R. Cole. Parallel merge sort. SIAM Journal Computing, 17(4):770-785, 1988.
[14] J. Dai, C. Chen, and Y. Li. A backdoor attack against LSTM--based text classification systems. IEEE Access, 7:138872-138878, 2019.
[15] E. Delage and Y. Ye. Distributionally robust optimization under moment uncertainty with application data-driven problems. Operations research, 58(3):595-612 612, 2010.
[16] Y. Deng, M. M. Kamani, and M. Mahdavi. Distributionally robust federated averaging. arXiv preprint 10t.:2002.12600, 2021.
[17] J. C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics,, (0(3):1378-144 2021.
[18] R. Gao and J. Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. arXiv preprint 17Xiv:160.002199 2016.
10

---

[19] G. Geraci, M. Wildemeersch, and T. Q. Quek. Energy efficiency of distributed signal processing in wireless networks: A cross-layer analysis. IEEE Transactions on Signal Processing, 64(4): 1034-1047, 2015.
[20] H. Gjoreski, M. Ciliberto, L. Wang, F. J. O. Morales, S. Mekki, S. Valentin, and D. Roggen. The university of oussex-huawel locomotion and transportation dataset for multimodal analyties with mobile devices. IEEE Access, 6:42592-426600 2018.
[21] B. L. Gorissen, I. Yankooll,, and D. den Hertog. A practical guide to robust optimization. Omega, 53:124-13 137, 2015.
[22] Haddadpour, M. M. Kamani, M. Mahdavi, and A. Karhasi. Learning distributionally robust models at scale via composite optimization. arXiv preprint arXiv:2203.09660 2022.
[23] Y. Hu, X. Chen, and N. He. On the bias-variance-com tradeoff of stochastic optimization. Advances in Neurul Information Processing Systems, 34, 2021.
[24] Jiang, W. Zhang, Gu, and W. Zhu. Asynchronous decentralized online learning. Advances in Neural Information Processing Systems, 34:20185-200 20196, 2021.
[25] C. Jin, P. Netrapalli, and M. Jordan. What is local optimality in nonconvex-nonconcaa minimax optimization? In International Conference on Machine Learning, pages 1880-4882 PMLR, 2020.
[26] B. Kaluža, V. Mirchevska, E. Dovgan, M. Luštrek, and M. Gams. An agent-based approach to care in independent living. International joint conference on ambient intelligence, pages 177-186. Springer, 2010.
[27] P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh. SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning. 2019.
[28] D. Kuhn, P. M. Esfahani, V. A. Nguyen, and Shafeezzdeh-
Wasserstein distribution- ally robust optimization: Theory and applications in machine learning. In Operations Research Management Science in the Age of Analytics, pages 130-166. INFORMS, 2019.
[29] D. Levy, Carmon, J. C. Duchi, and A. Sidford. Large- scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33:8847-8861 2020.
[30] W.-H. Liao and Y.-T. Huang. Investigation of DNN model robustness using heterogeneous datasets. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4393- 4397. IEFE, 2021.
[31] T. Lin, C. Jin, and M. Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Muchine Learning, pages 6083-6093. PMLR, 2020.
[32] S. Lu, I. Tsaknakis, M. Hong, and Chen. Hybrid block successive approximation for one- sided non-convex min-max problems: algorithms and applications. JEEE Transactions on Signal Processing, 68:3676-3691, 2020.
[33] B. McMahan, F. Moore, D. Ramage, S. Hampson, and B. A. y Areas. Communication-effcieenn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[34] S. Mehrotra and D. Papp. A cutting surface algorithm for semi-infinite convex programming with application to moment robust optimization. SIAM Journal on Optimization, 24(4): 1670-1697, 2014.
[35] M. Mohri, G. Sivek, and T. Suresh. Agnostic federated learning. International Conference on Machine Learning, pages 4615-4625, PMLR, 2019.
[36] A. Nedic and A. Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Trunsactions on Automatic Control, 54(1):48-61, 2009.
[37] Y. Nesterov. Introductory lectures on corvex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.

---

[38] Q. Qi, Z. Guo, Y. Xu, R. Jin, and T. Yang. An online method for a class of distributionally robust optimization with non-convex objectives. Advances in Neural Information Processing Systems, 34: 10000000700000 2021.
[39] J. Qian, X. Fafoutis, and L. K. Hansen. Towards federated learning: Robustness analyties to data heterogeneity. arXiv preprint arXiv:2002.05038, 2020.
[40] Qian, L. K. Hansen, X. Fafoutis, Tiwari, and H. M. Pandey. Robustness analytics to data heterogeneity in edge computing. Computer Communications,
2020.
[41] Q. Qian, S. Zhu, J. Tang, R. Jin, B. Sun, and H. Li. Robust optimization over multiple domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4739-4746, 2019.
[42] H. Rahimian and S. Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv: 05659, 2019.
[43] S. Sicari, A. Rizzardi, L. A. Grieco, and A. Coen-Porisini. Security, privacy and trust in Internet of Things: The road ahead. Computer networks, 76:146-164, 2015.
[44] K. Singhal, H. Sidahmed, Z. Garrett, S. Wu, J. Rush, and S. Prakash. Federated reconstruction: Partially local federated learning. Advances in Neural Information Processing Systems, 34, 2021.
[45] T. Subramanya and R. Riggio. Centralized and federated learning for predictive VNF autoscal- ing in multi-domain 5G networks and beyond. IEFE Transactions on Network and Service Management, 18(1 18(1):63-78, 2021.
[46] J. Sun, T. Chen, G. B. Giannakis, and Z. Yang. Communication-effccieent distributed learning via lazily aggregated quantized gradients. arXiv preprini 1rXiv:1909.07583 2019.
[47] K. K. Thekumparampil, P. Jain, Netrapalli, and S. Oh. Efficient algorithms for smooth minimax optimization. Advances in Neural Information Processing Systems, 32, 2019.
[48] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEFE, 2019.
[49] Z. Wang, W. Yan, and T. Oates. Time series classification from scratch with deep neural networks: strong baseline. 2017 International joint conference on neural networks (IJCNN), pages 1578-1585. IEER, 2017.
[50] W. Wiesemann, D. Kuhn, and B. Rustem. Robust Markov decision processes. Mathematics of Operations Reseurch, 381))1533-183 2013.
[51] H. Xiao, K. Rasul, and R. Vollgraf. Dashion-MNIST. novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: / 170..07747, 2017.
[52] 7.. Xu, H. Zhang, Y. Xu, and G. Lan. A unified single-loop alternating gradient projection algorithm for nonconvex-come concave and convex-noneoncave minimax problems. arXiv preprint arXiv:2006.02032 2020.
[53] Z. Xu, J. Shen, Z. Wang, and Y. Dai. Zeroth-order alternating randomized gradient projection algorithms for general nonconvex-concave minimax problems. arXiv preprint arXiv:2108.00 004733 2021.
[54] K. Yang, Y. Wu, J. Huang, X. Wang, and S. Verdú. Distributed robust optimization for communi- cation networks. IEEE INFOCOM 2008-The 27th Conference on Computer Communications, pages 1157-1165. IEEE, 2008
[55] K. Yang, J. Huang, Y. Wu, X. Wang, and M. Chiang. Distributed robust optimization (DRO), part I: Framework and example. Optimization and Engineering, 15(1):35-67, 2014.
[56] Í. Yanıkoğlu, B. I.. Gorissen, and D. den Hertog. survey of adjustable robust optimization. European Journal of Operational Research, 279 2019.
12

---

[57] S. Zawad, A. Ali, P.-Y. Chen, A. Anwar, Y. Zhou, N. Baracaldo, Y. Tian, and F. Yan. Curse or redemption? how data heterogeneity affects the robustness federated learning. arXiv preprint arXiv:2102.00633 2021.
[58] R. Zhang and J. Kwok. Asynchronous distributed ADMM for consensus optimization. In International conference on machine learning, pages 1701-1709. PMLR, 2014.
[59] X. Zhou. On the fenchel duality between strong convexity and lipschitz continuous gradient. arXiv preprint 1r83vv 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative societal impacts of your work? [N/A] There is no potential negative societal impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 5
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A and
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] The references
of the data used in this paper added in Section 6.1
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Section C.2
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section 6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 6.1
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "fL-qjfk8Ecp", "reviewer": "Reviewer_CFN7", "summary": "The authors propose three challenges that existing DRO techniques face. To this end, they propose an asynchronous distributed algorithm to tackle the distributed distributionally robust optimization problem. Moreover, they propose a new uncertainty set which is developed to effectively leverage the prior distribution and flexibly control the degree of robustness. They provide theoretical guatantees for convergence of the algorithm. Extensive empirical studies corroborate their theoretical results.", "questions": "1. Can the authors elaborate more on the assumptions in the theoretical analysis?\n2. Can the authors explain to what aspects the proposed algorithm address the challenge stated in line 6?", "limitations": "1. The authors did not explain whether the assumptions made in theoretical analysis is practical.\n2. To my knowledge, I do not understand why the proposed algorithm address the challenge stated in line 6. Can the authors explain more clearly?", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:  \n1. They propose an algorithm with theoretical guarantees.\n2. They conduct experiments to verify their statement. \n\nWeakness:  \nThey make  numerous assumptions in the theoretical analysis, but did not explain whether these assumptions are practical.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "ZfBa06hCWAK", "reviewer": "Reviewer_X3TP", "summary": "In this paper the authors introduce an algorithm to solve the Distributionally Robust Optimization problem in an asynchronous and distributed fashion where the data has been spread across multiple workers.  \nSpecifically, they focus on a DRO problem where the distribution has finite support (number of workers) and formulate it as a single master problem with finite number of constraints. They then use an augmented lagrangian algorithm to solve the problem where each worker updates the components of the augmented lagrangian using the data available to it. \nThe provide a convergence analysis for their method and illustrate their results through numerical experiments. ", "questions": "1. What is the novel about the CD-norm uncertainty set. It seems essentially the same as the regular budget uncertainty set. The probability constraints sum p_i = 1, don't seem to have any impact on the analysis. \n2. I think the authors to better emphasize the asynchronous aspect of their algorithm. I believe this is hte novel aspect of their method but is not emphasized especially in the numerical methods. ", "limitations": "I think the authors can better address the limitations of their work especially the limitations imposed by the  the specific budget uncertainty set that they use. It would also be interesting to see if there are any differences in the type of solution obtained by there method as compared to other methods. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The provide a method for solving DRO that can leverage the parallelism present in modern computer systems. \n2. Their assumptions and numerical experiments allow their method to be applicable to a wide variety of settings including popular machine learning methods. \n3. They provide a convergence analysis which shows that their method will converge to a stationary point. \n\nWeaknesses\n1. The uncertainty set used is quite common and the analysis using it is pretty standard. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AY4LdSoOQoj", "reviewer": "Reviewer_fDdQ", "summary": "This paper proposed a distributed distributionally robust optimization method. Both theoretical convergence rate and empirical results are provided. \n\nHowever, the convergence rate is much worse than existing methods. ", "questions": "- In recent years, many optimization algorithms for nonconvex problems have been proposed and they can achieve a much faster convergence rate than this method, e.g. [1] [2].  However, the authors failed to discuss and compare them. \n- It would be good if the authors could use a table to compare the convergence rate of this method and existing methods, including the centralized ones. \n- As for asynchronous optimization, a critical hyperparameter is the communication delay, which heavily affects the convergence rate. But in your convergence rate, it is not clear how the communication delay affects it.\n- It would be good if the authors could discuss the challenges in the theoretical analysis compared with the centralized method. \n- What benefits to introducing $h$? Is it always nonnegative? In Table 1, what loss values did you report? $h$ or $\\sum p_j f(w_j)$? Similarly, in figure 2, if it is $\\sum p_j f(w_j)$, which is a min-max problem, why is it always decreasing?\n- In table 1, FedAvg and DRFA-Prox perform periodic communication. But your method performs communication in each iteration. Hence, the comparison is not fair. \n \n\n\n[1] An online method for a class of distributionally robust optimization with non-convex objectives\n\n[2] Learning Distributionally Robust Models at Scale via Composite Optimization", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- This paper is well organized. \n- Provide extensive experimental results. \n\nCons:\n- In recent years, many optimization algorithms for nonconvex problems have been proposed and they can achieve a much faster convergence rate than this method, e.g. [1] [2].  However, the authors failed to discuss and compare them. \n- It would be good if the authors could use a table to compare the convergence rate of this method and existing methods, including the centralized ones. \n- As for asynchronous optimization, a critical hyperparameter is the communication delay, which heavily affects the convergence rate. But in your convergence rate, it is not clear how the communication delay affects it.\n- It would be good if the authors could discuss the challenges in the theoretical analysis compared with the centralized method. \n- What benefits to introducing $h$? Is it always nonnegative? In Table 1, what loss values did you report? $h$ or $\\sum p_j f(w_j)$? Similarly, in figure 2, if it is $\\sum p_j f(w_j)$, which is a min-max problem, why is it always decreasing?\n- In table 1, FedAvg and DRFA-Prox perform periodic communication. But your method performs communication in each iteration. Hence, the comparison is not fair. \n \n\n\n[1] An online method for a class of distributionally robust optimization with non-convex objectives\n\n[2] Learning Distributionally Robust Models at Scale via Composite Optimization", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept