Title: Zeroth-Order Methods for Nondifferentiable, Nonconvex, and Hierarchical Federated Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Nonsmooth Optimization

Nonconvex Optimization

Bilevel Optimization
Abstract: Federated learning (FL) has emerged as an enabling framework for communication-efficient decentralized training. We study three broadly applicable problem classes in FL: (i) Nondifferentiable nonconvex federated optimization; (ii) Federated bilevel optimization; (iii) Federated minimax problems. Notably, in an implicit sense, both (ii) and (iii) are instances of (i). However, the hierarchical problems in (ii) and (iii) are often complicated by the absence of a closed-form expression for the implicit objective function. Unfortunately, research on these problems has been limited and afflicted by reliance on strong assumptions, including the need for differentiability and L-smoothness of the implicit function. We address this shortcoming by making the following contributions. In (i), by leveraging convolution-based smoothing and Clarke’s subdifferential calculus, we devise a randomized smoothing-enabled zeroth-order FL method and derive communication and iteration complexity guarantees for computing an approximate Clarke stationary point.  To contend with (ii) and (iii), we devise a unified randomized implicit zeroth-order FL framework, equipped with explicit communication and iteration complexities. Importantly, our method utilizes delays during local steps to skip making calls to the inexact lower-level FL oracle. This results in significant reduction in communication overhead when addressing hierarchical problems. We empirically validate the theory on nonsmooth and hierarchical ML problems.

Zeroth-Order Methods for Nondifferentiable, Nonconvex, and Hierarchical Federated Optimization

Abstract

Federated learning (FL) has emerged as an enabling framework for communicationefficient decentralized training. We study three broadly applicable problem classes in FL: (i) Nondifferentiable nonconvex federated optimization; (ii) Federated bilevel optimization; (iii) Federated minimax problems. Notably, in an implicit sense, both (ii) and (iii) are instances of (i). However, the hierarchical problems in (ii) and (iii) are often complicated by the absence of a closed-form expression for the implicit objective function. Unfortunately, research on these problems has been limited and afflicted by reliance on strong assumptions, including the need for differentiability and L-smoothness of the implicit function. We address this shortcoming by making the following contributions. In (i), by leveraging convolution-based smoothing and Clarke's subdifferential calculus, we devise a randomized smoothing-enabled zeroth-order FL method and derive communication and iteration complexity guarantees for computing an approximate Clarke stationary point. To contend with (ii) and (iii), we devise a unified randomized implicit zeroth-order FL framework, equipped with explicit communication and iteration complexities. Importantly, our method utilizes delays during local steps to skip making calls to the inexact lower-level FL oracle. This results in significant reduction in communication overhead when addressing hierarchical problems. We empirically validate the theory on nonsmooth and hierarchical ML problems.

1 Introduction

Federated learning (FL) has recently emerged as a promising enabling framework for learning predictive models from a multitude of distributed, privacy-sensitive, and possibly, heterogeneous datasets. This is accomplished through the use of efficiently devised periodic communications between a central server and a collection of clients. The FL algorithmic framework allows for addressing several key obstacles in the development and implementation of standard machine learning methods in a distributed and parallel manner. For instance, the conventional parallel stochastic gradient descent (SGD) method requires the exchange of information among the computing nodes at every single time step, resulting in excessive communication overhead. In contrast, FL methods including FedAvg [34] and Local SGD [46] overcome this onerous communication bottleneck by provably attaining the linear speedup of parallel SGD by using a significantly fewer communication rounds [18, 54, 24, 7]. These guarantees have been further complemented by recent efforts [26,

38] where the presence of both data heterogeneity (i.e., variability of local datasets) and device heterogeneity (i.e., variability of edge devices in computational power, memory, and bandwidth) have been addressed. Despite recent advances, much needs to be understood about designing communication-efficient decentralized methods for resolving three broadly applicable problem classes, each of which is presented next.

(a) Nondifferentiable nonconvex locally constrained FL. Consider the prototypical FL setting:

where f is a nonsmooth nonconvex function and is associated with a group of m clients indexed by i ∈ [ m ] ≜ { 1 , . . . , m } , D i denotes the local dataset, ˜ f i : R n ×D i → R is the local loss function, and X i ⊆ R n is an easy-to-project local constraint set. Notably, local datasets may vary across clients, allowing for data heterogeneity. We also consider client-specific local sets to induce personalization.

(b) Nondifferentiable nonconvex bilevel FL. Overlaying a bilevel term in ( FL nn ) leads to

where f ( · ) ≜ 1 m ∑ m i =1 E ξ i ∈D i [ ˜ f i ( · , y ( · ) , ξ i ) ] denotes the implicit objective function and y ( · ) : R n → R ˜ n is a single-valued map returning the unique solution to the lower-level problem at x .

(c) Nondifferentiable nonconvex minimax FL. Finally, we consider the minimax setting, defined as

where we assume that y ( x ) ∈ arg max y ∈ R ˜ n 1 m ∑ m i =1 E ζ i ∈ ˜ D i [ ˜ f i ( x, y, ξ i )] is unique for all x . Let f ( · ) ≜ 1 m ∑ m i =1 E ξ i ∈D i [ ˜ f i ( · , y ( · ) , ξ i ) ] denote the implicit objective function. Indeed, problem ( FL bl ) subsumes this minimax formulation when we choose ˜ h i := -˜ f i and ˜ D i := D i .

Notably, in an implicit sense, both (b) and (c) are instances of problem (a). However, these hierarchical problems are often complicated by the absence of a closed-form expression for the implicit objective, denoted by f ( · ) . Indeed, f ( · ) is often nonsmooth, nonconvex, and unavailable. As such, the absence of both zeroth and first-order information of f ( · ) in problems (b) and (c) makes the design and analysis of FL methods for these problems more challenging than that for (a).

Gaps. To the best of our knowledge, there are no known efficient FL algorithms that can contend with both nonsmoothness and nonconvexity in an unstructured sense. Generalizations that can accommodate either a bilevel or a minimax structure also remain unaddressed in FL.

Goal. To develop a unified FL framework accommodating nondifferentiable nonconvex settings with extensions allowing for bilevel or minimax interactions . We now describe the proposed framework.

1.1 A Smoothed Sampled Zeroth-order Framework

We consider a smoothed framework for contending with constrained, nonsmooth, and nonconvex regimes. Specifically, given that f is an expectation-valued function and X is a closed and convex set, both of which are defined in ( FL nn ), a smoothed unconstrained approximation is given as follows.

If f is as defined in ( FL nn ) and d ( x ) ≜ 1 m ∑ m i =1 I X i ( x ) , then f and its smoothing f η are defined as

(i) Clarke-stationarity . Consider the original problem ( FL nn ). Under the assumption that the objective of ( FL nn ) is Lipschitz continuous, then Clarke-stationarity of x w.r.t. ( FL nn ) requires that x satisfies 0 ∈ ∂f ( x ) + N ∩ i m =1 X i ( x ) , where ∂f ( x ) represents the Clarke generalized gradient [3] of f at x . However, a negative result has been provided regarding the efficient computability of an ϵ -stationary solution in nonsmooth nonconvex regimes [56]. Consequently, we focus on the smoothed counterpart ( FL η nn ), a smooth nonconvex problem. In fact, under suitable conditions, it can be shown that stationary point of ( FL η nn ) is a 2 η -Clarke stationary point of the original problem, i.e.

where ∂ 2 η f ( x ) represents the 2 η -Clarke generalized gradient of f at x .

(ii) Meta-scheme for efficient resolution of ( FL η nn ) . We develop zeroth-order stochastic gradient schemes for resolving ( FL η nn ). This requires a zeroth-order gradient estimator for f η ( x ) , denoted by 1 m ∑ m i =1 g η i ( x, ξ i , v i ) where v i ∈ η S for i ∈ [ m ] and S denotes the surface of the unit ball. Note that the Moreau smoothing of the indicator function of X i , denoted by I η X i ( x ) , admits a gradient, defined as ∇ x I η X i ( x ) = 1 η ( x -P X i ( x )) , where P X i ( x ) ≜ arg min y ∈ X i ∥ y -x ∥ 2 . The resulting meta-scheme is defined next.

(iii) Inexact implicit generalizations for ( FL bl ) and ( FL mm ) . In addressing the bilevel problem, unlike in ( FL nn ), the clients in ( FL bl ) may not have access to the exact evaluation of the implicit local objective ˜ f i ( · , y ( · ) , ξ i ) . This makes a direct extension of FL schemes challenging. This is because the evaluation of the implicit local function necessitates exact resolution of the lower-level problem. We address this challenge by developing inexact implicit variants of the zeroth-order scheme, where clients compute only an ε -approximation of y ( x ) , denoted by y ε ( x ) in a federated fashion. This inexact framework, described next, is crucial in addressing hierarchy in bilevel FL formulations. Let f η i ( · ) denote the smoothed implicit local function. We estimate ∇ x f η i ( x ) by approximating the expectation by sampling in ( a. 1) , as follows, while in ( a. 2) , we replace y ( x ) by an inexact form y ε ( x ) . This leads to the introduction of g η,ε i ( x, ξ, v ) as captured below.

where k is the local time index, T r is the global time index of communication round r ( k ≥ T r ), and g η,ε i ( x, ξ, v ) ≜ n η ( ˜ f i ( x + v, y ε ( x + v ) , ξ ) -˜ f i ( x, y ε ( x ) , ξ )) v ∥ v ∥ denotes an inexact implicit zeroth-order gradient. Note that at each round of communication at the upper level, y ε ( x ) can be computed using calls to a standard FL method, e.g., FedAvg, in the lower level. Notably, such calls to an FL oracle should be made only at the global step to preserve the communication efficiency of the scheme. It follows that g η,ε i ( x, ξ i,k , v T r ) = ∇ x f η i ( x ) + ˜ e i,ε where the approximation error ˜ e i,ε is a possibly biased random variable. This bias can be then controlled by carefully by updating the accuracy level ε at each communication round, as we will address in this work.

1.2 Contributions

Our goal lies in extending ( Meta-ZO ) to federated nonsmooth nonconvex optimization and then provide generalizations to bilevel and minimax regimes. In each instance, we intend to provide iteration and communication-complexity bounds for computing an ϵ -accurate η -Clarke stationary point of the original problem. Accordingly, we make the following contributions.

(i) FL for nondifferentiable nonconvex problems. To address ( FL nn ) with heterogeneous datasets, we develop a Randomized Zeroth-Order Locally-Projected Federated Averaging method (FedRZO nn ). We derive iteration complexity of O ( 1 mϵ 2 ) and communication complexity of O ( m 3 / 4 K 3 / 4 ) for computing an approximate Clarke stationary point. Such guarantees appear to be new in the context of resolving nondifferentiable nonconvex FL problems, e.g. in training of ReLU neural networks (see Table 2). This is distinct from existing zeroth-order methods, including FedZO [12], that rely on differentiability and L -smoothness of the local loss functions.

(ii) Federated bilevel optimization. In addressing ( FL bl ), we develop FedRZO bl , an inexact implicit extension of FedRZO nn . By skipping local calls to the lower-level FL oracle, FedRZO bl is a

novel communication-efficient FL scheme with single-timescale local steps, resulting in significant reduction in communication overhead. Table 1 summarizes the communication complexity of this scheme. In all cases, we assume heterogeneous data at the upper level. In the lower level, depending on which conventional FL scheme is employed, we obtain the communication complexity accordingly.

(iii) Federated minimax optimization. FedRZO bl can be employed for addressing ( FL mm ) where ˜ h i := -˜ f i . As such, the complexity results in (ii) hold for solving (nondifferentiable nonconvex)(strongly concave) FL minimax problems. Such results are new for this class of FL problems.

Remark 1. There has been recent progress in addressing bilevel and minimax problems in FL, including Local SGDA and FedNest [48, 43]. Our work in (ii) and (iii) has two main distinctions with existing FL methods, described as follows. (1) We do not require the differentiability and L -smoothness of the implicit objective function. This assumption may fail to hold, e.g., in constrained hierarchical FL problems. (2) The existing FL methods for bilevel and minimax problems assume that the lower-level problem is unconstrained. In fact, even in centralized regimes, addressing hierarchical problems where the lower-level constraints depend on x have remained challenging. For example, consider the problem min x ∈ [ -1 , 1] max y ∈ [ -1 , 1] , x + y ≤ 0 x 2 + y that admits a unique solution ( x ∗ , y ∗ ) = (0 . 5 , -0 . 5) . Now consider a reversal of min and max in this problem, i.e., max y ∈ [ -1 , 1] min x ∈ [ -1 , 1] , x + y ≤ 0 x 2 + y, admitting the unique solution ( x ∗ , y ∗ ) = ( -1 , 1) . As a consequence, the well-known primal-dual gradient methods, that have been extensively employed for addressing minimax problems with independent constraint sets, may fail to converge to a saddle-point in minimax problems with coupling constrains. Our proposed algorithmic framework allows for accommodating these challenging problems in FL.

2 Related work

(i) Nondifferentiable nonconvex optimization. Nonsmooth and nonconvex optimization has been studied extensively with convergence guarantees to Clarke-stationary points via gradient sampling [1, 2] and difference-of-convex approaches [5]. Most complexity and rate guarantees necessitate smoothness of the nonconvex term [14, 51, 8, 9, 28] or convexity of the nonsmooth term [13], while only a few results truly consider nonsmooth nonconvex objective function [32, 4, 42]. (ii) Nondifferentiable nonconvex federated learning. The research on FL was initially motivated by decentralized neural networks where local functions are nondifferentiable and nonconvex [34]. Nevertheless, theoretical guarantees that emerged after FedAvg required either nonsmooth convex or smooth nonconvex local costs, under either iid [46, 58, 50, 47] or non-iid datasets [30, 26], while provable guarantees for FL methods under nonconvexity [58, 53, 19] require L -smoothness of local functions. Unfortunately, these assumptions do not hold either for ReLU neural networks or risk-averse learning and necessitate the use of Clarke calculus [3]. Moreover, existing work on zeroth-order FL methods in convex [31] and nonconvex settings [12] rely on the smoothness properties of the objective function. However, there appear to be no provably convergent FL schemes with complexity guarantees for computing approximate Clarke stationary points of nondifferentiable nonconvex problems. (iii) Federated bilevel optimization. Hyperparameter tuning [21] and its federated counterpart [23] is a crucial, and yet, computationally complex integrant of machine learning (ML) pipeline. Bilevel models

where the lower-level is a parameterized training model while the upper-level requires selecting the best configuration for the unknown hyperparameters [15, 22, 49]. Solving such hierarchical problems is challenging because of nondifferentiable nonconvex terms and absence of an analytical form for the implicit objective. These challenges exacerbate the development of provable guarantees for privacy-aware and communication-efficient schemes. (iv) Federated minimax optimization. Minimax optimization has assumed relevance in adversarial learning [17, 40, 44] and fairness in ML [57], amongst other efforts. Recently, FL was extended to distributed minimax problems [36, 10, 43], but relatively little exists in nonconvex-strongly concave settings [48, 43].

3 A Zeroth-order FL Framework for Nondifferentiable Nonconvex Settings

In this section, we introduce our framework for ( FL nn ), where we impose the following assumption. Assumption 1. Consider problem ( FL nn ). The following hold.

(i) The function f i is Lipschitz continuous with parameter L 0 > 0 for all i ∈ [ m ] .

(ii) For any i ∈ [ m ] , client i has access to a zeroth-order oracle ˜ f i ( x, ξ i ) satisfying the following for every x in an almost-sure sense:

(iii) The set X i is nonempty, closed, and convex for all i ∈ [ m ] . In addition, the following bounded set-dissimilarity condition holds for all x ∈ R n and some scalars B 1 and B 2 .

We note that the bounded set-dissimilarity condition is naturally analogous to the so-called bounded gradient-dissimilarity condition that has been employed in the literature, e.g., in [25]. In particular, when the bounded gradient-dissimilarity condition is stated for the Moreau smoothing of the indicator function of X i , denoted by I η X i ( x ) , we reach to (3). Notably, condition (3) holds for the generated iterates by the algorithm when, for example, the iterates remain bounded.

Nonsmooth unconstrained reformulation. Consider an unconstrained reformulation of ( FL nn ) given by min x ∈ R n f ( x ) (see (1)), where the nonsmoothness of f arises from that of f and the local indicator functions I X i . The minimization of f is challenging, as noted by recent findings on nonsmooth analysis where it is shown [56] that for a suitable class of nonsmooth functions, computing an ϵ -stationary point, i.e., a point ¯ x for which dist (0 n , ∂ f (¯ x )) ≤ ϵ , is impossible in finite time.

Approximate Clarke stationarity. To circumvent this challenge, as a weakening of ϵ -stationarity, a notion of ( δ, ϵ ) -stationarity is introduced [56] for a vector ¯ x when dist (0 n , ∂ δ f (¯ x )) ≤ ϵ , where the set

denotes the δ -Clarke generalized gradient of f at x [16]; i.e. if x is ( δ, ϵ ) -stationary, then there exists a convex combination of gradients in a δ -neighborhood of x that has a norm of at most ϵ [41].

This discussion naturally leads to the following key question: Can we devise provably convergent FL methods for computing approximate Clarke stationary points of minimization of f ? The aim of this section is to provide an affirmative answer to this question by proposing a zeroth-order FL method that employs smoothing. To contend with the nonsmoothness, we employ the Moreau-smoothed variant of I X ( x ) , where X ≜ ⋂ m i =1 X i , and a randomized smoothed variant of f , as shown next.

Randomized smoothing of loss function. To smoothen the loss function f , we employ a randomized smoothing approach where the smoothing parameter is maintained as sufficiently small. This framework is rooted in the seminal work by Steklov [45], leading to progress in both convex [27, 52, 11] and nonconvex [37] regimes. We consider a smoothing of f , given by f η defined as f η ( x ) ≜ E u ∈ B [ f ( x + ηu )] , where u is a random vector in the unit ball B , defined as B ≜ { u ∈ R n | ∥ u ∥ ≤ 1 } . Further, we let S denote the surface of the ball B , i.e., S ≜ { v ∈ R n | ∥ v ∥ = 1 } and η B and η S denote a ball with radius η and its surface, respectively.

Lemma 1 (Randomized spherical smoothing) . Let h : R n → R be a given continuous function and define h η ( x ) ≜ E u ∈ B [ h ( x + ηu )] . Then, the following hold.

(i) h η is continuously differentiable and ∇ h η ( x ) = ( n η ) E v ∈ η S [ h ( x + v ) v ∥ v ∥ ] for any x ∈ R n .

Suppose h is Lipschitz continuous with parameter L 0 > 0 . Then, the following statements hold.

The discussion leads to the consideration of the following smoothed federated problem.

Definition 1 (Unconstrained smoothed approximate problem) . Given η > 0 , consider an unconstrained smoothed problem given as

Algorithm 1 Randomized Zeroth-Order Locally-Projected Federated Averaging (FedRZO nn )

1: input: Server chooses a random initial point ˆ x 0 ∈ X , stepsize γ , smoothing parameter η , synchronization indices T 0 := 0 and T r ≥ 1 , where r ≥ 1 is the communication round index

2: for r = 0 , 1 , . . . do

3: Server broadcasts ˆ x r to all clients: x i,T r := ˆ x r , ∀ i ∈ [ m ]

4: for k = T r , . . . , T r +1 -1 do in parallel by clients

5: Client i generates the random replicates ξ i,k ∈ D i and v i,k ∈ η S

6: g η i,k := n η 2 ( ˜ f i ( x i,k + v i,k , ξ i,k ) -˜ f i ( x i,k , ξ i,k ) ) v i,k

7: Client i does a local update as x i,k +1 := x i,k -γ ( g η i,k + 1 η ( x i,k -P X i ( x i,k )) )

8: end for

10: end for

9: Server receives x i,T r +1 from all clients and aggregates, i.e., ˆ x r +1 := 1 m ∑ m i =1 x i,T r +1

To address (4), we propose FedRZO nn given by Algorithm 1. Here, client i employs a zeroth-order stochastic gradient of the form g η i,k ≜ n η 2 ( ˜ f i ( x i,k + v i,k , ξ i,k ) -˜ f i ( x i,k , ξ i,k ) ) v i,k , augmented by the gradient of the Moreau smoothed function. The random sample v i,k ∈ η S is locally generated by each client i , allowing for randomized smoothing. This is indeed in view of Lemma 1 (i) that facilitates the development of a randomized zeroth-order gradient.

We define ¯ x k ≜ ∑ i m =1 x i,k m as an auxiliary sequence to denote the averaged iterates of the clients.

Definition 2. Consider Algorithm 1. Let H > 0 denote an upper bound on the number of local steps per round, i.e., H ≥ max r =0 , 1 ,... | T r +1 -T r | . Throughout, we assume that H is finite.

Proposition 1. Consider Algorithm 1. Let Assumption 1 hold.

(i) [Error bound] Suppose γ ≤ min { η 4 L 0 n , 1 4 H , η 12 √ 3 B 2 ( L 0 n +1) H } . Let k ∗ denote an integer drawn uniformly at random from { 0 , . . . , K } and f η, ∗ ≜ inf x f η ( x ) . Then,

(ii) [Iteration complexity] Let γ := √ m K and H := ⌈ 4 √ K m 3 ⌉ where η > 0 . Let ϵ > 0 be an arbitrary scalar and K ϵ denote the number of iterations such that E [ ∥∇ f η (¯ x k ∗ ) ∥ 2 ] ≤ ϵ . Then, the iteration complexity is K ϵ := O ( ( L 0 n 3 ν 2 η 3 + L 3 0 n 3 η + L 2 0 n 4 ν 2 η 4 + L 2 0 n 2 B 2 1 η 4 + B 2 2 L 4 0 n 4 η 2 ) 2 1 mϵ 2 ) .

(iii) [Communication complexity] Suppose K ϵ ≥ m 3 . Then, the number of communication rounds to achieve the accuracy level in (ii) is R := O ( ( mK ϵ ) 3 / 4 ) .

We now formally relate the original nonsmooth problem and its smoothed counterpart.

Proposition 2. Consider problem (4) and let Assumption 1 hold.

(i) Assume that X i = R n for all i ∈ [ m ] . Then, for any η > 0 , we have ∇ f η ( x ) ∈ ∂ 2 η f ( x ) .

(ii) Assume that the sets X i are identical for all i ∈ [ m ] . Let δ > 0 be an arbitrary scalar. If ∇ f η ( x ) = 0 and η ≤ δ max { 2 ,nL 0 } , then 0 n ∈ ∂ δ ( f + I X ) ( x ) .

4 Extensions to Bilevel and Minimax FL

4.1 Nondifferentiable nonconvex bilevel FL. In this section, we consider the federated bilevel optimization problem defined earlier as ( FL bl ). We consider the following smoothed implicit problem.

Definition 3 (Unconstrained smoothed implicit problem) . Given η > 0 , consider an unconstrained smoothed implicit problem given as

Assumption 2. Consider problem ( FL bl ). Let the following assumptions hold.

(ii) For all i ∈ [ m ] , for any x , h i ( x, · ) is L h 1 ,y -smooth and µ h -strongly convex. Further, for any y , the map ∇ y h i ( · , y ) is Lipschitz continuous with parameter L ∇ h 0 ,x .

(iii) The sets X i , for i ∈ [ m ] , satisfy Assumption 1 (iii).

The outline of FedRZO bl is presented in Algorithm 2. We make the following remarks: (i) At each global step, the server makes two calls to a lower-level FL oracle to inexactly compute y (ˆ x r + v T r ) and y (ˆ x r ) . These lower-level FL calls are performed by the same clients, on the lower-level FL problem. (ii) The inexactness error is carefully controlled by terminating the lower-level FL oracle after O ( r ) number of iterations, where r denotes the upper-level communication round index. (iii) FedRZO bl skips the calls to the lower-level FL oracle during the local steps. To accommodate this, unlike in FedRZO nn , here we employ a global randomized smoothing denoted by v T r during the communication round r in the upper level.

Algorithm 2 Randomized Implicit Zeroth-Order Federated Averaging (FedRZO bl )

1: input: Server chooses a random ˆ x 0 ∈ X , stepsize γ , smoothing parameter η , synchronization indices T 0 := 0 and T r ≥ 1 , where r ≥ 1 is the upper-level communication round index

2: for r = 0 , 1 , . . . do

3: Server generates a random replicate v T r ∈ η S

4: Server calls FedAvg to receive y ε r (ˆ x r + v T r ) and y ε r (ˆ x r ) , denoting the inexact evaluations of y (ˆ x r + v T r ) and y (ˆ x r ) , respectively.

5: Server broadcasts ˆ x r , ˆ x r + v T r , y ε r (ˆ x r ) , and y ε r (ˆ x r + v T r ) to all clients; x i,T r := ˆ x r , ∀ i

6: for k = T r , . . . , T r +1 -1 do in parallel by clients

7: Client i generates the random replicates ξ i,k ∈ D i

10: end for

11: m i =1 +1

Server receives x i,T r +1 from all clients and aggregates, i.e., ˆ x r +1 := 1 ∑ m x i,T r

12: end for

obtaining the lower-level solution, i.e., E [ ∥ y ε r ( x ) -y ( x ) ∥ 2 | x ] ≤ ε r for x ∈ ∪ R r =0 { ˆ x r , ˆ x r + v T r } .

(i) [Error bound] We have

(ii) [Iteration complexity] Let γ := √ m K and H := ⌈ 4 √ K m 3 ⌉ where η > 0 . Let ϵ > 0 be an arbitrary scalar and K ϵ denote the number of iterations such that E [ ∥∇ f η (¯ x k ∗ ) ∥ 2 ] ≤ ϵ . Also, suppose we employ an FL method in the lower level that achieves a sublinear convergence rate with a linear speedup in terms of the number of clients, i.e., ε r := ˜ O ( 1 m ˜ T ˜ Rr ) where ˜ R r denotes the number of communica- tion rounds performed in the lower-level FL method when it is called in round r of FedRZO bl and ˜ T ˜ R r denotes the number of iterations performed in the lower-level FL scheme to do ˜ R r rounds of upperlevel communication. Further, suppose ˜ T ˜ R r := ˜ O ( m -1 ( r +1) 2 3 ) . Then, the iteration complexity of FedRZO bl (upper level) is K ϵ := ˜ O (( Θ 2 1 +max { Θ 2 , Θ 3 } 2 Θ 2 5 +max { Θ 2 , Θ 3 } 2 Θ 2 4 +Θ 2 3 ) 1 mϵ 2 ) .

(iii) [Communication complexity] Suppose K ϵ ≥ m 3 . Then, the number of communication rounds in FedRZO bl (upper-level only) to achieve the accuracy level in (ii) is R := O ( ( mK ϵ ) 3 / 4 ) .

Remark 2. (i) Importantly, Theorem 1 is equipped with explicit communication complexity R := O ( ( mK ϵ ) 3 / 4 ) , matching that of single-level nonsmooth nonconvex problems in Proposition 1. This implies that as long as the lower-level FL oracle has a rate of ε r := ˜ O ( 1 m ˜ T ˜ Rr ) , the inexactness does not affect the communication complexity bounds of the method in the upper level.

(ii) As noted in the assumptions, in the upper level, we allow for heterogeneity. To elaborate on overall communication complexity of FedRZO bl , we provide detailed complexity results in Table 1 for three cases, where we employ Local SGD [26], FedAC [54], and LFD [20] for the lower-level scheme. All these schemes meet the linear speedup condition in Theorem 1. Notably, among these schemes, the last scheme allows for the presence of heterogeneity. As an example, we present in Algorithm 3, the outline of FedAvg, if employed in step 4 of Algorithm 2.

Algorithm 3 FedAvg ( x, r, y 0 ,r , m, ˜ γ, ˜ H, ˜ T ˜ ) for lower level

R 1: input: x , r , server chooses a random initial point ˆ y 0 := y 0 ,r ∈ R ˜ n , a r := max { m, 4 κ h , r } +1 where κ h := L h 1 ,y µ h , ˜ γ := 1 µ h a r , ˜ T ˜ R r := 2 a r ln( a r ) , and ˜ H := ⌈ ˜ T ˜ Rr m ⌉ 2: for ˜ r = 0 , 1 , . . . , ˜ R r -1 do 3: Server broadcasts ˆ y ˜ r to all agents: y i, ˜ T ˜ r := ˆ y ˜ r , ∀ i 4: for t = ˜ T ˜ r , . . . , ˜ T ˜ r +1 -1 do in parallel by agents 5: Agent i does a local update as y i,t +1 := y i,t -˜ γ ∇ y h i ( x, y i,t , ˜ ξ i,t ) 6: Agent i sends x i,T r +1 to the server 7: end for 8: Server aggregates, i.e., ˆ y ˜ r +1 := 1 m ∑ m i =1 y i,T ˜ r +1 9: end for

(iii) We use L imp 0 ( ξ i ) to denote the Lipschitz continuity constant of the random local implicit function ˜ f i ( x, y ( x ) , ξ i ) , and let L imp 0 ≜ max i =1 ,...,m √ E [( L imp 0 ( ξ i )) 2 ] < ∞ . As shown in supplementary material, L imp 0 ( ξ i ) can be obtained explicitly in terms of problem parameters.

Remark 3. Atechnical challenge in designing Algorithm 2 is that an inexact evaluation of y ( x ) must be avoided during the local steps. This is because we consider bilevel problems of the form ( FL bl )

where both levels are distributed. Because of this, the inexact evaluation of y ( x ) by each client in the local step in the upper level would require significant communications that is undesirable in the FL framework. We carefully address this challenge by introducing delayed inexact computation of y ( x ) . In step 8 of Algorithm 2, we note how y ε is evaluated at ˆ x r + v T r which is a different than the vector used by the client, i.e., x i,k + v T r . At each communication round in the upper level, we only compute y ( x ) inexactly twice in the global step and then use this delayed information in the local steps. This delayed inexact computation of y renders a challenge in the convergence analysis which makes the design and analysis of Algorithm 2 a non-trivial extension of Algorithm 1.

4.2 Nondifferentiable nonconvex-strongly concave minimax FL. Next, we consider the decentralized federated minimax problem of the form ( FL mm ) introduced earlier. This problem is indeed a zero-sum game and can be viewed as an instance of the non-zero sum game ( FL bl ) where ˜ h i := -˜ f i .

Corollary 1. Consider Algorithm 2 for solving ( FL mm ). Let Assumption 2 hold for ˜ h i := -˜ f i and D i := ˜ D i . Then, all the results in Theorem 1 hold true.

5 Experiments

We present three sets of experiments to validate the performance of the proposed algorithms. In Section 5.1, we implement Algorithm 1 on ReLU neural networks (NNs) and compare it with some recent FL methods. In Sections 5.2 and 5.3 we implement Algorithm 2 on federated hyperparameter learning and a minimax formulation in FL. Throughout, we use the MNIST dataset. Additional experiments on a higher dimensional dataset (i.e., Cifar-10) are presented in supplementary material.

Setting

(S1) with

η

= 0

.

1

(S2) with

η

= 0

.

01

(S3) with

η

= 0

.

001

Log of average loss

0

200

400

600

800

1000

Communication rounds

3.6

3.5

3.4

3.3

3.2

3.1

1 local step

5 local steps

10 local steps

20 local steps

0

200

400

600

800

1000

Communication rounds

3.5

3.4

3.3

3.2

3.1

1 local step

5 local steps

10 local steps

20 local steps

0

200

400

600

800

1000

Communication rounds

3.6

3.5

3.4

3.3

3.2

3.1

1 local step

5 local steps

10 local steps

20 local steps

5.1 Federated training of ReLU NNs. We implement FedRZO nn for federated training in a single-layer ReLU NN with N 1 neurons. This is a nondifferentiable nonconvex optimization problem, aligning with ( FL nn ) and taking the form min x :=( Z,w ) ∈X 1 2 m ∑ m i =1 ∑ ℓ ∈D i ( v i,ℓ -∑ N 1 q =1 w q σ ( Z · ,q U i,ℓ )) 2 + λ 2 ( ∥ Z ∥ 2 F + ∥ w ∥ 2 ) , where m denotes the number of clients, Z ∈ R N 1 × N 0 , w ∈ R N 1 , N 0 is the feature dimension, U i,ℓ ∈ R N 0 and v i,ℓ ∈ {-1 , 1 } are the ℓ th input and output training sample of client i , respectively, σ ( x ) := max { 0 , x } , and λ is the regularization parameter.

Setup. We distribute the training dataset among m := 5 clients and implement FedRZO nn for the FL training with N 1 := 4 neurons under three different settings for the smoothing parameter, η ∈ { 0 . 1 , 0 . 01 , 0 . 001 } , γ := 10 -5 , and λ := 0 . 01 . We study the performance of the method under different number of local steps with H ∈ { 1 , 5 , 10 , 20 } .

Results and insights. Figure 1 presents the first set of numerics for FedRZO nn under the aforementioned settings. In terms of communication rounds, we observe that the performance of the method improves by using a larger number of local steps. In fact, in the case where H := 1 , FedRZO nn is equivalent to a parallel zeroth-order SGD that employs communication among clients at each iteration, resulting in a poor performance, motivating the need for the FL framework. In terms of η , while we observe robustness of the scheme in terms of the original loss function, we also note a slight improvement in the empirical speed of convergence in early steps, as η increases. This is indeed aligned with the dependence of convergence bound in Proposition 1 on η .

Comparison with other FL methods. While we are unaware of other FL methods for addressing nondifferentiable nonconvex problems, we compare FedRZO nn with other FL methods including FedAvg [34], FedProx [29], FedMSPP [55], and Scaffnew [35] when applied on a NN with a smooth rectifier. Details of these experiments are provided in the supplementary material.

5.2 Federated hyperparameter learning. To validate FedRZO bl , we consider the following FL hyperparameter learning problem for binary classification using logistic loss.

where m is number of clients, x denotes the regularization parameter for client i , U i,ℓ ∈ R n and v i,ℓ ∈ {-1 , 1 } are the ℓ th input and output testing sample of client i , respectively, U i,ℓ ′ ∈ R n and v i,ℓ ′ ∈ {-1 , 1 } are the ℓ ′ th input and output training sample of client i , respectively. The constraint set X is considered as X := { x ∈ R m | x ≥ ¯ µ 1 m } , where ¯ µ > 0 . This problem is an instance of ( FL bl ), where the lower-level problem is ℓ 2 -regularized and the regularization parameter is a decision variable of the upper-level FL problem. The convergence results are presented in Fig. 2 (left).

5.3 Fair classification learning. Here, we study the convergence of FedRZO bl in minimax FL. We consider solving an FL minimax formulation of the fair classification problem [39] of the form

where c denotes the class index and D i,c denotes the a portion of local dataset associated with client i that is comprised of class c samples. The loss function follows the same formulation in Section 5.1, where an ReLU neural network is employed. This problem is nondifferentiable and nonconvex-strongly concave, fitting well with the assumptions in our work in addressing minimax FL problems. The performance of our algorithm is presented in Figure 2 (right).

0

200

400

600

800

1000

Communication rounds

0

1000

2000

3000

4000

5000

Objective function

H=1

H=10

800

1200

Communication rounds

6 Concluding Remarks

Federated learning has assumed growing relevance in ML. However, most practical problems are characterized by the presence of local objectives, jointly afflicted by nonconvexity and nondifferentiability, precluding resolution by most FL schemes, which can cope with nonconvexity in only smooth settings. We resolve this gap via a zeroth-order communication-efficient FL framework that can contend with both nondifferentiability and nonsmoothness with rate and complexity guarantees for computing approximate Clarke-stationary points. Extensions to nonconvex bilevel and nonconvex-strongly concave minimax settings are developed via inexact generalizations.

7 Acknowledgments

We acknowledge the funding support from the U.S. Department of Energy under grant #DESC0023303, and the U.S. Office of Naval Research under grants #N00014-22-1-2757 and #N0001422-1-2589. We also would like to thank the four anonymous referees for their constructive suggestions.

0.5

0.4

0.3

0.2

0.1

Test accuracy

0

400

H=1

H=10

1600

2000

References

J. V. Burke, A. S. Lewis, and M. L. Overton. Approximating subdifferentials by random sampling of gradients. Math. Oper. Res. , 27(3):567-584, 2002.

J. V. Burke, A. S. Lewis, and M. L. Overton. A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. , 15(3):751-779, 2005.

F. H. Clarke. Generalized gradients and applications. Transactions of the American Mathematical Society , 205:247-262, 1975.

S. Cui, U. V. Shanbhag, and F. Yousefian. Complexity guarantees for an implicit smoothingenabled method for stochastic mpecs. Math. Program. , 198(2):1153-1225, 2023.

Y. Cui and J. S. Pang. Modern nonconvex nondifferentiable optimization , volume 29 of MOS-SIAM Series on Optimization . Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA; Mathematical Optimization Society, Philadelphia, PA, [2022] ©2022.

S. Dafermos. Sensitivity analysis in variational inequalities. Mathematics of Operations Research , 13(3):421-434, 1988.

R. Das, A. Acharya, A. Hashemi, S. Sanghavi, I. S. Dhillon, and U. Topcu. Faster non-convex federated learning via global and local momentum, 2021.

D. Davis and D. Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM J. Optim. , 29(1):207-239, 2019.

D. Davis and B. Grimmer. Proximally guided stochastic subgradient method for nonsmooth, nonconvex problems. SIAM J. Optim. , 29(3):1908-1930, 2019.

Y. Deng, M. M. Kamani, and M. Mahdavi. Distributionally robust federated averaging. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15111-15122. Curran Associates, Inc., 2020.

J. C. Duchi, P. L. Bartlett, and Martin J. Wainwright. Randomized smoothing for stochastic optimization. SIAM J. on Optimization , 22(2):674-701, 2012.

W. Fang, Z. Yu, Y. Jiang, Y. Shi, C. N Jones, and Y. Zhou. Communication-efficient stochastic zeroth-order optimization for federated learning. IEEE Transactions on Signal Processing , 70:5058-5073, 2022.

A. Gasnikov, A. Novitskii, V. Novitskii, F. Abdukhakimov, D. Kamzolov, A. Beznosikov, M. Takáˇ c, P. Dvurechensky, and B. Gu. The power of first-order smooth optimization for black-box non-smooth problems. arXiv preprint arXiv:2201.12289 , 2022.

S. Ghadimi, G. Lan, and H. Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Math. Programming , 155(1-2):267-305, 2016.

S. Ghadimi and M. Wang. Approximation methods for bilevel programming. arXiv preprint: https://arxiv.org/abs/1802.02246.

A. A. Goldstein. Optimization of Lipschitz continuous functions. Math. Programming , 13(1):1422, 1977.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

F. Haddadpour, M. M. Kamani, M. Mahdavi, and V. Cadambe. Local SGD with periodic averaging: Tighter analysis and adaptive synchronization. Advances in Neural Information Processing Systems , 32, 2019.

F. Haddadpour, M. M. Kamani, M. Mahdavi, and V. Cadambe. Trading redundancy for communication: Speeding up distributed SGD for non-convex optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 25452554. PMLR, 09-15 Jun 2019.

F. Haddadpour and M. Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425 , 2019.

E. Hazan, A. Klivans, and Y. Yuan. Hyperparameter optimization: A spectral approach. 6th International Conference on Learning Representations (ICLR) , 2018.

K. Ji, J. Yang, and Y . Liang. Bilevel optimization: Convergence analysis and enhanced design. arXiv preprint: https://arxiv.org/pdf/2010.07962.pdf.

P. Kairouz, H. Brendan McMahan, et al. Advances and open problems in federated learning. CoRR , abs/1912.04977, 2019.

S. P. Karimireddy, M. Jaggi, S. Kale, M. Mohri, S. J. Reddi, S. U Stich, and A. T. Suresh. Breaking the centralized barrier for cross-device federated learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

A. Khaled, K. Mishchenko, and P. Richtárik. Tighter theory for local SGD on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 4519-4529. PMLR, 2020.

H. Lakshmanan and D. Farias. Decentralized recourse allocation in dynamic networks of agents. SIAM Journal on Optimization , 19(2):911-940, 2008.

J. Lei and U. V. Shanbhag. Asynchronous variance-reduced block schemes for composite non-convex stochastic optimization: block-specific steplengths and adapted batch-sizes. Optimization Methods and Software , pages 1-31, 2020.

T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of FedAvg on Non-IID data. In International Conference on Learning Representations , 2020.

Zan Li and Li Chen. Communication-efficient decentralized zeroth-order method on heterogeneous data. In 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP) , pages 1-6. IEEE, 2021.

T. Lin, Z. Zheng, and M. Jordan. Gradient-free methods for deterministic and stochastic nonsmooth nonconvex optimization. Advances in Neural Information Processing Systems , 35:26160-26175, 2022.

D. Q. Mayne and E. Polak. Nondifferential optimization via adaptive smoothing. Journal of Optimization Theory and Applications , 43(4):601-613, 1984.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 1273-1282. PMLR, 20-22 Apr 2017.

K. Mishchenko, G. Malinovsky, S. Stich, and P. Richtarik. ProxSkip: Yes! Local gradient steps provably lead to communication acceleration! Finally! In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 15750-15769. PMLR, 17-23 Jul 2022.

M. Mohri, G. Sivek, and A. T. Suresh. Agnostic federated learning. In International Conference on Machine Learning , pages 4615-4625. PMLR, 2019.

Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions. Found. Comput. Math. , 17(2):527-566, 2017.

J. Nguyen, K. Malik, H. Zhan, A. Yousefpour, M. Rabbat, M. Malek, and D. Huba. Federated learning with buffered asynchronous aggregation. CoRR , abs/2106.06639, 2021.

M. Nouiehed, M. Sanjabi, T. Huang, J. D. Lee, and M. Razaviyayn. Solving a class of nonconvex min-max games using iterative first order methods. Advances in Neural Information Processing Systems , 32, 2019.

M. Sanjabi, J. Ba, M. Razaviyayn, and J. D. Lee. On the convergence and robustness of training gans with regularized optimal transport. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

O. Shamir. Can we find near-approximately-stationary points of nonsmooth nonconvex functions? In OPT2020: 12th Annual Workshop on Optimization for Machine Learning , 2021.

U. V. Shanbhag and F. Yousefian. Zeroth-order randomized block methods for constrained minimization of expectation-valued lipschitz continuous functions. In 2021 Seventh Indian Control Conference (ICC) , pages 7-12. IEEE, 2021.

P. Sharma, R. Panda, G. Joshi, and P. Varshney. Federated minimax optimization: Improved convergence analyses and algorithms. In International Conference on Machine Learning , pages 19683-19730. PMLR, 2022.

A. Sinha, H. Namkoong, and J. Duchi. Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations , 2018.

V. A. Steklov. Sur les expressions asymptotiques decertaines fonctions définies par les équations différentielles du second ordre et leers applications au problème du dévelopement d'une fonction arbitraire en séries procédant suivant les diverses fonctions. Comm. Charkov Math. Soc. , 2(10):97-199, 1907.

S. U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations , 2019.

S. U. Stich and S. P. Karimireddy. The error-feedback framework: Better rates for SGD with delayed gradients and compressed updates. Journal of Machine Learning Research , 21:1-36, 2020.

D. A. Tarzanagh, M. Li, C. Thrampoulidis, and S. Oymak. Fednest: Federated bilevel, minimax, and compositional optimization. In International Conference on Machine Learning , pages 21146-21179. PMLR, 2022.

R. Tibshirani, M. Saunders, S. Rosset, J. Zhu, and K. Knight. Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 67(1):91-108, 2005.

J. Wang and G. Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. Journal of Machine Learning Research , 22(213):1-50, 2021.

Y. Xu and W. Yin. A globally convergent algorithm for nonconvex optimization based on block coordinate update. Journal of Scientific Computing , pages 1-35, 2017.

F. Yousefian, A. Nedi´ c, and U. V . Shanbhag. On stochastic gradient and subgradient methods with adaptive steplength sequences. Automatica , 48(1):56-67, 2012.

H. Yu, S. Yang, and S. Zhu. Parallel restarted SGD with faster convergence and less communication: demystifying why model averaging works for deep learning. In AAAI Conference on Artificial Intelligence , 2019.

H. Yuan and T. Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems , 33:5332-5344, 2020.

X. Yuan and P. Li. On convergence of fedprox: Local dissimilarity invariant bounds, nonsmoothness and beyond. Advances in Neural Information Processing Systems , 35:10752-10765, 2022.

J. Zhang, H. Lin, S. Jegelka, S. Sra, and A. Jadbabaie. Complexity of finding stationary points of nonconvex nonsmooth functions. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 11173-11182. PMLR, 13-18 Jul 2020.

L. Zhang, D. Xu, S. Yuan, and X. Wu. FairGAN: Fairness-aware generative adversarial networks. In CoRR , 2018.

F. Zhou and G. Cong. On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of nondifferentiable nonconvex locally constrained federated learning (Eq. $FL_{nn}$), and the bilevel variant (Eq. $FL_{bl}$). The minimax settings (Eq. $FL_{mm}$) is a special case of the latter. \n\nThe authors provide error, iteration and communication complexity for these settings where both non-convexity and non-differentiability is assumed.\n\nSTRENGTHS:\nThe reasoning for obtaining the algorithm is clearly explained in Sec. 1. \n\nThe experiment section is relevant and provides a clear comparison with other methods.\n\nWEAKNESSES:\nContrary to the reasoning for obtaining the algorithm, the results of Thm. 1 and 2 are not discussed. Doing so would help understand their significance. Typically, do lower bounds exist in some specific settings? What are the interpretation of the different terms? What are the dominant ones? \n\nOf lower importance, the reading is a bit complicated: the content is dense, with lots of technicalities and equation in the main text.\n\nQUESTIONS:\n* Why is the final loss so big in Fig. 2? \n\n**Clarity.**\n\n* A reader not acquainted with Moreau smoothing cannot understand Eq. (1) without further research. Defining $\\mathbb B$ and $I_X^\\eta$ is advised (or at least point to `l. 158`). \n* Likewise, a reference for Eq. (4) is advised (or at least point to `l. 147`). \n* Lemma 1 in `l. 161`, Prop. 1 at `l. 171`, Thm. 1 at `l. 182`, Thm. 2 at `l. 209`: refer to the appendix' location of the proof. \n\n**Presentation.**\n\n* Tables 1 and 2 are useful summaries of the result. The font is however smaller than the rest of the text. \n* Framed environments `l. 48` and `l. 90` have italic or bold titles. Choose one of them?  \n* It is a matter of taste, but I would emphasize the text of Definition and Proposition environments so that they stand out of the main text.\n* Fig. 1 and 2 require bigger legends (or moved to the caption)\n* `l. 270`: `\\exp` instead of `exp`\n\nLIMITATIONS:\nDiscussing the significance of the results of Thm. 1 and 2 would be helpful.\n\nThe comparison with other FL scheme in Table 2 is done with different metric of accuracy. Relating those metrics would provide insight on the theoretical benefits of Alg. 1, and would complements the finding of the experiment section. Notably, highlighting the role of $\\eta$ in Table 2 and in Thm 1 and 2 would be welcomed too, as it is an important hyperparameter (complementary to the small remark `l. 263-265`).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the federated optimization problem with the non-smooth non-convex target function. The authors use a zero-order oracle, which allows to approximate the gradient. This approximation is related not only to the original target function, but also to its smoothed version (an additional theoretical object obtained using the random smoothing technique). For a new smooth target function, there are methods in the literature for solving it, and this is what the authors use. Mini-max problems and bilevel optimization are also considered. Experiments are given.\n\nSTRENGTHS:\nFor me, the article is nicely written and easy to follow.\n\nWEAKNESSES:\nI ask the authors to pay attention to this work:\n\nGasnikov, A., Novitskii, A., Novitskii, V., Abdukhakimov, F., Kamzolov, D., Beznosikov, A., ... & Gu, B. (2022). The power of first-order smooth optimization for black-box non-smooth problems. arXiv preprint arXiv:2201.12289.\n\nIt gives a more or less unified scheme of how by taking any practical first-order stochastic method for smooth problems to obtain a gradient-free method for non-smooth problems. It seems that using this scheme one can obtain the results of this paper as well. Moreover, this scheme is not new; it has been used for 10 years in most papers on gradient-free optimization.  In particular, the facts that are proven in the article under review are also used in the article I cited and there are links from where they are taken. \n\nPlease also note the refs within the article I cited. For example, there are references to works with gradient-free methods of solving saddle point problems. \n\nFinally, the contribution of this paper looks rather technical and at the moment I am not ready to accept the work. But perhaps the authors will change my mind in the process of rebuttal.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nThe paper is theoretical, therefore there is no need to discuss the social negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary:** The paper develops zero-th order methods for solving non-smooth and non-convex federated problems. In addition, the paper also develops zero-th order methods for solving bilevel and minimax optimization problems in a federated setting. The authors develop a randomized smoothing-based approach and present convergence guarantees for the proposed algorithms. Experiments on training ReLU neural networks and hyperparameter optimization tasks are presented to evaluate the performance of the proposed algorithms.\n\nSTRENGTHS:\n**Strengths:** The paper considers an important problem class of solving non-convex and non-smooth problems with zero-th order oracle. The paper is very well written with ideas explained clearly.\n\nWEAKNESSES:\n**Weaknesses:** Here, I list some points the authors should consider addressing:\n\n1.\tThere is some existing literature on federated learning where randomized smoothing-based zero-th order methods have already been developed. The authors have not mentioned/or compared their approach against such methods Please see [R1], [R2] (strongly convex) below. \n \n\n     [R1] Fang et al., Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning   \n\n     [R2] Li et al., Communication-efficient decentralized zeroth-order method on heterogeneous data\n\n\n2.\tThe authors should make dependence on the problem dimension explicit in the tables and the communication complexities listed in all the theorems. The current presentation gives an indication to the reader that zero-th order methods can achieve the same guarantees as first-order methods which is certainly not true.   \n\n3.\tThe authors should consider including some discussion on the bounded set dissimilarity since the assumption is relatively new in the context of FL. \n\n4.\tIs the notion of Clarke stationarity point related to Goldstein stationary point (or are the two notions the same)? Also, the discussion on Clarke's generalized gradient of the original problem compared to the gradient of the smoothed objective appears before the definition of Clarke's stationarity point. Please consider moving the definition before the discussion. \n\n5.\tIn Line 172 and Prop. 1 please characterize sufficiently small $\\eta$. \n\n6.\tTheorem 2 ignores the additional communication complexity because of solving the lower-level problem to $\\epsilon_r$ accuracy. The authors should report the complete communication complexity incurred by the algorithm for solving the bilevel optimization problem not only the upper-level complexity.  \n\n7.\tFinally, the experiments considered to evaluate Algorithm 1 are weak. Specifically, the authors have considered a very low-dimensional problem. It would be advisable to include training on Cifar-10/MINST datasets with practically sized neural networks to evaluate the actual performance of the proposed scheme. Moreover, for hyperparameter learning there are many baseline bilevel algorithms including BSA, TTSA, ALSET, SUSTAIN, stocBiO, etc. whose performance should be compared with the proposed scheme.\n\nQUESTIONS:\nPlease see the weakness section above.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting federated optimization algorithms usually rely on the assumption of differentiability and smoothness, which may fail to hold in practical settings. To this end, this paper employs randomized smoothing approach and zeroth-order optimization techniques for the development of FedRZO algorithm to address this kind of problem. Theoretical analyses on convergence, iteration complexity and communication complexity have also been provided. This paper further extend the idea behind the newly developed algorithm to bilevel and minimax federated optimization problems with sound theoretical guarantees.\n\nSTRENGTHS:\n1. This paper has studied three different federated setting that existing works seldomly consider with both theoretical guarantees and empirical justifications.\n2. The combination of randomized smoothing technique with zeroth-order optimization appeals to me, which may inspires the future work on non-smooth zeroth-order optimization.\n3. Empirical results are interesting with adequate interpretation.\n\nWEAKNESSES:\n1. This paper may give more intuitive explanation or interpretation for its equations to ease reader's understanding. E.g., the authors may need to provide certain motivations for the study of the three different cases in the introduction section instead of directly putting out these cases without any explanation. Besides, the authors may also need to provide intuitive explanation for eq. 3 to help justify the reasonability of this assumption as well as its connection with real-world examples.\n2. This paper may need to compare with more federated optimization algorithms to verify the efficacy of their algorithms, e.g., SCAFFOLD, FedZO [R1], from both theoretical and empirical perspectives.\n\n[R1] Fang, Wenzhi, Ziyi Yu, Yuning Jiang, Yuanming Shi, Colin N. Jones, and Yong Zhou. 2022. “Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning.”.\n\nQUESTIONS:\n1. What's the difference between line 6 of Algorithm 1 and the gradient estimation (or even the algorithm) in FedZO [R1]? They seem to share similar calculation with only different scale that has been multiplied.\n2. Are the baselines in Figure 2 able to access the gradient? If so, shouldn't they converge much faster compared with your zeroth-order optimization algorithm?\n\nLIMITATIONS:\nLimitations haven't been mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}