Title: Fine-Grained Theoretical Analysis of Federated Zeroth-Order Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated zeroth-order optimization

stability analysis

theoretical guarantee

non-convex optimization

sub-Weibull distribution
Abstract: Federated zeroth-order optimization (FedZO) algorithm enjoys the advantages of both zeroth-order optimization and federated learning, and has shown exceptional performance on black-box attack and softmax regression tasks. However, there is no generalization analysis for FedZO, and its analysis on computing convergence rate is slower than the corresponding first-order optimization setting. This paper aims to establish systematic theoretical assessments of FedZO by developing the analysis technique of on-average model stability. We establish the first generalization error bound of FedZO under the Lipschitz continuity and smoothness conditions. Then, refined generalization and optimization bounds are provided by replacing bounded gradient with heavy-tailed gradient noise and utilizing the second-order Taylor expansion for gradient approximation. With the help of a new error decomposition strategy, our theoretical analysis is also extended to the asynchronous case. For FedZO, our fine-grained analysis fills the theoretical gap on the generalization guarantees and polishes the convergence characterization of the computing algorithm.

Fine-Grained Theoretical Analysis of Federated Zeroth-Order Optimization

Abstract

Federated zeroth-order optimization (FedZO) algorithm enjoys the advantages of both zeroth-order optimization and federated learning, and has shown exceptional performance on black-box attack and softmax regression tasks. However, there is little generalization analysis for FedZO, and its analysis on computing convergence rate is slower than the corresponding first-order optimization setting. This paper aims to establish systematic theoretical assessments of FedZO by developing the analysis technique of on-average model stability. We establish the first generalization error bound of FedZO under the Lipschitz continuity and smoothness conditions. Then, refined generalization and optimization bounds are provided by replacing bounded gradient with heavy-tailed gradient noise and utilizing the second-order Taylor expansion for gradient approximation. With the help of a new error decomposition strategy, our theoretical analysis is also extended to the asynchronous case. For FedZO, our fine-grained analysis fills the theoretical gap on the generalization guarantees and polishes the convergence characterization of the computing algorithm.

1 Introduction

Federated learning collaborates multiple local clients to train a global model without sharing local raw data, which often enjoys great ability in protecting data privacy [1]. The core training steps of federated learning include local clients receiving the global model from the central server, the local models being updated by the global information and local data, and the updated local models being uploaded to renew the global model. Based on this building-block process, rich federated learning algorithms have been formulated to match different motivations, where their properties on privacy protection [2, 3, 4, 5] and convergence rate [6, 7, 8] are investigated. In general, the existing algorithms of federated learning depend heavily on the gradient information of loss function, see e.g., [1, 6, 7, 8]. Indeed, there are some learning scenarios where the gradient or Hessian information

is either unobtainable or too expensive to obtain, such as reinforcement learning [9, 10, 11], meta learning [12], black-box adversarial attacks [13, 14] and hyperparameter tuning [15]. To alleviate the dependence on the gradient or Hessian information, a federated zeroth-order optimization (FedZO) algorithm is proposed in [16] by integrating zeroth-order optimization into federated learning. In theory, the analysis of optimization convergence of FedZO has been established in [16], which shows the linear speedup in terms of the numbers of local iterations and clients participating in the update for the global iteration. However, there is little generalization analysis for federated zeroth-order optimization.

The goal of generalization analysis is to evaluate the capability of the empirical risk to approach the theoretical optimal risk on a given dataset. To characterize the generalization performance theoretically, there are four popular analysis techniques including uniform convergence approaches [17, 18, 19, 20], operator approximation techniques [21, 22], information-theoretic tools [23, 24, 25, 26, 27], and algorithmic stability analysis [28, 29, 30, 31]. Usually, stability-based generalization assessment enjoys some benefits over other tools. Firstly, in contrast to uniform convergence approaches, the theoretical guarantees derived from algorithmic stability analysis are independent of the capacity measurement of hypothesis function space [32]. Secondly, algorithmic stability analysis is available in a wide range of applications rather than only some models enjoying operator representation [33, 34]. Finally, data distribution does not affect the results of algorithmic stability analysis, whereas information-theoretic tools are typically sensitive to data distribution [25, 35]. To the best of our knowledge, there is only one work on the stability-based generalization analysis for the zeroth-order optimization, i.e., zeroth-order stochastic search (ZoSS) method [36]. Following this line, it is natural to further investigate the generalization guarantees of the FedZO algorithm. However, due to the essential difference between FedZO and ZoSS, the previous analysis technique in [36] can not be used for federated learning directly. In this paper, we develop the fine-grained error decomposition and estimations to overcome the challenge induced by the federated algorithmic formulation.

As a general assumption for loss function, Lipschitz continuity has been employed in many stabilitybased generalization assessments, see e.g., [36, 37, 38, 39, 40]. However, the index of Lipschitz continuity is likely too large or even infinite for some learning problems, which makes the previous results invalid [32]. With the help of the on-average model stability tool, [32] established the finegrained generalization analysis of stochastic gradient descent (SGD) by removing the Lipschitz continuity and the convexity of each loss function, and relaxing the smoothness to Holder continuity. Moreover, [38] additionally considered the bounded variance of the stochastic gradient to get the generalization bounds of non-convex SGD with high probability. Inspired by [32] and [38], this paper considers the heavy-tailed gradient noise (Assumption 2) as a refined version of bounded variance of gradient and adopts the second-order Taylor expansion for gradient approximation to remove bounded gradient condition. Conclusively, we fill the theoretical gap in the generalization analysis of the FedZO algorithm [16] and its asynchronous version. The main contributions of this paper are outlined as follows.

Generalization bounds of FedZO . We provide the first generalization bound of general FedZO after building the relationships between generalization error and ℓ 1 on-average model stability. To alleviate the restriction of bounded gradient, we further get a refined generalization bound and an optimal optimization bound under the condition of heavy-tailed gradient noise and the second-order Taylor expansion of gradient approximation.

Learning guarantees of asynchronous FedZO . For asynchronous FedZO, we design a new error decomposition strategy to bridge the relationships between each local model parameter and the global model parameter in each iteration. Then, the generalization and optimization bounds are derived for the asynchronous case. In particular, our fine-grained error bounds are tight even compared with the previous results for SGD implemented by the first-order optimization [32, 37, 38] and zeroth-order optimization [36].

2 Preliminaries

This section introduces some notations and definitions preparing for our theoretical analysis. Besides, we also give some detailed explanations for every symbol in Appendix A .

Considering a zeroth-order optimization algorithm for federated learning, we rely on N clients to independently train a global model. In each iteration, a subset of M clients is randomly selected to use their local data to obtain the corresponding gradient information. Then, this information is transmitted to the central server to update the global model parameter. For any i ∈ [ N ] , let z i be a random variable obeying data distribution D i associated with the i -th client. Denote S := { S i } N i =1 as the total dataset, where S i := { z i 1 , ..., z in } is the i -th local dataset and each sample z ij is drawn independently from the distribution D i , i ∈ [ N ] .

Federated learning aims to optimize model parameters with the collaboration among all local clients, i.e., minimizing the following population risk

where w,w i ∈ W ∈ R d denotes the training parameters for the global model and the i -th local model in a d -dimensional hypothesis space respectively, F ( w ) and F i ( w i ) indicate the population risks for the global model and the local model of the i -th client respectively, f i denotes the loss function of the i -th local client, and E z i ∼D i [ · ] denotes the conditional expectation with respect to (w.r.t.) the sample z i . Generally, the unattainability of the local population risk F i ( w i ) forces us to train the model by minimizing the following empirical approximation of population risk F ( w )

where F S ( w ) and F S i ( w i ) indicate the empirical risks for the global model and the local model of the i -th client respectively. Note that, for the update of the global model, the contribution of each client is treated equally. This paper considers the learning scenarios where the gradients (or Hessian information) of local loss functions are either unobtainable or too expensive to obtain [16]. Naturally, the first-order gradient estimation ˜ ∇ f i , defined as, for any t, b 2 ∈ N , m ∈ [ b 1 ] , b 1 ∈ [ n ]([ n ] := { 1 , ..., n } ) ,

is chosen to update the model parameters of federated learning, where w t i is the local model parameter of the i -th client at the t -th iteration, { z t i,m } b 1 m =1 and { v t i,l } b 2 l =1 are two sets of independent and identically distributed (i.i.d.) random samples and i.i.d. random direction vectors (satisfying the d -dimensional uniform distribution), and µ represents the distance between two model parameters ( w t i + µv t i,l and w t i ) used to estimate gradient in the l -th direction. In our analysis, the following second-order Taylor expansion is employed to approximate ˜ ∇ f i ,

Note that, the number b 2 of random direction vectors is set to be greater than the hypothesis space dimension d in this paper, such as the requirement of [13]( b 2 = 2 d ).

The update process of FedZO is formulated as follows,

where η t and M t ∈ [ N ]( |M t | = M ) denote the step size and the collection of selected client indices in the t -th iteration, respectively. In particular, we also consider the asynchronous FedZO algorithm.

Let

where F ( w ) , F S ( w ) are defined in (1) and (2). Denote A ( S ) = w T as the output of the global model after T iterations with any federated learning algorithm A (including Algorithm 1, i.e.,

synchronous FedZO). Typically, the excess risk of A ( S ) is measured by E [ F ( A ( S )) -F ( w ∗ )] and can be decomposed as

where E [ · ] denotes the expectation w.r.t. all randomness. The first part of (5) (called generalization error) measures the expected gap between population risk and empirical risk, while the second part of (5) (called optimization error) measures the divergence between the trained model and the theoretically optimal model. In this paper, we simultaneously bound the generalization error and optimization error of FedZO.

Remark 1. The update strategy (3) is associated with a minibatch of i.i.d. samples. Due to the interaction among samples, it is hard to establish the stability-based generalization analysis of the minibatch case directly. Fortunately, an equivalent formula can tackle this difficulty by means of the properties of binomial distribution (see Appendix B.3 ).

Remark 2. Compared with synchronous FedZO ([16] in Algorithm 1), there are several essential differences for its asynchronous one described in Section 4. Firstly, we assume that all clients participate throughout the entire update process of the global model parameters. Secondly, once some client finishes gradient computation, this gradient will be transmitted to update the global model immediately without waiting for other clients. Then, the updated parameters of the global model will be back to the corresponding client, not the other ones. As a result, the local model of some client may be inconsistent with other clients within the same iteration.

Algorithm 1 Synchronous FedZO

Require: w 1 : initial global model; η 1 : initial learning rate; b 1 , b 2 : minibatch sizes for samples and direction vectors respectively; µ : positive step size in the definition of the derivative; M : number of clients selected to update the global model in each iteration

Randomly select a clients set M t , let η t = η 1 /t for all

i

∈ M

in parallel do

t

Generate

Compute

{

z

t

i,m

}

b

1

m

∑

end for

Update

w

t

to

w

=1

t

+1

b

1

m

=1

{

t

}

b

2

i,l and

v

from

D

and

d

-dimensional uniform distribution

l

=1

i

˜

∇

f

i

and upload it to global model by Eq. (3)

end for

Ensure: Final global model w T

Inspired by Definition 4 in [32], we introduce a new definition of ℓ 1 on-average model stability for federated learning.

Definition 1. The federated learning algorithm A is ℓ 1 on-average model ϵ -stable if

where ∥·∥ is the Euclidean distance ∥·∥ 2 , S = { S i } N i =1 , S ( j i ) = { S 1 , ..., S i -1 , S ( j ) i , S i +1 , ..., S N } , S i = { z i 1 , ..., z in } , S ( j ) i = { z i 1 , ..., z i ( j -1) , z ′ ij , z i ( j +1) , ..., z in } ( z ′ ij is drawn from D i , ∀ i ∈ [ N ] , j ∈ [ n ]) .

In Definition 1, we assume that only one sample of one client is perturbed, not other clients. Particularly, as N = 1 , the current definition is consistent with the standard on-average model stability in [32, 38]. Compared with several other stability tools, the advantages of on-average model stability are listed as follows. First, on-average model stability is a weaker stability tool than uniform (model) stability. Second, on-average model stability measures the stability of model parameters w instead of function values f ( w ) like on-average stability, which can improve our analysis.

Definition 2. [41] For some θ, a, b > 0 and all x > 0 , if a random variable X satisfying

we call it sub-Weibull random variable (denoted as X ∼ subW ( θ ) ), where θ is a tail parameter.

Sub-Weibull distribution is a heavier-tailed (longer-tailed) distribution than the sub-Gaussian ( θ = 1 2 ) and sub-Exponential ( θ = 1) distribution, which matches some real applications [42] and has been used for learning theory analysis recently [43, 44]. This paper utilizes some sub-Weibull properties, derived from the moment generating function (MGF), to relax the bounded gradient assumption used in [36, 37, 38, 39, 40].

3 Evaluating the Generalization of FedZO via Stability

This section establishes the relationships between the generalization error of FedZO and the onaverage model stability in Theorem 1. Detailed proofs are provided in Appendix B.2 .

This paper makes the following assumptions for the loss function f i of i -th local client ( i ∈ [ N ] ).

Assumption 1. (a) ( L -Lipschitz continuity). For any w i , ¯ w i ∈ W , i ∈ [ N ] and some L > 0 , f i satisfies ∇ f i ( w i ; z i ) ≤ L , that is

(b) ( β -Smoothness). For any w i , ¯ w i ∈ W , i ∈ [ N ] and some β > 0 , f i satisfies

The requirement of Lipschitz continuity and smoothness is general in statistical learning theory, see e.g., [37, 38, 39, 40]. Usually, it appears to be justifiable when the hypothesis function space is uniformly bounded. However, the Lipschitz constant L may be infinite for some learning tasks with unbounded hypothesis function space [32]. In this paper, two attempts are proposed to remove the Lipschitz assumption. Firstly, we focus on the zeroth-order optimization problem, where the gradient information is unavailable. We approximate the first-order gradient using a second-order Taylor expansion to avoid the dependence on the Lipschitz constant (see the proofs of Theorems 2, 3 and 5). Secondly, we use a weaker gradient-related assumption, i.e., the heavy-tailed gradient noise assumption, to replace the Lipschitz assumption in our generalization analysis (see the proof of Theorem 1 (b)).

Assumption 2. (Heavy-tailed gradient noise). Let the tail parameter θ > 1 2 , the number of iterations t > 0 and the client index i ∈ [ N ] . For any w t i ∈ W , z t i ∈ S i , S i ∈ D n i , we assume that the gradient noise ∇ f i ( w t i ; z t i ) - ∇ F S i ( w t i ) is a sub-Weibull random vector, i.e., ∇ f i ( w t i ; z t i ) - ∇ F S i ( w t i ) ∼ subW ( θ ) .

From Theorem 2.1 in [41], we deduce that the MGF of ∥∇ f i ( w t i ; z t i ) -∇ F S i ( w t i ) ∥ 1 θ can be bounded at some point, i.e., E [ exp ( ∥ ∇ f i ( w t i ; z t i ) -∇ F S i ( w t i ) ∥ K ) 1 θ ] ≤ 2 for some K > 0 . Moreover, a random variable X is defined as K -sub-Weibull( θ ) if E [ exp( | X | /K ) 1 /θ ] ≤ 2 [45]. Therefore, the gradient noise in Assumption 2 can be also denoted as ∇ f i ( w t i ; z t i ) - ∇ F S i ( w t i ) ∼ subW ( θ, K ) . In this paper, a lemma (Lemma 2 in Appendix B.1 ) related to the bounded p -th norm of the sub-Weibull variable is used to build a novel relationship (Theorem 1 (b)) between ℓ 1 on-average model stability and generalization error.

Assumption 3. (PL condition). For any w ∈ W , S ∈ ⋃ N i =1 D n i and some α > 0 , the empirical risk F S ( w ) satisfies

Assumption 3 elucidates that the quadratic gradient of the empirical risk enjoys a linearly decreasing lower bound [46]. Without the setting of convexity, the gradient of empirical risk ∥∇ F S ( w ) ∥ = 0 is just a sufficient condition for a local optimal model instead of the guarantee to find a global optimal

parameter. This assumption implies that all local optimal parameters are global optimal parameters, that is, ∥∇ F S ( w ) ∥ = 0 implies that E [ F S ( w )] = E [ F ( w ( S ))] = E [ F ( w ∗ )] holds in expectation. For this reason, we can study the optimization error with the form F S ( w ) -F S ( w ( S )) rather than |∇ F S ( w ) | [44] under the non-convex condition.

Now we state two quantitative relationships between generalization error and ℓ 1 on-average model stability.

Theorem 1. Let S, S ( j i ) be given in Definition 1 and A be a federated learning algorithm.

(a) Let Assumption 1 (a) hold and A be ℓ 1 on-average model ϵ -stable. Then, for some L > 0 ,

(b) Let Assumption 2 hold and A be ℓ 1 on-average model ϵ -stable. Then, for some θ > 1 2 , K > 0 ,

Theorem 1 provides the generalization bounds of the FedZO algorithm by the ℓ 1 on-average model stability. Essentially, Theorem 1 (a) is consistent with Theorem 2 (a) in [32] and Theorem 1 (b) states a refined upper bound independent of the Lipschitz constant L . Explicitly, in Theorem 1 (b), the tail parameter θ in the first term is typically bounded [41] and the second term E [ F S ( A ( S ))] has no significant negative impact on the upper bound [32].

In the sequel, we derive the generalization bounds by integrating Theorem 1 and estimations of ℓ 1 on-average model stability. Meanwhile, some optimization bounds for FedZO are also provided. Table 1 and Table 2 report the comparisons of our results with the related theoretical analysis for ZoSS [36], AD-SGD [47], AD-PSGD [48], EF-ZO-SGD, FED-ZO-SGD [49], FedZO [16] and SGD [37, 38, 39, 40, 50] without the convexity requirement of loss function.

3.1 Generalization Analysis of General FedZO

This subsection considers the FedZO under the general setting.

Theorem 2. Let { w t } and { ¯ w t } be produced by FedZO (3) on S and S ( j i ) respectively, where η t = η 1 t -1 , η 1 ≤ (2 a 1 ) -1 with a 1 = ( 1 + √ d/b 2 ) β . After T iterations, we get the global parameters A ( S ) = w T and A ( S ( j i ) ) = ¯ w T respectively. Under Assumption 1, there holds

where a 2 = 2( nN ) -1 L + µβ +2( nN ) -1 L √ d/b 2 .

Theorem 2 states a ℓ 1 on-average model stability bound with order O ( ( ( nN ) -1 L + µ ) T 1 2 log T ) under mild conditions of parameter selection. When µ = O ( ( nN ) -1 ) , the upper bound is equal to O ( ( nN ) -1 LT 1 2 log T ) , which is comparable with the existing stability bounds for ZoSS [36] and SGD [37, 39, 40, 51, 52] under similar choices of step sizes.

The following corollary is derived by integrating Theorem 1 (a) and Theorem 2.

Corollary 1. Under the same conditions of Theorem 2, for FedZO (3) , there holds

When µ = O ( ( nN ) -1 ) , it provides the generalization bound O ( ( nN ) -1 L 2 T 1 2 log T ) for the general FedZO algorithm. When βc/ ( βc +1) ≥ 1 2 for some positive constant c , Corrollary 1

aligns with the previous generalization bounds of SGD algorithms for pointwise learning (Theorem 3.12 in [37]) and pairwise learning (Theorem 3.5 in [39] and Theorem 15 in [40]). [36] provided the first generalization error analysis for the minibatch ZoSS algorithm with both unbounded and bounded non-convex loss functions. Specifically, they presented the generalization bound n -1 (2 + C ) L 2 ( eT ) βc min { c + β -1 , c log( eT ) } ( C is a positive constant) with the decreasing step size η t ≤ c/ ( t Γ d b 2 ) , where Γ d b 2 = ( √ (3 d -1) /b 2 +1 ) . Under the constant step sizes η t ≤ log(1 + βc ) / ( Tβ Γ d b 2 ) and η t ≤ c/ ( T Γ d b 2 ) , they also showed the generalization bound n -1 (2 + C ) cL 2 and ( nβ ) -1 L 2 (2 + C ) ( e βc -1 ) . When βc ≥ 1 2 , our generalization bound can match their first result. Indeed, we also can get similar bounds as [36] for the special setting of constant step size. Detail comparisons are also provided in Table 1.

3.2 Learning Guarantees of FedZO with Heavy tails

Inspired by [43, 44], we further investigate the learning guarantees of FedZO with the smooth, heavy-tailed loss function and PL condition.

Theorem 3. Let { w t } and { ¯ w t } be produced by FedZO (3) on S and S ( j i ) respectively, where η t = η 1 t -1 , η 1 ≤ 1 -d/b 2 4 a 3 with a 3 = (1 + d/b 2 ) β , and 0 ≥ 1 -d/b 2 2 αη 1 -1 . After T iterations, we get the global parameters A ( S ) = w T and A ( S ( j i ) ) = ¯ w T respectively. Under Assumptions 1 (b), 2 and 3, there holds

Theorem 3 states the stability bound O ( ( ( nN ) -1 √ log T +1 ) ( nN ) -1 T 1 4 log T ) if taking µ = O ( ( nN ) -1 ) and a 1 (1 -d/b 2 ) 4 a 3 ≤ 1 4 . Compared with Theorem 2, the current result removes the dependence on the Lipschitz parameter L while the dependence on µ is improved from the partial dependence L/ ( nN ) + µ to the full dependence ( √ log T/ ( nN ) + 1) µ . The reason is that, motivated by [36], we decompose the gradient approximation into two parts, i.e., the difference between the unknown gradient and its expected estimator, the divergence between the expected and its empirical version. With the help of second-order Taylor expansion, the first part is bounded by Lemmas 4 and 5 in Appendix B.1 . Meanwhile, the second part is bounded by the PL condition. The detailed proof is provided by Equation (8) in Appendix B.4 .

Combining Theorem 1 (b) with Theorem 3, we derive the following generalization bound for the heavy-tailed FedZO.

Corollary 2. Under the same conditions of Theorem 3, for FedZO (3) , there holds

When µ = O ( ( nN ) -1 ) , a 1 (1 -d/b 2 ) 4 a 3 ≤ 1 4 and E [ F S ( w T ) ] = O ( ( nN ) -1 ) , Corollary 2 shows the generalization bound O ( ( ( nN ) -1 √ log T + 1 ) (4 θ ) θ ( nN ) -1 T 1 4 log T ) . With bounded variance of gradient and bounded loss function, [38] stated the generalization bound

O ( ( nδ ) -1 2 L 1 2 ( ( E [ v 2 ]) 1 4 + √ log T ) ) for SGD with high probability. Here, we further developed the analysis technique associated with the bounded variance of gradient to the federated learning setting by the fine-grained error analysis (see the proof of Theorem 1 (b)).

Theorem 4. Let { w t } be produced by FedZO (3) on S , where η t = η 1 t -1 , η 1 ≤ 1 -d/b 2 4 a 3 with a 3 = (1 + d/b 2 ) β and 0 ≥ 1 -d/b 2 2 αη 1 -1 . Under Assumptions 1 (b), 2 and 3, there hold

and

where w ( S ) , w ∗ are defined in (4) .

The PL condition can guarantee the identification of global minimizers. Therefore, we regard E [ F S ( w T ) -F S ( w ∗ ) ] , instead of ∥ ∥ ∇ F S ( w T ) ∥ ∥ , as the measure of optimization error in this paper. [40] provided the optimal optimization bound O ( 1 / ( Tα 2 ) ) with uniform stability tool for gradientdominated pairwise SGD. [16] characterized the convergence rate O ( ( MT ) -1 2 + dµ 2 ) of the FedZO algorithm with partial device participation. As shown in Table 2, Theorem 4 guarantees the optimal decay rate O ( T -1 ) on the optimization error as µ = O ( T -1 2 ) without the dependence of the dimension of hypothesis function space d and the random direction number b 2 . Note that, our optimization bounds rely on the quality of the initial model like many previous work (e.g. [16]).

4 Learning Guarantees of Asynchronous FedZO

Following the line of Section 3.2, we further study theoretical foundations for the asynchronous case of the FedZO algorithm. Considering the asynchrony among the local model parameters of different clients in the same iteration, we modify Equation (3) as follows

where t -t i ∈ [ t 0 ] denotes the delay of the i -th client in the t -th iteration, and t 0 = max t ∈ [ T ] { max i ∈ [ N ] t i - min i ∈ [ N ] t i } denotes the maximum delay for all clients in the whole update process of the global model.

Note that, if t i = t for some i , the parameter w t i i will be updated to w t +1 at the end of the t -th iteration. We state the differences between Equations (3) and (7) in Remark 2.

Theorem 5. Given S and S ( j i ) in Definition 1, let { w t } and { ¯ w t } be produced by asynchronous FedZO (7) on S and S ( j i ) respectively, where η t = η 1 t -1 , η 1 ≤ 1 -d/b 2 4 a 3 with a 3 = (1 + d/b 2 ) β and 0 ≥ 1 -d/b 2 2 αη 1 -1 . After T iterations, we get the global parameters A ( S ) = w T and A ( S ( j i ) ) = ¯ w T respectively. Under Assumptions 1 (b), 2 and 3, there holds

Asynchrony can cause discrepancies among the local models of various clients w t i , i = 1 , ..., N and the global model w t in the t -th iteration. Therefore, the primary challenge of asynchronous learning theory is to establish the relationship between ∥ ∥ w t i i -¯ w t i i ∥ ∥ and ∥ w t -¯ w t ∥ . To overcome this bottleneck, we design a new decomposition ∥ ∥ w t i i -¯ w t i i ∥ ∥ ≤ ∥ ∥ w t i i -w t -( ¯ w t i i -¯ w t )∥ ∥ + ∥ w t -¯ w t ∥

and then use second-order Taylor expansion to give an upper bound of the first term on the right-hand side of the inequality. When µ = O ( ( nN ) -1 ) and β (1 -d/b 2 ) 2 a 3 ≤ 1 2 , Theorem 5 provide a stability bound O ( (( 1 + ( nN ) -1 √ log T ) log T + √ log Tt 0 ) ( nN ) -1 T 1 2 ) .

Corollary 3. Under the same conditions of Theorem 5, for asynchronous FedZO (7) , there holds

Combining Theorem 1 (b) with Theorem 5, when µ = E [ F S ( w T )] = O ( ( nN ) -1 ) , β (1 -d/b 2 ) 2 a 3 ≤ 1 2 , Corollary 3 yields the generalization bound O ( (( 1 + ( nN ) -1 √ log T ) log T + √ log Tt 0 ) (4 θ ) θ ( nN ) -1 T 1 2 ) which appears to be the first generalization bound developed for asynchronous federated learning algorithms. It should be noted that, due to the more complex communication structure, using the condition t 0 = 0 (the synchronous case) can not recover the generalization bound in Corollary 2.

Theorem 6. Let { w t } be produced by FedZO (7) on S , where η t = η 1 t -1 , η 1 ≤ 1 -d/b 2 4 a 3 with a 3 = (1 + d/b 2 ) β and 0 ≥ 1 -d/b 2 2 αη 1 -1 . Under Assumptions 1 (b), 2 and 3, there hold

and

where w ( S ) , w ∗ are defined in (4) .

Theorem 6 also develops the first optimal optimization bound O ( T -2 + µ 2 ) for the asynchronous FedZO. Based on Equation (5), the excess risk bound O ( T -2 + ( ( 1 + ( nN ) -1 √ log T ) log T + t 0 √ log T ) (4 θ ) θ ( nN ) -1 T 1 2 ) can be directly derived by integrating Corollary 3 and this optimization bound when µ = E [ F S ( w T )] = O ( ( nN ) -1 ) .

5 Conclusion

This paper fills the gap of theoretical guarantees for both synchronous and asynchronous FedZO algorithms. We develop the first generalization bound for general FedZO after bridging the quantitative relationships between generalization error and ℓ 1 on-average model stability. Moreover, fine-grained learning theory analysis is established by means of the heavy-tailed condition and the second-order Taylor expansion, where the derived error bounds are satisfactory even compared with the previous results for traditional SGD [37, 32, 38] and recent ZoSS [36].

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (Nos. 12071166 and 62376104), the Fundamental Research Funds for the Central Universities of China (No. 2662023LXPY005), and HZAU-AGIS Cooperation Fund (No. SZYJY2023010).

References

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 54, pages 1273-1282, 2017.

Lichao Sun and Lingjuan Lyu. Federated model distillation with noise-free differential privacy. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI) , pages 1563-1570, 2021.

Lichao Sun, Jianwei Qian, and Xun Chen. LDP-FL: practical private aggregation in federated learning with local differential privacy. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI) , pages 1571-1578, 2021.

Robin C. Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective, 2017.

Peter Kairouz, Ziyu Liu, and Thomas Steinke. The distributed discrete gaussian mechanism for federated learning with secure aggregation. In Proceedings of the 38th International Conference on Machine Learning (ICML) , volume 139, pages 5201-5212, 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Feddane: A federated newton-type method. In 53rd Asilomar Conference on Signals, Systems, and Computers (ACSCC) , pages 1227-1231, 2019.

Xinwei Zhang, Mingyi Hong, Sairaj V. Dhople, Wotao Yin, and Yang Liu. Fedpd: A federated learning framework with adaptivity to non-iid data. IEEE Transactions on Signal Processing , 69:6055-6070, 2021.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. A novel framework for the analysis and design of heterogeneous federated learning. IEEE Transactions on Signal Processing , 69:5234-5249, 2021.

Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017.

Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard E. Turner, and Adrian Weller. Structured evolution with cmpact architectures for scalable policy optimization. In Proceedings of the 35th International Conference on Machine Learning (ICML) , volume 80, pages 969-977, 2018.

Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1805-1814, 2018.

Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems (NIPS) , pages 3981-3989, 2016.

Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security , pages 15--26, 2017.

Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning (ICML) , volume 80, pages 2137-2146, 10-15 Jul 2018.

Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NIPS) , pages 2960-2968, 2012.

Wenzhi Fang, Ziyi Yu, Yuning Jiang, Yuanming Shi, Colin N. Jones, and Yong Zhou. Communication-efficient stochastic zeroth-order optimization for federated learning. IEEE Transactions on Signal Processing , 70:5058-5073, 2022.

V. N. Vapnik. Statistical learning theory. Encyclopedia of the Sciences of Learning , 41(4):31853185, 1998.

Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. In Conference on Computational Learning Theory (COLT) , pages 224-240, 2001.

Felipe Cucker and Ding Xuan Zhou. Learning theory: An approximation theory viewpoint . Cambridge University Press, 2007.

Hong Chen, Xiaoqian Wang, Cheng Deng, and Heng Huang. Group sparse additive machine. In Advances in Neural Information Processing Systems (NeurIPS) , pages 198-208, 2017.

Stephen Smale and Ding-Xuan Zhou. Learning theory estimates via integral operators and their approximations. Constructive Approximation , 26:153-172, 2007.

Lorenzo Rosasco, Mikhail Belkin, and Ernesto De Vito. On learning with integral operators. Journal of Machine Learning Research , 11:905-934, 2010.

Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage, 2015.

Jiantao Jiao, Yanjun Han, and Tsachy Weissman. Dependence measures bounding the exploration bias for general measurements. In 2017 IEEE International Symposium on Information Theory (ISIT) , pages 1475-1479, 2017.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems (NIPS) , pages 2524-2533, 2017.

Amir R. Asadi, Emmanuel Abbe, and Sergio Verdú. Chaining mutual information and tightening generalization bounds. In Advances in Neural Information Processing Systems (NeurIPS) , pages 7245-7254, 2018.

Yuheng Bu, Shaofeng Zou, and Venugopal V. Veeravalli. Tightening mutual information based bounds on generalization error. In IEEE International Symposium on Information Theory (ISIT) , pages 587-591, 2019.

William Rogers and T. Wagner. A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statistics , 6:506-514, 1978.

Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research , 2:499-526, 2002.

André Elisseeff, Theodoros Evgeniou, and Massimiliano Pontil. Stability of randomized learning algorithms. Journal of Machine Learning Research , 6:55-79, 2005.

Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research , 11:2635-2670, 2010.

Yunwen Lei and Yiming Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning (ICML) , volume 119, pages 5809-5819, 2020.

Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics , 44(4):1363 - 1399, 2016.

Aymeric Dieuleveut, Nicolas Flammarion, and Francis R. Bach. Harder, better, faster, stronger convergence rates for least-squares regression. Journal of Machine Learning Research , 18:101:1101:51, 2017.

Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M. Roy. Information-theoretic generalization bounds for SGLD via data-pependent estimates. In Advances in Neural Information Processing Systems (NeurIPS) , pages 11013-11023, 2019.

Konstantinos Nikolakakis, Farzin Haddadpour, Dionysis Kalogerias, and Amin Karbasi. Blackbox generalization: Stability of zeroth-order learning. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pages 31525-31541, 2022.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning (ICML) , volume 48, pages 1225-1234, 2016.

Yi Zhou, Yingbin Liang, and Huishuai Zhang. Understanding generalization error of SGD in nonconvex optimization. Machine Learning , 111(1):345-375, 2022.

Wei Shen, Zhenhuan Yang, Yiming Ying, and Xiaoming Yuan. Stability and optimization error of stochastic gradient descent for pairwise learning, 2019.

Yunwen Lei, Mingrui Liu, and Yiming Ying. Generalization guarantee of SGD for pairwise learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 2121621228, 2021.

Mariia Vladimirova, Stéphane Girard, Hien Nguyen, and Julyan Arbel. Sub-Weibull distributions: Generalizing sub-Gaussian and sub-Exponential properties to heavier-tailed distributions. Stat , 9(1):e318:1-8, 2020.

Arun Kumar Kuchibhotla and Abhishek Chakrabortty. Moving beyond sub-Gaussianity in highdimensional statistics: Applications in covariance estimation and linear regression. Information and Inference: A Journal of the IMA , 11(4):1389-1456, 06 2022.

Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding priors in bayesian neural networks at the unit level. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning (ICML) , volume 97, pages 6458-6467, 2019.

Shaojie Li and Yong Liu. High probability guarantees for nonconvex stochastic gradient descent with heavy tails. In International Conference on Machine Learning (ICML) , volume 162, pages 12931-12963, 2022.

Liam Madden, Emiliano Dall'Anese, and Stephen Becker. High-probability convergence bounds for non-convex stochastic gradient descent, 2020.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the polyak-łojasiewicz condition. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) , volume 9851, pages 795-811, 2016.

Xiaoge Deng, Tao Sun, Shengwei Li, and Dongsheng Li. Stability-based generalization analysis of the asynchronous decentralized SGD. In the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI) , pages 7340-7348, 2023.

Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In Proceedings of the 35th International Conference on Machine Learning (ICML) , volume 80, pages 3049-3058, 2018.

Ege C. Kaya, Mehmet Berk Sahin, and Abolfazl Hashemi. Communication-efficient zerothorder distributed online optimization: Algorithm, theory, and applications. IEEE Access , 11:61173-61191, 2023.

Yunwen Lei and Yiming Ying. Sharper generalization bounds for learning with gradientdominated objective functions. In International Conference on Learning Representations (ICLR) , pages 1-23, 2021.

Yunwen Lei, Antoine Ledent, and Marius Kloft. Sharper generalization bounds for pairwise learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 2123621246, 2020.

Zhenhuan Yang, Yunwen Lei, Puyu Wang, Tianbao Yang, and Yiming Ying. Simple stochastic and online gradient descent algorithms for pairwise learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 20160-20171, 2021.

Peer reviews:
comments: SUMMARY:
This paper provides generalization analysis of federated zeroth order optimization.

STRENGTHS:
The paper is well-written and addresses a relevant problem. The theoretical results appear correct, although I haven't thoroughly checked them.

WEAKNESSES:
Since I'm not super familiar with this direction, I just have a few clarification questions I have asked below. The one limitation in my opinion is the lack of multiple-local updates at the clients.

QUESTIONS:
Theory:
- Line 105: what is the intuitive reason for the requirement $b_2 \geq d$?
- In (3), aren't we considering multiple local steps at the clients?
- In (5), shouldn't there be an additional difference $\mathbb E[F_S(w(S))] - F(w^*)$?
- In the paragraph following Theorem 1 in lines 187-88, it is said that $\mathbb E[F_S(A(S))]$ has no adverse impact on the upper bound. Why so? If this term is not close to zero, doesn't it introduce a bias, irrespective of $\epsilon$?
- Line 206: one can always find constant $c$ such that $\beta c \geq 1$ holds. Shouldn't there be some more conditions for this statement to hold?
- Line 228 "while introducing the dependence $(nN)^{-1}$": but theorem 2 also has $(nN)^{-1}$
- Line 249: what is $\alpha$ in this bound?
- Are all the results stated assuming full-client participation, since only $N$ appears in the bounds, even though (3) is with partial client participation ($M$).

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The analysis of Federated Zeroth-Order Optimization is limited now. This work considers the zeroth-order optimization in federated learning and establishes the generalization error bound of FedZO under the Lipschitz continuity and smoothness conditions.

STRENGTHS:
1. The analysis of Federated Zeroth-Order Optimization is limited now. This work provides generalization bounds with theoretical analysis and asynchronous FedZO.

WEAKNESSES:
There is no experimental analysis. 

1. In deep learning, the first-order stochastic optimizer is very popular. Although some cases are mentioned where gradient information is expensive to obtain, it is necessary to use experiments to verify the necessity of the zero-order algorithm. 

2. Experiment about the efficiency of algorithms are also missing.

QUESTIONS:
1. Could you provide more motivation to study zeroth-order optimization in federated learning? In practice, we have many stochastic optimizers in federated learning. Does zeroth-order optimization have any advantage compared with these optimizers? 

2. What is the key challenge of zeroth-order optimization in federated learning compared with the optimization in the single-machine setting?

LIMITATIONS:
No experiments are provided to verify their theoretical analysis
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a detailed analysis of Federated Zeroth-Order Optimization (FedZO) by developing the analysis technique of on-average model stability. The authors establish generalization error bounds for FedZO and refine them using heavy-tailed gradient noise and second-order Taylor expansion. They extend the analysis to the asynchronous case and contribute to systematic assessments, on-average model stability technique, and improved bounds for practical FedZO applications. In short, the main contributions of the paper are the establishment of systematic theoretical assessments of FedZO, the development of the analysis technique of on-average model stability, and the refinement of generalization and optimization bounds for practical applications of FedZO.\n\nSTRENGTHS:\n1. To my best knowledge, the authors provide the first generalization error bound of FedZO under the Lipschitz continuity and smoothness conditions, and also refine generalization and optimization bounds by replacing bounded gradient with heavy-tailed gradient noise and utilizing the second-order Taylor expansion for gradient approximation.\n2. This paper has also provided the theoretical analysis for asynchronous FedZO, which has in fact seldomly considered in the federated zeroth-order optimization field.\n3. The theoretical results are generally sound and can be inspiring for the follow-up works on federated zeroth order optimization.\n\nWEAKNESSES:\n1. While the authors provide a new error decomposition strategy for the asynchronous case, they do not compare their results with existing error bounds for other asynchronous optimization algorithms. \n2. This paper does not provide any experimental results to validate the theoretical findings, which may limit its practical relevance. Empirical verification may make this paper more sound. So, I encourage the authors to provide certain empirical experiments to validate their main theoretical results.\n\nQUESTIONS:\n1. Can the authors provide more intuition behind the on-average model stability analysis technique? It may be helpful for readers who are not familiar with this concept to have a more intuitive understanding of how it relates to the generalization error.\n2. The authors mention that their refined generalization and optimization bounds have important implications for practical applications of FedZO. Can they provide some examples of these practical applications and how their results could be used in these scenarios?\n3. The paper focuses on the theoretical analysis of FedZO. Can the authors provide some insights into how their results could be used in practical applications of federated learning? For example, how could their results be used to improve the performance of real-world federated learning systems?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studies the theoretical analysis for federated zeroth-order optimization (FedZO). The main contributions of this paper include 1) deriving the generalization bound of synchronous FedZO with different assumptions (bound gradient, heavy tail gradient noise). The main technical is to establish the relationship between generalization error and $\ell_1$ on-average model stability; 2) deriving the generalization and optimization bounds for asynchronous FedZO.

STRENGTHS:
1. This paper derives the generalization bound of synchronous FedZO with different assumptions (bound gradient, heavy tail gradient noise). The main technical is to establish the relationship between generalization error and $\ell_1$ on-average model stability; 

2. This paper derives the generalization and optimization bounds for asynchronous FedZO with novel technique by a new error decomposition strategy.

3. The paper is very well written. In particular, it explains the main technical challenges, main techniques and the implication of the results very well.

WEAKNESSES:
I didn't find major weaknesses in this paper. Perhaps, it will be beneficial if the authors could provide some experimental study to validate the theories.

QUESTIONS:
1. It seems that (5) does not equal to $\mathbb{E}[F(A(S)) - F(\omega^*)]$. Could you please clarify?

2. In the paragraph below (4), do you mean Algorithm 1 instead of Algorithm A?

3. In Assumption 2, does $z_i^t$ means a data sample?

4. In Theorems 2, 3 and so on, what is the physical meaning of $\mu$?

LIMITATIONS:
There are no much discussion on the limitation of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

comments: SUMMARY:
This paper fills the gap of theoretical guarantee for the Federated zeroth-order optimization (FedZO) algorithm. It provides the initial generalization error bound for FedZO and presents refined generalization and optimization bounds. The structure of the paper is logical and the accompanying theoretical proofs in the supplementary material are comprehensive and robust.

STRENGTHS:
1. The need for filling the theoretical gap is important.
2. The Author's contribution is solid.
3. This paper is well-organized and easy to follow.

WEAKNESSES:
This paper addresses a topic that isn't directly within my area of expertise, so my comments may not fully capture the nuances specific to this field, but I hope my remarks will prove useful to the authors.

**1.** In Assumption 3, the authors refer to the PL condition but they do not explain what PL stands for earlier in the text.

**2.** Regarding the assumptions made throughout the paper, it would be helpful if the authors could elaborate on whether each assumption is considered strong, mild, or weak, or if it is a relaxed version of a concept from previous work.

**3.** The term "heavy-tailed" is used but not clearly defined. It would benefit readers if the authors could briefly describe what they mean by "heavy-tailed" early in the paper. In the context of imbalanced regression problems, is "heavy-tailed" equivalent to "long-tailed"? In my understanding, both terms refer to highly biased or imbalanced data.

**4.** There are minor grammatical errors that need to be corrected.

In conclusion, this paper is well grounded and presents detailed, solid findings.

QUESTIONS:
See above

LIMITATIONS:
Authors did not provide the limitation section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization and optimization analysis of the Federated zeroth-order optimization (FedZO) algorithm. It develops tailored techniques for the federated setting to establish generalization bounds. One bound relies on Lipschitz continuity, and the second removes this dependency. The optimization error in the order of $\\mathcal{O}(1/T)$ is also developed for FedZO. The authors extend results to the asynchronous federated setting, where all the workers participate throughout the update process and asynchrony may cause inconsistency in local workers within the same iteration.\n\nSTRENGTHS:\nThe paper is well-crafted, and the authors effectively elucidate the distinctions from prior methods.\n\nThe theoretical results are quite comprehensive. Existing techniques in the literature may not be directly applicable to the FedZO algorithm. The authors fill in this gap by first developing error decomposition and estimation techniques and then presenting the first algorithmic stability-based generalization analysis for FedZO.\n\nWEAKNESSES:\nThe paper demonstrates a thorough analysis of the existing federated learning algorithm, FedZO, which is commendable. However, to further enhance its novelty and impact, it would have been valuable if the authors had introduced a new algorithm and conducted a comparative analysis against FedZO.\n\nFurthermore, to gain deeper insights into the analysis, it is recommended to include empirical evaluations exploring the effects of different parameters on the bounds. This empirical investigation would provide valuable practical implications and strengthen the overall findings.\n\nQUESTIONS:\nUnlike other existing papers, all the findings in this paper rely on the $\\theta$ tail parameter of the Sub-Weibull distribution. The extent of this dependence's impact is uncertain in practical settings, and it would be helpful if the authors could provide insights or comments on this matter.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides theoretical guarantees for both synchronous and asynchronous FedZO algorithms. It first establishes a generalization error bound of FedZO under conventional assumptions. Then, the bounds are further improved by using the second-order Taylor expansion and heavy-tailed gradient noise. The theoretical results seem promising; however, some technical concerns remain unresolved.\n\nSTRENGTHS:\nThis paper focuses on providing theoretical guarantees for the newly proposed FedZO algorithms. The mathematical tools used in the analysis are relatively advanced in recent literature. In addition to analyzing the original FedZO algorithm, this paper also extends its analysis to include asynchronous FedZO.\n\nWEAKNESSES:\nThis paper primarily focuses on providing further theoretical guarantees for a recently proposed algorithm, which addresses a minor problem. As a result, the contributions and impacts to the federated learning community are limited. Furthermore, some assumptions and results are not well-justified, as discussed below.\n\n[Related to federated learning]\n\nIn federated optimization literature, heterogeneous data distribution across clients is a significant challenge. In the original FedZO paper [16], Assumptions 3 and 4 were used to describe the impacts of such heterogeneity. However, this paper seems to have overlooked this necessary aspect in the analysis, which could significantly affect the proof of Theorem 2.\n\n[Experiment]\n\nThis paper introduces asynchronous FedZO and provides learning guarantees compared to FedZO. However, the original FedZO does not evaluate the asynchronous version. It is crucial to include experimental evaluations of both synchronous FedZO and asynchronous FedZO in this paper to support the proposed theories.\n\n[Organization]\n\nOne key aspect of this paper is the approximation of the first-order gradient using a second-order Taylor expansion and Assumption 2. However, the details of these methods are missing in the main paper, making it challenging to follow the arguments.\n\nQUESTIONS:\n[Comparison of generalization bounds]\n\nThis paper has mentioned, \"Due to the essential difference between FedZO and ZoSS, the previous analysis technique in [36] cannot be used for federated learning directly.\" So, why are the generalization bounds comparable with ZoSS in Table 1? It seems more reasonable to compare them with distributed zero-order optimization.\n\n[Comparison of optimization bounds]\n\nA vital challenge of zero-order optimization is the dimension issue of model parameters, as shown in Table 1 in FedZO. In this paper, how does it remove the dimension $d$ from the optimization bound? Is this improvement applicable to arbitrary zero-order optimization?\n\nLIMITATIONS:\nNo limitations are discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}