SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jun Chen $^{1}$ , Hong Chen $^{1,2,5,6*}$ , Bin Gu $^{3,4}$ , Hao Deng $^{1*}$
<sup>1</sup>College of Informatics, Huazhong Agricultural University, China
$^{2}$ Engineering Research Center of Intelligent Technology for Agriculture, China
$^{3}$ School of Artificial Intelligence, Jilin University, China
<sup>4</sup>Mohamed bin Zayed University of Artificial Intelligence
Shenzhen Institute of Nutrition and Health, Huazhong Agricultural University, China
Shenzhen Branch, Guangdong Laboratory for Lingnan Modern Agriculture,
Genome Analysis Laboratory of the Ministry of Agriculture,
Agricultural Genomics Institute at Shenzhen, Chinese Academy of Agricultural Sciences, China
cj850487243@163.com, chenh@mail.hzau.edu.cn,
jsgubin@gmail.com, dengh@mail.hzau.edu.cn
Federated zeroth-order optimization (FedZO) algorithm enjoys the advantages of both zeroth-order optimization and federated learning, and has shown exceptional performance on black-box attack and softmax regression tasks. However, there is little generalization analysis for FedZO, and its analysis on computing convergence rate is slower than the corresponding first-order optimization setting. This paper aims to establish systematic theoretical assessments of FedZO by developing the analysis technique of on-average model stability. We establish the first generalization error bound of FedZO under the Lipschitz continuity and smoothness conditions. Then, refined generalization and optimization bounds are provided by replacing bounded gradient with heavy-tailed gradient noise and utilizing the second-order Taylor expansion for gradient approximation. With the help of a new error decomposition strategy, our theoretical analysis is also extended to the asynchronous case. For FedZO, our fine-grained analysis fills the theoretical gap on the generalization guarantees and polishes the convergence characterization of the computing algorithm.
Federated learning collaborates multiple local clients to train a global model without sharing local raw data, which often enjoys great ability in protecting data privacy [1]. The core training steps of federated learning include local clients receiving the global model from the central server, the local models being updated by the global information and local data, and the updated local models being uploaded to renew the global model. Based on this building-block process, rich federated learning algorithms have been formulated to match different motivations, where their properties on privacy protection [2, 3, 4, 5] and convergence rate [6, 7, 8] are investigated. In general, the existing algorithms of federated learning depend heavily on the gradient information of loss function, see e.g., [1, 6, 7, 8]. Indeed, there are some learning scenarios where the gradient or Hessian information
is either unobtainable or too expensive to obtain, such as reinforcement learning [9, 10, 11], meta learning [12], black-box adversarial attacks [13, 14] and hyperparameter tuning [15]. To alleviate the dependence on the gradient or Hessian information, a federated zeroth-order optimization (FedZO) algorithm is proposed in [16] by integrating zeroth-order optimization into federated learning. In theory, the analysis of optimization convergence of FedZO has been established in [16], which shows the linear speedup in terms of the numbers of local iterations and clients participating in the update for the global iteration. However, there is little generalization analysis for federated zeroth-order optimization.
The goal of generalization analysis is to evaluate the capability of the empirical risk to approach the theoretical optimal risk on a given dataset. To characterize the generalization performance theoretically, there are four popular analysis techniques including uniform convergence approaches [17, 18, 19, 20], operator approximation techniques [21, 22], information-theoretic tools [23, 24, 25, 26, 27], and algorithmic stability analysis [28, 29, 30, 31]. Usually, stability-based generalization assessment enjoys some benefits over other tools. Firstly, in contrast to uniform convergence approaches, the theoretical guarantees derived from algorithmic stability analysis are independent of the capacity measurement of hypothesis function space [32]. Secondly, algorithmic stability analysis is available in a wide range of applications rather than only some models enjoying operator representation [33, 34]. Finally, data distribution does not affect the results of algorithmic stability analysis, whereas information-theoretic tools are typically sensitive to data distribution [25, 35]. To the best of our knowledge, there is only one work on the stability-based generalization analysis for the zeroth-order optimization, i.e., zeroth-order stochastic search (ZoSS) method [36]. Following this line, it is natural to further investigate the generalization guarantees of the FedZO algorithm. However, due to the essential difference between FedZO and ZoSS, the previous analysis technique in [36] can not be used for federated learning directly. In this paper, we develop the fine-grained error decomposition and estimations to overcome the challenge induced by the federated algorithmic formulation.
As a general assumption for loss function, Lipschitz continuity has been employed in many stability-based generalization assessments, see e.g., [36, 37, 38, 39, 40]. However, the index of Lipschitz continuity is likely too large or even infinite for some learning problems, which makes the previous results invalid [32]. With the help of the on-average model stability tool, [32] established the fine-grained generalization analysis of stochastic gradient descent (SGD) by removing the Lipschitz continuity and the convexity of each loss function, and relaxing the smoothness to Holder continuity. Moreover, [38] additionally considered the bounded variance of the stochastic gradient to get the generalization bounds of non-convex SGD with high probability. Inspired by [32] and [38], this paper considers the heavy-tailed gradient noise (Assumption 2) as a refined version of bounded variance of gradient and adopts the second-order Taylor expansion for gradient approximation to remove bounded gradient condition. Conclusively, we fill the theoretical gap in the generalization analysis of the FedZO algorithm [16] and its asynchronous version. The main contributions of this paper are outlined as follows.
- Generalization bounds of FedZO. We provide the first generalization bound of general FedZO after building the relationships between generalization error and  $\ell_1$  on-average model stability. To alleviate the restriction of bounded gradient, we further get a refined generalization bound and an optimal optimization bound under the condition of heavy-tailed gradient noise and the second-order Taylor expansion of gradient approximation.
- Learning guarantees of asynchronous FedZO. For asynchronous FedZO, we design a new error decomposition strategy to bridge the relationships between each local model parameter and the global model parameter in each iteration. Then, the generalization and optimization bounds are derived for the asynchronous case. In particular, our fine-grained error bounds are tight even compared with the previous results for SGD implemented by the first-order optimization [32, 37, 38] and zeroth-order optimization [36].
This section introduces some notations and definitions preparing for our theoretical analysis. Besides, we also give some detailed explanations for every symbol in Appendix A.
Considering a zeroth-order optimization algorithm for federated learning, we rely on  $N$  clients to independently train a global model. In each iteration, a subset of  $M$  clients is randomly selected to use their local data to obtain the corresponding gradient information. Then, this information is transmitted to the central server to update the global model parameter. For any  $i \in [N]$ , let  $z_i$  be a random variable obeying data distribution  $\mathcal{D}_i$  associated with the  $i$ -th client. Denote  $S := \{S_i\}_{i=1}^N$  as the total dataset, where  $S_i := \{z_{i1}, \dots, z_{in}\}$  is the  $i$ -th local dataset and each sample  $z_{ij}$  is drawn independently from the distribution  $\mathcal{D}_i$ ,  $i \in [N]$ .
Federated learning aims to optimize model parameters with the collaboration among all local clients, i.e., minimizing the following population risk
$$
F (w) := \frac {1}{N} \sum_ {i = 1} ^ {N} F _ {i} \left(w _ {i}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {z _ {i} \sim \mathcal {D} _ {i}} \left[ f _ {i} \left(w _ {i}; z _ {i}\right) \right], \tag {1}
$$
where  $w, w_i \in \mathcal{W} \in \mathbb{R}^d$  denotes the training parameters for the global model and the  $i$ -th local model in a  $d$ -dimensional hypothesis space respectively,  $F(w)$  and  $F_i(w_i)$  indicate the population risks for the global model and the local model of the  $i$ -th client respectively,  $f_i$  denotes the loss function of the  $i$ -th local client, and  $\mathbb{E}_{z_i \sim \mathcal{D}_i}[\cdot]$  denotes the conditional expectation with respect to (w.r.t.) the sample  $z_i$ . Generally, the unattainability of the local population risk  $F_i(w_i)$  forces us to train the model by minimizing the following empirical approximation of population risk  $F(w)$
$$
F _ {S} (w) := \frac {1}{N} \sum_ {i = 1} ^ {N} F _ {S _ {i}} \left(w _ {i}\right) = \frac {1}{n N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} f _ {i} \left(w _ {i}; z _ {i j}\right), \tag {2}
$$
where  $F_{S}(w)$  and  $F_{S_i}(w_i)$  indicate the empirical risks for the global model and the local model of the  $i$ -th client respectively. Note that, for the update of the global model, the contribution of each client is treated equally. This paper considers the learning scenarios where the gradients (or Hessian information) of local loss functions are either unobtainable or too expensive to obtain [16]. Naturally, the first-order gradient estimation  $\tilde{\nabla} f_i$ , defined as, for any  $t, b_2 \in \mathbb{N}, m \in [b_1], b_1 \in [n]$  ( $[n] := \{1, \dots, n\}$ ),
$$
\tilde {\nabla} f _ {i} \left(w _ {i} ^ {t}; z _ {i, m} ^ {t}, \{v _ {i, l} ^ {t} \} _ {l = 1} ^ {b _ {2}}, \mu\right) = \frac {1}{b _ {2}} \sum_ {l = 1} ^ {b _ {2}} \frac {v _ {i , l} ^ {t}}{\mu} \left(f _ {i} \left(w _ {i} ^ {t} + \mu v _ {i, l} ^ {t}; z _ {i, m} ^ {t}\right) - f _ {i} \left(w _ {i} ^ {t}; z _ {i, m} ^ {t}\right)\right),
$$
is chosen to update the model parameters of federated learning, where  $w_{i}^{t}$  is the local model parameter of the  $i$ -th client at the  $t$ -th iteration,  $\left\{z_{i,m}^{t}\right\}_{m=1}^{b_{1}}$  and  $\left\{v_{i,l}^{t}\right\}_{l=1}^{b_{2}}$  are two sets of independent and identically distributed (i.i.d.) random samples and i.i.d. random direction vectors (satisfying the  $d$ -dimensional uniform distribution), and  $\mu$  represents the distance between two model parameters  $(w_{i}^{t} + \mu v_{i,l}^{t}$  and  $w_{i}^{t})$  used to estimate gradient in the  $l$ -th direction. In our analysis, the following second-order Taylor expansion is employed to approximate  $\tilde{\nabla} f_{i}$ ,
$$
\tilde {\nabla} f _ {i} \left(w _ {i} ^ {t}; z _ {i j} ^ {t}\right) = \frac {1}{b _ {2}} \sum_ {l = 1} ^ {b _ {2}} \left(\langle \nabla f _ {i} (w _ {i} ^ {t}; z _ {i, j} ^ {t}), v _ {i, l} ^ {t} \rangle v _ {i, l} ^ {t} + \left(\frac {\mu}{2} (v _ {i, l} ^ {t}) ^ {\top} \nabla_ {w _ {i}} ^ {2} f _ {i} (w _ {i}; z _ {i j} ^ {t}) | _ {w _ {i} = w _ {i, l} ^ {t *}} v _ {i, l} ^ {t}\right) v _ {i, l} ^ {t}\right).
$$
Note that, the number  $b_{2}$  of random direction vectors is set to be greater than the hypothesis space dimension  $d$  in this paper, such as the requirement of [13]  $(b_{2} = 2d)$ .
The update process of FedZO is formulated as follows,
$$
w ^ {t + 1} = w ^ {t} - \frac {\eta_ {t}}{b _ {1} M} \sum_ {i \in \mathcal {M} _ {t}} \sum_ {m = 1} ^ {b _ {1}} \tilde {\nabla} f _ {i} \left(w _ {i} ^ {t}; z _ {i, m} ^ {t}, \left\{v _ {i, l} ^ {t} \right\} _ {l = 1} ^ {b _ {2}}, \mu\right), \quad w _ {i} ^ {t} = w ^ {t}, \tag {3}
$$
where  $\eta_t$  and  $\mathcal{M}_t \in [N](|\mathcal{M}_t| = M)$  denote the step size and the collection of selected client indices in the  $t$ -th iteration, respectively. In particular, we also consider the asynchronous FedZO algorithm.
$$
w ^ {*} \in \arg \min  _ {w \in \mathcal {W}} F (w) \text {a n d} w (S) \in \arg \min  _ {w \in \mathcal {W}} F _ {S} (w), \tag {4}
$$
where  $F(w), F_S(w)$  are defined in (1) and (2). Denote  $A(S) = w^T$  as the output of the global model after  $T$  iterations with any federated learning algorithm  $A$  (including Algorithm 1, i.e.,
synchronous FedZO). Typically, the excess risk of  $A(S)$  is measured by  $\mathbb{E}[F(A(S)) - F(w^{*})]$  and can be decomposed as
$$
\mathbb {E} [ F (A (S)) - F \left(w ^ {*}\right) ] \leq \mathbb {E} [ F (A (S)) - F _ {S} (A (S)) ] + \mathbb {E} [ F _ {S} (A (S)) - F _ {S} (w (S)) ], \tag {5}
$$
where  $\mathbb{E}[\cdot ]$  denotes the expectation w.r.t. all randomness. The first part of (5) (called generalization error) measures the expected gap between population risk and empirical risk, while the second part of (5) (called optimization error) measures the divergence between the trained model and the theoretically optimal model. In this paper, we simultaneously bound the generalization error and optimization error of FedZO.
Remark 1. The update strategy (3) is associated with a minibatch of i.i.d. samples. Due to the interaction among samples, it is hard to establish the stability-based generalization analysis of the minibatch case directly. Fortunately, an equivalent formula can tackle this difficulty by means of the properties of binomial distribution (see Appendix B.3).
Remark 2. Compared with synchronous FedZO ([16] in Algorithm 1), there are several essential differences for its asynchronous one described in Section 4. Firstly, we assume that all clients participate throughout the entire update process of the global model parameters. Secondly, once some client finishes gradient computation, this gradient will be transmitted to update the global model immediately without waiting for other clients. Then, the updated parameters of the global model will be back to the corresponding client, not the other ones. As a result, the local model of some client may be inconsistent with other clients within the same iteration.
Algorithm 1 Synchronous FedZO
Require:  $w^{1}$  : initial global model;  $\eta_{1}$  : initial learning rate;  $b_{1},b_{2}$  : minibatch sizes for samples and direction vectors respectively;  $\mu$  : positive step size in the definition of the derivative;  $M$  : number of clients selected to update the global model in each iteration   
for all  $t = 1,\dots,T - 1$  do Randomly select a clients set  $\mathcal{M}_t$  , let  $\eta_t = \eta_1 / t$    
for all  $i\in \mathcal{M}_t$  in parallel do Let  $w_i^t = w^t$  Generate  $\left\{z_{i,m}^t\right\}_{m = 1}^{b_1}$  and  $\left\{v_{i,l}^t\right\}_{l = 1}^{b_2}$  from  $\mathcal{D}_i$  and  $d$  -dimensional uniform distribution Compute  $\sum_{m = 1}^{b_1}\tilde{\nabla} f_i$  and upload it to global model   
end for Update  $w^{t}$  to  $w^{t + 1}$  by Eq. (3)   
end for   
Ensure: Final global model  $w^{T}$
Inspired by Definition 4 in [32], we introduce a new definition of  $\ell_1$  on-average model stability for federated learning.
Definition 1. The federated learning algorithm  $A$  is  $\ell_1$  on-average model  $\epsilon$ -stable if
$$
\mathbb {E} \left[ \frac {1}{n N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \left\| A \left(S ^ {\left(j _ {i}\right)}\right) - A (S) \right\| \right] \leq \epsilon , \tag {6}
$$
where  $\| \cdot \|$  is the Euclidean distance  $\| \cdot \| _2$ ,  $S = \{S_i\}_{i = 1}^N$ ,  $S^{(j_i)} = \left\{S_1,\dots,S_{i - 1},S_i^{(j)},S_{i + 1},\dots,S_N\right\}$ ,  $S_{i} = \{z_{i1},\ldots ,z_{in}\}$ ,  $S_{i}^{(j)} = \{z_{i1},\dots,z_{i(j - 1)},z_{ij}'',z_{i(j + 1)},\dots,z_{in}\}$  ( $z_{ij}'$  is drawn from  $\mathcal{D}_i, \forall i \in [N], j \in [n]$ ).
In Definition 1, we assume that only one sample of one client is perturbed, not other clients. Particularly, as  $N = 1$ , the current definition is consistent with the standard on-average model stability in [32, 38]. Compared with several other stability tools, the advantages of on-average model stability are listed as follows. First, on-average model stability is a weaker stability tool than uniform (model) stability. Second, on-average model stability measures the stability of model parameters  $w$  instead of function values  $f(w)$  like on-average stability, which can improve our analysis.
Definition 2. [41] For some  $\theta, a, b > 0$  and all  $x > 0$ , if a random variable  $X$  satisfying
$$
\mathbb {P} \left(| X | \geq x\right) \leq a \exp \left(- b x ^ {\frac {1}{\theta}}\right),
$$
we call it sub-Weibull random variable (denoted as  $X \sim \text{sub}W(\theta)$ ), where  $\theta$  is a tail parameter.
Sub-Weibull distribution is a heavier-tailed (longer-tailed) distribution than the sub-Gaussian  $\left(\theta = \frac{1}{2}\right)$  and sub-Exponential  $(\theta = 1)$  distribution, which matches some real applications [42] and has been used for learning theory analysis recently [43, 44]. This paper utilizes some sub-Weibull properties, derived from the moment generating function (MGF), to relax the bounded gradient assumption used in [36, 37, 38, 39, 40].
3 Evaluating the Generalization of FedZO via Stability
This section establishes the relationships between the generalization error of FedZO and the on-average model stability in Theorem 1. Detailed proofs are provided in Appendix B.2.
This paper makes the following assumptions for the loss function  $f_{i}$  of  $i$ -th local client ( $i \in [N]$ ).
Assumption 1. (a) ( $L$ -Lipschitz continuity). For any  $w_i, \bar{w}_i \in \mathcal{W}, i \in [N]$  and some  $L > 0$ ,  $f_i$  satisfies  $\nabla f_i(w_i; z_i) \leq L$ , that is
$$
\left| f _ {i} \left(w _ {i}; z _ {i}\right) - f _ {i} \left(\bar {w} _ {i}; z _ {i}\right) \right| \leq L \left\| w _ {i} - \bar {w} _ {i} \right\|, \forall z _ {i} \sim \mathcal {D} _ {i}.
$$
(b) ( $\beta$ -Smoothness). For any  $w_i, \bar{w}_i \in \mathcal{W}, i \in [N]$  and some  $\beta > 0$ ,  $f_i$  satisfies
$$
\left\| \nabla f _ {i} \left(w _ {i}; z _ {i}\right) - \nabla f _ {i} \left(\bar {w} _ {i}, z _ {i}\right) \right\| \leq \beta \left\| w _ {i} - \bar {w} _ {i} \right\|, \forall z _ {i} \sim \mathcal {D} _ {i}.
$$
The requirement of Lipschitz continuity and smoothness is general in statistical learning theory, see e.g., [37, 38, 39, 40]. Usually, it appears to be justifiable when the hypothesis function space is uniformly bounded. However, the Lipschitz constant  $L$  may be infinite for some learning tasks with unbounded hypothesis function space [32]. In this paper, two attempts are proposed to remove the Lipschitz assumption. Firstly, we focus on the zeroth-order optimization problem, where the gradient information is unavailable. We approximate the first-order gradient using a second-order Taylor expansion to avoid the dependence on the Lipschitz constant (see the proofs of Theorems 2, 3 and 5). Secondly, we use a weaker gradient-related assumption, i.e., the heavy-tailed gradient noise assumption, to replace the Lipschitz assumption in our generalization analysis (see the proof of Theorem 1 (b)).
Assumption 2. (Heavy-tailed gradient noise). Let the tail parameter  $\theta >\frac{1}{2}$ , the number of iterations  $t > 0$  and the client index  $i\in [N]$ . For any  $w_{i}^{t}\in \mathcal{W},z_{i}^{t}\in S_{i},S_{i}\in \mathcal{D}_{i}^{n}$ , we assume that the gradient noise  $\nabla f_i(w_i^t;z_i^t) - \nabla F_{S_i}(w_i^t)$  is a sub-Weibull random vector, i.e.,  $\nabla f_{i}(w_{i}^{t};z_{i}^{t}) - \nabla F_{S_{i}}(w_{i}^{t})\sim$  subW(θ).
From Theorem 2.1 in [41], we deduce that the MGF of  $\| \nabla f_i(w_i^t;z_i^t) - \nabla F_{S_i}(w_i^t)\|^{\frac{1}{\theta}}$  can be bounded at some point, i.e.,  $\mathbb{E}\left[\exp \left(\frac{\|\nabla f_i(w_i^t;z_i^t) - \nabla F_{S_i}(w_i^t)\|}{K}\right)^{\frac{1}{\theta}}\right] \leq 2$  for some  $K > 0$ . Moreover, a random variable  $X$  is defined as  $K$ -sub-Weibull( $\theta$ ) if  $\mathbb{E}\left[\exp \left(|X| / K\right)^{1/\theta}\right] \leq 2$  [45]. Therefore, the gradient noise in Assumption 2 can be also denoted as  $\nabla f_i(w_i^t;z_i^t) - \nabla F_{S_i}(w_i^t) \sim subW(\theta,K)$ . In this paper, a lemma (Lemma 2 in Appendix B.1) related to the bounded  $p$ -th norm of the sub-Weibull variable is used to build a novel relationship (Theorem 1 (b)) between  $\ell_1$  on-average model stability and generalization error.
Assumption 3. (PL condition). For any  $w \in \mathcal{W}$ ,  $S \in \bigcup_{i=1}^{N} \mathcal{D}_i^n$  and some  $\alpha > 0$ , the empirical risk  $F_S(w)$  satisfies
$$
\mathbb {E} \left[ \| \nabla F _ {S} (w) \| ^ {2} \right] \geq 2 \alpha \mathbb {E} \left[ \left(F _ {S} (w) - F _ {S} (w (S))\right) \right].
$$
Assumption 3 elucidates that the quadratic gradient of the empirical risk enjoys a linearly decreasing lower bound [46]. Without the setting of convexity, the gradient of empirical risk  $\| \nabla F_S(w)\| = 0$  is just a sufficient condition for a local optimal model instead of the guarantee to find a global optimal
parameter. This assumption implies that all local optimal parameters are global optimal parameters, that is,  $\| \nabla F_S(w)\| = 0$  implies that  $\mathbb{E}[F_S(w)] = \mathbb{E}[F(w(S))] = \mathbb{E}[F(w^*)]$  holds in expectation. For this reason, we can study the optimization error with the form  $F_{S}(w) - F_{S}(w(S))$  rather than  $|\nabla F_S(w)|$  [44] under the non-convex condition.
Now we state two quantitative relationships between generalization error and  $\ell_1$  on-average model stability.
Theorem 1. Let  $S, S^{(j_i)}$  be given in Definition 1 and  $A$  be a federated learning algorithm.
(a) Let Assumption 1 (a) hold and  $A$  be  $\ell_1$  on-average model  $\epsilon$ -stable. Then, for some  $L > 0$ ,
$$
\left| \mathbb {E} [ F (A (S)) - F _ {S} (A (S)) ] \right| \leq \frac {L}{n N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \mathbb {E} \left[ \left\| A (S ^ {(j _ {i})}) - A (S) \right\| \right] \leq L \epsilon .
$$
(b) Let Assumption 2 hold and  $A$  be  $\ell_1$  on-average model  $\epsilon$ -stable. Then, for some  $\theta > \frac{1}{2}, K > 0$ ,
$$
\begin{array}{l} \left| \mathbb {E} [ F (A (S)) - F _ {S} (A (S)) ] \right| \leq \frac {(4 \theta) ^ {\theta} K}{n N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \mathbb {E} \left[ \left\| A \left(S ^ {\left(j _ {i}\right)}\right) - A (S) \right\| \right] + 2 \mathbb {E} [ F _ {S} (A (S)) ] \\ \leq (4 \theta) ^ {\theta} K \epsilon + 2 \mathbb {E} [ F _ {S} (A (S)) ]. \\ \end{array}
$$
Theorem 1 provides the generalization bounds of the FedZO algorithm by the  $\ell_1$  on-average model stability. Essentially, Theorem 1 (a) is consistent with Theorem 2 (a) in [32] and Theorem 1 (b) states a refined upper bound independent of the Lipschitz constant  $L$ . Explicitly, in Theorem 1 (b), the tail parameter  $\theta$  in the first term is typically bounded [41] and the second term  $\mathbb{E}\left[F_S(A(S))\right]$  has no significant negative impact on the upper bound [32].
In the sequel, we derive the generalization bounds by integrating Theorem 1 and estimations of  $\ell_1$  on-average model stability. Meanwhile, some optimization bounds for FedZO are also provided. Table 1 and Table 2 report the comparisons of our results with the related theoretical analysis for ZoSS [36], AD-SGD [47], AD-PSGD [48], EF-ZO-SGD, FED-ZO-SGD [49], FedZO [16] and SGD [37, 38, 39, 40, 50] without the convexity requirement of loss function.
3.1 Generalization Analysis of General FedZO
This subsection considers the FedZO under the general setting.
Theorem 2. Let  $\{w^t\}$  and  $\{\bar{w}^t\}$  be produced by FedZO (3) on  $S$  and  $S^{(j_i)}$  respectively, where  $\eta_t = \eta_1 t^{-1}$ ,  $\eta_1 \leq (2a_1)^{-1}$  with  $a_1 = \left(1 + \sqrt{d / b_2}\right) \beta$ . After  $T$  iterations, we get the global parameters  $A(S) = w^T$  and  $A(S^{(j_i)}) = \bar{w}^T$  respectively. Under Assumption 1, there holds