SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{n N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \mathbb {E} \left[ \left\| w ^ {T} - \bar {w} ^ {T} \right\| \right] \leq \left(e (T - 1)\right) ^ {a _ {1} \eta_ {1}} a _ {2} \eta_ {1} \log \left(e (T - 1)\right),
$$
where  $a_2 = 2(nN)^{-1}L + \mu \beta + 2(nN)^{-1}L\sqrt{\frac{d}{b_2}}$ .
Theorem 2 states a  $\ell_{1}$  on-average model stability bound with order  $\mathcal{O}\left(\left((nN)^{-1}L + \mu\right)T^{\frac{1}{2}}\log T\right)$  under mild conditions of parameter selection. When  $\mu = \mathcal{O}\left((nN)^{-1}\right)$ , the upper bound is equal to  $\mathcal{O}\left((nN)^{-1}LT^{\frac{1}{2}}\log T\right)$ , which is comparable with the existing stability bounds for ZoSS [36] and SGD [37, 39, 40, 51, 52] under similar choices of step sizes.
The following corollary is derived by integrating Theorem 1 (a) and Theorem 2.
Corollary 1. Under the same conditions of Theorem 2, for FedZO (3), there holds
$$
\left. \left| \mathbb {E} \left[ F \left(w ^ {T}\right) - F _ {S} \left(w ^ {T}\right) \right] \right| \leq \mathcal {O} \left(L \left(\left(n N\right) ^ {- 1} L + \mu\right) T ^ {\frac {1}{2}} \log T\right). \right.
$$
When  $\mu = \mathcal{O}\left((nN)^{-1}\right)$ , it provides the generalization bound  $\mathcal{O}\Big((nN)^{-1}L^2 T^{\frac{1}{2}}\log T\Big)$  for the general FedZO algorithm. When  $\beta c / (\beta c + 1) \geq \frac{1}{2}$  for some positive constant  $c$ , Corollary 1
Table 1: Comparisons of stability-based generalization bounds under the non-convex condition (Thm.-Theorem; Cor.-Corollary;  $c$ -a positive constant;  $v^2$ -the upper bound of the variance of gradient;  $\lambda$ -a parameter characterizing the properties of decentralized topology;  $\delta$ -high probability; *-high probability bound; B.-bounded loss function; Uni.-uniform stability;  $\ell_1 - \ell_1$  on-average model stability;  $\sqrt{}$ -has such a property;  $\times$ -hasn't such a property;  $\Gamma_{b_2}^d = \left(\sqrt{(3d - 1) / b_2} + 1\right)$ ;  $\hat{a}(N, T, t_0) = (1 + \sqrt{\log T} / (nN)) \sqrt{\log T} + t_0$ ).
Algorithm Result	Generalization bound	Tool	Assumptions
L	θ	v2	B.
SGD (ηt≤c/t) [37] (Thm. 3.12)	O(2/βc+1Tβc+1/n)	Uni.	√	×	×	√
SGD (ηt≤c/(t+2) log(t+2)) [38] (Thm. 3)	*O(√L(√E[v2]+logT)/nδ)	l1	√	×	√	√
SGD (ηt≤c/t) [39] (Thm. 3.5)	O(2/βc+1Tβc+1/n)	Uni.	√	×	×	√
SGD (ηt≤2t+1/2α(t+1)2) [40] (Thm. 15)	O(β/Tβ+α/n)	Uni.	√	×	×	√
ZoSS (ηt≤c/(tΓd/b2)) [36] (Thm. 5)	O(L2/βc/Tβc+1/n)	Uni.	√	×	×	√
ZoSS (ηt≤c/t) [36] (Cor. 6)	O(L2/T/n)	Uni.	√	×	×	√
ZoSS (ηt≤c/(tΓd/b2)) [36] (Thm. 8)	O(L2/Tβc/n min{c+β-1,c log(eT)}	Uni.	√	×	×	×
ZoSS (ηt≤c/(TTΓd/b2)) [36] (Thm. 7)	O(L2/n)	Uni.	√	×	×	×
AD-SGD (ηt=η1) [47] (Cor. 2)	O(nη1-λ/n(1-λ)L2(1+βη1/M)T)	Uni.	√	×	×	×
AD-SGD (ηt=M/β(t+1)) [47] (Cor. 2)	O(nM-λ/n(1-λ)L2T)	Uni.	√	×	×	×
FedZO (ηt≤η1/t) Ours (Cor. 1)	O((L/nN+μ) LT1/2 log T)	l1	√	×	×	×
FedZO (ηt≤η1/t) Ours (Cor. 2)	O((√logT/nN+1)(4θ)θμT1/4 log T)	l1	×	√	×	×
FedZO (ηt≤η1/t) Ours (Cor. 3)	O(ˆa(N,T,t0)(4θ)θμT1/2√logT)	l1	×	√	×	×
aligns with the previous generalization bounds of SGD algorithms for pointwise learning (Theorem 3.12 in [37]) and pairwise learning (Theorem 3.5 in [39] and Theorem 15 in [40]). [36] provided the first generalization error analysis for the minibatch ZoSS algorithm with both unbounded and bounded non-convex loss functions. Specifically, they presented the generalization bound  $n^{-1}(2 + C)L^2 (eT)^{\beta c}\min \left\{c + \beta^{-1},c\log (eT)\right\}$  ( $C$  is a positive constant) with the decreasing step size  $\eta_t\leq c / \left(t\Gamma_{b_2}^d\right)$ , where  $\Gamma_{b_2}^d = \left(\sqrt{(3d - 1) / b_2} +1\right)$ . Under the constant step sizes  $\eta_t\leq \log (1 + \beta c) / \left(T\beta \Gamma_{b_2}^d\right)$  and  $\eta_t\leq c / \left(T\Gamma_{b_2}^d\right)$ , they also showed the generalization bound  $n^{-1}(2 + C)cL^2$  and  $(n\beta)^{-1}L^{2}(2 + C)\left(e^{\beta c} - 1\right)$ . When  $\beta c\geq \frac{1}{2}$ , our generalization bound can match their first result. Indeed, we also can get similar bounds as [36] for the special setting of constant step size. Detail comparisons are also provided in Table 1.
3.2 Learning Guarantees of FedZO with Heavy tails
Inspired by [43, 44], we further investigate the learning guarantees of FedZO with the smooth, heavy-tailed loss function and PL condition.
Table 2: Comparisons of optimization bounds under the non-convex condition (Thm.-Theorem; Cor.-Corollary; D-ZO-PD (Distributed ZO Primal-Dual);  $r, C_{\lambda}$ -some constants;  $\lambda$ -a parameter characterizing the properties of decentralized topology;  $\sigma$ -the upper bound of the square of gradient; B.-bounded loss function;  $\sqrt{}$ -has such a property;  $\times$ -hasn't such a property).
Algorithm Result	Optimization bound	Step size	Assumptions
L	θ	β	B.	σ
AD-SGD [47] (Thm. 8)	O((r + Cλ/λt0 + t0/M)(logT)-1)	ηt = O(M/t+1)	√	×	√	×	×
AD-PSGD [48](Cor. 2)	O(T-1/2)	ηt = O(n/b1(√T+1))	×	×	√	×	√
EF-ZO-SGD [49](Thm. 1)	O(T-1/2d1/2 + T-1d)	ηt = O(√1/dT)	√	×	√	×	×
FED-EF -ZO-SGD [49](Thm. 2)	O(T-1/2d1/2 + T-3/2d3/2)	ηt = O(√1/dT)	√	×	√	×	×
FedZO [16] (Cor. 2)	O((MT)-1/2 + dμ2)	ηt ≤ √Mb1b2/dT	×	×	√	×	√
FedZO Ours (Thm. 4)	O(T-2 + μ2)	ηt ≤ ηt/t	×	√	√	×	×
FedZO Ours (Thm. 6)	O(T-2 + μ2)	ηt ≤ ηt/t	×	√	√	×	×
Theorem 3. Let  $\{w^t\}$  and  $\{\bar{w}^t\}$  be produced by FedZO (3) on  $S$  and  $S^{(j_i)}$  respectively, where  $\eta_t = \eta_1 t^{-1}$ ,  $\eta_1 \leq \frac{1 - d / b_2}{4a_3}$  with  $a_3 = (1 + d / b_2)\beta$ , and  $0 \geq \frac{1 - d / b_2}{2}\alpha\eta_1 - 1$ . After  $T$  iterations, we get the global parameters  $A(S) = w^T$  and  $A(S^{(j_i)}) = \bar{w}^T$  respectively. Under Assumptions 1 (b), 2 and 3, there holds
$$
\frac {1}{n N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \mathbb {E} \left[ \left\| w ^ {T} - \bar {w} ^ {T} \right\| \right] \leq \left(e (T - 1)\right) ^ {a _ {1} \eta_ {1}} a _ {4} (T - 1) \eta_ {1} \log \left(e (T - 1)\right),
$$
where  $a_4(T - 1) = 2(nN)^{-1}\sqrt{\tau(T - 1)} + \mu\beta + 2(nN)^{-1}\sqrt{\tau(T - 1)}\sqrt{d / b_2}$  and  $\tau(T - 1) = \mathcal{O}(\mu^2 \log T)$ .
Theorem 3 states the stability bound  $\mathcal{O}\left((nN)^{-1}\sqrt{\log T} +1\right)(nN)^{-1}T^{\frac{1}{4}}\log T)$  if taking  $\mu = \mathcal{O}\left((nN)^{-1}\right)$  and  $\frac{a_1(1 - d / b_2)}{4a_3}\leq \frac{1}{4}$ . Compared with Theorem 2, the current result removes the dependence on the Lipschitz parameter  $L$  while the dependence on  $\mu$  is improved from the partial dependence  $L / (nN) + \mu$  to the full dependence  $(\sqrt{\log T} /(nN) + 1)\mu$ . The reason is that, motivated by [36], we decompose the gradient approximation into two parts, i.e., the difference between the unknown gradient and its expected estimator, the divergence between the expected and its empirical version. With the help of second-order Taylor expansion, the first part is bounded by Lemmas 4 and 5 in Appendix B.1. Meanwhile, the second part is bounded by the PL condition. The detailed proof is provided by Equation (8) in Appendix B.4.
Combining Theorem 1 (b) with Theorem 3, we derive the following generalization bound for the heavy-tailed FedZO.
Corollary 2. Under the same conditions of Theorem 3, for FedZO (3), there holds
$$
\left. \right.\left| \mathbb {E} \left[ F \left(w ^ {T}\right) - F _ {S} \left(w ^ {T}\right)\right]\right| \leq \mathcal {O} \left(\left(\left(n N\right) ^ {- 1} \sqrt {\log T} + 1\right)\left(4 \theta\right) ^ {\theta} \mu T ^ {a _ {1} \eta_ {1}} \log T + \mathbb {E} \left[ F _ {S} \left(w ^ {T}\right)\right]\right).
$$
When  $\mu = \mathcal{O}\left((nN)^{-1}\right),\frac{a_1(1 - d / b_2)}{4a_3} \leq \frac{1}{4}$  and  $\mathbb{E}\left[F_S(w^T)\right] = \mathcal{O}\left((nN)^{-1}\right)$ , Corollary 2 shows the generalization bound  $\mathcal{O}\Bigl ((nN)^{-1}\sqrt{\log T} +1\Bigr)(4\theta)^{\theta}(nN)^{-1}T^{\frac{1}{4}}\log T\Bigr)$ . With bounded variance of gradient and bounded loss function, [38] stated the generalization bound
$\mathcal{O}\left((n\delta)^{-\frac{1}{2}}L^{\frac{1}{2}}\bigl (\left(\mathbb{E}\left[v^{2}\right]\right)^{\frac{1}{4}} + \sqrt{\log T}\bigr)\right)$  for SGD with high probability. Here, we further developed the analysis technique associated with the bounded variance of gradient to the federated learning setting by the fine-grained error analysis (see the proof of Theorem 1 (b)).
Theorem 4. Let  $\{w^t\}$  be produced by FedZO (3) on  $S$ , where  $\eta_t = \eta_1 t^{-1}$ ,  $\eta_1 \leq \frac{1 - d / b_2}{4a_3}$  with  $a_3 = (1 + d / b_2)\beta$  and  $0 \geq \frac{1 - d / b_2}{2}\alpha \eta_1 - 1$ . Under Assumptions 1 (b), 2 and 3, there hold
$$
\mathbb {E} \left[ F _ {S} \left(w ^ {T}\right) - F _ {S} \left(w (S)\right) \right] \leq \mathcal {O} \left(T ^ {- 2} + \mu^ {2}\right)
$$
$$
\begin{array}{l} | \mathbb {E} [ F (w ^ {T}) - F (w ^ {*}) ] | \\ \leq \mathcal {O} \left(T ^ {- 2} + \mu^ {2} + \left((n N) ^ {- 1} \sqrt {\log T} + 1\right) (4 \theta) ^ {\theta} \mu T ^ {a _ {1} \eta_ {1}} \log T + \mathbb {E} [ F _ {S} (w ^ {T}) ]\right), \\ \end{array}
$$
where  $w(S),w^{*}$  are defined in (4).
The PL condition can guarantee the identification of global minimizers. Therefore, we regard  $\mathbb{E}\left[F_S(w^T) - F_S(w^*)\right]$ , instead of  $\| \nabla F_S(w^T)\|$ , as the measure of optimization error in this paper. [40] provided the optimal optimization bound  $\mathcal{O}\left(1 / (T\alpha^2)\right)$  with uniform stability tool for gradient-dominated pairwise SGD. [16] characterized the convergence rate  $\mathcal{O}\left((MT)^{-\frac{1}{2}} + d\mu^2\right)$  of the FedZO algorithm with partial device participation. As shown in Table 2, Theorem 4 guarantees the optimal decay rate  $\mathcal{O}\left(T^{-1}\right)$  on the optimization error as  $\mu = \mathcal{O}(T^{-\frac{1}{2}})$  without the dependence of the dimension of hypothesis function space  $d$  and the random direction number  $b_{2}$ . Note that, our optimization bounds rely on the quality of the initial model like many previous work (e.g. [16]).
4 Learning Guarantees of Asynchronous FedZO
Following the line of Section 3.2, we further study theoretical foundations for the asynchronous case of the FedZO algorithm. Considering the asynchrony among the local model parameters of different clients in the same iteration, we modify Equation (3) as follows
$$
w ^ {t + 1} = w ^ {t} - \frac {\eta_ {t}}{b _ {1} N} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {b _ {1}} \tilde {\nabla} f _ {i} \left(w _ {i} ^ {t _ {i}}; z _ {i, m} ^ {t _ {i}}, \left\{v _ {i, l} ^ {t _ {i}} \right\} _ {l = 1} ^ {b _ {2}}, \mu\right), \tag {7}
$$
where  $t - t_i \in [t_0]$  denotes the delay of the  $i$ -th client in the  $t$ -th iteration, and  $t_0 = \max_{t \in [T]} \left\{ \max_{i \in [N]} t_i - \min_{i \in [N]} t_i \right\}$  denotes the maximum delay for all clients in the whole update process of the global model. Note that, if  $t_i = t$  for some  $i$ , the parameter  $w_i^{t_i}$  will be updated to  $w^{t+1}$  at the end of the  $t$ -th iteration. We state the differences between Equations (3) and (7) in Remark 2.
Theorem 5. Given  $S$  and  $S^{(j_i)}$  in Definition 1, let  $\{w^t\}$  and  $\{\bar{w}^t\}$  be produced by asynchronous FedZO (7) on  $S$  and  $S^{(j_i)}$  respectively, where  $\eta_t = \eta_1 t^{-1}$ ,  $\eta_1 \leq \frac{1 - d / b_2}{4a_3}$  with  $a_3 = (1 + d / b_2) \beta$  and  $0 \geq \frac{1 - d / b_2}{2} \alpha \eta_1 - 1$ . After  $T$  iterations, we get the global parameters  $A(S) = w^T$  and  $A(S^{(j_i)}) = \bar{w}^T$  respectively. Under Assumptions 1 (b), 2 and 3, there holds
$$
\frac {1}{n N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \mathbb {E} \left[ \left\| w ^ {T} - \bar {w} ^ {T} \right\| \right] \leq \left(e (T - 1)\right) ^ {2 \beta \eta_ {1}} \left(a _ {5} (T - 1) \eta_ {1} \log (e (T - 1)) + 4 a _ {6} (T - 1) \eta_ {1} ^ {2}\right),
$$
where  $a_5(T - 1) = \mu \beta +4(nN)^{-1}\sqrt{\hat{\tau}(T - 1)}\sqrt{d / b_2}, a_6(T - 1) = \mu \beta^2\big((b_1N)^{-1}\left(1 + \frac{b_1 - 1}{n}\right) + 2t_0\big) + \beta \left(2 + 2\sqrt{d / b_2}\right)\left((b_1N)^{-1}\left(1 + \frac{b_1 - 1}{n}\right) + 2t_0\right)\sqrt{\hat{\tau}(T - 1)}$  and  $\hat{\tau} (T - 1) = \mathcal{O}\left(\mu^2\log T\right)$ .
Asynchrony can cause discrepancies among the local models of various clients  $w^{t_i}$ ,  $i = 1, \dots, N$  and the global model  $w^t$  in the  $t$ -th iteration. Therefore, the primary challenge of asynchronous learning theory is to establish the relationship between  $\left\| w_{i}^{t_{i}} - \bar{w}_{i}^{t_{i}} \right\|$  and  $\| w^{t} - \bar{w}^{t}\|$ . To overcome this bottleneck, we design a new decomposition  $\left\| w_{i}^{t_{i}} - \bar{w}_{i}^{t_{i}} \right\| \leq \left\| w_{i}^{t_{i}} - w^{t} - \left(\bar{w}_{i}^{t_{i}} - \bar{w}^{t}\right) \right\| + \| w^{t} - \bar{w}^{t}\|$
and then use second-order Taylor expansion to give an upper bound of the first term on the right-hand side of the inequality. When  $\mu = \mathcal{O}\left((nN)^{-1}\right)$  and  $\frac{\beta(1 - d / b_2)}{2a_3} \leq \frac{1}{2}$ , Theorem 5 provide a stability bound  $\mathcal{O}\left(\left((1 + (nN)^{-1}\sqrt{\log T}\right)\log T + \sqrt{\log T} t_0\right)(nN)^{-1}T^{\frac{1}{2}}$ .
Corollary 3. Under the same conditions of Theorem 5, for asynchronous FedZO (7), there holds
$$
\begin{array}{l} \left. \left| \mathbb {E} \left[ F \left(w ^ {T}\right) - F _ {S} \left(w ^ {T}\right) \right] \right| \right. \\ \leq \mathcal {O} \left(\left(\left(1 + (n N) ^ {- 1} \sqrt {\log T}\right) \log T + \sqrt {\log T} t _ {0}\right) (4 \theta) ^ {\theta} \mu T ^ {2 \beta \eta_ {1}} + \mathbb {E} [ F _ {S} (w ^ {T}) ]\right). \\ \end{array}
$$
Combining Theorem 1 (b) with Theorem 5, when  $\mu = \mathbb{E}[F_S(w^T)] = \mathcal{O}\left((nN)^{-1}\right),\frac{\beta(1 - d / b_2)}{2a_3} \leq \frac{1}{2},$  Corollary 3 yields the generalization bound  $\mathcal{O}\Big(\big((1 + (nN)^{-1}\sqrt{\log T})\log T + \sqrt{\log T} t_0\big)(4\theta)^{\theta}(nN)^{-1}T^{\frac{1}{2}}\Big)$  which appears to be the first generalization bound developed for asynchronous federated learning algorithms. It should be noted that, due to the more complex communication structure, using the condition  $t_0 = 0$  (the synchronous case) can not recover the generalization bound in Corollary 2.
Theorem 6. Let  $\{w^t\}$  be produced by FedZO (7) on  $S$ , where  $\eta_t = \eta_1 t^{-1}$ ,  $\eta_1 \leq \frac{1 - d / b_2}{4a_3}$  with  $a_3 = (1 + d / b_2)\beta$  and  $0 \geq \frac{1 - d / b_2}{2}\alpha \eta_1 - 1$ . Under Assumptions 1 (b), 2 and 3, there hold
$$
\mathbb {E} \left[ F _ {S} \left(w ^ {T}\right) - F _ {S} (w (S)) \right] \leq \mathcal {O} \left(T ^ {- 2} + \mu^ {2}\right)
$$
$$
\begin{array}{l} | \mathbb {E} [ F (w ^ {T}) - F (w ^ {*}) ] | \\ \leq \mathcal {O} \left(T ^ {- 2} + \mu^ {2} + \left(\left(1 + (n N) ^ {- 1} \sqrt {\log T}\right) \log T + \sqrt {\log T} t _ {0}\right) (4 \theta) ^ {\theta} \mu T ^ {2 \beta \eta_ {1}} + \mathbb {E} [ F _ {S} (w ^ {T}) ]\right), \\ \end{array}
$$
Theorem 6 also develops the first optimal optimization bound  $\mathcal{O}\left(T^{-2} + \mu^2\right)$  for the asynchronous FedZO. Based on Equation (5), the excess risk bound  $\mathcal{O}\Big(T^{-2} + \big((1 + (nN)^{-1}\sqrt{\log T})\log T + t_0$ $\sqrt{\log T}\big)(4\theta)^{\theta}(nN)^{-1}T^{\frac{1}{2}}\Big)$  can be directly derived by integrating Corollary 3 and this optimization bound when  $\mu = \mathbb{E}[F_S(w^T)] = \mathcal{O}\left((nN)^{-1}\right)$ .
This paper fills the gap of theoretical guarantees for both synchronous and asynchronous FedZO algorithms. We develop the first generalization bound for general FedZO after bridging the quantitative relationships between generalization error and  $\ell_1$  on-average model stability. Moreover, fine-grained learning theory analysis is established by means of the heavy-tailed condition and the second-order Taylor expansion, where the derived error bounds are satisfactory even compared with the previous results for traditional SGD [37, 32, 38] and recent ZoSS [36].
This work was supported in part by the National Natural Science Foundation of China (Nos. 12071166 and 62376104), the Fundamental Research Funds for the Central Universities of China (No. 2662023LXPY005), and HZAU-AGIS Cooperation Fund (No. SZYJY2023010).
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 54, pages 1273–1282, 2017.