SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Randomized smoothing of loss function. To smoothen the loss function  $f$ , we employ a randomized smoothing approach where the smoothing parameter is maintained as sufficiently small. This framework is rooted in the seminal work by Steklov [45], leading to progress in both convex [27, 52, 11] and nonconvex [37] regimes. We consider a smoothing of  $f$ , given by  $f^{\eta}$  defined as  $f^{\eta}(x) \triangleq \mathbb{E}_{u \in \mathbb{B}}[f(x + \eta u)]$ , where  $u$  is a random vector in the unit ball  $\mathbb{B}$ , defined as  $\mathbb{B} \triangleq \{u \in \mathbb{R}^n \mid \| u \| \leq 1\}$ . Further, we let  $\mathbb{S}$  denote the surface of the ball  $\mathbb{B}$ , i.e.,  $\mathbb{S} \triangleq \{v \in \mathbb{R}^n \mid \| v \| = 1\}$  and  $\eta \mathbb{B}$  and  $\eta \mathbb{S}$  denote a ball with radius  $\eta$  and its surface, respectively.
Lemma 1 (Randomized spherical smoothing). Let  $h: \mathbb{R}^n \to \mathbb{R}$  be a given continuous function and define  $h^\eta(x) \triangleq \mathbb{E}_{u \in \mathbb{B}}[h(x + \eta u)]$ . Then, the following hold.
(i)  $h^\eta$  is continuously differentiable and  $\nabla h^\eta (x) = \left(\frac{n}{\eta}\right)\mathbb{E}_{v\in \eta \mathbb{S}}[h(x + v)\frac{v}{\|v\|}]$  for any  $x\in \mathbb{R}^n$ .
Suppose  $h$  is Lipschitz continuous with parameter  $L_0 > 0$ . Then, the following statements hold.
(ii)  $|h^{\eta}(x) - h^{\eta}(y)| \leq L_0 \| x - y \|$  for all  $x, y \in \mathbb{R}^n$ ; (iii)  $|h^{\eta}(x) - h(x)| \leq L_0 \eta$  for all  $x \in \mathbb{R}^n$ ; (iv)  $\| \nabla h^{\eta}(x) - \nabla h^{\eta}(y) \| \leq \frac{L_0 n}{\eta} \| x - y \|$  for all  $x, y \in \mathbb{R}^n$ .
The discussion leads to the consideration of the following smoothed federated problem.
Definition 1 (Unconstrained smoothed approximate problem). Given  $\eta >0$ , consider an unconstrained smoothed problem given as
$$
\min  _ {x \in \mathbb {R} ^ {n}} \mathbf {f} ^ {\eta} (x) \left\{\triangleq \frac {1}{m} \sum_ {i = 1} ^ {m} \mathbf {f} _ {i} ^ {\eta} (x) \right\}, \text {w h e r e} \mathbf {f} _ {i} ^ {\eta} (x) \triangleq \mathbb {E} _ {\xi_ {i}, u _ {i} \in \mathbb {B}} [ \tilde {f} _ {i} (x + \eta u _ {i}, \xi_ {i}) ] + \frac {1}{2 \eta} \operatorname {d i s t} ^ {2} (x, X _ {i}). \tag {4}
$$
Algorithm 1 Randomized Zeroth-Order Locally-Projected Federated Averaging (FedRZO<sub>nn</sub>)
1: input: Server chooses a random initial point  $\hat{x}_0 \in X$ , stepsize  $\gamma$ , smoothing parameter  $\eta$ , synchronization indices  $T_0 \coloneqq 0$  and  $T_r \geq 1$ , where  $r \geq 1$  is the communication round index
2: for  $r = 0,1,\ldots$  do
3: Server broadcasts  $\hat{x}_r$  to all clients:  $x_{i,T_r} \coloneqq \hat{x}_r$ ,  $\forall i \in [m]$
4: for  $k = T_r, \dots, T_{r+1} - 1$  do in parallel by clients
5: Client  $i$  generates the random replicates  $\xi_{i,k} \in \mathcal{D}_i$  and  $v_{i,k} \in \eta \mathbb{S}$
6:  $g_{i,k}^{\eta} := \frac{n}{\eta^{2}}\left(\tilde{f}_{i}(x_{i,k} + v_{i,k},\xi_{i,k}) - \tilde{f}_{i}(x_{i,k},\xi_{i,k})\right)v_{i,k}$
7: Client  $i$  does a local update as  $x_{i,k + 1} \coloneqq x_{i,k} - \gamma \left(g_{i,k}^{\eta} + \frac{1}{\eta}\left(x_{i,k} - \mathcal{P}_{X_i}(x_{i,k})\right)\right)$
8: end for
9: Server receives  $x_{i,T_{r+1}}$  from all clients and aggregates, i.e.,  $\hat{x}_{r+1} := \frac{1}{m}\sum_{i=1}^{m}x_{i,T_{r+1}}$
To address (4), we propose  $\mathrm{FedRZO_{nn}}$  given by Algorithm 1. Here, client  $i$  employs a zeroth-order stochastic gradient of the form  $g_{i,k}^{\eta} \triangleq \frac{n}{\eta^{2}} \left( \tilde{f}_{i}(x_{i,k} + v_{i,k}, \xi_{i,k}) - \tilde{f}_{i}(x_{i,k}, \xi_{i,k}) \right) v_{i,k}$ , augmented by the gradient of the Moreau smoothed function. The random sample  $v_{i,k} \in \eta \mathbb{S}$  is locally generated by each client  $i$ , allowing for randomized smoothing. This is indeed in view of Lemma 1 (i) that facilitates the development of a randomized zeroth-order gradient.
We define  $\bar{x}_k \triangleq \frac{\sum_{i=1}^m x_{i,k}}{m}$  as an auxiliary sequence to denote the averaged iterates of the clients.
Definition 2. Consider Algorithm 1. Let  $H > 0$  denote an upper bound on the number of local steps per round, i.e.,  $H \geq \max_{r = 0,1,\ldots} |T_{r + 1} - T_r|$ . Throughout, we assume that  $H$  is finite.
Proposition 1. Consider Algorithm 1. Let Assumption 1 hold.
(i) [Error bound] Suppose  $\gamma \leq \min \left\{\frac{\eta}{4L_0n},\frac{1}{4H},\frac{\eta}{12\sqrt{3}B_2(L_0n + 1)H}\right\}$ . Let  $k^{*}$  denote an integer drawn uniformly at random from  $\{0,\dots ,K\}$  and  $\mathbf{f}^{\eta ,*}\triangleq \inf_{x}\mathbf{f}^{\eta}(x)$ . Then,
$$
\begin{array}{l} \mathbb {E} \left[ \| \nabla \mathbf {f} ^ {\eta} (\bar {x} _ {k ^ {*}}) \| ^ {2} \right] \leq \frac {8 \left(\mathbb {E} [ \mathbf {f} ^ {\eta} (\bar {x} _ {0}) ] - \mathbf {f} ^ {\eta , *}\right)}{\gamma (K + 1)} + \frac {1 2 \gamma L _ {0} n ^ {3}}{\eta m} \left(\frac {2 \nu^ {2}}{\eta^ {2}} + L _ {0} ^ {2}\right) \\ + \frac {3 6 H ^ {2} \gamma^ {2} (L _ {0} n + 1) ^ {2}}{\eta^ {2}} \left(\frac {6 n ^ {2} \nu^ {2} + 2 B _ {1} ^ {2}}{\eta^ {2}} + (3 + 4 B _ {2} ^ {2}) L _ {0} ^ {2} n ^ {2}\right). \\ \end{array}
$$
(ii) [Iteration complexity] Let  $\gamma \coloneqq \sqrt{\frac{m}{K}}$  and  $H\coloneqq \left\lceil \sqrt[4]{\frac{K}{m^3}}\right\rceil$  where  $\eta >0$ . Let  $\epsilon >0$  be an arbitrary scalar and  $K_{\epsilon}$  denote the number of iterations such that  $\mathbb{E}\left[\| \nabla f^{\eta}(\bar{x}_{k^{*}})\|^{2}\right]\leq \epsilon$ . Then, the iteration complexity is  $K_{\epsilon}\coloneqq \mathcal{O}\left(\left(\frac{L_0n^3\nu^2}{\eta^3} +\frac{L_0^3n^3}{\eta} +\frac{L_0^2n^4\nu^2}{\eta^4} +\frac{L_0^2n^2B_1^2}{\eta^4} +\frac{B_2^2L_0^4n^4}{\eta^2}\right)^2\frac{1}{m\epsilon^2}\right)$ .
(iii) [Communication complexity] Suppose  $K_{\epsilon} \geq m^{3}$ . Then, the number of communication rounds to achieve the accuracy level in (ii) is  $R \coloneqq \mathcal{O}\left((mK_{\epsilon})^{3/4}\right)$ .
We now formally relate the original nonsmooth problem and its smoothed counterpart.
Proposition 2. Consider problem (4) and let Assumption 1 hold.
(i) Assume that  $X_{i} = \mathbb{R}^{n}$  for all  $i\in [m]$ . Then, for any  $\eta >0$ , we have  $\nabla f^{\eta}(x)\in \partial_{2\eta}f(x)$
(ii) Assume that the sets  $X_{i}$  are identical for all  $i\in [m]$ . Let  $\delta >0$  be an arbitrary scalar. If  $\nabla \mathbf{f}^{\eta}(x) = 0$  and  $\eta \leq \frac{\delta}{\max\{2,nL_0\}}$ , then  $0_{n}\in \partial_{\delta}\left(f + \mathbb{I}_{X}\right)(x)$ .
4 Extensions to Bilevel and Minimax FL
4.1 Nondifferentiable nonconvex bilevel FL. In this section, we consider the federated bilevel optimization problem defined earlier as  $(\mathbf{FL}_{bl})$ . We consider the following smoothed implicit problem.
Definition 3 (Unconstrained smoothed implicit problem). Given  $\eta >0$ , consider an unconstrained smoothed implicit problem given as
$$
\left. \min  _ {x \in \mathbb {R} ^ {n}} \mathbf {f} ^ {\eta} (x) \left\{\triangleq \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\mathbb {E} _ {\xi_ {i}, u \in \mathbb {B}} [ \tilde {f} _ {i} (x + \eta u, y (x + \eta u), \xi_ {i}) ] + \frac {1}{2 \eta} \operatorname {d i s t} ^ {2} (x, X _ {i})\right) \right\}. \right. \tag {5}
$$
Assumption 2. Consider problem  $(\mathbf{FL}_{bl})$ . Let the following assumptions hold.
(i) For all  $i \in [m]$ ,  $\tilde{f}_i(\bullet, y, \xi_i)$  is  $L_{0,x}^f(\xi_i)$ -Lipschitz for any  $y$  and  $\tilde{f}_i(x, \bullet, \xi_i)$  is  $L_{0,y}^f(\xi_i)$ -Lipschitz for any  $x$ , where  $L_{0,x}^f \triangleq \max_{i=1,\dots,m} \sqrt{\mathbb{E}[(L_{0,x}^f(\xi_i))^2]} < \infty$  and  $L_{0,y}^f \triangleq \max_{i=1,\dots,m} \sqrt{\mathbb{E}[(L_{0,y}^f(\xi_i))^2]} < \infty$ .
(ii) For all  $i \in [m]$ , for any  $x$ ,  $h_i(x, \bullet)$  is  $L_{1,y}^h$ -smooth and  $\mu_h$ -strongly convex. Further, for any  $y$ , the map  $\nabla_y h_i(\bullet, y)$  is Lipschitz continuous with parameter  $L_{0,x}^{\nabla h}$ .
(iii) The sets  $X_{i}$ , for  $i \in [m]$ , satisfy Assumption 1 (iii).
The outline of  $\mathrm{FedRZO_{bl}}$  is presented in Algorithm 2. We make the following remarks: (i) At each global step, the server makes two calls to a lower-level FL oracle to in exactly compute  $y(\hat{x}_r + v_{T_r})$  and  $y(\hat{x}_r)$ . These lower-level FL calls are performed by the same clients, on the lower-level FL problem. (ii) The inexactness error is carefully controlled by terminating the lower-level FL oracle after  $\mathcal{O}(r)$  number of iterations, where  $r$  denotes the upper-level communication round index. (iii)  $\mathrm{FedRZO_{bl}}$  skips the calls to the lower-level FL oracle during the local steps. To accommodate this, unlike in  $\mathrm{FedRZO_{nn}}$ , here we employ a global randomized smoothing denoted by  $v_{T_r}$  during the communication round  $r$  in the upper level.
Algorithm 2 Randomized Implicit Zeroth-Order Federated Averaging (FedRZO<sub>b1</sub>)
1: input: Server chooses a random  $\hat{x}_0\in X$  , stepsize  $\gamma$  , smoothing parameter  $\eta$  , synchronization indices  $T_0\coloneqq 0$  and  $T_{r}\geq 1$  , where  $r\geq 1$  is the upper-level communication round index   
2: for  $r = 0,1,\ldots$  do   
3: Server generates a random replicate  $v_{T_r}\in \eta \mathbb{S}$    
4: Server calls FedAvg to receive  $y_{\varepsilon_r}(\hat{x}_r + v_{T_r})$  and  $y_{\varepsilon_r}(\hat{x}_r)$  , denoting the inexact evaluations of  $y(\hat{x}_r + v_{T_r})$  and  $y(\hat{x}_r)$  , respectively.   
5: Server broadcasts  $\hat{x}_r,\hat{x}_r + v_{T_r},y_{\varepsilon_r}(\hat{x}_r)$  , and  $y_{\varepsilon_r}(\hat{x}_r + v_{T_r})$  to all clients;  $x_{i,T_r}\coloneqq \hat{x}_r$ $\forall i$    
6: for  $k = T_r,\dots ,T_{r + 1} - 1$  do in parallel by clients   
7: Client  $i$  generates the random replicates  $\xi_{i,k}\in \mathcal{D}_i$    
8:  $g_{i,k}^{\eta ,\varepsilon_r}\coloneqq \frac{n}{\eta^2}\left(\tilde{f}_i(x_{i,k} + v_{T_r},y_{\varepsilon_r}(\hat{x}_r + v_{T_r}),\xi_{i,k}) - \tilde{f}_i(x_{i,k},y_{\varepsilon_r}(\hat{x}_r),\xi_{i,k})\right)v_{T_r}$    
9: Client  $i$  does a local update as  $x_{i,k + 1}\coloneqq x_{i,k} - \gamma \left(g_{i,k}^{\eta ,\varepsilon_r} + \frac{1}{\eta}\left(x_{i,k} - \mathcal{P}_{X_i}(x_{i,k})\right)\right)$    
10: end for   
11: Server receives  $x_{i,T_{r + 1}}$  from all clients and aggregates, i.e.,  $\hat{x}_{r + 1}\coloneqq \frac{1}{m}\sum_{i = 1}^{m}x_{i,T_{r + 1}}$    
12: end for
Theorem 1 (FedRZOb1 when using an arbitrary inexact FL method for lower-level). Consider Algorithm 2. Let Assumption 2 hold,  $k^{*}$  be chosen uniformly at random from  $0,\ldots ,K\coloneqq T_R - 1$  and  $\gamma \leq \min \left\{\frac{\max\left\{2,\sqrt{0.1\Theta_3},4B_2\sqrt{3\Theta_2},4B_2\sqrt{3\Theta_3}\right\}^{-1}}{4H},\frac{\eta}{24(L_0^{\mathrm{imp}}n + 1)}\right\}$ . Let  $\varepsilon_r$  denote the inexactness in obtaining the lower-level solution, i.e.,  $\mathbb{E}\left[\| y_{\varepsilon_r}(x) - y(x)\| ^2\mid x\right]\leq \varepsilon_r$  for  $x\in \cup_{r = 0}^{R}\{\hat{x}_r,\hat{x}_r + v_{T_r}\}$ .
(i) [Error bound] We have
$$
\begin{array}{l} \mathbb {E} \left[ \| \nabla \mathbf {f} ^ {\eta} (\bar {x} _ {k ^ {*}}) \| ^ {2} \right] \leq 8 (\gamma K) ^ {- 1} (\mathbb {E} \left[ \mathbf {f} ^ {\eta} (x _ {0}) \right] - \mathbf {f} ^ {\eta , *}) + \frac {8 \gamma \Theta_ {1}}{m} + 8 H ^ {2} \gamma^ {2} \max  \{\Theta_ {2}, \Theta_ {3} \} \Theta_ {5} \\ + 8 \left(H ^ {2} \gamma^ {2} \max  \{\Theta_ {2}, \Theta_ {3} \} \Theta_ {4} + \Theta_ {3}\right) H \frac {\sum_ {r = 0} ^ {R - 1} \varepsilon_ {r}}{K}, \\ \end{array}
$$
$$
\begin{array}{l} \text {w h e r e} \Theta_ {1} := \frac {3 (L _ {0} ^ {\operatorname* {i m p}} n + 1) n ^ {2}}{2 \eta} (L _ {0} ^ {\operatorname* {i m p}}) ^ {2}, \Theta_ {2} := \frac {5 (L _ {0} ^ {\operatorname* {i m p}} n + 1) ^ {2}}{8 \eta^ {2}}, \Theta_ {3} := \left(\frac {L _ {0 , x} ^ {\nabla h}}{\mu h}\right) ^ {2} \frac {2 0 n ^ {2}}{\eta^ {2}} (L _ {0, y} ^ {f}) ^ {2}, \\ \Theta_ {4} := \frac {9 6 n ^ {2}}{\eta^ {2}} (L _ {0, y} ^ {f}) ^ {2}, \text {a n d} \Theta_ {5} := \frac {4 8 B _ {1} ^ {2}}{\eta^ {2}} + (9 6 B _ {2} ^ {2} + 1) (L _ {0} ^ {\text {i m p}}) ^ {2} n ^ {2}. \\ \end{array}
$$
(ii) [Iteration complexity] Let  $\gamma \coloneqq \sqrt{\frac{m}{K}}$  and  $H\coloneqq \left\lceil \sqrt[4]{\frac{K}{m^3}}\right\rceil$  where  $\eta >0$ . Let  $\epsilon >0$  be an arbitrary scalar and  $K_{\epsilon}$  denote the number of iterations such that  $\mathbb{E}\left[\| \nabla \mathbf{f}^{\eta}(\bar{x}_{k^{*}})\|^{2}\right]\leq \epsilon$ . Also, suppose we employ an FL method in the lower level that achieves a sublinear convergence rate with a linear speedup in terms of the number of clients, i.e.,  $\varepsilon_r\coloneqq \tilde{\mathcal{O}} (\frac{1}{m\tilde{T}_{\tilde{R}_r}})$  where  $\tilde{R}_r$  denotes the number of communication rounds performed in the lower-level FL method when it is called in round  $r$  of FedRZOb1 and  $\tilde{T}_{\tilde{R}_r}$  denotes the number of iterations performed in the lower-level FL scheme to do  $\tilde{R}_r$  rounds of upper-level communication. Further, suppose  $\tilde{T}_{\tilde{R}_r}\coloneqq \tilde{\mathcal{O}}\left(m^{-1}(r + 1)^{\frac{2}{3}}\right)$ . Then, the iteration complexity of FedRZOb1 (upper level) is  $K_{\epsilon}\coloneqq \tilde{\mathcal{O}}\left((\Theta_1^2 +\max \{\Theta_2,\Theta_3\} ^2\Theta_5^2 +\max \{\Theta_2,\Theta_3\} ^2\Theta_4^2 +\Theta_3^2\right)\frac{1}{m\epsilon^2})$
(iii) [Communication complexity] Suppose  $K_{\epsilon} \geq m^{3}$ . Then, the number of communication rounds in FedRZO<sub>b1</sub> (upper-level only) to achieve the accuracy level in (ii) is  $R \coloneqq \mathcal{O}\left((mK_{\epsilon})^{3/4}\right)$ .
Remark 2. (i) Importantly, Theorem 1 is equipped with explicit communication complexity  $R \coloneqq \mathcal{O}\left((mK_{\varepsilon})^{3 / 4}\right)$ , matching that of single-level nonsmooth nonconvex problems in Proposition 1. This implies that as long as the lower-level FL oracle has a rate of  $\varepsilon_r \coloneqq \tilde{\mathcal{O}}\left(\frac{1}{m\tilde{T}_{R_r}}\right)$ , the inexactness does not affect the communication complexity bounds of the method in the upper level.
(ii) As noted in the assumptions, in the upper level, we allow for heterogeneity. To elaborate on overall communication complexity of  $\mathrm{FedRZO_{bl}}$ , we provide detailed complexity results in Table 1 for three cases, where we employ Local SGD [26], FedAC [54], and LFD [20] for the lower-level scheme. All these schemes meet the linear speedup condition in Theorem 1. Notably, among these schemes, the last scheme allows for the presence of heterogeneity. As an example, we present in Algorithm 3, the outline of FedAvg, if employed in step 4 of Algorithm 2.
Algorithm 3 FedAvg  $(x,r,y_{0,r},m,\tilde{\gamma},\tilde{H},\tilde{T}_{\tilde{R}})$  for lower level
1: input:  $x, r$ , server chooses a random initial point  $\hat{y}_0 \coloneqq y_{0,r} \in \mathbb{R}^{\tilde{n}}$ ,  $a_r \coloneqq \max \{m, 4\kappa_h, r\} + 1$  where  $\kappa_h \coloneqq \frac{L_{1,y}^h}{\mu_h}$ ,  $\tilde{\gamma} \coloneqq \frac{1}{\mu_h a_r}$ ,  $\tilde{T}_{\tilde{R}_r} \coloneqq 2a_r \ln(a_r)$ , and  $\tilde{H} \coloneqq \lceil \frac{\tilde{T}_{\tilde{R}_r}}{m} \rceil$
2: for  $\tilde{r} = 0,1,\dots,R_r - 1$  do
3: Server broadcasts  $\hat{y}_{\tilde{T}}$  to all agents:  $y_{i,\tilde{T}_{\tilde{T}}} \coloneqq \hat{y}_{\tilde{T}}, \forall i$
4: for  $t = \tilde{T}_{\tilde{r}},\dots ,\tilde{T}_{\tilde{r} +1} - 1$  do in parallel by agents
5: Agent  $i$  does a local update as  $y_{i,t + 1}\coloneqq y_{i,t} - \tilde{\gamma}\nabla_yh_i(x,y_{i,t},\tilde{\xi}_{i,t})$
6: Agent  $i$  sends  $x_{i,T_r + 1}$  to the server
7: end for
8: Server aggregates, i.e.,  $\hat{y}_{\tilde{r} +1}\coloneqq \frac{1}{m}\sum_{i = 1}^{m}y_{i,T_{\tilde{r} +1}}$
9: end for
(iii) We use  $L_0^{\mathrm{imp}}(\xi_i)$  to denote the Lipschitz continuity constant of the random local implicit function  $\tilde{f}_i(x,y(x),\xi_i)$ , and let  $L_0^{\mathrm{imp}} \triangleq \max_{i = 1,\dots,m} \sqrt{\mathbb{E}[(L_0^{\mathrm{imp}}(\xi_i))^2]} < \infty$ . As shown in supplementary material,  $L_0^{\mathrm{imp}}(\xi_i)$  can be obtained explicitly in terms of problem parameters.
Remark 3. A technical challenge in designing Algorithm 2 is that an inexact evaluation of  $y(x)$  must be avoided during the local steps. This is because we consider bilevel problems of the form  $(\mathbf{F}\mathbf{L}_{bl})$
where both levels are distributed. Because of this, the inexact evaluation of  $y(x)$  by each client in the local step in the upper level would require significant communications that is undesirable in the FL framework. We carefully address this challenge by introducing delayed inexact computation of  $y(x)$ . In step 8 of Algorithm 2, we note how  $y_{\varepsilon}$  is evaluated at  $\hat{x}_r + v_{T_r}$  which is a different than the vector used by the client, i.e.,  $x_{i,k} + v_{T_r}$ . At each communication round in the upper level, we only compute  $y(x)$  in exactly twice in the global step and then use this delayed information in the local steps. This delayed inexact computation of  $y$  renders a challenge in the convergence analysis which makes the design and analysis of Algorithm 2 a non-trivial extension of Algorithm 1.
4.2 Nondifferentiable nonconvex-strongly concave minimax FL. Next, we consider the decentralized federated minimax problem of the form  $(\mathbf{FL}_{mm})$  introduced earlier. This problem is indeed a zero-sum game and can be viewed as an instance of the non-zero sum game  $(\mathbf{FL}_{bl})$  where  $\tilde{h}_i \coloneqq -\tilde{f}_i$ .
Corollary 1. Consider Algorithm 2 for solving  $(\mathbf{FL}_{mm})$ . Let Assumption 2 hold for  $\tilde{h}_i \coloneqq -\tilde{f}_i$  and  $\mathcal{D}_i \coloneqq \hat{\mathcal{D}}_i$ . Then, all the results in Theorem 1 hold true.
We present three sets of experiments to validate the performance of the proposed algorithms. In Section 5.1, we implement Algorithm 1 on ReLU neural networks (NNs) and compare it with some recent FL methods. In Sections 5.2 and 5.3 we implement Algorithm 2 on federated hyperparameter learning and a minimax formulation in FL. Throughout, we use the MNIST dataset. Additional experiments on a higher dimensional dataset (i.e., Cifar-10) are presented in supplementary material.
Figure 1: Performance of  $\mathrm{FedRZO_{nn}}$  on a single-layer ReLU NN in terms of communication rounds for different no. of local steps and different values of the smoothing parameter  $\eta$ .  $\mathrm{FedRZO_{nn}}$  benefits from larger number of local steps and shows robustness with respect to the choice of  $\eta$ .
5.1 Federated training of ReLU NNs. We implement FedRZO<sub>nn</sub> for federated training in a single-layer ReLU NN with  $N_{1}$  neurons. This is a nondifferentiable nonconvex optimization problem, aligning with  $(\mathbf{FL}_{nn})$  and taking the form  $\min_{x:=(Z,w)\in\mathcal{X}}\frac{1}{2m}\sum_{i=1}^{m}\sum_{\ell\in\mathcal{D}_{i}}(v_{i,\ell}-\sum_{q=1}^{N_{1}}w_{q}\sigma(Z_{\bullet,q}U_{i,\ell}))^{2}+\frac{\lambda}{2}\left(\|Z\|_{F}^{2}+\|w\|^{2}\right)$ , where  $m$  denotes the number of clients,  $Z\in\mathbb{R}^{N_{1}\times N_{0}}$ ,  $w\in\mathbb{R}^{N_{1}}$ ,  $N_{0}$  is the feature dimension,  $U_{i,\ell}\in\mathbb{R}^{N_{0}}$  and  $v_{i,\ell}\in\{-1,1\}$  are the  $\ell$ th input and output training sample of client  $i$ , respectively,  $\sigma(x):=\max\{0,x\}$ , and  $\lambda$  is the regularization parameter.
Setup. We distribute the training dataset among  $m \coloneqq 5$  clients and implement  $\mathrm{FedRZO_{mn}}$  for the FL training with  $N_{1} \coloneqq 4$  neurons under three different settings for the smoothing parameter,  $\eta \in \{0.1, 0.01, 0.001\}$ ,  $\gamma \coloneqq 10^{-5}$ , and  $\lambda \coloneqq 0.01$ . We study the performance of the method under different number of local steps with  $H \in \{1, 5, 10, 20\}$ .
Results and insights. Figure 1 presents the first set of numerics for  $\mathrm{FedRZO_{nn}}$  under the aforementioned settings. In terms of communication rounds, we observe that the performance of the method improves by using a larger number of local steps. In fact, in the case where  $H\coloneqq 1$ ,  $\mathrm{FedRZO_{nn}}$  is equivalent to a parallel zeroth-order SGD that employs communication among clients at each iteration, resulting in a poor performance, motivating the need for the FL framework. In terms of  $\eta$ , while we observe robustness of the scheme in terms of the original loss function, we also note a slight improvement in the empirical speed of convergence in early steps, as  $\eta$  increases. This is indeed aligned with the dependence of convergence bound in Proposition 1 on  $\eta$ .
Comparison with other FL methods. While we are unaware of other FL methods for addressing nondifferentiable nonconvex problems, we compare  $\mathrm{FedRZO_{nn}}$  with other FL methods including FedAvg [34], FedProx [29], FedMSPP [55], and Scaffnew [35] when applied on a NN with a smooth rectifier. Details of these experiments are provided in the supplementary material.
5.2 Federated hyperparameter learning. To validate  $\mathrm{FedRZO_{b1}}$ , we consider the following FL hyperparameter learning problem for binary classification using logistic loss.
$$
\min _ {x \in X, y \in \mathbb {R} ^ {n}} f (x, y) \triangleq \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {\ell \in \mathcal {D} _ {i}} \log \left(1 + \exp (- v _ {i, \ell} U _ {i, \ell} ^ {T} y)\right)
$$
subject to  $y \in \arg \min_{y \in \mathbb{R}^n} h(x, y) \triangleq \frac{1}{m} \sum_{i=1}^{m} \left( \sum_{\tilde{\ell} \in \tilde{D}_i} \log \left( 1 + \exp(-v_{i,\tilde{\ell}} U_{i,\tilde{\ell}}^T y) \right) + x_i \frac{\|y\|^2}{2} \right)$ ,
where  $m$  is number of clients,  $x$  denotes the regularization parameter for client  $i$ ,  $U_{i,\ell} \in \mathbb{R}^n$  and  $v_{i,\ell} \in \{-1,1\}$  are the  $\ell$ th input and output testing sample of client  $i$ , respectively,  $U_{i,\ell'} \in \mathbb{R}^n$  and  $v_{i,\ell'} \in \{-1,1\}$  are the  $\ell'$ th input and output training sample of client  $i$ , respectively. The constraint set  $X$  is considered as  $X := \{x \in \mathbb{R}^m \mid x \geq \underline{\mu} \mathbf{1}_m\}$ , where  $\underline{\mu} > 0$ . This problem is an instance of  $(\mathbf{FL}_{bl})$ , where the lower-level problem is  $\ell_2$ -regularized and the regularization parameter is a decision variable of the upper-level FL problem. The convergence results are presented in Fig. 2 (left).
5.3 Fair classification learning. Here, we study the convergence of  $\mathrm{FedRZO_{b1}}$  in minimax FL. We consider solving an FL minimax formulation of the fair classification problem [39] of the form
$$
\min _ {x \in \mathbb {R} ^ {n}} \max _ {y \in \mathbb {R} ^ {c}} \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {c = 1} ^ {C} \sum_ {\ell \in \mathcal {D} _ {i, c}} (v _ {i, \ell} - \sum_ {q = 1} ^ {N _ {1}} w _ {q} \sigma (Z _ {\bullet , q} U _ {i, \ell})) ^ {2} - \frac {\lambda}{2} \| y \| ^ {2},
$$
where  $c$  denotes the class index and  $\mathcal{D}_{i,c}$  denotes the a portion of local dataset associated with client  $i$  that is comprised of class  $c$  samples. The loss function follows the same formulation in Section 5.1, where an ReLU neural network is employed. This problem is nondifferentiable and nonconvex-strongly concave, fitting well with the assumptions in our work in addressing minimax FL problems. The performance of our algorithm is presented in Figure 2 (right).
Figure 2: (Left) Convergence of  $\mathrm{FedRZO_{b1}}$  in hyperparameter FL for  $\ell_2$  regularized logistic loss, where we plot the loss function on test data for different values of local steps with  $95\%$  CIs. (Right) Convergence of  $\mathrm{FedRZO_{b1}}$  in minimax FL, where we present test results in solving a nondifferentiable nonconvex-strongly concave FL minimax formulation of the fair classification problem [39].