SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the stochastic gradient of  $f_{i_k}(\mathbf{x}_{i_k}')$  at  $\mathbf{x}_{i_k}'$ . The stochastic approximation can tremendously reduce memory consumption and save computational costs in each iteration. By setting the subgradient of the objective function in Eq. (10) to zero, we can derive the closed-form solution in Eq. (11).
$$
\mathbf {x} _ {i _ {k}} = \mathbf {y} _ {i _ {k}} ^ {\prime} + \frac {1}{\beta} \mathbf {z} _ {i _ {k}} ^ {\prime} \odot \operatorname {s g n} \left(\mathbf {t} ^ {\prime}\right) - \frac {1}{\beta} \operatorname {s g n} \left(\mathbf {t} ^ {\prime}\right) \odot \left(\varepsilon_ {i} + g _ {i _ {k}} \left(\mathbf {x} _ {i _ {k}} ^ {\prime}, \boldsymbol {\xi} _ {i _ {k}}\right)\right) = \mathbf {y} _ {i _ {k}} ^ {\prime} + \frac {1}{\beta} \operatorname {s g n} \left(\mathbf {t} ^ {\prime}\right) \odot \left(\mathbf {z} _ {i _ {k}} ^ {\prime} - \varepsilon_ {i} - g _ {i _ {k}} \left(\mathbf {x} _ {i _ {k}} ^ {\prime}, \boldsymbol {\xi} _ {i _ {k}}\right)\right) \tag {11}
$$
where the signum function  $\text{sgn}(\cdot)$  extracts the signs of a vector and  $\mathbf{t}_{i_k}' = \mathbf{y}_{i_k}' - \mathbf{x}_{i_k}'$ .
Updating  $\mathbf{y}_{i_k}$ : We solve the following problem
$$
\min  _ {\mathbf {y} _ {i _ {k}}} \left\langle \mathbf {Z} _ {\mathcal {N} (i _ {k})}, \left| \mathbf {1} \otimes \mathbf {y} _ {i _ {k}} - \mathbf {X} _ {\mathcal {N} (i _ {k})} \right| - \mathbf {1} \otimes \varepsilon_ {i _ {k}} \right\rangle + \frac {\beta}{2} \| \left| \mathbf {1} \otimes \mathbf {y} _ {i _ {k}} - \mathbf {X} _ {\mathcal {N} (i _ {k})} \right| - \mathbf {1} \otimes \varepsilon_ {i _ {k}} \| _ {F} ^ {2} \tag {12}
$$
one can readily derive a closed-form solution for the problem (12) as:
$$
\mathbf {y} _ {i _ {k}} = \frac {1}{n _ {i _ {k}}} \sum_ {j \in \mathcal {N} _ {i _ {k}}} \left[ \mathbf {x} _ {i _ {k}} - \left(\frac {\mathbf {Z} _ {i _ {k}}}{\beta} + \varepsilon_ {i _ {k}}\right) \odot s g n \left(\mathbf {t} _ {i _ {k}}\right) \right] \tag {13}
$$
where  $\mathbf{t}_{i_k} = \mathbf{y}_{i_k}' - \mathbf{x}_{i_k}$  is similar to that of Eq. (11) except the updated  $\mathbf{x}$ . Specifically, via mathematical induction, we can attain the new updated form of  $\mathbf{y}_{i_k}$  below, which can also reduce the communication cost from  $O(n)$  to  $O(1)$ :
$$
\mathbf {y} _ {i _ {k}} = \mathbf {y} _ {i _ {k}} ^ {\prime} + \frac {1}{n _ {i _ {k}}} \left[ \mathbf {x} _ {i _ {k}} - \left(\frac {\mathbf {z} _ {i _ {k}}}{\beta} + \varepsilon_ {i _ {k}}\right) \odot s g n \left(\mathbf {t} _ {i _ {k}}\right) \right] - \left[ \mathbf {x} _ {i _ {k}} ^ {\prime} - \left(\frac {\mathbf {z} _ {i _ {k}} ^ {\prime}}{\beta} + \varepsilon_ {i _ {k}}\right) \odot s g n \left(\mathbf {t} _ {i _ {k}} ^ {\prime}\right) \right] \tag {14}
$$
Updating  $\mathbf{z}_{i_k}$ : The Lagrangian multiplier  $\mathbf{z}_{i_k}$  can be updated strictly following the standard ADMM scheme below:
$$
\mathbf {z} _ {i _ {k}} = \mathbf {z} _ {i _ {k}} ^ {\prime} + \kappa \beta \left[ \mathbf {x} _ {i _ {k}} - \mathbf {y} _ {i _ {k}} ^ {\prime} - \varepsilon_ {i _ {k}} \right] \tag {15}
$$
The  $\kappa$  coefficient used in Eq. (15) is decayed in each process step to achieve better convergence.
Please refer to Appendix A for the entire RWSADMM algorithm framework.
In this section, we present the theoretical convergence guarantee of RWSADMM. To ensure its convergence, certain common assumptions are made regarding the properties of the loss functions. The assumptions are as follows:
Assumption 4.1. The objective function  $f(\mathbf{x})$  is bounded from below and coercive over  $\mathbb{R}^p$ , that is, for any sequence  $\{\mathbf{x}^k\}_{k\geq 0}\subset \mathbb{R}^p$
$$
\text {i f} \| \mathbf {x} ^ {k} \| \xrightarrow {k \rightarrow \infty} \infty \Rightarrow \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (\mathbf {x}) \rightarrow \infty \tag {16}
$$
Assumption 4.2. The objective function  $f_{i}(\mathbf{x})$ 's are L-smooth, that is,  $f_{i}$  are differentiable, and its gradients are L-Lipschitz, that is,  $\forall \mathbf{u}, \mathbf{v} \in \mathbb{R}^{p}$  [39],
$$
\left\| \nabla f _ {i} (\mathbf {u}) - \nabla f _ {i} (\mathbf {v}) \right\| \leq L \| \mathbf {u} - \mathbf {v} \|, \forall i = 1, \dots , n \tag {17}
$$
Remark: In consequence it also holds that  $\forall \mathbf{u},\mathbf{v}\in \mathbb{R}^{p}$
$$
f _ {i} (\mathbf {u}) - f _ {i} (\mathbf {v}) \leq \nabla f _ {i} (\mathbf {v}) ^ {T} (\mathbf {u} - \mathbf {v}) + \frac {L}{2} \| \mathbf {u} - \mathbf {v} \| ^ {2}, \forall i = 1, \dots , n. \tag {18}
$$
Assumption 4.3. The objective function  $f$  is M-Lipschitz, that is,  $\forall \mathbf{u}, \mathbf{v} \in \mathbb{R}^p$  [41],
$$
| f (\mathbf {u}) - f (\mathbf {v}) | \leq M \| \mathbf {u} - \mathbf {v} \| \tag {19}
$$
Assumption 4.4. The first-order stochastic gradient is sampled, which returns a noisy but unbiased estimate of the gradient of  $f$  at any point  $\mathbf{x} \in \mathbb{R}^p$ , that is,  $\forall \mathbf{x} \in \mathbb{R}^p$ ,
$$
\mathbb {E} _ {\xi} [ g (\mathbf {x}, \xi) ] = \nabla f (\mathbf {x}) \tag {20}
$$
Remark: Substituting Eq. (20) into Eq. (17), one can obtain that for  $i = 1, \dots, n$ , we have
$$
\left\| \mathbb {E} _ {\xi} [ g (\mathbf {u}, \xi) ] - \mathbb {E} _ {\xi} [ g (\mathbf {v}, \xi) ] \right\| \leq L \| \mathbf {u} - \mathbf {v} \| \tag {21}
$$
Substituting Eq. (20) into Eq. (18), for  $i = 1, \dots, n$ , we can obtain
$$
f _ {i} (\mathbf {u}) - f _ {i} (\mathbf {v}) \leq \mathbb {E} _ {\xi} [ g (\mathbf {v}, \xi) ] ^ {T} (\mathbf {u} - \mathbf {v}) + \frac {L}{2} \| \mathbf {u} - \mathbf {v} \| ^ {2}, \tag {22}
$$
Assumption 4.5. The noise variance of the stochastic gradient is bounded as:
$$
\mathbb {E} _ {\xi} \left(\| \nabla f (\mathbf {x}) - g (\mathbf {x}, \xi) \| ^ {2}\right) \leq \exp (1), \text {f o r a l l} \mathbf {x}. \tag {23}
$$
This condition bounds the expectation of  $\| \nabla f(\mathbf{x}_t) - g(\mathbf{x}_t,\xi_t)\| ^2$ . Using Jensen's inequality, this condition implies a bounded variance [41].
We revisit the related crucial properties of the Markov Chain. The first time that the Markov Chain  $(i_k)_{k\geq 0}$  hits agent  $i$  is denoted as  $T_{i}:= \min \{k: i_{k} = i\}$ , and maximum value of  $T$  over all clients is defined as  $T:= \max \{T_1,\dots ,T_n\}$ . For  $k > T$ , let  $\tau (k,i)$  denote the iteration of the last visit to agent  $i$  before  $k$ , mathematically we have
$$
\tau (k, i) = \max  \left\{k ^ {\prime}: i _ {k ^ {\prime}} = i, k ^ {\prime} <   k \right\}. \tag {24}
$$
To prove the convergence of our proposed algorithm, two Lyapunov functions defined for RWSADMM are required to be investigated:
$$
L _ {\beta} ^ {k} := L _ {\beta} \left(\mathbf {y} ^ {k}, \mathbf {X} ^ {k}; \mathbf {Z} ^ {k}\right), M _ {\beta} ^ {k} := L _ {\beta} ^ {k} + \frac {L ^ {2}}{n} \sum_ {i = 1} ^ {n} \left\| \mathbf {y} _ {i} ^ {\tau (k, i) + 1} - \mathbf {y} _ {i} ^ {\tau (k, i)} \right\| ^ {2} \tag {25}
$$
where  $L_{\beta}(\mathbf{y}^k,\mathbf{X}^k;\mathbf{Z}^k)$  is defined in Eq. (8). The  $M_{\beta}^{k}$  is utilized in the convergence analysis. To guarantee the convergence of our algorithm, first, we refer to the asymptotic analysis of the nonhomogeneous Markov chain presented in [42]. Define  $\Phi (k,l)$  with  $k\geq l$  as the product of the transition probability matrices for the Markov chain from time  $l$  to  $k$ , i.e.,  $\Phi (k,l) = \mathbf{P}(k)\dots \mathbf{P}(l)$  with  $k\geq l$ . Then we have the following convergence result:
Lemma 4.6. Consider
1.  $\forall s,\lim_{k\to \infty}\Phi (k,l) = \frac{1}{n}\mathbf{e}^T.$
2. The convergence of  $\Phi$  is geometric and the rate of convergence considering  $\forall k, l$ , with  $k \geq l \geq 0$ , is given by
$$
\left| \left[ \Phi (k, l) \right] _ {i, j} - \frac {1}{n} \right| \leq \left(1 - \frac {\eta}{4 n ^ {2}}\right) ^ {\lceil \frac {k - l + 1}{Q} \rceil - 2} \tag {26}
$$
Using Lemma 4.6, the convergence analysis of the algorithm is as follows.
Lemma 4.7. Under Assumptions 4.1 and 4.2, if  $\beta > 2L^2 + L + 2$ ,  $(M_{\beta}^{k})_{k \geq 0}$  is lower bounded and convergent, the iterates  $(\mathbf{y}^k, \mathbf{X}^k, \mathbf{Z}^k)_{k \geq 0}$  generated by RWSADMM is bounded.
The proof sketch and the detailed convergence proof are presented in Appendix B. Using Lemma 4.7 and B.6, we can present the convergence of RWSADMM in Theorem 4.8.
Theorem 4.8. Let Assumption 4.5 hold. For  $\beta > 2L^2 + L + 2$ , it holds that any limit point  $(\mathbf{y}^*, \mathbf{X}^*, \mathbf{Z}^*)$  of the sequence  $(\mathbf{y}^k, \mathbf{X}^k, \mathbf{Z}^k)$  generated by RWSADMM satisfies  $\mathbf{y}^* = \mathbf{x_i}^*$ ,  $i = 1, \ldots, n$  where  $\mathbf{y}^*$  is a stationary point of Eq. (7), with probability 1, that is,
$$
P r \left(0 \in \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} \left(\mathbf {y} ^ {*}\right)\right) = 1 \tag {27}
$$
If the objective function of Eq. (7) is convex, then  $\mathbf{y}^*$  is a minimizer.
Next, Theorem 4.9 further presents that the algorithm converges sublinearly. This is comparable to the convergence rate of other FL methods [43, 44, 24, 25], but the existing methods didn't consider the dynamic graph and infrastructure-less environment. The detailed proof is offered in Appendix C.
Theorem 4.9. Under Assumptions (3.1), (4.1), and (4.2), with given  $\beta$  in Lemma 4.7, and local variables initiated as  $\nabla f_{i}(\mathbf{x}_{i}^{0}) = \beta \mathbf{x}_{i}^{0} = \mathbf{z}_{i}^{0},\forall i\in \{1,\ldots ,n\}$ , there exists a sequence  $\{g^k\}_{k\geq 0}$  with  $\{g^k\} \in \partial L_\beta^{k + 1}$  satisfying
$$
\min  _ {k \leq K} \mathbb {E} \left\| g ^ {k} \right\| ^ {2} \leq \frac {C}{K} \left(L _ {\beta} ^ {0} - f\right), \quad \forall K > \tau (\delta) + 2 \tag {28}
$$
where  $C$  is a constant depending on  $\beta, L, \text{and} \gamma, n, \text{and} \tau(\delta)$ .
Communication Complexity Using Theorem 4.9, the communication complexity of RWSADMM for nonconvex nonsmooth problems is as follows. To achieve ergodic gradient deviation  $E_{t} \coloneqq \min_{k \leq K} \mathbb{E}\|g^{k}\|^{2} \leq \omega$  for any  $K > \tau(\delta) + 2$ , it is sufficient to have
$$
\frac {C}{K} \left(L _ {\beta} ^ {0} - f\right) \leq \omega \xrightarrow {(a)} K \sim O \left(\frac {1}{\omega} \cdot \frac {\tau (\delta) ^ {2} + 1}{(1 - \delta) n \pi_ {*}}\right) \tag {29}
$$
(a) is achieved by taking  $L_{\beta}^{0}$  and  $f$  as constants and independent of  $n$  and the network structure. Using the  $\tau(\delta)$  definition from (6), by setting  $\delta = 1/2$  and assuming the reversible Markov chain with  $P(k)^{T} = P(k)$ , the communication complexity is
$$
O \left(\frac {1}{\omega} \cdot \frac {\ln^ {2} n}{\left(1 - \lambda_ {2} (\mathbf {P} (k))\right) ^ {2}}\right) \tag {30}
$$
Communication Comparison Among the baseline frameworks, Per-FedAvg [22] and APFL [25] have addressed the communication complexity of their respective frameworks. By assuming that Assumption 3.1 holds and utilizing Eq. (30), we can determine the communication complexity of RWSADMM as  $O(\omega^{-1})$  for  $K$  iterations. In comparison, Per-FedAvg exhibits a higher communication complexity of  $O(\omega^{-3/2})$ . In the case of APFL, all clients are assumed to be used in each computation round to ensure convergence in nonconvex settings. The communication complexity of APFL is determined as  $O(n^{3/4}\omega^{-3/4})$ , where  $n$  represents the total number of clients. Consequently, when  $n$  is large, APFL exhibits a significantly higher communication rate than RWSADMM. Overall, the communication complexity analysis suggests that RWSADMM offers superior scalability and communication efficiency compared to existing methods.
Setup We evaluate the performance of RWSADMM using heterogeneous data distributions. All the experiments are conducted on a workstation with Threadripper Pro 5955WX, 64GB DDR4 RAM, and NVidia 4090 GPU. All frameworks are performed on standard FL benchmark datasets (MNIST [45], Synthetic [23], and CIFAR10 [46]) with 10-class labels and convex and non-convex models. Multinomial logistic regression (MLR), multilayer perceptron network (MLP), and convolutional neural network (CNN) models are utilized for strongly convex and two non-convex settings, respectively. We create a moderately dynamic connected graph of randomly placed nodes where each node has at least 5 neighboring nodes at  $k$ -th update. We set the probability transition matrix  $\mathbf{P}(k)$  as  $[\mathbf{P}(k)]ij = 1 / deg(i_k)$  and set up the experiments for  $N = 20$  clients with a regeneration frequency of 10 steps for the dynamic graph. The data is split among clients using a pathological non-IID setting. The data on each client contains a portion of labels (two out of ten labels), and the allocated data size for each client is variable. For the Synthetic data, we use the same data generative procedures of [23] with 60 features and 100 clients. All local datasets are split randomly with  $75\%$  and  $25\%$  for training and testing, respectively. The models' details, the rationale behind graph construction, and hyperparameter tuning for  $\beta$ ,  $\kappa$ , and selected  $\varepsilon$  value are further described in Appendix D.
Performance Comparison The performance of RWSADMM is compared with FedAvg [1] as a benchmark and several state-of-the-art personalized FL algorithms such as Per-FedAvg [22], pFedMe [23], APFL [25], and Ditto [24]. The test accuracy and training loss for the MNIST dataset is depicted in Fig. 2. (Synthetic and CIFAR10 figures are presented in Appendix D). Test accuracy and time cost for all the datasets are reported in Table 1.
The test accuracy progress curves of RWSADMM for all the models (2a-2c) have a significantly faster convergence. For the non-convex models (2b), RWSADMM reaches convergence after 200 iterations, while the rest of the algorithms, except Ditto, work toward convergence until 600 iterations. The performance curves are shown for 100 iterations for consistency. When tested on MNIST with MLP, RWSADMM demonstrated comparable performance against pFedMe. In the test on Synthetic data with MLR models, RWSADMM exhibited a significant advantage over the other methods, with an improved margin of  $14.95\%$ . Regarding computational efficiency, RWSADMM is slower than FedAvg and Per-FedAvg, but faster than pFedMe. Furthermore, RWSADMM converges in fewer iterations (200 iterations) than pFedMe (600 iterations). RWASDMM is also run for more extensive networks with 50 and 100 nodes as a separate set of experiments. The performance comparison results and diagrams are also in Appendix D.
This study proposes a novel approach called RWSADMM, designed for systems with isolated nodes connected via wireless links to the mobile server without relying on pre-existing communication
(a) MLR acc
(b) MLP acc
(c) CNN acc
(d) MLR loss
(e) MLP loss
(f) CNN loss
Figure 2: Performance comparison (test accuracy and training loss) of RWSADMM, pFedMe, PerAvg, FedAvg, APFL, and Ditto for MNIST dataset for the MLR (2a, 2d), MLP (2b, 2e), and CNN (2c, 2f) models. The first 100 iterations are plotted to show the convergence progress better.
Frameworks	MNIST	Synthetic
MLR	MLP	CNN	MLR	MLP
acc(%)	t(s)	acc(%)	t(s)	acc(%)	t(s)	acc(%)	t(s)	acc(%)	t(s)
FedAvg	93.96 ± 0.02	128	98.79 ± 0.03	155	97.83 ± 0.15	2655	77.62 ± 0.11	592	83.64 ± 0.22	680
PerAvg	94.37 ± 0.04	154	98.90 ± 0.02	203	98.97 ± 0.08	2432	81.49 ± 0.09	267	85.01 ± 0.10	269
pFedMe	95.62 ± 0.04	448	99.46 ± 0.01	699	99.05 ± 0.06	5541	83.20 ± 0.06	254	86.36 ± 0.15	1413
Ditto	97.37 ± 0.02	276	97.79 ± 0.03	423	99.20 ± 0.11	3273	86.24 ± 0.03	72	85.26 ± 0.10	79
APFL	92.64 ± 0.03	304	97.74 ± 0.02	533	98.58 ± 0.03	5933	83.40 ± 0.04	95	82.52 ± 0.15	111
RWSADMM (our method)	98.63 ± 0.01	167	99.29 ± 0.02	295	99.52 ± 0.04	3857	96.44 ± 0.12	473	97.17 ± 0.18	692
Frameworks	CIFAR10
MLR	MLP	CNN
acc(%)	t(s)	acc(%)	t(s)	acc(%)	t(s)
FedAvg	40.84 ± 0.01	160	41.02 ± 0.05	69	38.65 ± 0.05	78
PerAvg	47.43 ± 0.09	192	60.25 ± 0.07	253	83.52 ± 0.01	800
pFedMe	67.53 ± 0.34	515	78.12 ± 0.38	340	83.56 ± 0.05	3480
Ditto	75.2 ± 0.01	225	81.37 ± 0.13	259	83.86 ± 0.02	2189
APFL	75.17 ± 0.32	50	78.00 ± 0.18	55	66.23 ± 0.03	702
RWSADMM (our method)	80.72 ± 0.11	131	84.99 ± 0.20	253	87.08 ± 0.03	3759
Table 1: Performance comparisons of FedAvg, Per-FedAvg, pFedMe, Ditto, APFL, and RWSADMM frameworks on MNIST, Synthetic, and CIFAR10 datasets. Three models are utilized for each dataset, and each model's converged accuracy (\%) and time consumption (seconds) are reported. Each configuration is executed for ten iterations, and variance is calculated to compute the degree of confidence for test accuracy rates.
infrastructure. The algorithm enables the server to move randomly toward a local client, establishing local proximity among adjacent clients based on hard inequality constraints, addressing the challenge of data heterogeneity. Theoretical and experimental results demonstrate that RWSADMM is fast-converging and communication-efficient, surpassing current state-of-the-art FL frameworks. This study primarily focuses on the methodological framework for RWSADMM. Future research directions should explore essential techniques such as incorporating differential privacy techniques and examining scalability in more extensive network and dataset scenarios. Further investigation is needed to assess the implementation in physical networks and evaluate the effect of communication delays in the real world.
This research was partially supported by the NSF 2122309 and NSF 2104337.