SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\eta_{\alpha}$  and  $\eta$  are the meta-step-size parameters. Practical parameter selection strategies with meta-step are shown as  $\mu_{i,r}^{(t)}$  in Eq. (14). The three of  $\mu_{i}$ , i.e.  $\lg$ ,  $\mathrm{meg}$  and  $\mathrm{mh}$ , represent loss gradient, memorized envelope gradient and memorized hybrid respectively.
Convergence Analysis we analyze the convergence of pFedBreD with RMD on a uniform client sampling  $\mathbf{E}_i = \frac{1}{N}\sum_{i=1}^{N}$  setting for simplification. Other sampling methods can be obtained with client sampling expectation  $\mathbf{E}_i[F_i] = F$ , by changing sampling weights. The assumptions, proof sketch and detailed notations are in Appendix A.1 and Appendix D.
Theorem 1 (pFedBreD's global bound). Under settings in Section 5 and Appendix  $D$ , at global epoch  $T \geq \frac{2}{\hat{\mu}_F \cdot \hat{\alpha}}$ , by properly choose  $\tilde{\alpha}_m = \alpha_m \beta R$ ,  $\exists \tilde{\alpha}_m \leq \min \left\{ \frac{\beta}{\sqrt{2\hat{c}}}, \frac{2}{\hat{\mu}_F}, \hat{\alpha}_m \right\}$ , where  $A = [\frac{\hat{L}_{g^*}}{\hat{\mu}_F} (\hat{u}_m + \eta \hat{\gamma}_\Phi)]^2 (\frac{\hat{\gamma}_f^2}{|\hat{d}_i|} + \hat{\epsilon}^2)$ ,  $B = [\hat{L}_{\mathcal{E}} \hat{\gamma}_\Phi (1 + \sigma_\Phi)(\hat{u}_m + \eta \hat{\gamma}_\Phi)]^2$ ,  $C = \frac{\sigma_\Phi^2 \hat{L}_{\mathcal{E}}^2 (\hat{u}_m + \eta \hat{\gamma}_\Phi)^2}{\hat{\mu}_F^3}$ ,  $\xi^{(t)} = (1 - \frac{\tilde{\alpha} \hat{\mu}_F}{2})^{-t-1}$ ,  $\bar{w}^{(T)} := \frac{\sum_{t=0}^{T-1} \xi^{(t)} w^{(t)}}{\sum_{t=0}^{T-1} \xi^{(t)}}$  and  $\hat{\alpha}_m := \frac{\hat{\mu}_F \cdot \beta R}{e(1 + \sigma_\Phi) \hat{L}_{\mathcal{E}} (\hat{u}_m + \eta \hat{\gamma}_\Phi) 2^{R+6\frac{1}{2}} (\frac{1}{R} + 2) + 18(\hat{\mu}_F \cdot \beta R) \hat{L}_F}$ , such that:
$$
\begin{array}{l} \mathcal {O} \left[ \mathcal {D} _ {F} \left(\bar {w} ^ {(T)}, w ^ {*}\right) \right] = \mathcal {O} \left(\hat {\mu} _ {F} e ^ {- \tilde {\alpha} _ {m} \hat {\mu} _ {F}. T / 2} \boldsymbol {\Delta} ^ {(0)}\right) + \mathcal {O} \left(\frac {A \lambda^ {2} + B}{\hat {\mu} _ {F}}\right) \\ + \mathcal {O} \left(\frac {(N / S - 1) \sigma_ {F , *} {} ^ {2}}{N T \hat {\mu} _ {F .}}\right) + \mathcal {O} \left(\frac {2 ^ {R} C}{T ^ {2} \beta^ {2} R ^ {2}} \left[ R \sigma_ {F, *} ^ {2} + A \lambda^ {2} + B \right]\right). \\ \end{array}
$$
$^{8}$ A variant of  $\mathbf{mh}$  is in Appendix C.2.
Theorem 2 (pFedBreDns's first-order personalization bound). Under the same conditions as in Theorem 1, with prior assumption of a spherical Gaussian and first-order approximation, the bound for the gap between the personalized approximate model and global model in the Euclidean space is:
$$
\mathbf {E} | | \tilde {\theta} _ {i} (\bar {w} ^ {T}) - w ^ {*} | | ^ {2} \leq \mathcal {O} (\dot {\delta} _ {p}) + \mathcal {O} [ \dot {c} _ {p} \mathcal {D} _ {F} (\bar {w} ^ {(T)}, w ^ {*}) ]
$$
where  $\dot{\delta}_p = \frac{2}{\hat{\mu}_{F_{i_1}}^2} \left( \frac{\hat{\gamma}_f^2}{|\dot{d}_i|} + \hat{\epsilon}^2 \right) + \frac{2}{\lambda^2} \epsilon_1^2 + \frac{4}{\lambda^2} \sigma_{F,*}^2 + \frac{1}{2} \eta^2 \mathcal{G}_{\Phi}^2$ , and  $\dot{c}_p = (\frac{32}{\lambda^2} \hat{L}_F + \frac{8}{\hat{\mu}_F})$ .
Remark 1. Theorem 1 shows the main factors that affect the convergence of a global model are as follows: random mini-batch size, client drift error, aggregation error, heterogeneous data, dual space selection, local approximation error, and selection strategy for exponential family prior mean and variance. These can be divided into four categories based on their computational complexity. The first and second term shows that the proper fixed  $\tilde{\alpha}_m$  can linearly reduce the influence of initial error  $\pmb{\Delta}^{(0)}$  and the global model converges to a ball near the optimal point. The radius of this ball is determined by the personalized strategy and local errors (including local data randomness and envelope approximation errors). The third term implies that a linear convergence rate  $\mathcal{O}(1/(NT))$  can be obtained w.r.t. the total global epoch  $NT$  in the presence of aggregation noise. Without client sampling  $N = S$ , according to the fourth term, the quadratic rate  $\mathcal{O}(1/(TNR)^2)$  can be obtained with  $\beta = \mathcal{O}(N)$  or  $\beta = \mathcal{O}(N\sqrt{R})$  (Note that the number of local epoch  $R$  cannot be too large due to client drift, according to  $2^R$ ). Theorem 2 shows that, with spherical Gaussian prior assumption and first-order methods, the radius of the neighborhood range for the minimum that includes the personalized model on  $i^{th}$  client,  $\mathcal{O}(C_{\Phi,F,f,d} + \frac{1}{\lambda^2} (\epsilon_1^2 + \sigma_{F,*}^2 + \frac{B\hat{L}_F}{\hat{\mu}_F}) + \lambda^2 \frac{A}{\hat{\mu}_F})$ , can be trade-off by  $\lambda$ , and is affected by the prior selection strategies and first-order approximate error besides the elements in Theorem 1. (Note that the Euclidean space is self-dual.)
6.1 General Settings
Tricks, Datasets and Models: our experiments include several tasks: CNN [28] on CIFAR-10 [18, 39], LSTM [27] on Sent140 [11] and MCLR/DNN on FEMNIST [11]/FMNIST [65, 67]/MNIST[65, 42]. The details of tricks (FT, AM), data heterogeneity and models are in Appendix C.
Baselines: we choose following algorithms as our baselines: FedAvg [53], Per-FedAvg [20], pFedMe [65], FedAMP [31], pFedBayes [77] and FedEM [52]. These baselines are respectively classical FL, MAML-based meta learning, regularization based, FTML methods, variational inference PFL and FMTL with EM.
Global Test and Local Test: the global and personalized model, represented by  $\mathbf{G}$  and  $\mathbf{P}$ , are evaluated with global and local tests respectively. Global test means all the test data is used in the test. Local test means only the local data is used for the local test and the weight of the sum in local test is the ratio of the number of data. The results of average accuracy per client are shown in Table 1. Each experiment is repeated 5 times. More details are in Appendix C. For readability, we only give the error bar in the main Table 1 and Table 2, and keep one decimal except for the main Table 1.
Hyperparameter Settings The step-size of the main problem,  $\alpha_{m}$ , and the personalized step-size,  $\alpha$ , for all methods are 0.01.  $\beta$  is 1, and the number of local epochs,  $R$ , is 20 for all datasets.  $\lambda$  is chosen from 15.0 to 60.0. The batch sizes of Sent140 and the other datasets are 400 and 20, respectively, and the aggregation strategy,  $\mathcal{I}$ , is uniform sampling. The ratios of aggregated clients per global epoch are  $40\%$ ,  $10\%$ , and  $20\%$  for Sent140, FEMNIST, and the other datasets, respectively. The numbers of total clients,  $N$ , are 10, 198, 20, and 100 for Sent140, FEMNIST, CIFAR-10, and other datasets. The number of proximal iterations is 5 for all settings with proximal mapping. In our implementations,  $\eta_{\alpha}$  and  $\eta$  are respectively 0.01 and 0.05.
Summarizing the Effects of Hyper-parameters We test the hyper-parameter effect of  $\eta$  and  $\lambda$  in our implementation  $pFedBreD_{ns,\mathbf{mh}}$ . The details are in Appendix C. From the results, we find that it will degrade the test accuracy if the values of  $\lambda$  or  $\eta$  are too large or too small. The test accuracy of personalized model is more sensitive than the ones of global model. The test accuracy of personalized
Table 1: Results of average testing accuracy (\%) per client of each settings. We mark the best and second best performance by **bold** and **underline*. Avg and **Std**: the average results and the standard deviation of them on all tasks; H.Avg and H.Std: the average results and the standard deviation of them on hard tasks (non-linear DNN with complex classification or architecture: DNN / CNN / LSTM on FEMNIST / CIFAR-10 / Sent140). The G and P are global and personalized model
Methods / Datasets	FEMNIST	FMNIST	MNIST	CIFAR-10	Sent140	Statistics
Names - Models	MCLR	DNN	MCLR	DNN	MCLR	DNN	CNN	LSTM	Avg	Std	H.Avg	H.Sd
FedAvg [53] -G	53.38±0.26	57.04±0.08	82.75±0.04	80.09±0.06	86.59±0.03	88.26±0.05	57.51±0.07	70.86±0.01	72.06	14.34	61.80	7.85
FedAvg+AM -G	55.34±0.05	59.03±0.10	82.58±0.03	81.03±0.12	86.74±0.03	89.31±0.05	57.07±0.12	71.27±0.01	72.80	14.01	62.46	7.70
FedEM [52] -G	40.75±0.32	45.47±0.04	95.78±0.03	96.42±0.03	85.75±0.01	86.49±0.02	57.67±0.16	66.72±0.03	71.88	22.28	56.62	10.66
pFedBayes [77] -P	49.66±0.46	-	98.46±0.05	98.67±0.05	89.64±0.06	90.48±0.12	-	-	-	-	-	-
FedAMP [31] -P	60.04±0.08	66.77±0.04	98.63±0.02	98.72±0.01	90.81±0.02	92.21±0.02	77.40±0.04	69.83±0.05	81.80	15.21	71.34	5.46
pFedMe [65] -P	50.74±0.10	53.56±0.12	97.60±0.03	98.63±0.01	88.20±0.05	90.51±0.01	72.24±0.05	69.36±0.02	77.61	18.96	65.05	10.06
pFedMe+FT -P	58.04±0.11	62.93±0.10	97.63±0.01	98.39±0.02	88.36±0.02	91.71±0.01	68.17±0.11	67.82±0.03	79.13	16.53	66.31	2.93
pFedMe+AM -P	55.56±0.09	60.08±0.05	97.57±0.02	98.67±0.00	88.46±0.02	91.22±0.00	73.35±0.09	70.93±0.05	79.48	16.79	68.12	7.07
Per-FedAvg [20] -P	54.34±0.14	62.72±0.03	94.28±0.05	97.46±0.04	87.09±0.01	90.96±0.02	78.87±0.05	70.05±0.03	79.47	15.74	70.54	8.09
Per-FedAvg+FT -P	55.34±0.15	63.34±0.01	95.76±0.07	98.10±0.01	87.56±0.03	89.58±0.01	79.68±0.04	70.20±0.01	79.95	15.61	71.07	8.20
Per-FedAvg+AM -P	56.66±0.09	65.74±0.02	92.08±0.10	98.24±0.02	86.91±0.04	90.85±0.02	78.97±0.03	70.73±0.05	80.02	14.54	71.81	6.68
mh (ours) -P	56.34±0.09	64.93±0.03	98.44±0.01	98.73±0.01	89.83±0.02	92.04±0.01	79.44±0.02	72.04±0.01	81.47	15.88	72.14	7.26
mh (ours)+FT -P	59.81±0.07	67.53±0.02	98.51±0.02	98.98±0.03	90.10±0.03	92.96±0.05	79.16±0.03	71.87±0.01	82.37	14.92	72.85	5.88
mh (ours)+AM -P	60.64±0.02	70.34±0.01	98.48±0.01	98.73±0.01	89.88±0.01	92.32±0.02	80.60±0.01	73.68±0.05	83.09	14.01	74.87	5.23
model is more sensitive to  $\eta$  than to  $\lambda$ . Note that the hyper-parameters are roughly tuned, which shows the insensitivity of  $\mathbf{m}\mathbf{h}$ , and better tuning could improve the performance in the Table 1.
6.2 Analysis
Comparative Analysis of Performance We compare our methods and the baselines from different perspectives, including convex or non-convex problems, easy or hard tasks, and text tasks. Additionally, we briefly discuss the absence of BNN on hard tasks.
Convex or non-convex: on non-convex problems, especially in hard tasks, our method significantly outperforms other methods by at least  $3.06\%$  employing some simple tricks. On convex problem, FedAMP outperforms our method somewhat on convex problems with simple data sets. One explanation is that the learning landscape is simple in shape for these problems and FedAMP converges faster for this case. One possible reason for this is that since FedAMP uses the distance between models as a similarity in the penalty point selection, giving greater weight to the model that is most similar to the local one. In the later stages of training, since there is only one global optimum, this penalty point tends not to change, and thus the method degenerates into a non-dynamic regular term. Compounding intuition, this method will not be as advantageous for non-convex problems and harder convex problems, as penalty point tends to fall into the local optimum and lead to degradation of the dynamic regular term.
From easy to difficult task: from the difference between the statistics of Avg and H.Avg in Table 1, it can be observed that meta-step methods perform most consistently, with all other methods dropping at least  $10\%$ . This is due to the simple and effective local loss design of MAML, with its learning-to-learning design philosophy that enables the method to be more stable in complex situations [19, 20].
Personalized prior on text: text tasks, as opposed to image tasks, generally have relatively rugged learning landscape. [55, 16, 13] This understanding is manifested in specific ways, such as parameter sensitivity, slow convergence, and struggling during the process. Thus, the overlooked prior information seems to be more important, which means that each local iteration not only obtains local knowledge from the data, but also the prior itself already contains some local knowledge. Therefore, there is no need to re-obtain this knowledge from scratch solely from the data during training.
Absence of BNN on hard tasks: complex BNN is not in Table 1, such as LSTM in pFedBayes, because it is difficult to conduct comparative experiments by fixing elements, e.g., inferential models, tricks and optimization methods. In pFedBayes, training often crashes on hard tasks and large datasets, as mentioned in [77]. Our one-step-further research shows that it may be caused by the reparameterization tricks and vanilla Gaussian sampling. If we add tricks on it, the implementation will be very different from the original pFedBayes, and it is beyond this analysis.
Ablation Analysis of Personalized Prior We conduct ablation experiments by dropping the gradient of the Bregman-Moreau envelope, the local loss function, or both, from the personalized
Table 2: Average local test accuracy of personalized model (\%) in ablation experiments.  $(\uparrow/\downarrow$  : average accuracy is increased/reduced; AC4PP: Additional cost for personalized prior; Grad. and Add.: cost about calculate gradient and addition; Other notations are the same in Table 1.)
Methods	FEMNIST	FMNIST	MNIST	CIFAR-10	Sent140	Statistics	AC4PP
MCLR	DNN	MCLR	DNN	MCLR	DNN	CNN	LSTM	Avg	H.Avg
Non-PP	50.7±0.10	53.6±0.12	97.6±0.03	98.6±0.01	88.2±0.05	90.5±0.01	72.2±0.05	69.4±0.02	77.6	65.1	None
lg (ours)	50.8±0.05	49.1±0.53	98.3±0.02	98.4±0.02	88.4±0.01	91.0±0.00	65.7±0.46	60.7±0.41	75.3↓	58.5↓	Grad. × R
meg (ours)	50.3±0.07	53.9±0.06	97.8±0.00	98.6±0.01	88.4±0.01	90.6±0.01	73.8±0.06	69.4±0.02	77.9↑	65.7↑	Add. × R
mh (ours)	56.3±0.09	64.9±0.03	98.4±0.01	98.7±0.01	89.8±0.02	92.0±0.01	79.4±0.02	72.0±0.01	81.5↑	72.1↑	Both above
CIFAR-10 with CNN
Epoch Index
MNIST with DNN
Figure 2: The results of GCE( $\{\nabla F_i(w_i^{(t)})\}$ ) at each global epoch  $t$  after Savitzky-Golay filtering [62].
strategy mh as shown in Table 2. The relationship among the three strategies mentioned in Eq. (14) is that mh consists of lg and meg. Moreover, pFedMe can be regarded in our framework as the one which takes the spherical Gaussian as prior and uses vanilla prior selection strategy  $\mu_{i} = I$  without personalization. Thus, pFedMe and the three implementations of pFedBreD are compared. The results reveal the instability of our implementation lg and the introduction of meg on difficult tasks is about the same as not introducing it. However, introducing both lg and meg (i.e., mh) together shows remarkable performance. This indicates that lg and meg complement each other. To explain these results, by observing the error bars, in most of the settings, meg is significantly more stable compared to methods that do not use personalized priors, while lg is relatively less stable. Based on this observation, we have reason to believe that meg weakens the influence of potential noise, while lg introduces new noise. Therefore, we can infer that while the mean parameters are steadily biased towards the personalized model, the introduction of new noise finds a path that is more likely to escape from local optima or saddle points, based on implicit regularization [61, 58, 57].
Generalized Coherence Analysis of Information Injection and Extraction The generalized coherence estimate (GCE) [25] of vectors from personalized to local model (i.e., the envelope gradients in pFedMe and ours) among clients on each global epoch are shown in Figure 2. The smaller the GCE, the less coherent the envelope gradient between individual nodes and the greater the diversity of information in the global model update. As shown in Figure 2, we can observe that during the convergence phase, using a personalized prior method has significantly greater information diversity than not using a personalized prior method, which proves the success of injecting personalized prior knowledge into the global model and extracting local knowledge from the local training.
Variable-Control Analysis of Robustness We analyze the impact of aggregation noise and data heterogeneity [29] on our method, mainly mh, by controlling variables. Results are in Table 3 and Table 4. (Details are in Appendix C.8.) We test the performance of global model on different aggregation ratios, where all hyper-parameters except for the aggregation ratios are fixed. Meanwhile, we test the performance of both global and the personalized model on different data heterogeneity settings, where full aggregation (sample client equals total number of clients,  $S = N$ ) and one-step local update (local epoch  $R = 1$ ) are employed to get rid of the effects of aggregation noise and client drift. The experiments demonstrate the instability of the global model in mh at small aggregation ratios, which most of the other PFL methods have, by comparing their performances on different aggregation numbers. Comparing to the baselines, the experiments also demonstrate the relative robustness of our method to extreme data heterogeneity.
Table 3: The global test accuracy  $(\%)$  of the global model with different numbers of clients for aggregation  $S\in \{10,20,50,100\} .(\spadesuit :\mathrm{FEMNIST},$ $\diamondsuit$  :FMNIST)
Numbers	small	→	large	Std
♠-DNN	59.0	60.1	60.1	59.8	0.5
♠-MCLR	54.4	55.4	55.4	55.5	0.5
♦-DNN	75.1	79.6	79.4	79.3	2.2
♦-MCLR	80.0	82.6	81.8	82.7	1.3
Table 4: The local test accuracy  $(\%)$  of the personalized model on FMNIST-DNN setting with different data heterogeneity (Non-IID) settings  $\alpha \in$ $\{0.01, 0.1, 1, 10, 100, 1000\} (\alpha \downarrow, \text{Non-IID} \uparrow)$  [29]. The Bolded means the best.
Non-IID	small	→	large	Avg
FedAvg-G	18.2	14.8	14.5	11.9	11.3	11.2	13.7
pFedMe-P	89.5	58.2	24.2	12.3	11.8	10.6	34.4
pFedMe-G	17.0	14.3	14.1	12.3	10.8	10.9	13.2
mh(ours)-P	89.6	58.7	25.2	13.1	11.1	11.0	34.8
mh(ours)-G	17.1	14.6	14.6	12.4	11.9	11.9	13.8
MNIST-MCLR-G
MNIST-MCLR-L
MNIST-DDN-G
Figure 3: The loss deviation of experiments in Section 6 on the first client, whose major data are on  $0^{t\overline{h}}$  classes. The lower deviation of the available class on global tests and the higher deviation of the unavailable class on local tests demonstrate the superior personalization ability of our methods.
MNIST-DDN-L
**Deviation Analysis of Personalization** Deviation represents the difference between an individual and the mean value. We use the deviation of the loss function to reflect the personalization. On global test, the lower the deviation, the better the personalized model performance on the corresponding local data. On local test, the model is only tested on its own dataset, and because of multiple local iterations, the local test deviation converges to almost the same value, as shown in MNIST-MCLR-L and MNIST-DNN-L in Figure 3. Furthermore, since the local test has a loss of 0 on missing classes, a higher deviation on missing classes reflects a lower mean on these classes. Thus, the lower loss in local testing and better performance are reflected from both of the almost equal deviation in local testing and the higher deviation on missing class. **Summary:** based on Figure 3, we can see that our method has higher deviation on missing classes in local testing and lower deviation in global testing. This means that our method has better personalized performance.
Conclusion To address the issue of neglecting client-sampling information while providing prior knowledge to local training via direct use of a global model, we propose a general concept: the personalized prior. In this paper, we propose a general framework, pFedBreD, for exploring PFL strategies under the SX-family prior assumption and computation, the RMD to explicitly extract the prior information, and three optional meta-step strategies to personalize the prior. We analyze our proposal both theoretically and empirically. Our strategy mh shows remarkable improvement in personalization and robustness to data heterogeneity on non-i.i.d. datasets and the LEAF benchmark [11] with MCLR / DNN / CNN / LSTM as inferential model, which conduct convex / non-convex problems, and image / language benchmarks.
Limitations and Future Work Although mh shows remarkable performance and robustness, there is still instability in the global model with aggregation noise. Furthermore, it should be noted that the superficial reason for the improvement of mh seems to be that  $\eta_{\alpha}$  and  $\eta$  and (which are similar to each other) are used simultaneously, resulting in a magnitude in mh that is twice as large as the ones in the other two implementations and leading to better performance. However, empirically, simply doubling  $\eta_{\alpha}$  in lg or  $\eta$  in meg does not improve performance, and using one more meg step used in lg significant improvement. Our theoretical analysis cannot explain this phenomenon, and more detailed modeling is needed.
This work is supported by the Key Program of National Science Foundation of China (Grant No. 61836006) from College of Computer Science (Sichuan University) and Engineering Research Center of Machine Learning and Industry Intelligence (Ministry of Education), Chengdu 610065, P. R. China. This research is also supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-008).
[1] Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Proceedings of the NeurIPS, 34:8392-8406, 2021.
[2] Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to mcmc for machine learning. MACH LEARN, 50(1):5-43, 2003.
[3] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv:1912.00818, 2019.
[4] Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the ICALT, volume 117 of PMLR, pages 111-132, 08 Feb-11 Feb 2020.
[5] Ulrich Matchi Aïvodji, Sébastien Gambs, and Alexandre Martin. Iotfla: A secured and privacy-preserving smart home architecture implementing federated learning. In 2019 IEEE SPW, pages 175–180, 2019.
[6] Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh, and John Lafferty. Clustering with bregman divergences. JMLR, 6(10), 2005.
[7] Heinz Bauschke, Patrick Combettes, and Dominikus Noll. Joint minimization with alternating bregman proximity operators. PAC JOPTIM, 2006.
[8] Heinz H Bauschke, Jonathan M Borwein, and Patrick L Combettes. Bregman monotone optimization algorithms. SIAM J CONTROL OPTIM, 42(2):596-636, 2003.
[9] Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28. Princeton university press, 2009.
[10] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. COMP MATH MATH PHYS+, 7(3):200-217, 1967.
[11] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv:1812.01097, 2018.
[12] Gary Cheng, Karan Chadha, and John Duchi. Federated asymptotics: a model to compare federated learning algorithms. arXiv:2108.07313, 2022.
[13] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural Language Processing (Almost) from Scratch. NLP, 2011.
[14] Katalin Csilléry, Michael G.B. Blum, Oscar E. Gaggiotti, and Olivier François. Approximate bayesian computation (abc) in practice. TRENDS ÉCOL EVOL, 25(7):410-418, 2010.
[15] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. JR STAT SOC B, 39(1):1-22, 1977.
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In the Proceedings of the CVPR, pages 248–255, 2009.