PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning.
Authors: Mingjia Shi
Yuhao Zhou
Kai Wang
Huaizheng Zhang
Shudong Huang
Qing Ye
Jiancheng Lv
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Personalized Federated Learning
Expectation Maximization
Relaxed Mirror Descent
TL;DR: The paper proposes new PFL methods called pFedBreD that injects personalized prior knowledge into clients' global models to address the incomplete information problem with theoretical support, achieves state-of-the-art performance on 8 benchmarks.

Abstract: Classical federated learning (FL) enables training machine learning models without sharing data for privacy preservation, but heterogeneous data characteristic degrades the performance of the localized model. Personalized FL (PFL) addresses this by synthesizing personalized models from a global model via training on local data. Such a global model may overlook the specific information that the clients have been sampled. In this paper, we propose a novel scheme to inject personalized prior knowledge into the global model in each client, which attempts to mitigate the introduced incomplete information problem in PFL. At the heart of our proposed approach is a framework, the $\textit{PFL with Bregman Divergence}$ (pFedBreD), decoupling the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized scenarios. We also relax the mirror descent (RMD) to extract the prior explicitly to provide optional strategies. Additionally, our pFedBreD is backed up by a convergence analysis. Sufficient experiments demonstrate that our method reaches the $\textit{state-of-the-art}$ performances on 5 datasets and outperforms other methods by up to 3.5% across 8 benchmarks. Extensive analyses verify the robustness and necessity of proposed designs. The code will be made public.

PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning.
Mingjia Shil*
Yuhao Zhou'
Kai Wang"
Huaizheng Zhang
Shudong Huang'
Qing Ye
Jiangcheng Lv
Sichuan University
National University of Singapore
Abstract
Classical federated learning (FL) enables training machine learning models with- out sharing data for privacy preservation, but heterogeneous data characteristic degrades the performance of the localized model. Personalized FL (PFL) addresses this by synthesizing personalized models from a global model via training on local data. Such global model may overlook the specific information that the clients have been sampled. In this paper, we propose novel scheme to inject personalized prior knowledge into the global model in each client, which attempts to mitigate the introduced incomplete information problem in PFL. At the heart of our pro- posed approach is framework, the PFL with Bregman Divergence (pFedBreD), decoupling the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized scenarios. We also relax the mirror descent (RMD) to extract the prior explicitly to provide optional strategies. Additionally, our pFedBreD is backed up by a convergence analysis. Sufficient experiments demonstrate that our method reaches the state-of-the sfute-of-the-ar performances on 5 datasets and outperforms other methods by up to 3.5% across 8 benchmarks. Extensive analyses verify the robustness and necessity of proposed designs. https: //gts::/gtt thub. cwm/ com/BDeMo/ @eFeeBree_ppubll
Introduction
Federated learning (FL) [53] has achieved significant success in many fields [72, 43, 71, 76, 69, 59, 32, 5, 34, 49], which include recommendation systems utilized by e-commerce platforms [71], pro- phylactic maintenance for industrial machinery [76], disease prognosis employed in healthcare [69]. Data heterogeneity is fundamental characteristic of FL, leading to challenges such as inconsistent training and testing data (data drift) [36]. An efficient solution to these challenges is to fine-tune the global model locally for adaptation on local data [3, 74, 35]. This solution is straightforward and pioneering, but presents fundamental limitation when dealing with highly heterogeneous data, For examples, heterogeneous data drift may introduce substantial noise [29] and the resulted model may not generalize well to new sample [20, 12]. Thus, heterogeneous data in FL still challenging [66].
Recently, personalized FL (PFL) is proposed to mitigate the aforementioned negative impact of heterogeneous data [66]. To improve the straightforward solution mentioned above, Per- Per-FedAvg [20] is introduced to train a global model that is casier to fine-tune. Another paper on the similar topic, FedProx [45], aims to resolve the issue of personalized models drifting too far from the global model during training with dynamic regularizer in the objective during local training. This issue could occur especially in post-training fine-tuning methods without regularization (e.g., Per-FedAAg [20]). Moreover, PFedMe [65], another regularization method modeling local problems using Moreau
*3 i0s 13100iss@gmillcom gmail.com
'qqual Corresponding Authors (yeqing, lyjiancheng | @ @sscu.du.cn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Aspropatinn E,
Pric with Clee Cheeini
Extracion
Locil
BAIDE
Clobal Kinwiinte Transter
Pervonable Frice
Triteig (IOEAM
Lecd Kannicdar Trensier
Glebal Mocil
Looal Model
Ptier
Prreealled Model
Pesosdlod Price
Lood Deo
Opentioe: WithWitiet Clex Sanyling
Krondeipe Wibh Wib//Wbccu Sarpieee
Figure 1: PFedBreD framework: Global-MLE and Local-MAP. The personalized prior knowledge is injected into the global model of the global problem (MLE) in the 2nd step for local training. The local knowledge extracted from the local problem (MAP) in the 4th step for aggregation.
envelopes, replaces FedProx's personalized model aggregation method with an interpretable approach for aggregating local models [45]. It also accommodates first-order Per-FedAvg [20].
Although the existing PFL methods have achieved promising results, the prior knowledge from single global model for local training [66] hinders the development PFL. Specifically, we analyze the shortcomings of current PFL methods as follows: 1) utilizing the same global model for direct local training could potentially disregard the client's ' sampling information. As shown in Figure 1, single global model provides global knowledge directly for local training, which overlooks the client-sampling information when the global knowledge is transferred to specific clients. 2) Explicitly extracting prior knowledge can be challenging task. Most of the insightful works [17, 52] propose assumptions for recovering this incomplete information, but these assumptions are implicit, which limits the way use the information to develop personalized strategies.
To address the former issue above, we propose framework pFedBreD to inject personalized prior knowledge into the provided by global model. As shown in Figure 1, it is injected in the 2nd step and the local knowledge is transferred into global model via local models instead of directly aggregating personalized models [45]. To address the latter, introduce relaxed mirror descent RMD to explicitly extract the prior for exploring personalized strategies.
Our method is backed up with direct theoretical support from Bayesian modeling in Section 4 and a convergence analysis in Section which provides a linear bound (1/TTN) with aggregation noise and quadratic speedup O O1I/(TN without." Meanwhile, the existence and validity of injection and extraction aforementioned information verified Section 6.2. The remarkable performance of the implements of the proposed method is tested on 5 datasets and 8 benchmarks. Consistently, our method reach the state-of-the-are Especially, the improvement of accuracy on task DNN-FEMNIST [11] is up to 3.5%. Extensive ablation study demonstrate that parts of the hybrid strategy mh are complementary to each other. Our contributions can summarized as follows:
The problem of overlooking slient-sampling information at prior knowledge being trans- ferred is introduced in this paper, and we first investigate the possibility of explicitly expressing the prior knowledge the information and design personalized strategies on it.
To express the personalized prior, model PFL into Bayesian optimization problem, Global-MLE and Moral-MAP.... A novel framework, pFedBreD, is proposed for computing the modeled problem, and RMD is introduced explicitly extract prior information.
Sufficient experiments demonstrate our method surpasses most baselines on public bench- marks, thereby showcasing its robustness data heterogeneity, particularly in cases involv- ing small aggregation ratios and non-convex local objective settings.
TN and T'N R: total global local epoch in FL system. See Appendix A.I.

---

2 Related Works
Regularization Researchers have developed variety of approaches based on regularization to handle the PFL challenge in recent years (e.g., FedU [18]. pFedMe [65], FedAMP [31],
HeurFedAMP [31]). All of these approaches personalized objective functions can be expressed as ((] R(O; μ)) where (() is the loss function of the local problem and R(0; p)) is the regularization
term used to restrict the deviation between 0 and j (e.g., R(O; pb) = = μ in pFedMc).
Meta Learning One of the most representative meta- veta-learning based single-model PFL approach the well-known Per-FedAvg [20]. aiming to find an initialization that easy fine-tune. That is, the global model in the FL. setting is regarded
meta model in MAMI. [21, 19], where the objective function of the local problem is J(O v (O))). Researchers also show the connections between FedAvg [53] and Reptile [60], another meta learning framework. [35] shows how to improve
personalization of FL via Reptile. Proximal updating is also used in meta-learning based algorithms such as [79]. One of our strategies meg is the one motivated by MAML.
Expectation Maximization Two EM-based [15] methods are proposed, e.g., FedSparse in [50] and FedEM in [17]. Both of them focus on communication compression. The latter provides a variance reduce version and assumes complete information (or data) of the global model obeys distribution X-family. Another FedEM [52] combines Bayesian modeling, Federated Multi-task learning (FTML) and EM. Our framework pFedBreD a expectation maximuzatioin and maximum a posteriori estimate (EM-MAP) [17] algorithm with personalized prior specified.
Bayesian FL In recent years, studies of PFL with Bayesian learning have been proposed. In related approaches, FLOA [48] and pFedGP [1] are proposed with KL divergence regularization the loss function, which is comparable to applying specific assumption of X-family prior in pFedBreD see Appendix B.2 for details. Our implementation doesn't 't use Bayesian neural network (BNN) model as an inferential model as others do (e.g., pFedGP uses Gaussian process tree and pFedBayes [77] uses BNN). Instead, to eliminate weight sampling cost Bayesian methods, prior knowledge introduced through regularization term.
Preliminary
Overlooked Information in Prior Knowledge From a Bayesian and info. perspective, the global knowledge transferred in conventional method with single global model has no mutual information (MI) with client sampling i, i.c., formally, w = E,w, = E,wyli = MI [(w; i) = O, in particular (t) local init. w0o0 + where w, is the local model the hth client.
when applying reg. R(w(+).... This makes the specific model on each client have to re-obtain this information from scratch solely
from the data during training, especially impacted on hard-to-learn representations and datasets.
Bregman divergencee [10] is employed general regular term in our local objective that exactly satisfies the computational requirements and prior assumption, and is formally defined in Eq. (1).
Bregman-Moreat Envelope
: (V
Vg(u), y)
(1)
where is a convex function. For convenience, is assumed to be strictly convex, proper and differentiable such that Bregman divergence well-defined. To utilize the computational properties of Bregman Divergence in optimization problems, we introduce the following definition in Eq. (2) [7, 8]: Bregman proximal mapping, Bregman-Moreal envelope, and the relationship between them.
Dprox 9prox,,--1(et) = arg min( (0,
Denv ff == min (0,
VDDen (f=) = XV2g(>) -Drox a. (x)),
where À > 0 denotes the regular intensity in general and the variance of the prior in our modeling.
3

---

Exponential Family The regular exponential family (X (X-family) is a relatively large family that facilitates calculations. Therefore, to yield the prior, we employ the X-family [6]defined in Eq. (3).
Pe/ (V; s, g) h(V) exp( s) ((s)} h(V) exp y.- (V.p) + (V)),
(3)
where y is assumed to be convex, -)('s-) the Bregman divergence, and g" the Fenchel Conjugate of g. In Eq. (3), 5. h(V) and g(s) are respectively the natural parameter, potential measure and logarithmic normalization factor, where we have the mean parameter μ = Vg(s). Additionally, to highlight the variance, the scaled exponential family (SX-family) is introduced in Eq. (4)
Psef (V; X, 8, 9. hv) exp(>((V,s) s) x(s)]} = hx(V) exp(- - - 0-/00-1/ ()) + (V)),
(4)
where log (V) is the scaled potential measure, and the scale parameter À is employed to highlight the variance. Moreover, V is assumed to be the minimal sufficient statistic of the complete information
for local inference, details of which can be found in Section 4.
4 Methodology
In this section", we introduce missing -ampllgg information based on classic FL, use EM to reduce the computational cost of the information- introduced FL problem, and propose RMD, class of prior selection strategies, based on the E-step in EM. The general FL classification problem with KL divergence could be formulated Eq. (5) [53, 66].
information-introduueed
arg min E,E,, (L((P(M)a))) (y [r,, w)) arg max E, E,g, gyle, log ((ul w), (Tt, v1) E de. no
(5)
where we rewrite the discriminant model maximum likelihood estimation (MLE) problem [41] of y la; in the right hand side (R.H.S.) of Eq. (5). (It, y4) represent the pairs of input and label respectively in dataset d; on the jth client, and P(y [ri, w) is the inferential model parameterized by w. Each local data distribution presuppose to be unique, so using the global model with local data for inference and training could overlook the fact that the client has been sampled before transmitting the global model, and the prior knowledge transmitted directly via the global model as the local training prior knowledge (e.g. via initial points, penalty points in dynamic regular terms, etc.) has no mutual information with the client sampling, i.e., the global model w E,w; E,w;li. Thus, to reduce the potential impact of the overlooked information, the complete information θ, on the 7th client is introduced which turns Eq. (5) into Eq. (6).
arg max E Elog J.
P(y, O, w) ()O,O, d).
(6)
where E = E, E,E lu, and the direct calculation of this is computationally expensive [14].
Framework: Leveraging Expectation Maximization for Prior Parameter Extraction The integral term in Eq. (6) makes direct computation impossible [14], so we employ EM to approximate the likelihood with unobserved variables [15] as shown in Eq. (7), where Q(0,) is any probability measure.
Σ log P(y: w) Σ Eqre,) [log P(,,,, G w) Eyg log P(O,ld,, w)).
(7)
Assuming that prior O.|d,, w P sef (O,; (0; X, se(w; d)). g) and the local loss function on the zth client f,,,, w) is Ez, 1- log Pyy ja., O,, w)), we have the left hand side (L.H.S.) of the Eq. (8) from (7). Here is an assumption for simplification that O, contains all the information for local inference, i.e. 0, = O and ((y kr, O,, w) P (( O²). It happens when 0, is all the parameters of the personalized model and we only use the personalized model for inference. Thus, fa;)))
E2,1-log (s [x,, 0.)). Thus, we can optimize an upper bound as a bi-level optimization problem as shown in the R.H.S. of the Eq. (8) to solve Eq. (5) approximately, where mean parameter 14, V s; [9]. And, we can derivate our framework as shown in Section
max E, ((.))))))) - xD (0,, ((w))) S min E, min(o.) (f((.)) + XD. w,(
(8)
'oore details of equations in Appendix B.
A posteriori distribution for local client whose prior knowledge is from global model. See Appendix B.
The following di is omitted with the same footnote i in μi for simplification (pi(-) p.(;:d)))

---

To extract the prior strategies and implement μ1, E-step EM in close-form, we propose a method called relaxed mirror descent (RMD), where the mirror descent (MD) is EM in X-family [40]. MD can be generally written Eq. (9) from the old tú the new one ú+ each iteration [54, 40].
Strategies: Relaxing Mirror Descent for Prior Selection
w + arg (J + V f()), w) + (0,((0
0
According to the Lagrangian dual, we rewrite the problem into a more general variant shown in Eq. (10) with relaxed restrictions and superfluous parameter.
arg min (1 .b) (V fa w)) XD. (0. - = P
(10)
We can transform Eq. (10) back into Eq. (9) by setting (()) to satisfy V V f(),, and defining (0, vh)) as a function with f()) and a penalty term to make Ó and close as possible (e.g. xDy (8, w) This provides us a way to extract He the function to generate mean parameter of the prior, shown in Eq. (11), which is minimizing an upper bound of the problem in Eq. (10).
Denv 8* X-1 (1, (ux min (0, w) + XD (w(w))
(11)
(w(w) arg min ( (V μ w) + (2n) w1.3].
By optimality condition, we have (w) = w - 1j ( which can be specified by 4. The remaining part is Bregman-Moreat envelope. Thus, we optimize the upper bound with an EM-MAP method, alternately computing pa (w) and Dprox A (V,, ((o())
Framework Design
Problem Formulation that Highlights Personalized Prior Inspired by the aforementioned motivation, the personalized models O; and mean parameters are respectively the solution of g (((x(w) and ps(w) on the 7th client, where is the global model. We assume that personalized model contains all the local information required for inference on the 7th client, and satisfies 0 [d, w ~ P sef (0;; X, s;(w), g). The global problem can be written as Eq. (12).
min E, (w) == Denv A ff( p.(w) }}
(12)
w
The given g is strictly convex, À > 0, fa is the local loss function, s:(w) is the natural parameter and (www)= Eo = [w)s is the mean (or expectation) parameter in Eq. (12).
Framework: pFedBreD To solve the optimization problem in Eq. (12), we use gradient-bart methods to solve the global problem using the gradient of Fj:
VF(w)= = (w), (H)(w)) - Dprox, A f((((w)))
(13)
where D is the gradient operator of the vector value function, and 2 is the Hessian operator. The framework is shown as Algorithm 1, where I is the client selecting strategy for global model aggregation; Winst and Oinit are the initialization strategies the 7th client; a'm is the main problem step-size; T, R, N are respectively the total number iterations, local iterations, and clients. 3 is used in the same trick [37, 65]. The strategies derive the initialization points of uy and f at each local epoch are w0. < (t-1) and 0/0 + (!t-1)
R
Implementation: Maximum Entropy and Meta-Step Practically, two main parts of the pFedBreD are needed to be implemented:
g, the function used to derive the logarithmic normalization factor, determines the type of prior to be used;
{s.} or {uz). the functions used to derive the natural parameter and mean parameter for the personalized local prior, determine which particular prior is used.
The details of first-order methods is in Appendix A.7.
5

---

Algorithm Algorithm for pFedBreD
Input: i. I..NV
Parameter: Im....A.T.RR RR, Winst. Dis = 1...N
Output: (.( J.
I...N
1: Initialize (0 (
J. (Cr,nI:
2: for t=
3: Server sends w(t-1) to clients
4: i in parallel on each clients do
5: Initialize w and 8 with Winit and Amati
6:
for rl...R do
7:
Generate Mir f 1=(w,)--)...)
( 1r t- Dprox 1 (1))
8:
w f- (1) - am F. FF( 1-r-1
9:
end for 11: end for
12: Server collects {w(h) and calculate w(t) t- (1 - 3)wlt-1) + B i.17
13: end for return (
14:
We propose first-order implementations based on maximum entropy rule [22, 33]. In the SX-family, the Gaussian distribution has the maximum entropy among continuous distributions when g. μ4 (the first-order moment), and À (the parameter determining the second moment) are given. Thus, employ the scaled norm square g = 9" = alll 1|a to turn the prior into spherical Gaussian, in particular client. With this assumed prior, have Vg = Vgg = I, which means μ4 = $4. We can choose a different 4 as shown in Eq. (14)* to generate selection strategies according to Section 4, via ((w)) w vV),(w) (meta-step).
order to maximize the entropy of the prior
fr
w i-r-1 na ((((W
lg
4.
P fi Fi
(2)
w w 7( (
(1)
meg (14) 000 10 mh
W na V((WW())) nw (t-1)
(1)
where aa and 1) are the meta-step-size parameters. Practical parameter selection strategies with (t)
meta-step are shown as ui in Eq. (14). The three of μ4, i.e. lg. meg and mh, represent loss gradient, memorized envelope gradient and memorized hybrid respectively.
we analyze the convergence of pFedBreD with RMD on uniform client sampling E, N Em= setting for simplification. Other sampling methods can be obtained with client sampling expectation Eg [F.] F, by changing sampling weights. The assumptions, proof sketch and detailed notations in Appendix A.1 and Appendix D.
Convergence Analysis
Theorem 1 (pFedBreD's s global bound). Under settings in Section and Appendix D, at global epock
/28" be 2_ (,m)., where A Lg (ūm
T
Br by properly choose ām aR 3ām I minf
7))) 37 |d.1
+ +2)), B (L (1+ n/ C
L2 (6. +n
g(t) = &ûF ---1
(TT) = TT (()) and âm = e(1+a))
A BR
such that:
Eie g(t)
1800
AA +B
OlD ,,=)]] =0 amar. T/2 ammii)-m (0)
+ +
AF.
(N/S - 1)0F, 2
2
+ 00
+ O
Ro F. + AX + B)).
NTØF
T2 R2
'A variant of mh is in Appendix C.2.

---

Theorem 2 (pFedBreDns' 's first-order personalization bound), Under the same conditions as in Theorem with prior assumption of a spherical Gaussian and first-onder approximation, the bound for the gap between the personalized approximate model and global model in the Euclidean space is:
(l(0,(w") =²1 2 ((6p) + +O(,Dr((T()) ,wn)]
where 8,
37 [d
22 35 ,;, + 2 and Ep ( LF
pr.
Remark 1. Theorem shows the main factors that affect the convergence of a global model are as follows: random mini-batch size, client drift error, aggregation error, heterogeneous data, dual space selection, local approximation error, and selection strategy for exponential family prior mean and variance. These can be divided into four categories based on their computational complexity. The first and second term shows that the proper fixed öm can linearly reduce the inftuence of initial error A (0) and the global model converges to ball near the optimal point. The radius of this ball is determined by the personalized strategy and local errors (including local data randomness and envelope approximation errors). The third term implies that a linear convergence rate ((I)(NT) can be obtained w.r.l. the total global epoch NT in the presence of aggregation noise. Without client sampling. N = S, according to the fourth term, the quadratic rate ( R)²) can be obtained with 3 = O(N) or 3 = (NNR)) (Note that the number of local epoch R cannot be too large due to client drift, according 2R). Theorem 2 shows that, with spherical Gaussian prior assumption and first-order methods, the radius of the neighborhood range for the minimum that includes the personalized model 8th client, OCGo + ( + oz. BLF)) + X2 Pr.. by X, and is affected by the prior selection strategies and first-order approximate error besides the elements in Theorem I. (Note that the Euclidean spuce is self-dual.)
x2,A ) can be trade-off
6 Experiments
6.1
General Settings
Tricks, Datasets and Models: our experiments include several tasks: CNN [28] on CIFAR-10 [18,
391, LSTM [27] on Sent140 [11] and MCLR/DNN on FEMNIST [1 [1]//MNIS [65, 67//MNIST665,
42]. The details of tricks (FT, AM), data heterogeneity and models are in Appendix C.
Baselines: we choose following algorithms as our baselines: FedAvg [53], Per-FedAvg [20],
pFedMe [65], FedAMP [31], pFedBayes [77] and FedEM [52]. These baselines are respectively classical FL. MAML-based meta learning, regularization based, FTML methods, variational inference
PFL and FMTL with EM.
Global Test and Local Test: the global and personalized model, represented by G and P. are evaluated with global and local tests respectively. Global test means all the test data is used in the test. Local test means only the local data is used for the local test and the weight of the sum in local test is the ratio of the number of data. The results of average accuracy per client are shown in Table 1. Each experiment is repeated times. More details are in Appendix C. For readability, we only give the error bar in the main Table and Table 2, and keep one decimal except for the main Table 1.
Hyperparameter Settings The step-size of the main problem, amn and the personalized step-size, a, for all methods are 0.01. ,3 is 1, and the number of local epochs, R, is 20 for all datasets. X is chosen from 15.0 60.0. The batch sizes of 4ent140 and the other datasets are 400 and 20, respectively, and the aggregation strategy, I, is uniform sampling. The ratios of aggregated clients per global epoch are 40% 10%, and 20% for Sent140, 40, FEMNIST, and the other datasets, respectively. The numbers of total clients, N, are 10, 198, 20, and 100 for Sent14A 40, FEMNIST, CIFAR-10, 10, and other datasets. The number of proximal iterations for all settings with proximal mapping. In our implementations, To and 77 are respectively 0.01 and 0.05.
Summarizing the Effects of Hyper-parametery our implementation DFedBreD mm.h The details are in Appendix C. From the results, we find that it will degrade the test accuracy the values of or y) are too large or too small. The test accuracy of personalized model is more sensitive than the ones of global model. The test accuracy of personalized
We test the hyper-parameter effect of 7) and À in
7

---

Table I: Results of average testing accuracy (%) per client of each settings. We mark the best and second best performance by bold and underline. Avg and Std the average results and the standard deviation of them on all tasks; H.Avg and H.Std the average results and the standard deviation of them on hard tasks (non-linear DNN with complex classification or architecture: DNN CNN/ LSTM on FEMNIST / CIFAR-10 / 44)t The G and P are global and personalized model
Methods Duluses
PEMNIST
FSINITT
MINIST
CIEAK TO
Stanstics
Names Models
MCLR
DNN
MCLR
DNN
MCLR
DNN
CNN
LSTM
206
Std
ILAwg
11.Std
Fealvgg [ST] -11--
31.3822:
$6.59
38 88.20
57.31amm
70.56-0
7200
14:34
61:20
7:35
FodAvgrAM G
55. 55.34
19,03.110
32.58,0:15
81.033..:1
56.75 16.74_.01
89.33 40.3
57.0744.1
71.27-0m
72.80
14.01
62.46
7.70
FedFM (ST]- -6
20.751.32
45.47.044
05-78.0.2
96-42us.3
85.75_..
86 16.49
57.0718.06
66.73_.000
7138
32.75
56.62
10.06
Weddioes
09.66.5.55
98. 98.67.3u5
k9.64_..e
10.48.032
FedAMP [11] P
60.05:1.s
66.79.114
98.63.0.3
98. 98.72samm
90.81
92.01amme
75.00
69,85 .e
81.80
15.21
71.34
5.46
50.74.0.1
53.56u0.1
60
u8.63
72:24
69..66....
18.56
65.05
10.116
18.01
62.9330.00
59.36
91.71amu
68.17.0.11
67.82_.910
79.13
16.53
66.31
BFMMMrAAM P
15.56
60,05
97.57.m22
98.67.9.00
99,46
91.22
73.35
79.48
16.79
68.12
7.07
Ro-FeulAvg [20]-P
28
15.74
70.54
PosRolAvg--FT
10
79.95
15.61
PusFolAvggAM
56.6622mm
92.08an
24
56.91 mM
80.02
14.54
71.81
24.9
08.48.0
in
22.08
72.04_..m
7.25
(oumseFT-P
92.9610.1
_m
louns>AM
08.4
70.34.0m
98.48
15.75
92.32
80.6
73.68.91 .m
model is more sensitive to v than to A. Note that the hyper-parameters are roughly tuned, which shows the insensitivity of mh, and better tuning could improve the performance in the Table 1.
6.2 Analysis
Comparative Analysis of Performance We compare our methods and the baselines from dif- ferent perspectives, including convex or non-convex problems, easy or hard tasks, and text tasks. Additionally, we briefly discuss the absence of BNN on hard tasks.
Convex or non non-convex: on non-convex problems, especially in hard tasks, our method significantly outperforms other methods by at least 3.06% employing some simple tricks. On convex problem, FedAMP outperforms our method somewhat on convex problems with simple data sets. One explanation is that the learning lanscape is simple shape for these problems and FedAMP converges faster for this case. One possible reason for this is that since FedAMP uses the distance between models as a similarity in the penalty point selection, giving greater weight to the model that is most similar to the local one. In the later stages of training, since there is only one global optimum, this penalty point tends not to change, and thus the method degenerates into a non-dynamic regular term. Compounding intuition, this method will not be as advantageous for non-convex problems and harder convex problems, as penalty point tends fall into the local optimum and lead to degradation of the dynamic regular term.
From easy to difficult task: from the difference between the statistics of Avg and H.Avg in Table 1, it can be observed that meta-step methods perform most consistently, with all other methods dropping at least 10%. This is due to the simple and effective local loss design of MAML, with its learning- to- tealennn design philosophy that enables the method to be more stable in complex situations [19, 20].
Personalized prior on text: text tasks, as opposed to image tasks, generally have relatively rugged learning landscape. [55, 16, 13] This understanding manifested in specific ways, such as parameter sensitivity, slow convergence, and struggling during the process. Thus, the overlooked prior infor- mation seems to be more important, which means that each local iteration not only obtains local knowledge from the data, but also the prior itself already contains some local knowledge. Therefore, there is no need to re-obtain this knowledge from scratch solely from the data during training.
Absence of BNN on hard tasks: complex BNN is not in Table 1, such as LSTM in PFedBayes, because it is difficult to conduct comparative experiments by fixing elements, e.g., inferential models, tricks and optimization methods. In pFedBayes, training often crashes on hard tasks and large datasets, as mentioned in [77]. Our one-step-further research shows that it may be caused by the reparameterization tricks and vanilla Gaussian sampling. If we add tricks on it, the implementation will be very different from the original pFedBayes, and it is beyond this analysis.
Ablation Analysis of Personalized Prior
We conduct ablation experiments by dropping the gradient of the Bregman-Morcat envelope, the local loss function, or both, from the personalized
8

---

Table 2: Average local test accuracy of personalized model (%) in ablation experiments average aceuracy increased/reduced;; AC4PP: Additional cost for personalized prior: Grad. and Add.: cost about calculate gradient and addition; Other notations are the same in Table 1.)
Methods
FEMNIST
FMNIST
MNIST
CI
Sent140
Statistics
AC4PP
MCLR
MCLR
MCLR
LSTM
Aogg
H.Avg
Non-FF
50.720.00
$3.600.12
97.0e0.05
98.0g0.01
88, 2e0.00
50.5
72.220.06
09.4m0002
77.6
65.1
None
1g tours)
50.8
49. 20.3s
28.32
98. 28.4
28.4
91.0g000
65.720.40
60.7gm.:
75.3)
58.5
Grad. x
meg (ouns)
53.9 20.00
27,8
20.00
88. 2am
90.6
23.8.0.00
77.91
Add.
mh (ours)
56.3 0.c0
64.9 10.00
98.4 jo.0
98.7
89.8 Jame
92.0
0.0
79.4 10.00
72.0 b.c:
81.51
72.1
Both above
1.00
1.0
GCB
0.75
PeddMe FedAAg mh (ours)
GCE
0.50
0.5
mh (ours)
FedAvg
0.25
Peddlee
500
1000
1500
0
250
500
750
1000
Epoch Index
Epoch Index
100 with CNN
MNIST with DNN
Figure 2: The results of GCE((VF((()))) at each global epoch t after Savitzky- Golayy filtering [62].
strategy mh as shown in Table 2. The relationship among the three strategies mentioned in Eq. (14) is that mh consists of Ig and meg. Moreover, pFedMe can be regarded in our framework as the one which takes the spherical Gaussian as prior and uses vanilla prior selection strategy μ4, = I without personalization. Thus, pFedMe and the three implementations of pFedBreD are compared. The results reveal the instability of our implementation Ig and the introduction of meg on difficult tasks is about the same as not introducing it. However, introducing both Ig and meg (i.e., mh) together shows remarkable performance. This indicates that ig and meg complement each other. To explain these results, by observing the error bars, in most of the settings, meg is significantly more stable compared to methods that do not use personalized priors, while Ig is relatively less stable. Based on this observation, we have reason believe that meg weakens the influence of potential noise, while Ig introduces new noise. Therefore, we can infer that while the mean parameters are steadily biased towards the personalized model, the introduction of new noise finds path that is more likely to escape from local optima or saddle points, based on implicit regularization [61, 58, 57].
Generalized Coherence Analysis of Information Injection and Extraction The generalized coherence estimate (GCE) [25] of vectors from personalized to local model (i.e., the envelope gradients in PFedMe and ours) among clients on each global epoch are shown in Figure 2. The smaller the GCE, the less coherent the envelope gradient between individual nodes and the greater the diversity of information in the global model update. As shown in Figure 2, we can observe that during the convergence phase, using a personalized prior method has significantly greater information diversity than not using a personalized prior method, which proves the success of injecting personalized prior knowledge into the global model and extracting local knowledge from the local training.
Variable-Contre Analysis of Robustness We analyze the impact of aggregation noise and data heterogeneity [29]
method, mainly mh, by controlling variables. Results are in Table and Table 4. (Details are in Appendix C.8.) We test the performance of global model on different aggregation ratios, where all paper-parameters except for the aggregation ratios are fixed. Meanwhile, we test the performance of both global and the personalized model on different data heterogeneity settings, where full aggregation (sample client equals total number of clients, S = N) and one-step local update (local epoch R = 1) are employed to get rid of the effects of aggregation noise and client drift. The experiments demonstrate the instability of the global model in mh at small aggregation ratios, which most of the other PFL methods have, by comparing their performances on different aggregation numbers. Comparing the baselines, the experiments also demonstrate the relative robustness of our method to extreme data heterogeneity.

---

Table 3:
The global test accuracy Table 4: The local test accuracy (%) of the per- (%) of the global model with differ- sonalized model on FMNIST-DNN setting with dif- ent numbers of clients for aggregation ferent data heterogeneity (Non-IID) settings
E {10,20, 50, 100). FEMNIST. {0.01, 0.1, 1, 10, 100, (000) Non-IIDf) [29]. The
(:FMNIST)
Bolded means the best.
Numbers
small
large
Non-IID
small
3
large
Avg
DNN
59.0
60.1
60.1
59.8
0.5
FedAvg-G
18.2
14.8
14.5
11.9
11.3
11.2
13.7
MCLR
54.4
55.4
55.4
55.5
0.5
PFedMc-P
89.5
58.2
24.2
12.3
11.8
10.6
34.4
ODNN
75.1
79.6
79.4
79.3
2.2
PFedMe-G
17.0
14.3
14.1
12.3
10.8
10.9
13.2
-MCLR
80.0
81.8
82.7
1.3
mh(ours)-P
89.6
58.7
25.2
13.1
11.1
11.0
34,8
mb(ours)-G
17.1
14.6
14.6
12.4
11.9
13.8
Jiih Qyeand
+-
Futlediig ab pwrn
+
020
a33000
+ PeiiAAE pho00
calter cap Ian
hihhedeg *
asg
(plen))
Drestattionn
Dreeiittion
Law Pae Mattee
-085
-0a5
Class Indes
Class Indes
ClassIndes
Class Index
MNIST-MCLR-G
MNIST-MCLR-L
MNIST-DNN-G
MNIST-DNN-L
Figure 3: The loss deviation of experiments in Section 6 on the first client, whose major data 9th classes. The lower deviation of the available class on global tests and the higher deviation of the unavailable class local tests demonstrate the superior personalization ability of our methods.
Deviation Analysis of Personalization Deviation represents the difference between an individual and the mean value. We use the deviation of the loss function to reflect the personalization. On global test, the lower the deviation, the better the personalized model performance on the corresponding local data. On local test, the model is only tested on its own dataset, and because of multiple local iterations, the local test deviation converges to almost the same value, as shown in MNIST-MCLR-L and MNIST-DNN -L in Figure 3. Furthermore, since the local test has a loss of 0 on missing classes, a higher deviation on missing classes reflects a lower mean on these classes. Thus, the lower loss in local testing and better performance testing and the higher deviation on missing class. Summary: based on Figure 3, we can see that our method has higher deviation on missing classes in local testing and lower deviation in global testing. This means that our method has better personalized performance.
reflected from both of the almost equal deviation in local
7 Conclusion and Discussion
Conclusion To address the issue neglecting client-sampling information while providing prior knowledge to local training via direct use of global model, we propose general concept: the personalized prior. this paper, we propose a general framework, pFedBreD, for exploring PFL strategies under the SX-family prior assumption and computation, the RMD to explicitly extract the prior information, and three optional meta-step strategies to personalize the prior. We analyze our proposal both theoretically and empirically. Our strategy mh shows remarkable improvement in personalization and robustness to data heterogeneity on non-i.i.d. datasets and the LEAF benchmark [11] with MCLR / DNN CNN LLTM as inferential model, which conduct convex / non-convex problems, and image language benchmarks.
Linitations and Future Work Although mh shows remarkable performance and robustness, there still instability in the global model with aggregation noise. Furthermore, should be noted that the superficial reason for the improvement of mh seems be that 7/ and 17 and (which are similar to each other) are used simultaneously. resulting in a magnitude in mh that twice as large as the ones in the other implementations and leading better performance. However, empirically, simply doubling 7), in Ig or in meg does not improve performance, and using one more meg step used in ig significant improvement. Our theoretical analysis cannot explain this phenomenon, and more detailed modeling is needed.
10

---

Acknowledge
This work is supported by the Key Program of National Science Foundation of China (Grant No. 61836006) from College of Computer Science (Sichuan University) and Engineering Research Center of Machine Learning and Industry Intelligence (Ministry of Education), Chengdu 610065, R. China. This research is also supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG2-PHD-2021-00
References
[1] Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Proceedings of the NeurIPS, 14:8392-8406 2021.
[2] Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael Jordan. An introduction to meme for machine learning. MACH LEARN, 50(1):5-43, 2003.
[3] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv:1912.00813. 2019.
[4] Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the ICALT, volume 117 of PMLR, pages 111-132, 08 Feb-11 Feb 2020.
[5] Ulrich Matchi Aīvodji, Sébastien Gambs, and Alexandre Martin. Iotfla A secured and privacy- preserving smart home architecture implementing federated learning. In 2019 IEEE SPW, pages 175-180, 2019.
[6] Arindam Banerjee, Srujana Merugu, Inderjit Dhillon, Joydeep Ghosh, and John Lafferty. Clustering with bregman divergences. JMLR, 6(10), 2005.
[7] Heinz Bauschke, Patrick Combettes, and Dominikus Noll. Joint minimization with alternating bregman proximity operators. PAC J OPTIM, 2006.
[8] Heinz H Bauschke, Jonathan M Borwein, and Patrick L Combettes. Bregman monotone optimization algorithms. SIAM J CONTROL OPTIM, 42(2))596-6366 2003.
[9] Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28. Princeton university press, 2009.
[10] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application the solution of problems convex programming. COMP MATH MATH PHYS+, 7(3):200-211 1967.
[11] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv:1812.01009 2018
[12] Gary Cheng, Karan Chadha, and John Duchi. Federated asymptoties: model to compare federated learning algorithms. arXiv:2108.077 07311, 2022.
[13] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural Language Processing (Almost) from Scratch. NLP, 2011.
[14] Katalin Csilléry, Michael G.B. Blum, Oscar Gaggiotti. and Olivier François. Approximate bayesian computation (abc) in practice. TRENDS ECOL EVOL, 25(7):410-418, 2010.
[15] Arthur Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. J R STAT SOC B, 39(1):1-22, 1977.
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In the Proceedings of the CVPR, pages 248-255, 2009.

---

[17] Aymeric Dieuleveut, Gersende Fort, Eric Moulines, and Geneviève Robin. rederated-em with heterogeneity mitigation and variance reduction. Proceedings of the NeurIPS, 34:29553-29566. 2021.
[18] Canh T Dinh, Tung Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. Fedu: unified framework for federated multi-task learning with laplacian regularization. arXiv:2
2021.
[19] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-lcarning algorithms. In Proceedings of the AISTATS, pages 1082- 1092. PMLR, 2020.
[20] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Proceedings of the NeurlPS, 33:3557-3568, 2020.
[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings the 2640-3498.
the ICML, pages 1126-1135, July 2017. ISSN:
[22] Kenneth Friedman and Abner Shimony. Jaynes's maximum entropy prescription and probability theory. J STAT PHYS, 3(4):381-384 1971.
[23] Andrew Gelman and Cosma Rohilla Shalizi. Philosophy and the practice of bayesian statistics. BRIT MATH STAT PSY, 66(1):8-38, 2013.
[24] Christian Genest and James V Zidek. Combining probability distributions: A critique and an annotated bibliography. STAT SCI, 1(1):114-135, 1986.
[25] H. Gish and D. Cochran. Generalized coherence (signal detection). In ICASSP-88, pages 2745-20 2774 vol.5, 1988.
[26] Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General Analysis and Improved Rates. In Proceedings of the ICML, pages 5200-5209, May 2019. ISSN: 2640-3498.
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. NEURAL COMPUT, 9(3):1335-1800 1997.
[28] Jan Hosang, Mohamed Omran, Rodrigo Benenson, and Bernt Schiele. Taking deeper look at pedestrians. In Proceedings of the CVPR, pages 4073-4082, 2015.
[29] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the Effects of Non-ldentical Data Distribution for Federated Visual Classification. arXiv: 2019.
[30] Chenxi Huang, Liang Xie. Yibo Yang, Wenxiao Wang, Binbin Lin, and Deng Cai. Neural collapse inspired federated learning with non-iid data. arXiv:22303.16066 2023.
[31] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang. Jiangehuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. AAAI, 35:7865-7873, 2021.
[32] Ahmed Imteaj, Urmish Thakker, Shiqiang Wang, Jian Li, and M. Hadi Amini. survey federated learning for resource-constraine iot devices. IEEE INTERNET THINGS, 911: 2022.
[33] Edwin T Jaynes. Information theory and statistical mechanics. PHYS REV, 106(4):620, 1957.
[34] Chu Jiang, Burak Kantarci, Sema Oktug, and Tolga Soyata. Federated learning in smart city sensing: Challenges and opportunities. Sensors, 20(21), 2020.
[35] Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. 1.XXi:1909.12480 2019.
12

---

[36] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji. Kallista Bonawitz, Zachary Charles, Graham Cornode. Rachel Cummings. et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1 2021.
[37] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the ICML, pages 5132-5143, PMLR, 2020.
[38] Robert E Kass and Larry Wasserman. The selection of prior distributions by formal rules. J JAM STAT ASSOC, 91 (435):1343-1377), 1996.
[39] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[40] Frederik Kunstner, Raunak Kumar, and Mark Schmidt. Homeomorphic-ing invariinne of em: Non-asymptotic convergence kl divergence for exponential families via mirror descent. In Proceedings of the AISTATS, pages 3295-3303. PMLR, 2021.
[41] L. Le Cam, Maximum Likelihood: An Introduction. INT STAT REV, 58(2)153-177,, 1990. Wiley, ISI.
[42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2332 1998.
[43] Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE T KNOWL DATA EN, 35(4):3347-3366 2023.
[44] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Proceedings of the ICML, pages 6357-6368. PMLR, 2021.
[45] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated Optimization in Heterogeneous Networks. Proceedings of MLSys, 2:429-450, March 2020.
[46] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the Convergence of FedAvg Non-IID Data. In Proceedings of the ICLR, April 2020.
[47] Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. arXiv:2303. 10X00:2303.1008, 2023,
[48] Liangxi Liu, Feng Zheng, Hong Chen, Guo-Jun Qi, Heng Huang, and Ling Shao. A bayesian federated learning framework with online laplace approximation. 2 2021.
[49] Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated Learning for Open Banking. In Qiang Yang, Lixin Fan, and Han Yu, editors, Federated Learning: Privacy and Incentive, pages 240-254. Springer International Publishing, Cham, 2020.
[50] Christos Louizos, Matthias Reisser, Joseph Soriaga, and Max Welling. An expectation- maximization perspective on federated learning. 1. 10192, 2021.
[51] Andrew I. Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the ICMI., volume 30, page 3. Citeseer, 2013.
[52] Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Lactitia Kameni, and Richard Vidal. Federated Multi-Task Learning under a Mixture of Distributions. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Proceedings of the NeurIPS. volume 34, pages 15444,1 2021.
[53] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eefficen learning of deep networks from decentralized data, In Proceedings of the AISTATS, pages 1273-1282. PMLR, 2017.
13

---

[54] H Brendan McMahan. A survey of algorithms and analysis for adaptive online learning. The JMLR, 18(1:3111-3166, 2017.
[55] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Represen- tations of Words and Phrases and their Compositionality. In C. J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Proceedings of the NeurlPS, volume 26, 2013.
[56] Thomas P. Minka. arXiv: 2001.
Expectation propagation for approximate bayesian inference.
[57] Behnam Neyshabur. Implicit regularization in deep learning. arXiv: 2017.
[58] Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of optimization and implicit regularization deep learning. arXiv:1705.0300 2017.
[59] Dinh C. Nguyen, Ming Ding, Pubudu N. Pathirana, Aruna Seneviratne, Jun Li, and H. Vin- cent Poor. Federated learning for internet of things: A comprehensive survey, IEEE COMMUN 2021.
SURV TUT,
[60] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. 180.002900002999 2018.
[61] Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization tensor factorization. In Marina Meila and Tong Zhang, editors, Proceedings of the ICML, volume 139 of PMLR, pages 9013-89244 18-24 Jul 2021.
[62] Abraham. Savitzky and M. J. E. Golay. Smoothing and Differentiation of Data by Simplified Least Squares Procedures. ANAL CHEM, 36(8):1627-16333 July 1964. American Chemical Society.
[63] Teddy Seidenfeld. rosenkrantz. THEOR DECIS, (144):413444) 1979.
Why i am not objective bayesian; some reflections prompted by
[64] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In Proceedings of the ICML, pages 9489-9502. PMLR, 2021.
[65] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Proceedings of the NeurIPS, 33:21394-21405 2020.
[66] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. TNNLS, 2022.
[67] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. wVXiv:1708.07747,00 2017.
[68] Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In Proceedings of the ICLR, 2023.
[69] Jie Xu, Benjamin Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated Learning for Healthcare Informaties. JHIR, 5(1):1-19, March 2021.
[70] Beining Yang. Kai Wang, Qingyun Sun, Cheng Ji, Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? In Proceedings of the NeurIPS, 2023.
[71] Liu Yang, Ben Tan, Vincent W. Zheng, Kai Chen, and Qiang Yang. Federated Recommendation Systems. In Qiang Yang, Lixin Fan, and Han Yu, editors, Federated Learning: Privacy and Incentive, pages 225-239. Springer International Publishing, Cham, 2020.
[72] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. TIST, 10(2) jan 2019.
[73] Lei Zhang, Guanyu Gao, and Huaizheng Zhang. Spatial-temporal federated learning for lifelong person re-identification distributed edges. arXiv:2207.117500 2022.
14

---

[74] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. Proceedings of the CVPR, pages 10174-10183, June 2022.
[75] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. In Proceedings of the ICLR, 2020.
[76] Wei Zhang, Xiang Li, Hui Ma, Zhong Luo, and Xu Li. Federated learning for machinery fault diagnosis with dynamic validation and self-supervision. KBS, 213:106679, 2021.
[77] Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference. In Proceedings of the ICML, pages 26293-26310. PMLR, 2022.
[78] Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. arXiv:22/1.13977 2022.
[79] Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng. Efficient meta learning via minibatch proximal update. Proceedings of the NeurIPS, 32, 2019.
[80] Yuhao Zhou, Mingjia Shi, Yuanxi Li, Yanan Sun, Qing Ye, and Jiancheng Lv. Communication- efficient federated learning with single-step synthetic features compressor for faster convergence. In Proceedings of the ICCV, pages 5031-5040, 2023.
15


Reviews:
[{"review_id": "uuT3KKlAUX", "reviewer": "Reviewer_Q28a", "summary": "The paper introduces a novel scheme, pFedBreD, which addresses the incomplete information challenge in personalized federated learning by incorporating personalized prior knowledge into the global model of each client. This is achieved by decoupling the personalized prior from the local objective function and applying Bregman divergence regularization. The proposed approach leverages the Expectation-Maximization (EM) algorithm to approximate complete information from both global and local clients. Extensive empirical evaluations on multiple benchmarks demonstrate the effectiveness of the proposed method, and a theoretical analysis is provided to support its theoretical foundations.", "questions": "1. Is the method of calculating GCE mentioned in the paper? Does it depend on model parameters or extracted feature maps?\n2. Some abbreviation is not clearly explained in the paper. For example, what is MCLR, FT, AM, etc.?\n3. How does the computation cost of the proposed method compare to other approaches? Is the local training phase highly time-consuming? For example, what computational resources were utilized in the experiment, and what was the duration required to complete the experiment?\n4. Does the scalability of the proposed method remain unaffected? For instance, can the proposed method handle a scenario with more than 100 clients?", "limitations": "The authors have adequately addressed the limitations in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to read.\n2. This paper introduces a novel approach that incorporates Bregman divergence and leverages the Expectation-Maximization (EM) algorithm to approximate complete information from both global and local clients. It effectively enhance personalized federated learning.\n3. The effectiveness of the proposed method is validated through extensive empirical evaluations and thorough theoretical analysis in the paper.\n4. This work showcases superior performance across multiple benchmarks when compared to various baseline methods.", "weaknesses": "1. The authors discuss the concepts of prior knowledge injection and extraction in Figure 1, as well as analyze the process of information injection and extraction. To enhance clarity, it would be helpful if the authors explicitly indicate in the framework design section which step corresponds to these specific actions.\n2. Certain aspects of the paper would benefit from additional clarifications to enhance understanding. See below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Comment***:\n- We sincerely thank the reviewer Q28a for the appreciation and constructive comments to further improve clarity and readability, which are detailed in [general responses](https://openreview.net/forum?id=kuxu4lCRr5&noteId=WzbbMONDyW)[W1]. The concern raised by the reviewer, i.e., the further details of our experiments, most of which are in the Appendix, and we answer the reviewer's questions one by one in the hope that our response can address the concern and any questions asked.\n\n***Responses***:\n\n1. [W1] Modifications for improving readability.\n Thanks for the suggestion, and we will make the followings based on the suggestion. Specifically, the motivation and framework illustration, Fig. 1, will be more visually linked to the relevant parts of the math. methodology and computation framework. We will put the algorithm 1 in Appendix into the main body and highlight the relevant parts in the algorithm box with statements.\n\n2. [Q1, W2] How do we calculate the Generalized Coherence Estimate (GCE)?\n- **What is GCE?** GCE is a classic method [e, 24] in the field of signal processing used to measure the coherence among a batch of sources. This method is mentioned by citing reference [e, 24] in the paper. Formally, $GCE(X) := 1 - \\text{det}(\\hat{G}(X))$, where $\\text{det}(\\hat{G}(X))$ is the determinant of the Gram Matrix (Gramian) of $X$. The Gram Matrix consists of the cosine values of each pair of sources in the set $X$. *More details will be included* in the next version of the paper.\n- **How to calculate GCE in this paper**: The GCE in Fig. 2 is $GCE(\\{\\nabla F_{i}(w_{i})\\}_{i})$, as shown in the title of Fig. 2. The GCE analysis is on the first-order directions among each client, which are the gradients of the local objective function. The directions determine the update direction of the global model through aggregation, so our analysis depends on these directions.\n\n3. [Q3, W2] What about computation cost?\n- Computation cost on both of the local training and entire algorithm *are not highly time-consuming*, see details as follows:\n  - **Entire Alg. Complexity**: the complexity comparison among our baselines in Line 572-578 in Appendix is detailed in Table below, where $N$,$T$,$R$,$K$ and $M$ are respectively the number of clients, global epochs, local epochs, proximal solver iterations and components on each client. Our method *share the same complexity* with pFedMe and FedAMP.\n  - **Local Computation Complexity**: local computation cost is independent of the number of clients $N$. As shown in Table below and the last column of Table 2 in Line 269-271 of the paper, the local computation *only adds constant time* about $N$. Comparing with pFedMe and FedAMP, additional local computation is at most *one extra local gradient and addition computation* for our MAML-based personalized prior strategies. In practice, the local proximal solver iterations *$K$ is not too large*, e.g., $K=5$ in our settings.\n- Main details about Computation Environment for our computation and test:\n  - GPU: TitanX with 1417 Hz frequency, 12288MB mem. 480GB/s of bandwidth and 11 TFLOPs;\n  - System: Ubuntu 20.04.1;\n  - Torch: 1.11.0+cu102.\n\n [It can be seen that our method improves significantly without great costs. For ease-to-read, we sort by test performance.]\n |Complexity/Methods|FedEM|FedAvg|pFedMe|Per-FedAvg|FedAMP|pFedBreD (ours)|\n |-|-|-|-|-|-|-|\n |Entire Sys. Memory|$O(NM)$|$O(N)$|$O(N)$|$O(N)$|$O(N)$|$O(N)$|\n |Entire Sys. Time|$O(NTRM)$|$O(NTR)$|$O(NTRK)$|$O(NTR)$|$O(NTRK)$|$O(NTRK)$|\n |Local Computation Comparing to classic FedAvg|Each Local Loss and Weighted Sum|-|Addition|Gradient|Weighted Sum and L2-Distance|Gradient and Addition|\n|Time Cost of Each Global Epoch with $R=20$ on FMNIST-DNN with TITANX (sec.)|2.7|1.9|3.1|3.3|3.1|3.6|\n |Average Acc. on All Tasks|71.88|72.80|79.48|80.02|81.80|83.09|\n\n4. [Q4, W2] Does the scalability of our method remain unaffected, e.g., the case with more than 100 clients?\n- **In our Experiments**: the numbers of clients in the settings of our experiments *are above 100, as shown in Line 228*. The actual time comparison is shown in the Table above in Response 3 [Q3, W2], and the scalability about *algorithm complexity* of the proposed method *is affected linearly* by the number of clients $N$ (linear complexity, an acceptable system expansion cost.).\n- **Theoretically**: according to Theorem 1 in Line 711 in Appendix, we have the linear convergence rate $O(\\frac{\\hat{S}^{-1}-1}{NT})$, if the client sampling ratio $\\hat{S}\\le 1$ is fixed, and the convergence rate will *still increase linearly* by $N$.\n- **Conclusion**: the scalability about *both algorithm complexity and convergence* of the proposed method is affected linearly (acceptable linear complexity) by $N$.\n\n***Minors***:\n1. [Q2] Some abbreviation is not clearly explained in the paper. For example, what is MCLR, FT, AM, etc.?\n- Thanks for pointing out. They are in the Appendix C1 and C4, and for readability, we will put them in the main body of the revision as suggested.\n\n***References***:\n\n[e] An Invariance Property of the Generalized Coherence Estimate. TSP 45.4 (1997): 1065-1067."}, {"review_id": "OPW1ptTOA5", "reviewer": "Reviewer_HD18", "summary": "In this paper, the authors propose pFedBreD to decouple prior knowledge from each client. pFedBreD extracts the personalized prior with Bregman Divergence for better performing personalized tasks. The authors provide convergence analysis and experiments evaluated on 5 datasets.", "questions": "1.    The intuition of the work is not clear. Could the authors explain the motivation for using Bregman Divergence and Relaxing Mirror Descent in detail?   \n2.    In Table 1, why the second-best performance is 79.44 (mh) rather than 79.68 (Per-FedAvg+FT)?", "limitations": "The authors have discussed the potential limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.    The authors give a detailed convergence analysis for the proposed method.  \n2.    The theoretical derivation of the paper is comprehensive in the Appendix.  \n3.    The authors provided extensive experiments for different tasks and models and the results show the effectiveness of improving the local models' performance.  \n", "weaknesses": "1.    The motivation of the work is not clearly presented. It is hard to understand why the authors use Bregman Divergence and Relaxing Mirror Descent.     \n2.    It is confusing that the authors use the average results and the standard deviation of them on all tasks to represent the overall performance of methods. Could the authors give some intuition or explain the purpose of this?   \n3.    More related works need to be introduced. And we'd like to see experimental results compared to more related works, such as [1], [2], [3], and [4].   \n4.    This paper focuses on the specific label distribution problem. It would be better if the authors could discuss the performance of other heterogeneous data distribution problems (e.g., different number of classes is different).   \n\n[1] Personalized Federated Learning with Feature Alignment and Classifier Collaboration. ICLR 2023.   \n[2] Ditto: Fair and robust federated learning through personalization. ICML 2021.   \n[3] Personalized Federated Learning with First Order Model Optimization. ICLR 2021.   \n[4] Personalized federated learning using hypernetworks. ICML 2021.   ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Comment***:\n- We sincerely thank the reviewer HD18 for the appreciation and constructive comments. The reviewer raises concern about the motivation of using Bregman-Divergence (B-Div) and relaxing Mirror Descent (RMD), and we answer the questions one by one.\n\n***Responses***:\n1. [Q1, W1] The Motivation of B-Div and RMD.\n- **B-Div**:\n  - **Modeling**: the main problem studied in this paper is the overlooked information of client-sampling. In order *to widen the range of the main problem math. modeling*, we use the exponential family (X-family) prior assumption, due to its broad coverage in both practice and theoretical analysis.\n  - **Calculation**: according to the relationship between X-family and B-Div, as shown in Equation 3 and more in Line 601-612 in Appendix, the introduction of the B-Div brings about *the properties of easy calculation*, e.g., the first-order moment estimation point is the expected point with the closest Bregman distance (i.e., B-Div), in Line 507-509 in Appendix.\n  - **Computation**: these properties bring many easy-compute *closed-form solutions* instead of numerical approximations, e.g., the gradient of Bregman-Moreau envelope, the simpler form of both the Fenchel conjugate duality and transformation of the natural and expected param. of an X-family.\n- **RMD**:\n  - **Pre**：the introduction of Expectation Maximization (EM) and the X-family prior assumption in Line 124-134 transforms the optimization problem into an easy-compute one, from implicit into explicit assumption.\n  - **Mirror descent (MD)**: we use MD as the proposed computational framework with rigor, according to the relationship among EM, X-family and MD, i.e., EM under the X-family assumption is MD [39] in Line 142.\n  - **Relaxing**: we relax the constrains of MD *provides space for personalization* that MD would not otherwise have, in Line 140-148.\n2. [W2] Why do we put the avg. and the std. on all tasks in Tables?\n- Thanks for pointing out. Since we have a lot of benchmarks and *for ease-to-read*, to catch the overall results of the comparison, we provide the additional columns to provide statistics of the performance and stability. It's better to *differentiate between hard tasks and others*.\n3. [W3] *More related works and comparison*.\n- Thanks for the constructive suggestions. We add the comparison with the methods [a-d] in the revision. We provide the results in [*GENERAL RESPONSES*](forum?id=kuxu4lCRr5&noteId=WzbbMONDyW), and analysis as follows:\n  - **Results Sum.**:\n    - In comparison, as we mentioned in Line 65-67, our method still shows great accuracy and robustness to aggregation ratio, that our accuracy surpass the baselines at least 2.7/0.5/0.8 on three hard tasks and the avg. decrease is only 0.37 comparing to others about 0.6-1.5. \n    - In the additional experiments, FedPAC[a], the FA of which is the main module pull representations to the global feature centroid, is effective, but is not specifically designed for the overlooked client-sampling information, so it performs relatively not well on some scenarios, e.g., low-aggregation-rate settings, regression tasks, text tasks.\n  - **Analysis**:\n  \t- Our method with personalized prior does not have to re-obtain all local information from scratch solely from the data during training. These are especially important on hard-to-learn representations and datasets.\n    - The FA [a] might be affected by the aggregation noise from low ratio, due to the noise makes the global feature centroid [a], unstable and failed.\n    - For example, our method design is crucial for Sent140 from LEAF [11]. The task on Sent140 is a regression task (or binary cls.) on text dataset with heterogeneous input data obtained from social software.  The landscape of rep. on text task is more rugged [53, 16, 13], and the heterogeneous inputs makes it harder.\n4. [W4] This paper focuses on the specific label distribution problem. How about others?\n- **Clarification**:\n  - Theoretically:\nOur modeling about data $(x_i,y_i)$ in Line 113-119, *the pairs of input and label respectively in dataset $d_i$ on the $i^{th}$ client*, does not depend on a specific labeling distribution. We do not constraint the label in the methodology, framework and convergence analysis.\n  - Experiments:\nIn order to verify the effectiveness of pFedBreD on different distributions, we conduct the following experiments as shown in the following table. There are 4 data partition methods, shown in Appendix C.5, and the datasets of our experiments are not on specific label distribution, as follows:\n|Datasets|FEMNIST|Sent140|FMNIST/MNIST/CIFAR-10|$\\alpha$-FMNIST|\n|-|-|-|-|-|\n|Partition Methods|Uniform and dominant class [a, 11]|Heterogeneous input data [67, 11]|Rotation allocation|$\\alpha$-partition [28] in Table 4|\n|Our performance (Acc.)|70.3|73.7|99.0/93.0/80.6|Avg. 34.8|\n|The best performances among our baselines|66.8|71.3| 98.7/92.2/79.7|Avg. 34.4|\n\n***Minors***:\n1. [Q2] The second-best should be 79.68 (Per-FedAvg+FT).\n- Thanks for pointing out, and we will fix it and carefully proof reading of our paper in the revision.\n\n***References***:\n\n[a] Personalized Federated Learning with Feature Alignment and Classifier Collaboration. ICLR 2023.\n\n[b] Ditto: Fair and Robust Federated Learning Through Personalization. ICML 2021.\n\n[c] Personalized Federated Learning with First Order Model Optimization. ICLR 2021.\n\n[d] Personalized federated learning using hypernetworks. ICML 2021."}, {"review_id": "XWZD1FWgxw", "reviewer": "Reviewer_jGgs", "summary": "This paper proposes pFedBreD, which decouples the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized FL. Extensive experiments validate the effectiveness of the proposed method on 5 datasets.", "questions": "Q1. The assumption this paper makes is the uniform sampling of clients. How about other sampling methods? Is the proposed method applicable to other sampling scenarios?\n\nQ2. In Table 1, the accuracy of FedAMP is better than the proposed method on some datasets. In particular, the accuracy of the MCLR of FedAMP on different datasets is close to or even outperforms mh. The reason should be provided.\n\nQ3. How to determine the number of global/local epochs in your experiment setting?\n", "limitations": "The authors adequately pointed out the limitations of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. The problem of overlooking client-sampling information at prior knowledge being transferred is important.\nS2. A novel framework, pFedBreD, is proposed for computing the Bayesian optimization problem. The theorem is provided to analyze the convergence.\nS3. The experiments validate the effectiveness of pFedBreD on five widely-used datasets.\n", "weaknesses": "W1. The problem is not well-motivated. To be more specific, the shortcomings of existing PFL methods are not well explained. The client-sampling information should be clearly illustrated, and the reason why the prior knowledge extraction is challenging is missing.\n\nW2. The problem should be formally defined. It would be better to add a subsection to provide the statement of the research problem of this paper.\n\nW3. The efficiency of the proposed methods is not well analyzed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Comment***:\n- We sincerely thank the reviewer jGgs for appreciation recognizing the importance of the problem we introduced and suggestion for improving readability. We address the concern raised by the reviewer about motivations and challenges, and we answer these questions as follows.\n\n***Responses***:\n  \n1. [W1] More explanation about motivations.\n- Thank you for pointing out. We provide further explanation about the existing shortcomings in PFL that this paper is focusing on, and the two major shortcomings are as follows [Please see [general responses](https://openreview.net/forum?id=kuxu4lCRr5&noteId=WzbbMONDyW) for more about motivations]:\n   1. **What is exactly the effects of overlooked client-sampling information?**: the salient shortcoming of this overlooking is that *the specific global knowledege for each client is the same*, making the specific model on each client have to re-obtain the local information from scratch solely from the data during training which is particularly distinctive in text tasks as analyzed in Line 259-262. These are especially important on hard-to-learn representations and datasets.\n   2. **Why is the prior knowledge *explicit* extraction challenging?**: the main purpose of the explicit expressions in most classical methods is for parameterization, computational convenience, and closed-form solutions, e.g., variational inference for re-parameterization techniques, conjugate prior in Bayesian methods. In this paper, under implicit prior assumption, the complete information is *not parameterized and computationally expensive* in Line 125-134. The substitute is to tune this information by the way of regularization, representation or loss function design. Under explicit prior assumption, if all the parameters are given, the prior distribution is *completely determined*, and the local information that comes after that, *comes exclusively from the local data*. Thus, we can design theoretical-supported strategies directly with parameters.\n\n2. [W2] Formal definition and suggestion about additional subsection.\n- Thanks for the suggestion about a subsection about motivation to improve reading with formal definition of the overlooked information problem. We consider adding sections focusing on the followings: [Welcome to discuss in detail during the discussion period.]\n   - **The core of the problem**: global model $w=E_{i}w_{i}=E_{i}w_{i}|i \\Rightarrow$ Mutual Information $I(w;i)=0$.\n   - **Effects**: global model *has no mutual information with client-sampling $i$*, as shown in the above equations and discussed in Line 120-124, in particular when applying regularization $R(w^{(t)};...)$ or local initialization $w_{i,0}^{(t)}\\leftarrow w^{(t)}$  where $w_{i}$ is the local model on the $i^{th}$ client. This makes the specific model on each client have to re-obtain this information from scratch solely from the data during training which is particularly distinctive in text tasks as the analysis in Line 259-262.\n\n3. [W3] More analysis about efficiency.\n- Thanks for suggestion, and we will discuss more about efficiency as follows and in revision. Please see [Q28a rebuttal Q3](https://openreview.net/forum?id=kuxu4lCRr5&noteId=KKUTOVP2PR) for more details, and we'd like to provide more during the discussion period:\n  - We discuss *efficiency* in terms of time and space complexity. The complexity comparison is shown below, which is *in Appendix A and the last column in Table 2*. Our method *share the same complexity* as pFedMe and FedAMP, where $N$,$T$,$R$ and $K$ are respectively the number of clients, global epochs, local epochs and proximal solver iterations. *As shown below and in Table 2*, the local computation only adds limited computation with significant improvement.\n\n |Complexity/Methods|FedAvg|pFedMe|Per-FedAvg|FedAMP|pFedBreD (ours)|\n |-|-|-|-|-|-|\n |Entire Sys. Memory|$O(N)$|$O(N)$|$O(N)$|$O(N)$|$O(N)$|\n |Entire Sys. Time|$O(NTR)$|$O(NTRK)$|$O(NTR)$|$O(NTRK)$|$O(NTRK)$|\n |Additional Local Computation beyond FedAvg|-|Addition|Gradient|Weighted Sum and L2-Distance|Gradient and Addition|\n |Average Acc. on All Tasks|72.80|79.48|80.02|81.80|83.09|\n\n4. [Q1] How about other sampling scenarios?\n- The proposed method *is applicable to any scenarios* with a known or given sampling distribution if the expectation exists, as we mentioned *in Line 188-189* in Convergence Analysis (CA). We analyze on a *uniform* client sampling $E_{i}=\\frac{1}{N}\\sum_{i=1}^{N}$ setting for simplification. *Other sampling methods can be obtained* with client sampling expectation $E_{i} [F_{i}] = F$, by changing sampling weights. In the CA, we use the uniform sampling *only for simplification* of the theorem, and only the linearity of operator $\\frac{1}{N}\\sum_{i=1}^{N}$ in convexity, Jensen inequality and gradient calculation, which could be replaced by linear $E_{i}$.\n\n5. [Q2] Analysis about FedAMP.\n- We do provide an analysis about FedAMP *in Line 242-251*, which is fully consistent with our experimental results, and the main analysis provided are as follows:\n\t-  *On convex problem, FedAMP outperforms our method ... One possible reason ... FedAMP uses the distance between models as a similarity in the penalty point selection ... since there is only one global optimum, this penalty point tends not to change ... a non-dynamic regular term ... will not be as advantageous for non-convex problems ... as penalty point tends to fall into the local optimum ...*\n\n6. [Q3] The number of global/local epochs.\n- The number of global/local epochs $T$/$R$ are shown below and in Line 224. Empirically, different tasks require different numbers of training epochs to reach convergence, and it's recommended that smaller $R$ if global stability is wanted, and larger if personalization and convergence speed are wanted. \n \n| Dataset-Model| CIFAR-10 | Sent140-LSTM | FEMNIST-MCLR/DNN | FMNIST-MCLR/DNN | MNIST-MCLR/-DNN |\n|-|-|-|-|-|-|\n|Local Epochs $R$|20|20|20|20|20|\n|Global Epochs $T$|2000|800|800|200|200|"}, {"review_id": "KNVaTFMM8K", "reviewer": "Reviewer_qY3t", "summary": "The authors of the paper consider the problem of personalized federated learning. The main problem the authors attempt to tackle is the information on sampling of clients being overlooked. Specifically, they attempt to introduce two major steps in the training of personalized models: the first one is the injection of personalized prior knowledge into the global model before training, the second is the extraction of the prior before sending the global model updates. The authors formally present the problem of overlooking client-sampling information, then they utilize a framework for the problem called pFedBreD. Additionally, they provide extensive theoretical examination of the framework, as well as numerous experiments to validate the superior performance of their solution compared to a handful of baseline algorithms.", "questions": "The mentioned weaknesses need to be addressed. The main body of the paper should be improved to include the missing parts as well as the acronyms should be clearly defined.\n\nFurther, the paper needs to be checked for typos. For example: line 85, \"pFedBreD is a expectation\" -> \"pFedBreD is an expectation\", line 95 \"Bregman divergencee\" -> \"Bregman divergence\", line 213: \"we choose following\" -> \"we choose the following\".", "limitations": "The authors present the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is very well-written. It is clear and comprehensive, and its structure is sound.\n- The authors motivate the problem very well, and provide a sound formulation of it.\n- The theoretical examination of the problem, especially the results are well-presented.\n- The extensive experimental results, especially the comparisons with various baselines are useful for the completeness of the evaluation of the proposed framework. A number of datasets as well as different models are considered. Moreover, the analysis of the results is insightful.", "weaknesses": "I have a couple of comments on the weaknesses of the paper:\n- I think the main body of the paper is well-written as I mentioned in the weaknesses, but I also think it lacks fundamental parts that can be added to improve the readability of the paper. For example, the pFedBreD algorithm is not included in the main body but it is an important part that needs to be presented in the main body of the paper for completeness of the presentation. Similarly, the authors refer to Theorem 1 in the paper but it is not mentioned until the Appendix D. The paper should be self sufficient for the reader.\n- The authors use acronyms in the paper before mentioning what they are, for example the use of **mh**, **lg**, and **meg** before explicitly defining them or mentioning what they stand for. This can be confusing for the reader and needs to be rectified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Comment***:\n- We sincerely thank the reviewer qY3t for the appreciation and the constructive suggestion to further improve clarity and readability.\n\n***Responses***:\n1. [W1, Q1] Suggestions for improving readability.\n- Thanks for the constructive suggestion for readability, we will do the following modifications in the revision:\n  - The pFedBreD algorithm in Line 527-530 in Appendix A.5 will be put between Line 169 and Line 170 in the main body of the paper from the appendix.\n  - The Theorems in Appendix D.1 will be put in Section 5 before Remark 1 in the main body.\n- Other modifications for readability are listed as follows: (Thanks for all the suggestions from the reviewers)\n  - All the full names of the acronyms will be placed before or around the acronyms. [W2, which will be discussed in details later]\n  - The motivation and framework illustration, Fig. 1, will be more visually linked to the various relevant parts of the math. methodology and computation framework. [Suggested by reviewer [Q28a](https://openreview.net/forum?id=kuxu4lCRr5&noteId=uuT3KKlAUX)]\n  - We will add a subsection to formalize the overlooked information problem, focusing on the followings: [Suggested by reviewer [jGgs](https://openreview.net/forum?id=kuxu4lCRr5&noteId=XWZD1FWgxw)]\n       - **The core of the problem**: global model $w=E_{i}w_{i}=E_{i}w_{i}|i \\Rightarrow$ Mutual Information $I(w;i)=0$.\n       - **Effects**: global model *has no mutual information with client-sampling $i$*, as shown in the above equations and discussed in Line 120-124, in particular when applying regularization $R(w^{(t)};...)$ or local initialization $w_{i,0}^{(t)}\\leftarrow w^{(t)}$  where $w_{i}$ is the local model on the $i^{th}$ client. This makes the specific model on each client have to re-obtain this information from scratch solely from the data during training which is particularly distinctive on text tasks as the analysis in Line 259-262, and especially  impacted on hard-to-learn representations and datasets.\n\n2. [W2, Q1] The unclearly defined parts of acronyms.\n- Thanks for pointing out. All the full names of the acronyms will be placed before or around the acronyms. *In Line 634-638 in the Appendix*, we give the definitions and reasons about the full names of the acronyms. The three implementations of $\\mu_{i}$, i.e. lg, meg and mh, represent *loss gradient*, *memorized envelope gradient* and *memorized hybrid* respectively. *Memorized* means that we choose the gradient of Bregman-Moreau envelope  $\\nabla F_{i}(w_{i,r-1}^{(t)})$ as $\\eta[w_{i,R}^{(t-1)} - \\theta_{i,r-1}^{(t)}]$, where $\\eta \\ge 0$ is a step-size-like hyper-parameter. Each local client memorizes their own local part of the latest global model $w^{(t)}$ at the last global epochs $w_{i,R}^{(t-1)}$, instead of $w_{i,r-1}^{(t)}$ in practice.\n\n***Minors***:\n1. [Q2] Typos.\n- Thanks for pointing out, and we will fix all typos in the revision."}, {"review_id": "WHjNdN8en1", "reviewer": "Reviewer_JezB", "summary": "\nThis paper targets injecting personalized prior knowledge into the global model, which attempts to mitigate the introduced incomplete information problem in PFL. The idea is to decouple the personalized prior from the local objective function regularized by Bregman divergence. The mirror descent (RMD) is used to extract the prior.\nThe authors presented convergence analysis and showed many experimental results.\n", "questions": "**Background**\n1. In Lines 43-46, \"Most of the insightful works [17, 50] propose assumptions for recovering this incomplete information, but these assumptions are implicit, which limits the way to use the information to develop personalized strategies. To address the former issue above, ...\". \nWhat is the exact problem/issue of the previous assumption? Can the authors explain more about the motivation? What is \"limits the way to use the information\" exactly here? I did not catch the exact challenge that the authors target here.\n\n2. In lines 46,47, the major contribution comes from \"we propose framework pFedBreD to inject personalized prior knowledge (PPK) into the one provided by a global model.\" Is there a research line for injecting personalized prior knowledge? What is the advantage of pFedBreD compared with them? \nI see the authors elaborated on \"Ablation Analysis of Personalized Prior\" from Line 269 to Line 283. Can authors explain more about PPK-relevant research line in a more intuitive manner?\n\n**Method and Framework**\n\n3. In lines 103,104, \"Exponential Family The regular exponential family (X-family) is a relatively large family... Therefore, to yield the prior, we employ the X-family...\" Is the employment of X-family due to \"large family\"? Is there any other special advantage for X-family?\n\n4. I can follow the equations of Section 4 from a math view. Would the authors elaborate the intuitive logic of combining these equations? \n\n5. Section 4 is \"methodology\", meanwhile, Section 5 is \"framework\". Does the designed framework belong to the proposed methodology? Or, what is the relation between the two sections?\n\n6. In Line 156, \"Inspired by the aforementioned motivation,\" What is the \"motivation\"? I tried to search \"motivation,\" but \"motivation\" appears only once in the paper...What is the \"motivation\" here? Is it relevant to some types of math objective functions?\n\n7. In Line 163 and Line 164, \"To solve the optimization problem in Eq. (12), we use gradient-based methods to solve the global problem ...\" Based on my knowledge, the conventional method is \"gradient-based methods.\" Is there any other special optimization in your framework?\n\n**Experiment**\n\n8. In Line 220, \"The results of average accuracy per client are shown in Table 1.\" Could the authors summarize the insight conclusion of Table 1 in the main text? \n\n9. What is the relation between \"RMD\" and \"mh\"? Are any experiments relevant to \"RMD\"?\n\n**Minor**\n\n- In Line 85, \"a expectation\"\n- The clickable indices of reference/table/equation/etc do not work, which are not convenient for searching the relevant contents.\n- In Line 113 - Line 115, long sentence with grammar error.\n\n  \"In this section2, we introduce missing client-sampling information based on classic FL use EM to reduce the computational cost of the information-introduced FL problem, and propose RMD, a class of prior selection strategies, based on the E-step in EM\"\n- Equation 5 requires to introduce \"KL\" function before usage. \n- In Line 168, T,R,N appear suddenly by following the main body.\n", "limitations": "The authors try to empirically address the limitation of instability in the global model with aggregation noise. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors conducted many experiments and show improved results. For example, the proposed method has a higher deviation in missing classes in local testing and a lower deviation in global testing.\n2. The idea seems to be interesting.", "weaknesses": "1. Unclear motivation and literature review. \n2. Lack of an explainable summary of the main idea. Given the abstract, I think the idea seems to be interesting, but I got lost in the details and symbols when reading Section 4 and Section 5.\n3. The writing requires to be improved.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Comments***:\n- We sincerely thank reviewer JezB for appreciating our idea. The main concern of the reviewer is about main idea and our methodology, and we answer the questions one by one and make the structure and design of our paper clear.\n\n***Responses***:\n1. [W1, W2, and W3] Motivation and readability:\n- We set out the *main idea and structure* of the paper in short. This paper aims to address the *information overlooked problem* and all the followings are proposed for this. Please see [general responses](https://openreview.net/forum?id=kuxu4lCRr5&noteId=WzbbMONDyW) for detailed explanation about motivation [W1, W2] and readability improvement [W3].\n2. [Q1, W1] The issue of the implicit assumption? How does it limit the way?\n- The main purpose of the explicit expressions in most classical methods is for parameterization, computational convenience, and closed-form solutions, e.g., re-parameterization for variational inference, Bayesian conjugate prior.[f, g] \n- In this paper, under implicit prior assumption, the complete information is not parameterized and computationally expensive in Line 125-134. The substitute is to tune this information by the way of regularization, representation or loss function design. Under explicit prior assumption, if the parameters are given, the prior distribution is *completely determined*, and the local information that comes after that, *comes exclusively from the local data*. Thus, we can design theoretically supported strategies and directly with parameters.\n3. [Q2, W1] Is there a research line for injecting personalized prior knowledge (PPK)?\n- To the best of our knowledge, this is *the first paper* presenting this concept of PPK in PFL. The introduction of PPK is to address the problem of overlooked information, and this is the first time this problem has been formally discussed in PFL. For space reason, we can discuss this in the discussion period.\n4. [Q3] Why X-family? More discussion.\n- In order to expand the discussion of the main problem in this paper, we use the *X-family prior assumption*, for its *broad coverage* in both practice and theoretical analysis. According to the *relationship between X-family and B-Div*, as shown in Equation 3 and more in Line 601-612 in Appendix, the introduction of the B-Div brings about the properties of *easy calculation*, e.g., the first-order moment estimation point is the expected point with the closest Bregman distance (i.e. B-Div), in Line 507-509 in Appendix. the properties bring many *closed-form solutions that can be computed by replacing numerical approximations*, e.g., the gradient of Bregman-Moreau envelope, the simpler form of both the Fenchel conjugate duality and transformation of the natural and expected parameters of an X-family. This assumption makes the problem well-computable in Line 127-139 with optimization framework, and the computation method mirror descent is introduced with theoretical support [39]. If there is anything unclear in our responses, we'd like to discuss in the discussion period.\n5. [Q4] The intuitive logic of the equations in Section 4, as follows [Please see the motivation explanation in [general responses](https://openreview.net/forum?id=kuxu4lCRr5&noteId=WzbbMONDyW)]:\n  - Math. modeling. [Eq. 5]\n  - Implicit complete information $\\Theta_{i}$ is *not directly computable*. [Eq. 6]\n  - Expectation maximization (EM) *makes it computable* and the prior explicit. [Eq. 7-8]\n  - The optimization method Mirror Descent (MD) is used *to compute the EM problem*. [Eq. 9]\n  - Relaxing the constrains of MD so that there is *room for personalization*. [Eq. 10-11]\n6. [Q5] The relation between Sec. 4 and 5.\n- Section 4 is about *optimization* methodology, and Section 5 is about *computation* framework. The latter is a practical implementation of the former.\n7. [Q6] What does the *aforementioned motivation* in Line 156 stand for?\n- Typical motivation for a computation framework is *the modeled optimization problem*, which is in Line 148-153, the methodology in Section 4.\n8. [Q7] Why is *gradient-based methods* mentioned?\n- We did *not fix* exactly how to solve the bi-level optimization problem *until Section 5*. There are many computation methods for the optimization problem. In Section 5, we only employ gradient-based method, and other methods are also considerable, e.g., Newton, evolutionary, MCMC-Bayesian or other Approximate-Bayesian sampling-based computation methods discussed in Line 540-570 in Appendix.\n9. [Q8] The insightful conclusion of Table 1.\n- Table 1 is for comparison *in Line 241-262*, we provide insights from 3 perspectives, including convex or non-convex problems, easy or hard tasks, and text tasks. The most insightful one we think is *in Line 257-262*: the overlooked information makes the specific model on each client have to re-obtain this information from scratch solely from the data during training. Table 1 provides an overall evaluation and the tasks our method specializes in as claimed *in Line 65-67*.\n10. [Q9] What is the relation between \"RMD\" and \"mh\"? Are any experiments relevant to \"RMD\"?\n- RMD is a class of strategies determined by $\\Phi$.\n- With given $\\Phi = f_{i}+F_{i}$, mh is an implementation of RMD, or a strateiges in the class of strategies RMD.\n- Any experiments about mh are the experiments relevant to RMD.\n\n***Minors***:\nWe will carefully check the paper to correct similar issues.\n1. Yes, that should be *an*.\n2. Technical issue. We fix it.\n3. The sentence should be *...on classic FL, use EM...*. [missed comma]\n4. KL-divergence will be elaborated.\n5. We will make it better for reading.\n\n***References***:\n\n[f] Variational Autoencoder Based Anomaly Detection Using Reconstruction Probability. Special lecture on IE 2.1 (2015): 1-18.\n\n[g] A geometric View of Conjugate Priors. ML 81 (2010): 99-113."}]

Author rebuttal:
# General Responses:
We begin by making the following responses about the structure of our paper and results of additional experiments, and later to allow more space for responses to each author.

- More intuitive and high-level line of logic, which may help understanding and reading, are as follows [(j), the related section j]:
  - **Practical Problem (1): overlooked client-sampling information problem $\rightarrow$ Math. Problem Modeling (4): Bayesian modeling global problem as Maximize Likelihood Estimation (MLE) $\rightarrow$ Optimization Problem Modeling and Framework (4 & 4.1): Introduction of incomplete information and bi-level optimization with Expectation Maximization (EM) $\rightarrow$ Optional Implemental Strategies (4.2): Relaxing mirror descent $\rightarrow$ Computation Problem Formulation and Framework (5.1 & 5.2): Maximum A-Posteriori Estimation (MAP) as local problem and first-order methods $\rightarrow$ Implemental Principle and Computable Algorithms (5.3): Maximum entropy rule and meta step $\rightarrow$ Convergence Analysis (5.4): Bounded errors $\rightarrow$ Numerical Experiments and Analyses (6): 5 analyses for evaluation.**
- Detailed motivation logic is shown as follows:
  1. **Background**: we identify the problem of missing information in prior knowledge in *a single global model* and propose the concept, personalized prior, for this problem in Section 1. Formal discussion, global model $w=E_{i}w_{i}=E_{i}w_{i}|i \Rightarrow$ mutual information (MI) $I(w;i)=0$, is in Line 119-124.
	[From a Bayesian and info. perspective, the global knowledge transferred in conventional method with single global model *has no MI with client-sampling $i$*, in particular when applying reg. $R(w^{(t)};...)$ or local init. $w_{i,0}^{(t)}\leftarrow w^{(t)}$  where $w_{i}$ is the local model on the $i^{th}$ client. This makes the specific model on each client have to re-obtain this information from scratch solely from the data during training, especially impacted on hard-to-learn representations and datasets.]
  2. **Modeling**: we turn this practical problem into a math. one through Bayesian modeling.
  3. **Main Assumption**: *under X-family prior assumption for Bayesian modeling*, Bregman Divergence is introduced, in order to *widen the range of the theory and simplify the computation*.
  4. **Optim. Prob. and Framework**: based on EM, a bi-level optim. prob. and framework. pFedBreD is proposed.
  5. **Implemental Strategies**: a class of strategies called *RMD is proposed as a class of optional implementations* of pFedBreD for personalization.
  6. **Comp. Prob. and Framework**: maximum entropy rule and first-order method for computation is employed to implement the framework as pFedBreD$_{ns}$.
  7. **Experiment Results**: our methods with meta-step strategies reaches the SOTA on 8 public benchmarks and are especially great of our hybrid methods pFedBreD$_{ns, mh}$ *in hard tasks with small aggregation ratios and non-convex local objective settings*.
  8. **Further Impacts**: 
     - **Overall**, we introduce the problem of information overlooking during global knowledge transfer in conventional FL. 
     - **Theoretically**, our work provides a Bayesian explanation for regularization-based FL methods, modeling and convergence analysis for most of the methods that can be generalized to Bregman regularization.
     - **Practically**, our algorithm gives a new class of the SOTA methods and provides validation from more than 5 different perspectives to demonstrate the effectiveness of PFL in Section 6.2. 

- More comparison the reviewer HD18 interested in:
  - We compare the test accuracy between our method and the baselines recommended by reviewer HD18 on hard tasks mentioned in the paper as shown below. The settings are the same as the ones in the paper. We provide an extra column about average decrease for overall comparison about robustness to the noise caused by lowering aggregation ratio. (Note that *lower the aggregation ratio*, *larger the aggregation noise*.)
|Methods/Datasets & Models|FEMNIST & DNN|CIFAR-10 & CNN|Sent140 & LSTM|Average Decrease by Noise|
|:-|-|-|-|:-:|
|**Aggregation Ratio**|**10% $\rightarrow$ 5%**|**20% $\rightarrow$ 10%**|**40% $\rightarrow$ 20%**| - |
|FedPAC[a]|62.2 $\rightarrow$ 60.7|78.9 $\rightarrow$ 77.3|68.1 $\rightarrow$ 66.8|1.5|
|FedHN[d]|61.1 $\rightarrow$ 59.6|77.5 $\rightarrow$ 76.9|71.2 $\rightarrow$ 70.1|1.1|
|Fedfomo[c]|60.1 $\rightarrow$ 58.9|71.4 $\rightarrow$ 70.6 |70.1 $\rightarrow$ 68.9|1.1|
|Ditto[b]|52.9 $\rightarrow$ 52.2|72.4 $\rightarrow$ 72.1|71.0 $\rightarrow$ 70.3| 0.6 |
|mh (ours)|**64.9** $\rightarrow$ **64.3**|**79.4** $\rightarrow$ **79.1**|**72.0** $\rightarrow$ **71.8**| **0.37** |

- Modifications for further improvement of readability from constructive suggestions. More about this are welcomed to discuss in discussion period, and we consider the followings:
  - The pFedBreD alg. in Appendix A.5 will be put between Line 169 and Line 170 in the main body of the paper from the appendix. [[qY3t](forum?id=kuxu4lCRr5&noteId=KNVaTFMM8K)]
  - The Theorems in Appendix D.1 will be put in Section 5 before Remark 1. [[qY3t](forum?id=kuxu4lCRr5&noteId=KNVaTFMM8K)]
  - All the full names of the acronyms will be placed before or around the acronyms. [[qY3t](forum?id=kuxu4lCRr5&noteId=KNVaTFMM8K)]
  - The Fig. 1 will be more visually linked to the various relevant parts of the math. methodology and comp. framework. [[Q28a](forum?id=kuxu4lCRr5&noteId=uuT3KKlAUX)]
  - We will add a subsection to formalize the overlooked information problem. [[jGgs](forum?id=kuxu4lCRr5&noteId=XWZD1FWgxw)]

**References**:

[a] Personalized Federated Learning with Feature Alignment and Classifier Collaboration. ICLR 2023.

[b] Ditto: Fair and robust federated learning through personalization. ICML 2021.

[c] Personalized Federated Learning with First Order Model Optimization. ICLR 2021.

[d] Personalized federated learning using hypernetworks. ICML 2021.

Meta-review: This paper provides a new framework for personalization in federated learning. All reviewers appreciated the clarity of the paper, the novelty of the problem setup/approach, and the extensive experiments that demonstrated the effectiveness of the proposal. There were some concerns about baselines, and presentation issues that have been addressed in the rebuttal by the authors. All reviewers have reached consensus that the paper should be accepted. As such, the paper is recommended to be accepted subject to the authors incorporating the new results from rebuttal into the paper. Congratulations!
Decision: Accept (poster)