Title: PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning.
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Personalized Federated Learning

Expectation Maximization

Relaxed Mirror Descent
Abstract: Classical federated learning (FL) enables training machine learning models without sharing data for privacy preservation, but heterogeneous data characteristic degrades the performance of the localized model. Personalized FL (PFL) addresses this by synthesizing personalized models from a global model via training on local data. Such a global model may overlook the specific information that the clients have been sampled. In this paper, we propose a novel scheme to inject personalized prior knowledge into the global model in each client, which attempts to mitigate the introduced incomplete information problem in PFL. At the heart of our proposed approach is a framework, the $\textit{PFL with Bregman Divergence}$ (pFedBreD), decoupling the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized scenarios. We also relax the mirror descent (RMD) to extract the prior explicitly to provide optional strategies. Additionally, our pFedBreD is backed up by a convergence analysis. Sufficient experiments demonstrate that our method reaches the $\textit{state-of-the-art}$ performances on 5 datasets and outperforms other methods by up to 3.5% across 8 benchmarks. Extensive analyses verify the robustness and necessity of proposed designs. The code will be made public.

PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning.

Abstract

Classical federated learning (FL) enables training machine learning models without sharing data for privacy preservation, but heterogeneous data characteristic degrades the performance of the localized model. Personalized FL (PFL) addresses this by synthesizing personalized models from a global model via training on local data. Such a global model may overlook the specific information that the clients have been sampled. In this paper, we propose a novel scheme to inject personalized prior knowledge into the global model in each client, which attempts to mitigate the introduced incomplete information problem in PFL. At the heart of our proposed approach is a framework, the PFL with Bregman Divergence (pFedBreD), decoupling the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized scenarios. We also relax the mirror descent (RMD) to extract the prior explicitly to provide optional strategies. Additionally, our pFedBreD is backed up by a convergence analysis. Sufficient experiments demonstrate that our method reaches the state-of-the-art performances on 5 datasets and outperforms other methods by up to 3.5% across 8 benchmarks. Extensive analyses verify the robustness and necessity of proposed designs. https://github.com/BDeMo/pFedBreD_public

1 Introduction

Federated learning (FL) [53] has achieved significant success in many fields [72, 43, 71, 76, 69, 59, 32, 5, 34, 49], which include recommendation systems utilized by e-commerce platforms [71], prophylactic maintenance for industrial machinery [76], disease prognosis employed in healthcare [69]. Data heterogeneity is a fundamental characteristic of FL, leading to challenges such as inconsistent training and testing data (data drift) [36]. An efficient solution to these challenges is to fine-tune the global model locally for adaptation on local data [3, 74, 35]. This solution is straightforward and pioneering, but presents a fundamental limitation when dealing with highly heterogeneous data. For examples, heterogeneous data drift may introduce substantial noise [29] and the resulted model may not generalize well to new sample [20, 12]. Thus, heterogeneous data in FL is still challenging [66].

Recently, personalized FL (PFL) is proposed to mitigate the aforementioned negative impact of heterogeneous data [66]. To improve the straightforward solution mentioned above, Per-FedAvg [20] is introduced to train a global model that is easier to fine-tune. Another paper on the similar topic, FedProx [45], aims to resolve the issue of personalized models drifting too far from the global model during training with a dynamic regularizer in the objective during local training. This issue could occur especially in post-training fine-tuning methods without regularization ( e.g. , Per-FedAvg [20]). Moreover, pFedMe [65], another regularization method modeling local problems using Moreau

Operation: With/Without Client Sampling Info

Knowledge: With/Without Client Sampling Info

Likelihood

Posterior

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1

Local Training

(Prior + Likelihood )

Training (RMD-M)

Maximize

Data

3

Personalize Prior

Prior With Client-Sampling Info

GLYPH(cmap:d835)

GLYPH(cmap:d835)

2

Local Knowledge Transfer

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

4

GLYPH(cmap:d835)

= GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Aggregation

5

GLYPH(cmap:d835)

Injection

Extraction

RMD-E

envelopes, replaces FedProx's personalized model aggregation method with an interpretable approach for aggregating local models [45]. It also accommodates first-order Per-FedAvg [20].

Although the existing PFL methods have achieved promising results, the prior knowledge from single global model for local training [66] hinders the development of PFL. Specifically, we analyze the shortcomings of current PFL methods as follows: 1) utilizing the same global model for direct local training could potentially disregard the client's sampling information. As shown in Figure 1, a single global model provides global knowledge directly for local training, which overlooks the client-sampling information when the global knowledge is transferred to specific clients. 2) Explicitly extracting prior knowledge can be a challenging task. Most of the insightful works [17, 52] propose assumptions for recovering this incomplete information, but these assumptions are implicit, which limits the way to use the information to develop personalized strategies.

To address the former issue above, we propose framework pFedBreD to inject personalized prior knowledge into the one provided by a global model. As shown in Figure 1, it is injected in the 2 nd step and the local knowledge is transferred into global model via local models instead of directly aggregating personalized models [45]. To address the latter, we introduce relaxed mirror descent RMDto explicitly extract the prior for exploring personalized strategies.

Our method is backed up with direct theoretical support from Bayesian modeling in Section 4 and a convergence analysis in Section 5, which provides a linear bound O (1 /TN ) with aggregation noise and a quadratic speedup O (1 / ( TNR ) 2 ) without. 3 Meanwhile, the existence and validity of the injection and extraction aforementioned information is verified in Section 6.2. The remarkable performance of the implements of the proposed method is tested on 5 datasets and 8 benchmarks. Consistently, our method reach the state-of-the-art . Especially, the improvement of accuracy on task DNN-FEMNIST [11] is up to 3.5%. Extensive ablation study demonstrate that parts of the hybrid strategy mh are complementary to each other. Our contributions can be summarized as follows:

The problem of overlooking client-sampling information at prior knowledge being transferred is introduced in this paper, and we first investigate the possibility of explicitly expressing the prior knowledge of the information and design personalized strategies on it.

To express the personalized prior, we model PFL into a Bayesian optimization problem, Global-MLE and Local-MAP . A novel framework, pFedBreD, is proposed for computing the modeled problem, and RMD is introduced to explicitly extract prior information.

Sufficient experiments demonstrate our method surpasses most baselines on public benchmarks, thereby showcasing its robustness to data heterogeneity, particularly in cases involving small aggregation ratios and non-convex local objective settings.

2 Related Works

Regularization Researchers have developed a variety of approaches based on regularization to handle the PFL challenge in recent years ( e.g. , FedU [18], pFedMe [65], FedAMP [31], HeurFedAMP [31]). All of these approaches' personalized objective functions can be expressed as J ( θ ) + R ( θ ; µ ) where J ( θ ) is the loss function of the local problem and R ( θ ; µ ) is the regularization term used to restrict the deviation between θ and µ ( e.g. , R ( θ ; µ ) = 1 2 || θ -µ || 2 in pFedMe).

Meta Learning One of the most representative meta-learning based single-model PFL approach is the well-known Per-FedAvg [20], aiming to find an initialization that is easy to fine-tune. That is, the global model in the FL setting is regarded as a meta model in MAML [21, 19], where the objective function of the local problem is J ( θ -η ∇ J ( θ )) . Researchers also show the connections between FedAvg [53] and Reptile [60], another meta learning framework. [35] shows how to improve personalization of FL via Reptile. Proximal updating is also used in meta-learning based algorithms such as [79]. One of our strategies meg is the one motivated by MAML.

Expectation Maximization Two EM-based [15] methods are proposed, e.g. , FedSparse in [50] and FedEM in [17]. Both of them focus on communication compression. The latter provides a variance reduce version and assumes complete information (or data) of the global model obeys distribution in X-family. Another FedEM [52] combines Bayesian modeling, Federated Multi-task learning (FTML) and EM. Our framework pFedBreD is a expectation maximuzatioin and maximum a posteriori estimate (EM-MAP) [17] algorithm with personalized prior specified.

Bayesian FL In recent years, studies of PFL with Bayesian learning have been proposed. In related approaches, FLOA [48] and pFedGP [1] are proposed with KL divergence regularization in the loss function, which is comparable to applying specific assumption of X-family prior in pFedBreD see Appendix B.2 for details. Our implementation doesn't use a Bayesian neural network (BNN) model as an inferential model as others do ( e.g. , pFedGP uses a Gaussian process tree and pFedBayes [77] uses BNN). Instead, to eliminate weight sampling cost in Bayesian methods, prior knowledge is introduced through regularization term.

3 Preliminary

Overlooked Information in Prior Knowledge From a Bayesian and info. perspective, the global knowledge transferred in conventional method with single global model has no mutual information (MI) with client sampling i , i.e., formally, w = E i w i = E i w i | i ⇒ MI I ( w ; i ) = 0 , in particular when applying reg. R ( w ( t ) ; ... ) or local init. w ( t ) i, 0 ← w where w i is the local model on the i th client. This makes the specific model on each client have to re-obtain this information from scratch solely from the data during training, especially impacted on hard-to-learn representations and datasets.

Bregman-Moreau Envelope Bregman divergencee [10] is employed as a general regular term in our local objective that exactly satisfies the computational requirements and prior assumption, and is formally defined in Eq. (1).

where g is a convex function. For convenience, g is assumed to be strictly convex, proper and differentiable such that Bregman divergence is well-defined. To utilize the computational properties of Bregman Divergence in optimization problems, we introduce the following definition in Eq. (2) [7, 8]: Bregman proximal mapping, Bregman-Moreau envelope, and the relationship between them.

where λ > 0 denotes the regular intensity in general and the variance of the prior in our modeling.

Exponential Family The regular exponential family (X-family) is a relatively large family that facilitates calculations. Therefore, to yield the prior, we employ the X-family [6]defined in Eq. (3).

where g is assumed to be convex, D g ( · , · ) is the Bregman divergence, and g ∗ is the Fenchel Conjugate of g . In Eq. (3), s , h ( V ) and g ( s ) are respectively the natural parameter, potential measure and logarithmic normalization factor, where we have the mean parameter µ = ∇ g ( s ) . Additionally, to highlight the variance, the scaled exponential family (SX-family) is introduced in Eq. (4)

where log h λ ( V ) is the scaled potential measure, and the scale parameter λ is employed to highlight the variance. Moreover, V is assumed to be the minimal sufficient statistic of the complete information for local inference, details of which can be found in Section 4.

4 Methodology

In this section 4 , we introduce missing client-sampling information based on classic FL, use EM to reduce the computational cost of the information-introduced FL problem, and propose RMD, a class of prior selection strategies, based on the E-step in EM. The general FL classification problem with KL divergence could be formulated as Eq. (5) [53, 66].

where we rewrite the discriminant model as an maximum likelihood estimation (MLE) problem [41] of y i | x i in the right hand side (R.H.S.) of Eq. (5). ( x i , y i ) represent the pairs of input and label respectively in dataset d i on the i th client, and ˆ P ( y i | x i , w ) is the inferential model parameterized by w . Each local data distribution is presuppose to be unique, so using the global model with local data for inference and training could overlook the fact that the client has been sampled before transmitting the global model, and the prior knowledge transmitted directly via the global model as the local training prior knowledge ( e.g. via initial points, penalty points in dynamic regular terms, etc.) has no mutual information with the client sampling, i.e. , the global model w = E i w i = E i w i | i . Thus, to reduce the potential impact of the overlooked information, the complete information Θ i on the i th client is introduced which turns Eq. (5) into Eq. (6).

where E = E i E d i E y i | x i and the direct calculation of this is computationally expensive [14].

Framework: Leveraging Expectation Maximization for Prior Parameter Extraction The integral term in Eq. (6) makes direct computation impossible [14], so we employ EM to approximate the likelihood with unobserved variables [15] as shown in Eq. (7), where Q (Θ i ) is any probability measure.

Assuming that prior Θ i | d i , w ∼ ˆ P sef (Θ i ; λ, s i ( w ; d i ) , g ) 5 and the local loss function on the i th client f i (Θ i , w ) is E d i [ -log P ( y i | x i , Θ i , w )] , we have the left hand side (L.H.S.) of the Eq. (8) from (7). Here is an assumption for simplification that θ i contains all the information for local inference, i.e. θ i = Θ i and P ( y i | x i , Θ i , w ) = P ( y i | x i , Θ i ) . It happens when θ i is all the parameters of the personalized model and we only use the personalized model for inference. Thus, f i ( θ i ) = E d i [ -log ˆ P ( y i | x i , θ i )] . Thus, we can optimize an upper bound as a bi-level optimization problem as shown in the R.H.S. of the Eq. (8) to solve Eq. (5) approximately, where mean parameter µ i = ∇ g ◦ s i 6 [9]. And, we can derivate our framework as shown in Section 5.

Strategies: Relaxing Mirror Descent for Prior Selection To extract the prior strategies and implement µ i E-step of EM in close-form, we propose a method called relaxed mirror descent (RMD), where the mirror descent (MD) is EM in X-family [40]. MD can be generally written as Eq. (9) from the old ˆ w to the new one ˆ w + in each iteration [54, 40].

According to the Lagrangian dual, we rewrite the problem into a more general variant shown in Eq. (10) with relaxed restrictions and superfluous parameter.

We can transform Eq. (10) back into Eq. (9) by setting Φ( ˆ w ) to satisfy ∇ Φ( ˆ w ) = ∇ f ( ˆ w ) , and defining Ψ( ˆ θ, ˆ w ) as a function with f ( ˆ w ) and a penalty term to make ˆ θ and ˆ w close as possible ( e.g. , ˆ λ D ˆ g ( ˆ θ, ˆ w ) ). This provides us a way to extract µ Φ the function to generate mean parameter of the prior, as shown in Eq. (11), which is minimizing an upper bound of the problem in Eq. (10).

By optimality condition, we have µ Φ ( w ) = w -η ∇ Φ( w ) , which can be specified by Φ . The remaining part is a Bregman-Moreau envelope. Thus, we can optimize the upper bound with an EM-MAP method, alternately computing µ Φ ( w ) and D prox g ∗ ,λ -1 Ψ( · , w )( µ Φ ( w )) .

5 Framework Design

Problem Formulation that Highlights Personalized Prior Inspired by the aforementioned motivation, the personalized models θ i and mean parameters are respectively the solution of D env g ∗ ,λ -1 f i ( µ i ( w )) and µ i ( w ) on the i th client, where w is the global model. We assume that personalized model contains all the local information required for inference on the i th client, and satisfies θ i | d i , w ∼ P sef ( θ i ; λ, s i ( w ) , g ) . The global problem can be written as Eq. (12).

The given g is strictly convex, λ > 0 , f i is the local loss function, s i ( w ) is the natural parameter and µ i ( w ) = E θ i | x i ,w θ i = ∇ g ( s i ( w )) is the mean (or expectation) parameter in Eq. (12).

Framework: pFedBreD To solve the optimization problem in Eq. (12), we use gradient-based methods to solve the global problem using the gradient of F i :

where D is the gradient operator of the vector value function, and ∇ 2 is the Hessian operator. 7 The framework is shown as Algorithm 1, where I is the client selecting strategy for global model aggregation; w init and θ init are the initialization strategies on the i th client; α m is the main problem step-size; T , R , N are respectively the total number of iterations, local iterations, and clients. β is used in the same trick as [37, 65]. The strategies to derive the initialization points of w i and θ i at each local epoch are w ( t ) i, 0 ← w ( t -1) and θ ( t ) i, 0 ← θ ( t -1) i,R .

Implementation: Maximum Entropy and Meta-Step Practically, two main parts of the pFedBreD are needed to be implemented:

g , the function used to derive the logarithmic normalization factor, determines the type of prior to be used;

{ s i } or { µ i } , the functions used to derive the natural parameter and mean parameter for the personalized local prior, determine which particular prior is used.

We propose first-order implementations based on maximum entropy rule [22, 33]. In the SX-family, the Gaussian distribution has the maximum entropy among continuous distributions when g , µ i (the first-order moment), and λ (the parameter determining the second moment) are given. Thus, we employ the scaled norm square g = g ∗ = 1 2 || · || 2 to turn the prior into a spherical Gaussian, in order to maximize the entropy of the prior on a particular client. With this assumed prior, we have ∇ g = ∇ g ∗ = I , which means µ i = s i . We can choose a different Φ i as shown in Eq. (14) 8 to generate selection strategies according to Section 4, via µ i ( w ) = w -η ∇ Φ i ( w ) (meta-step).

where η α and η are the meta-step-size parameters. Practical parameter selection strategies with meta-step are shown as µ ( t ) i,r in Eq. (14). The three of µ i , i.e. lg , meg and mh , represent loss gradient , memorized envelope gradient and memorized hybrid respectively.

Convergence Analysis we analyze the convergence of pFedBreD with RMD on a uniform client sampling E i = 1 N ∑ N i =1 setting for simplification. Other sampling methods can be obtained with client sampling expectation E i [ F i ] = F , by changing sampling weights. The assumptions, proof sketch and detailed notations are in Appendix A.1 and Appendix D.

Theorem 1 (pFedBreD's global bound) . Under settings in Section 5 and Appendix D, at global epoch

.

Theorem 2 (pFedBreD ns 's first-order personalization bound) . Under the same conditions as in Theorem 1, with prior assumption of a spherical Gaussian and first-order approximation, the bound for the gap between the personalized approximate model and global model in the Euclidean space is:

Remark 1. Theorem 1 shows the main factors that affect the convergence of a global model are as follows: random mini-batch size, client drift error, aggregation error, heterogeneous data, dual space selection, local approximation error, and selection strategy for exponential family prior mean and variance. These can be divided into four categories based on their computational complexity. The first and second term shows that the proper fixed ˜ α m can linearly reduce the influence of initial error ∆ (0) and the global model converges to a ball near the optimal point. The radius of this ball is determined by the personalized strategy and local errors (including local data randomness and envelope approximation errors). The third term implies that a linear convergence rate O (1 / ( NT )) can be obtained w.r.t. the total global epoch NT in the presence of aggregation noise. Without client sampling N = S , according to the fourth term, the quadratic rate O (1 / ( TNR ) 2 ) can be obtained with β = O ( N ) or β = O ( N √ R ) (Note that the number of local epoch R cannot be too large due to client drift, according to 2 R ). Theorem 2 shows that, with spherical Gaussian prior assumption and first-order methods, the radius of the neighborhood range for the minimum that includes the personalized model on i th client, O ( C Φ ,F,f,d + 1 λ 2 ( ϵ 2 1 + σ 2 F, ∗ + B ˆ L F ˆ µ F · ) + λ 2 A ˆ µ F · ) , can be trade-off by λ , and is affected by the prior selection strategies and first-order approximate error besides the elements in Theorem 1. (Note that the Euclidean space is self-dual.)

6 Experiments

6.1 General Settings

Tricks, Datasets and Models: our experiments include several tasks: CNN [28] on CIFAR-10 [18, 39], LSTM [27] on Sent140 [11] and MCLR/DNN on FEMNIST [11]/FMNIST [65, 67]/MNIST[65, 42]. The details of tricks (FT, AM), data heterogeneity and models are in Appendix C.

Baselines: we choose following algorithms as our baselines: FedAvg [53], Per-FedAvg [20], pFedMe [65], FedAMP [31], pFedBayes [77] and FedEM [52]. These baselines are respectively classical FL, MAML-based meta learning, regularization based, FTML methods, variational inference PFL and FMTL with EM.

Global Test and Local Test: the global and personalized model, represented by G and P , are evaluated with global and local tests respectively. Global test means all the test data is used in the test. Local test means only the local data is used for the local test and the weight of the sum in local test is the ratio of the number of data. The results of average accuracy per client are shown in Table 1. Each experiment is repeated 5 times. More details are in Appendix C. For readability, we only give the error bar in the main Table 1 and Table 2, and keep one decimal except for the main Table 1.

Hyperparameter Settings The step-size of the main problem, α m , and the personalized step-size, α , for all methods are 0.01. β is 1, and the number of local epochs, R , is 20 for all datasets. λ is chosen from 15.0 to 60.0. The batch sizes of Sent140 and the other datasets are 400 and 20, respectively, and the aggregation strategy, I , is uniform sampling. The ratios of aggregated clients per global epoch are 40%, 10%, and 20% for Sent140, FEMNIST, and the other datasets, respectively. The numbers of total clients, N , are 10, 198, 20, and 100 for Sent140, FEMNIST, CIFAR-10, and other datasets. The number of proximal iterations is 5 for all settings with proximal mapping. In our implementations, η α and η are respectively 0.01 and 0.05.

Summarizing the Effects of Hyper-parameters We test the hyper-parameter effect of η and λ in our implementation pFedBreD ns, mh . The details are in Appendix C. From the results, we find that it will degrade the test accuracy if the values of λ or η are too large or too small. The test accuracy of personalized model is more sensitive than the ones of global model. The test accuracy of personalized

Methods /

Datasets

FEMNIST

FMNIST

MNIST

CIFAR-10

Sent140

Statistics

Names -

Models

MCLR

DNN

MCLR

DNN

MCLR

DNN

CNN

LSTM

Avg

H.Avg

H.Std

FedAvg [53] -

G

53.38

±

0

.

26

57.04

±

0

.

08

82.75

±

0

.

04

80.09

±

0

.

06

86.59

±

0

.

03

88.26

±

0

.

05

57.51

±

0

.

07

70.86

±

0

.

01

72.06

14.34

61.80

7.85

FedAvg+AM -

G

55.34

±

0

.

05

59.03

±

0

.

10

82.58

±

0

.

03

81.03

±

0

.

12

86.74

±

0

.

03

89.31

±

0

.

05

57.07

±

0

.

12

71.27

±

0

.

01

72.80

14.01

62.46

7.70

FedEM [52] -

G

40.75

±

0

.

32

45.47

±

0

.

04

95.78

±

0

.

03

96.42

±

0

.

03

85.75

±

0

.

01

86.49

±

0

.

02

57.67

±

0

.

16

66.72

±

0

.

03

71.88

22.28

56.62

10.66

pFedBayes [77] -

P

49.66

±

0

.

46

-

98.46

±

0

.

05

98.67

±

0

.

05

89.64

±

0

.

06

90.48

±

0

.

12

-

-

-

-

-

-

FedAMP [31] -

P

60.04

±

0

.

08

66.79

±

0

.

04

98.63

±

0

.

02

98.72

±

0

.

01

90.81

±

0

.

02

92.21

±

0

.

02

77.40

±

0

.

04

69.83

±

0

.

05

81.80

15.21

71.34

5.46

pFedMe [65] -

P

50.74

±

0

.

10

53.56

±

0

.

12

97.60

±

0

.

03

98.63

±

0

.

01

88.20

±

0

.

05

90.51

±

0

.

01

72.24

±

0

.

05

69.36

±

0

.

02

77.61

18.96

65.05

10.06

pFedMe+FT -

P

58.04

±

0

.

11

62.93

±

0

.

10

97.63

±

0

.

01

98.39

±

0

.

02

88.36

±

0

.

02

91.71

±

0

.

01

68.17

±

0

.

11

67.82

±

0

.

03

79.13

16.53

66.31

2.93

pFedMe+AM -

P

55.56

±

0

.

09

60.08

±

0

.

05

97.57

±

0

.

02

98.67

±

0

.

00

88.46

±

0

.

02

91.22

±

0

.

00

73.35

±

0

.

09

70.93

±

0

.

05

79.48

16.79

68.12

7.07

Per-FedAvg [20] -

P

54.34

±

0

.

14

62.72

±

0

.

03

94.28

±

0

.

05

97.46

±

0

.

04

87.09

±

0

.

01

90.96

±

0

.

02

78.87

±

0

.

05

70.05

±

0

.

03

79.47

15.74

70.54

8.09

Per-FedAvg+FT -

P

55.34

±

0

.

15

63.34

±

0

.

01

95.76

±

0

.

07

98.10

±

0

.

01

87.56

±

0

.

03

89.58

±

0

.

01

79.68

±

0

.

04

70.20

±

0

.

01

79.95

15.61

71.07

8.20

Per-FedAvg+AM -

P

56.66

±

0

.

09

65.74

±

0

.

02

92.08

±

0

.

10

98.24

±

0

.

02

86.91

±

0

.

04

90.85

±

0

.

02

78.97

±

0

.

03

70.73

±

0

.

05

80.02

14.54

71.81

6.68

mh

(ours) -

P

56.34

±

0

.

09

64.93

±

0

.

03

98.44

±

0

.

01

98.73

±

0

.

01

89.83

±

0

.

02

92.04

±

0

.

01

79.44

±

0

.

02

72.04

±

0

.

01

81.47

15.88

72.14

7.26

mh

(ours)+FT -

P

59.81

±

0

.

07

67.53

±

0

.

02

98.51

±

0

.

02

98.98

±

0

.

03

90.10

±

0

.

03

92.96

±

0

.

05

79.16

±

0

.

03

71.87

±

0

.

01

82.37

14.92

72.85

5.88

mh

(ours)+AM -

P

60.64

±

0

.

02

70.34

±

0

.

01

98.48

±

0

.

01

98.75

±

0

.

01

89.88

±

0

.

01

92.32

±

0

.

02

80.60

±

0

.

01

73.68

±

0

.

01

83.09

14.01

74.87

5.23

model is more sensitive to η than to λ . Note that the hyper-parameters are roughly tuned, which shows the insensitivity of mh , and better tuning could improve the performance in the Table 1.

6.2 Analysis

Comparative Analysis of Performance We compare our methods and the baselines from different perspectives, including convex or non-convex problems, easy or hard tasks, and text tasks. Additionally, we briefly discuss the absence of BNN on hard tasks.

Convex or non-convex: on non-convex problems, especially in hard tasks, our method significantly outperforms other methods by at least 3.06% employing some simple tricks. On convex problem, FedAMP outperforms our method somewhat on convex problems with simple data sets. One explanation is that the learning lanscape is simple in shape for these problems and FedAMP converges faster for this case. One possible reason for this is that since FedAMP uses the distance between models as a similarity in the penalty point selection, giving greater weight to the model that is most similar to the local one. In the later stages of training, since there is only one global optimum, this penalty point tends not to change, and thus the method degenerates into a non-dynamic regular term. Compounding intuition, this method will not be as advantageous for non-convex problems and harder convex problems, as penalty point tends to fall into the local optimum and lead to degradation of the dynamic regular term.

From easy to difficult task: from the difference between the statistics of Avg and H.Avg in Table 1, it can be observed that meta-step methods perform most consistently, with all other methods dropping at least 10%. This is due to the simple and effective local loss design of MAML, with its learning-to-learning design philosophy that enables the method to be more stable in complex situations [19, 20].

Personalized prior on text: text tasks, as opposed to image tasks, generally have relatively rugged learning landscape. [55, 16, 13] This understanding is manifested in specific ways, such as parameter sensitivity, slow convergence, and struggling during the process. Thus, the overlooked prior information seems to be more important, which means that each local iteration not only obtains local knowledge from the data, but also the prior itself already contains some local knowledge. Therefore, there is no need to re-obtain this knowledge from scratch solely from the data during training.

Absence of BNN on hard tasks: complex BNN is not in Table 1, such as LSTM in pFedBayes, because it is difficult to conduct comparative experiments by fixing elements, e.g. , inferential models, tricks and optimization methods. In pFedBayes, training often crashes on hard tasks and large datasets, as mentioned in [77]. Our one-step-further research shows that it may be caused by the reparameterization tricks and vanilla Gaussian sampling. If we add tricks on it, the implementation will be very different from the original pFedBayes, and it is beyond this analysis.

Ablation Analysis of Personalized Prior We conduct ablation experiments by dropping the gradient of the Bregman-Moreau envelope, the local loss function, or both, from the personalized

Methods

FEMNIST

FMNIST

MNIST

CIFAR-10

Sent140

Statistics

AC4PP

MCLR

DNN

MCLR

DNN

MCLR

DNN

CNN

LSTM

Avg

H.Avg

Non-PP

50.7

±

0

.

10

53.6

±

0

.

12

97.6

±

0

.

03

98.6

±

0

.

01

88.2

±

0

.

05

90.5

±

0

.

01

72.2

±

0

.

05

69.4

±

0

.

02

77.6

65.1

None

lg

(ours)

50.8

↑

±

0

.

05

49.1

↓

±

0

.

53

98.3

↑

±

0

.

02

98.4

↓

±

0

.

02

88.4

↑

±

0

.

01

91.0

↑

±

0

.

00

65.7

↓

±

0

.

46

60.7

↓

±

0

.

41

75.3

↓

58.5

↓

Grad.

×

R

meg

(ours)

50.3

↓

±

0

.

07

53.9

↑

±

0

.

06

97.8

↑

±

0

.

00

98.6

↓

±

0

.

01

88.4

↑

±

0

.

01

90.6

↑

±

0

.

01

73.8

↑

±

0

.

06

69.4

↑

±

0

.

02

77.9

↑

65.7

↑

Add.

×

R

mh

(ours)

56.3

↑

±

0

.

09

64.9

↑

±

0

.

03

98.4

↑

±

0

.

01

98.7

↑

±

0

.

01

89.8

↑

±

0

.

02

92.0

↑

±

0

.

01

79.4

↑

±

0

.

02

72.0

↑

±

0

.

01

81.5

↑

72.1

↑

Both above

0

500

1000

1500

0.25

0.50

0.75

1.00

GCE

pFedMe

FedAvg

mh (ours)

0

250

500

750

1000

0.5

1.0

GCE

mh (ours)

FedAvg

pFedMe

CIFAR-10 with CNN

MNIST with DNN

strategy mh as shown in Table 2. The relationship among the three strategies mentioned in Eq. (14) is that mh consists of lg and meg . Moreover, pFedMe can be regarded in our framework as the one which takes the spherical Gaussian as prior and uses vanilla prior selection strategy µ i = I without personalization. Thus, pFedMe and the three implementations of pFedBreD are compared. The results reveal the instability of our implementation lg and the introduction of meg on difficult tasks is about the same as not introducing it. However, introducing both lg and meg (i.e., mh ) together shows remarkable performance. This indicates that lg and meg complement each other. To explain these results , by observing the error bars, in most of the settings, meg is significantly more stable compared to methods that do not use personalized priors, while lg is relatively less stable. Based on this observation, we have reason to believe that meg weakens the influence of potential noise, while lg introduces new noise. Therefore, we can infer that while the mean parameters are steadily biased towards the personalized model, the introduction of new noise finds a path that is more likely to escape from local optima or saddle points, based on implicit regularization [61, 58, 57].

Generalized Coherence Analysis of Information Injection and Extraction The generalized coherence estimate (GCE) [25] of vectors from personalized to local model ( i.e. , the envelope gradients in pFedMe and ours) among clients on each global epoch are shown in Figure 2. The smaller the GCE, the less coherent the envelope gradient between individual nodes and the greater the diversity of information in the global model update. As shown in Figure 2, we can observe that during the convergence phase, using a personalized prior method has significantly greater information diversity than not using a personalized prior method, which proves the success of injecting personalized prior knowledge into the global model and extracting local knowledge from the local training.

Variable-Control Analysis of Robustness We analyze the impact of aggregation noise and data heterogeneity [29] on our method, mainly mh , by controlling variables. Results are in Table 3 and Table 4. (Details are in Appendix C.8.) We test the performance of global model on different aggregation ratios, where all hyper-parameters except for the aggregation ratios are fixed. Meanwhile, we test the performance of both global and the personalized model on different data heterogeneity settings, where full aggregation (sample client equals total number of clients, S = N ) and one-step local update (local epoch R = 1 ) are employed to get rid of the effects of aggregation noise and client drift. The experiments demonstrate the instability of the global model in mh at small aggregation ratios, which most of the other PFL methods have, by comparing their performances on different aggregation numbers. Comparing to the baselines, the experiments also demonstrate the relative robustness of our method to extreme data heterogeneity.

0

1

2

3

4

5

6

7

8

9

Class Index

0.25

0.20

0.15

0.10

0.05

0.00

0.05

0.10

0.15

Loss Deviation

pFedMe

lg (ours)

meg (ours)

PerFedAvg

mh (ours)

0

1

2

3

4

5

6

7

8

9

Class Index

0.8

0.6

0.4

0.2

0.0

Loss Deviation

pFedMe

lg (ours)

meg (ours)

PerFedAvg

mh (ours)

0

1

2

3

4

5

6

7

8

9

Class Index

0.10

0.05

0.00

0.05

0.10

Loss Deviation

mh (ours)

meg (ours)

lg (ours)

PerFedAvg

pFedMe

0

1

2

3

4

5

6

7

8

9

Class Index

0.6

0.5

0.4

0.3

0.2

0.1

0.0

Loss Deviation

mh (ours)

meg (ours)

lg (ours)

PerFedAvg

pFedMe

MNIST-MCLR-G

MNIST-MCLR-L

MNIST-DNN-G

MNIST-DNN-L

Deviation Analysis of Personalization Deviation represents the difference between an individual and the mean value. We use the deviation of the loss function to reflect the personalization. On global test , the lower the deviation, the better the personalized model performance on the corresponding local data. On local test , the model is only tested on its own dataset, and because of multiple local iterations, the local test deviation converges to almost the same value, as shown in MNIST-MCLR-L and MNIST-DNN-L in Figure 3. Furthermore, since the local test has a loss of 0 on missing classes, a higher deviation on missing classes reflects a lower mean on these classes. Thus, the lower loss in local testing and better performance are reflected from both of the almost equal deviation in local testing and the higher deviation on missing class. Summary: based on Figure 3, we can see that our method has higher deviation on missing classes in local testing and lower deviation in global testing. This means that our method has better personalized performance.

7 Conclusion and Discussion

Conclusion To address the issue of neglecting client-sampling information while providing prior knowledge to local training via direct use of a global model, we propose a general concept: the personalized prior. In this paper, we propose a general framework, pFedBreD, for exploring PFL strategies under the SX-family prior assumption and computation, the RMD to explicitly extract the prior information, and three optional meta-step strategies to personalize the prior. We analyze our proposal both theoretically and empirically. Our strategy mh shows remarkable improvement in personalization and robustness to data heterogeneity on non-i.i.d. datasets and the LEAF benchmark [11] with MCLR / DNN / CNN / LSTM as inferential model, which conduct convex / non-convex problems, and image / language benchmarks.

Limitations and Future Work Although mh shows remarkable performance and robustness, there is still instability in the global model with aggregation noise. Furthermore, it should be noted that the superficial reason for the improvement of mh seems to be that η α and η and (which are similar to each other) are used simultaneously, resulting in a magnitude in mh that is twice as large as the ones in the other two implementations and leading to better performance. However, empirically, simply doubling η α in lg or η in meg does not improve performance, and using one more meg step used in lg significant improvement. Our theoretical analysis cannot explain this phenomenon, and more detailed modeling is needed.

References

Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Proceedings of the NeurIPS , 34:8392-8406, 2021.

Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to mcmc for machine learning. MACH LEARN , 50(1):5-43, 2003.

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv:1912.00818 , 2019.

Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the ICALT , volume 117 of PMLR , pages 111-132, 08 Feb-11 Feb 2020.

Ulrich Matchi Aïvodji, Sébastien Gambs, and Alexandre Martin. Iotfla : A secured and privacypreserving smart home architecture implementing federated learning. In 2019 IEEE SPW , pages 175-180, 2019.

Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh, and John Lafferty. Clustering with bregman divergences. JMLR , 6(10), 2005.

Heinz Bauschke, Patrick Combettes, and Dominikus Noll. Joint minimization with alternating bregman proximity operators. PAC J OPTIM , 2006.

Heinz H Bauschke, Jonathan M Borwein, and Patrick L Combettes. Bregman monotone optimization algorithms. SIAM J CONTROL OPTIM , 42(2):596-636, 2003.

Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization , volume 28. Princeton university press, 2009.

Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. COMP MATH MATH PHYS+ , 7(3):200-217, 1967.

Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Koneˇ cn` y, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv:1812.01097 , 2018.

Gary Cheng, Karan Chadha, and John Duchi. Federated asymptotics: a model to compare federated learning algorithms. arXiv:2108.07313 , 2022.

Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural Language Processing (Almost) from Scratch. NLP , 2011.

Katalin Csilléry, Michael G.B. Blum, Oscar E. Gaggiotti, and Olivier François. Approximate bayesian computation (abc) in practice. TRENDS ECOL EVOL , 25(7):410-418, 2010.

Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. J R STAT SOC B , 39(1):1-22, 1977.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In the Proceedings of the CVPR , pages 248-255, 2009.

Aymeric Dieuleveut, Gersende Fort, Eric Moulines, and Geneviève Robin. Federated-em with heterogeneity mitigation and variance reduction. Proceedings of the NeurIPS , 34:29553-29566, 2021.

Canh T Dinh, Tung T Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. Fedu: A unified framework for federated multi-task learning with laplacian regularization. arXiv:2102.07148 , 2021.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradientbased model-agnostic meta-learning algorithms. In Proceedings of the AISTATS , pages 10821092. PMLR, 2020.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Proceedings of the NeurIPS , 33:3557-3568, 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the ICML , pages 1126-1135, July 2017. ISSN: 2640-3498.

Kenneth Friedman and Abner Shimony. Jaynes's maximum entropy prescription and probability theory. J STAT PHYS , 3(4):381-384, 1971.

Andrew Gelman and Cosma Rohilla Shalizi. Philosophy and the practice of bayesian statistics. BRIT J MATH STAT PSY , 66(1):8-38, 2013.

Christian Genest and James V Zidek. Combining probability distributions: A critique and an annotated bibliography. STAT SCI , 1(1):114-135, 1986.

H. Gish and D. Cochran. Generalized coherence (signal detection). In ICASSP-88 , pages 2745-2748 vol.5, 1988.

Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General Analysis and Improved Rates. In Proceedings of the ICML , pages 5200-5209, May 2019. ISSN: 2640-3498.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. NEURAL COMPUT , 9(8):1735-1780, 1997.

Jan Hosang, Mohamed Omran, Rodrigo Benenson, and Bernt Schiele. Taking a deeper look at pedestrians. In Proceedings of the CVPR , pages 4073-4082, 2015.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. arXiv:1909.06335 , 2019.

Chenxi Huang, Liang Xie, Yibo Yang, Wenxiao Wang, Binbin Lin, and Deng Cai. Neural collapse inspired federated learning with non-iid data. arXiv:22303.16066 , 2023.

Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. AAAI , 35:7865-7873, 2021.

Ahmed Imteaj, Urmish Thakker, Shiqiang Wang, Jian Li, and M. Hadi Amini. A survey on federated learning for resource-constrained iot devices. IEEE INTERNET THINGS , 9(1):1-24, 2022.

Edwin T Jaynes. Information theory and statistical mechanics. PHYS REV , 106(4):620, 1957.

Ji Chu Jiang, Burak Kantarci, Sema Oktug, and Tolga Soyata. Federated learning in smart city sensing: Challenges and opportunities. Sensors , 20(21), 2020.

Yihan Jiang, Jakub Koneˇ cn` y, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv:1909.12488 , 2019.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the ICML , pages 5132-5143. PMLR, 2020.

Robert E Kass and Larry Wasserman. The selection of prior distributions by formal rules. J AM STAT ASSOC , 91(435):1343-1370, 1996.

AKrizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront , 2009.

Frederik Kunstner, Raunak Kumar, and Mark Schmidt. Homeomorphic-invariance of em: Non-asymptotic convergence in kl divergence for exponential families via mirror descent. In Proceedings of the AISTATS , pages 3295-3303. PMLR, 2021.

L. Le Cam. Maximum Likelihood: An Introduction. INT STAT REV , 58(2):153-171, 1990. Wiley, ISI.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE T KNOWL DATA EN , 35(4):3347-3366, 2023.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Proceedings of the ICML , pages 6357-6368. PMLR, 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated Optimization in Heterogeneous Networks. Proceedings of MLSys , 2:429-450, March 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the Convergence of FedAvg on Non-IID Data. In Proceedings of the ICLR , April 2020.

Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. arXiv:2303.10058 , 2023.

Liangxi Liu, Feng Zheng, Hong Chen, Guo-Jun Qi, Heng Huang, and Ling Shao. A bayesian federated learning framework with online laplace approximation. arXiv:2102.01936 , 2021.

Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated Learning for Open Banking. In Qiang Yang, Lixin Fan, and Han Yu, editors, Federated Learning: Privacy and Incentive , pages 240-254. Springer International Publishing, Cham, 2020.

Christos Louizos, Matthias Reisser, Joseph Soriaga, and Max Welling. An expectationmaximization perspective on federated learning. arXiv:2111.10192 , 2021.

Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the ICML , volume 30, page 3. Citeseer, 2013.

Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated Multi-Task Learning under a Mixture of Distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Proceedings of the NeurIPS , volume 34, pages 15434-15447, 2021.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the AISTATS , pages 1273-1282. PMLR, 2017.

H Brendan McMahan. A survey of algorithms and analysis for adaptive online learning. The JMLR , 18(1):3117-3166, 2017.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality. In C. J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Proceedings of the NeurIPS , volume 26, 2013.

Thomas P. Minka. Expectation propagation for approximate bayesian inference. arXiv:1301.2294 , 2001.

Behnam Neyshabur. Implicit regularization in deep learning. arXiv:1709.01953 , 2017.

Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of optimization and implicit regularization in deep learning. arXiv:1705.03071 , 2017.

Dinh C. Nguyen, Ming Ding, Pubudu N. Pathirana, Aruna Seneviratne, Jun Li, and H. Vincent Poor. Federated learning for internet of things: A comprehensive survey. IEEE COMMUN SURV TUT , 23(3):1622-1658, 2021.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv:1803.02999 , 2018.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In Marina Meila and Tong Zhang, editors, Proceedings of the ICML , volume 139 of PMLR , pages 8913-8924, 18-24 Jul 2021.

Abraham. Savitzky and M. J. E. Golay. Smoothing and Differentiation of Data by Simplified Least Squares Procedures. ANAL CHEM , 36(8):1627-1639, July 1964. American Chemical Society.

Teddy Seidenfeld. Why i am not an objective bayesian; some reflections prompted by rosenkrantz. THEOR DECIS , 11(4):413-440, 1979.

Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In Proceedings of the ICML , pages 9489-9502. PMLR, 2021.

Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Proceedings of the NeurIPS , 33:21394-21405, 2020.

Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. TNNLS , 2022.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747 , 2017.

Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In Proceedings of the ICLR , 2023.

Jie Xu, Benjamin S. Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated Learning for Healthcare Informatics. JHIR , 5(1):1-19, March 2021.

Beining Yang, Kai Wang, Qingyun Sun, Cheng Ji, Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? In Proceedings of the NeurIPS , 2023.

Liu Yang, Ben Tan, Vincent W. Zheng, Kai Chen, and Qiang Yang. Federated Recommendation Systems. In Qiang Yang, Lixin Fan, and Han Yu, editors, Federated Learning: Privacy and Incentive , pages 225-239. Springer International Publishing, Cham, 2020.

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. TIST , 10(2), jan 2019.

Lei Zhang, Guanyu Gao, and Huaizheng Zhang. Spatial-temporal federated learning for lifelong person re-identification on distributed edges. arXiv:2207.11759 , 2022.

Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the CVPR , pages 10174-10183, June 2022.

Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. In Proceedings of the ICLR , 2020.

Wei Zhang, Xiang Li, Hui Ma, Zhong Luo, and Xu Li. Federated learning for machinery fault diagnosis with dynamic validation and self-supervision. KBS , 213:106679, 2021.

Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference. In Proceedings of the ICML , pages 26293-26310. PMLR, 2022.

Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. arXiv:2211.13976 , 2022.

Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng. Efficient meta learning via minibatch proximal update. Proceedings of the NeurIPS , 32, 2019.

Yuhao Zhou, Mingjia Shi, Yuanxi Li, Yanan Sun, Qing Ye, and Jiancheng Lv. Communicationefficient federated learning with single-step synthetic features compressor for faster convergence. In Proceedings of the ICCV , pages 5031-5040, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets injecting personalized prior knowledge into the global model, which attempts to mitigate the introduced incomplete information problem in PFL. The idea is to decouple the personalized prior from the local objective function regularized by Bregman divergence. The mirror descent (RMD) is used to extract the prior.\nThe authors presented convergence analysis and showed many experimental results.\n\nSTRENGTHS:\n1. The authors conducted many experiments and show improved results. For example, the proposed method has a higher deviation in missing classes in local testing and a lower deviation in global testing.\n2. The idea seems to be interesting.\n\nWEAKNESSES:\n1. Unclear motivation and literature review. \n2. Lack of an explainable summary of the main idea. Given the abstract, I think the idea seems to be interesting, but I got lost in the details and symbols when reading Section 4 and Section 5.\n3. The writing requires to be improved.\n\nQUESTIONS:\n**Background**\n1. In Lines 43-46, \"Most of the insightful works [17, 50] propose assumptions for recovering this incomplete information, but these assumptions are implicit, which limits the way to use the information to develop personalized strategies. To address the former issue above, ...\". \nWhat is the exact problem/issue of the previous assumption? Can the authors explain more about the motivation? What is \"limits the way to use the information\" exactly here? I did not catch the exact challenge that the authors target here.\n\n2. In lines 46,47, the major contribution comes from \"we propose framework pFedBreD to inject personalized prior knowledge (PPK) into the one provided by a global model.\" Is there a research line for injecting personalized prior knowledge? What is the advantage of pFedBreD compared with them? \nI see the authors elaborated on \"Ablation Analysis of Personalized Prior\" from Line 269 to Line 283. Can authors explain more about PPK-relevant research line in a more intuitive manner?\n\n**Method and Framework**\n\n3. In lines 103,104, \"Exponential Family The regular exponential family (X-family) is a relatively large family... Therefore, to yield the prior, we employ the X-family...\" Is the employment of X-family due to \"large family\"? Is there any other special advantage for X-family?\n\n4. I can follow the equations of Section 4 from a math view. Would the authors elaborate the intuitive logic of combining these equations? \n\n5. Section 4 is \"methodology\", meanwhile, Section 5 is \"framework\". Does the designed framework belong to the proposed methodology? Or, what is the relation between the two sections?\n\n6. In Line 156, \"Inspired by the aforementioned motivation,\" What is the \"motivation\"? I tried to search \"motivation,\" but \"motivation\" appears only once in the paper...What is the \"motivation\" here? Is it relevant to some types of math objective functions?\n\n7. In Line 163 and Line 164, \"To solve the optimization problem in Eq. (12), we use gradient-based methods to solve the global problem ...\" Based on my knowledge, the conventional method is \"gradient-based methods.\" Is there any other special optimization in your framework?\n\n**Experiment**\n\n8. In Line 220, \"The results of average accuracy per client are shown in Table 1.\" Could the authors summarize the insight conclusion of Table 1 in the main text? \n\n9. What is the relation between \"RMD\" and \"mh\"? Are any experiments relevant to \"RMD\"?\n\n**Minor**\n\n- In Line 85, \"a expectation\"\n- The clickable indices of reference/table/equation/etc do not work, which are not convenient for searching the relevant contents.\n- In Line 113 - Line 115, long sentence with grammar error.\n\n  \"In this section2, we introduce missing client-sampling information based on classic FL use EM to reduce the computational cost of the information-introduced FL problem, and propose RMD, a class of prior selection strategies, based on the E-step in EM\"\n- Equation 5 requires to introduce \"KL\" function before usage. \n- In Line 168, T,R,N appear suddenly by following the main body.\n\nLIMITATIONS:\nThe authors try to empirically address the limitation of instability in the global model with aggregation noise.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of the paper consider the problem of personalized federated learning. The main problem the authors attempt to tackle is the information on sampling of clients being overlooked. Specifically, they attempt to introduce two major steps in the training of personalized models: the first one is the injection of personalized prior knowledge into the global model before training, the second is the extraction of the prior before sending the global model updates. The authors formally present the problem of overlooking client-sampling information, then they utilize a framework for the problem called pFedBreD. Additionally, they provide extensive theoretical examination of the framework, as well as numerous experiments to validate the superior performance of their solution compared to a handful of baseline algorithms.\n\nSTRENGTHS:\n- The paper is very well-written. It is clear and comprehensive, and its structure is sound.\n- The authors motivate the problem very well, and provide a sound formulation of it.\n- The theoretical examination of the problem, especially the results are well-presented.\n- The extensive experimental results, especially the comparisons with various baselines are useful for the completeness of the evaluation of the proposed framework. A number of datasets as well as different models are considered. Moreover, the analysis of the results is insightful.\n\nWEAKNESSES:\nI have a couple of comments on the weaknesses of the paper:\n- I think the main body of the paper is well-written as I mentioned in the weaknesses, but I also think it lacks fundamental parts that can be added to improve the readability of the paper. For example, the pFedBreD algorithm is not included in the main body but it is an important part that needs to be presented in the main body of the paper for completeness of the presentation. Similarly, the authors refer to Theorem 1 in the paper but it is not mentioned until the Appendix D. The paper should be self sufficient for the reader.\n- The authors use acronyms in the paper before mentioning what they are, for example the use of **mh**, **lg**, and **meg** before explicitly defining them or mentioning what they stand for. This can be confusing for the reader and needs to be rectified.\n\nQUESTIONS:\nThe mentioned weaknesses need to be addressed. The main body of the paper should be improved to include the missing parts as well as the acronyms should be clearly defined.\n\nFurther, the paper needs to be checked for typos. For example: line 85, \"pFedBreD is a expectation\" -> \"pFedBreD is an expectation\", line 95 \"Bregman divergencee\" -> \"Bregman divergence\", line 213: \"we choose following\" -> \"we choose the following\".\n\nLIMITATIONS:\nThe authors present the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes pFedBreD, which decouples the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized FL. Extensive experiments validate the effectiveness of the proposed method on 5 datasets.

STRENGTHS:
S1. The problem of overlooking client-sampling information at prior knowledge being transferred is important.
S2. A novel framework, pFedBreD, is proposed for computing the Bayesian optimization problem. The theorem is provided to analyze the convergence.
S3. The experiments validate the effectiveness of pFedBreD on five widely-used datasets.

WEAKNESSES:
W1. The problem is not well-motivated. To be more specific, the shortcomings of existing PFL methods are not well explained. The client-sampling information should be clearly illustrated, and the reason why the prior knowledge extraction is challenging is missing.

W2. The problem should be formally defined. It would be better to add a subsection to provide the statement of the research problem of this paper.

W3. The efficiency of the proposed methods is not well analyzed.

QUESTIONS:
Q1. The assumption this paper makes is the uniform sampling of clients. How about other sampling methods? Is the proposed method applicable to other sampling scenarios?

Q2. In Table 1, the accuracy of FedAMP is better than the proposed method on some datasets. In particular, the accuracy of the MCLR of FedAMP on different datasets is close to or even outperforms mh. The reason should be provided.

Q3. How to determine the number of global/local epochs in your experiment setting?

LIMITATIONS:
The authors adequately pointed out the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose pFedBreD to decouple prior knowledge from each client. pFedBreD extracts the personalized prior with Bregman Divergence for better performing personalized tasks. The authors provide convergence analysis and experiments evaluated on 5 datasets.\n\nSTRENGTHS:\n1.    The authors give a detailed convergence analysis for the proposed method.  \n2.    The theoretical derivation of the paper is comprehensive in the Appendix.  \n3.    The authors provided extensive experiments for different tasks and models and the results show the effectiveness of improving the local models' performance.\n\nWEAKNESSES:\n1.    The motivation of the work is not clearly presented. It is hard to understand why the authors use Bregman Divergence and Relaxing Mirror Descent.     \n2.    It is confusing that the authors use the average results and the standard deviation of them on all tasks to represent the overall performance of methods. Could the authors give some intuition or explain the purpose of this?   \n3.    More related works need to be introduced. And we'd like to see experimental results compared to more related works, such as [1], [2], [3], and [4].   \n4.    This paper focuses on the specific label distribution problem. It would be better if the authors could discuss the performance of other heterogeneous data distribution problems (e.g., different number of classes is different).   \n\n[1] Personalized Federated Learning with Feature Alignment and Classifier Collaboration. ICLR 2023.   \n[2] Ditto: Fair and robust federated learning through personalization. ICML 2021.   \n[3] Personalized Federated Learning with First Order Model Optimization. ICLR 2021.   \n[4] Personalized federated learning using hypernetworks. ICML 2021.\n\nQUESTIONS:\n1.    The intuition of the work is not clear. Could the authors explain the motivation for using Bregman Divergence and Relaxing Mirror Descent in detail?   \n2.    In Table 1, why the second-best performance is 79.44 (mh) rather than 79.68 (Per-FedAvg+FT)?\n\nLIMITATIONS:\nThe authors have discussed the potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel scheme, pFedBreD, which addresses the incomplete information challenge in personalized federated learning by incorporating personalized prior knowledge into the global model of each client. This is achieved by decoupling the personalized prior from the local objective function and applying Bregman divergence regularization. The proposed approach leverages the Expectation-Maximization (EM) algorithm to approximate complete information from both global and local clients. Extensive empirical evaluations on multiple benchmarks demonstrate the effectiveness of the proposed method, and a theoretical analysis is provided to support its theoretical foundations.\n\nSTRENGTHS:\n1. The paper is well-written and easy to read.\n2. This paper introduces a novel approach that incorporates Bregman divergence and leverages the Expectation-Maximization (EM) algorithm to approximate complete information from both global and local clients. It effectively enhance personalized federated learning.\n3. The effectiveness of the proposed method is validated through extensive empirical evaluations and thorough theoretical analysis in the paper.\n4. This work showcases superior performance across multiple benchmarks when compared to various baseline methods.\n\nWEAKNESSES:\n1. The authors discuss the concepts of prior knowledge injection and extraction in Figure 1, as well as analyze the process of information injection and extraction. To enhance clarity, it would be helpful if the authors explicitly indicate in the framework design section which step corresponds to these specific actions.\n2. Certain aspects of the paper would benefit from additional clarifications to enhance understanding. See below.\n\nQUESTIONS:\n1. Is the method of calculating GCE mentioned in the paper? Does it depend on model parameters or extracted feature maps?\n2. Some abbreviation is not clearly explained in the paper. For example, what is MCLR, FT, AM, etc.?\n3. How does the computation cost of the proposed method compare to other approaches? Is the local training phase highly time-consuming? For example, what computational resources were utilized in the experiment, and what was the duration required to complete the experiment?\n4. Does the scalability of the proposed method remain unaffected? For instance, can the proposed method handle a scenario with more than 100 clients?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}