Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning

Authors: pouya m ghari
yanning  shen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:22:05

Abstract: Federated learning is renowned for its efficacy in distributed model training, ensuring that users, called clients, retain data privacy by not disclosing their data to the central server that orchestrates collaborations. Most previous work on federated learning assumes that clients possess static batches of training data. However, clients may also need to make real-time predictions on streaming data in non-stationary environments. In such dynamic environments, employing pre-trained models may be inefficient, as they struggle to adapt to the constantly evolving data streams. To address this challenge, clients can fine-tune models online, leveraging their observed data to enhance performance. Despite the potential benefits of client participation in federated online model fine-tuning, existing analyses have not conclusively demonstrated its superiority over local model fine-tuning. To bridge this gap, the present paper develops a novel personalized federated learning algorithm, wherein each client constructs a personalized model by combining a locally fine-tuned model with multiple federated models learned by the server over time. Theoretical analysis and experiments on real datasets corroborate the effectiveness of this approach for real-time predictions and federated model fine-tuning.

Main paper: 
 introduction: 
 Federated learning enables a group of learners, known as clients, to collaborate and collectively train a model under the coordination of a central server, without revealing their data. In this framework, clients perform local model updates and share these updates with the server. By aggregating these local updates, the server globally updates the model. Many prior works in the literature have assumed that each client stores a batch of training data and updates models locally based on this stored data (see e.g., [38, 29, 47, 5, 34, 53] ). However, in some cases, clients may need to make real-time predictions, and streams of data arrive sequentially, making it challenging to store and process data in batch. Furthermore, if clients operate in a non-stationary and dynamic environment, employing pre-trained models may fall short in prediction accuracy, requiring clients to fine-tune their models to adapt to their data.A federated learning framework, commonly referred to as online federated learning [37, 26, 19] , specifically addresses situations where clients engage in real-time predictions using a shared global model. After making predictions, clients collaborate with the server to update the global model for subsequent predictions in the future. Specifically, after making predictions, clients incur losses and based on the incurred losses, clients update the model locally and send their local updates to the server. The server updates the global model upon aggregating local updates and then distributes the updated global model to clients for use in the ongoing online prediction task. In this context, the models. With Fed-Rep [8] , each client generates a representation using the global model and learns its own local head for prediction. Algorithms in [10, 32, 57] enable clients to achieve personalized models by combining local and global models. However, none of these works have addressed the problem of online prediction while clients collaborate on training their personalized models.Federated Learning with Streaming Data. Several studies have explored the problem of federated learning when clients receive a stream of data in real time. While [7, 36, 35] investigate federated learning scenarios where clients receive new training data in each learning round, they do not address the aspect of online prediction by clients. Consequently, these works cannot provide regret guarantees for online prediction. Additionally, [9] studies the issue of staleness in online federated learning to both train and make prediction with the model; however, it lacks rigorous theoretical analysis. In [37] , an online federated learning algorithm is introduced, utilizing online mirror descent, with a proven sublinear regret bound. Similarly, [42, 16] propose online federated learning algorithms with guaranteed sublinear regret. Furthermore, [43] analyzes the benefits of collaboration in online federated learning, particularly in scenarios where only loss values at queried points are available to clients, without access to loss gradients. Regarding online federated kernel learning, algorithms are introduced in [17, 23] , albeit without accompanying regret analysis, leading to an absence of guaranteed regret bounds. In [26, 19] , multiple kernel-based models and random feature-based online federated kernel learning algorithms are proposed, with demonstrated sublinear regret. 
preliminaries: 
 This section explains the problem of federated learning for real-time prediction and model training. The present section studies the cases where clients either collaborate in federated learning or employ online gradient descent methods to locally train the model in real-time. This study highlights the motivation behind the proposed ensemble approach to federated learning.Let there are N clients interact with a server to train a model f (•; •). Also, let [N ] := {1, . . . , N }. At each time step t, client i, ∀i ∈ [N ] receives a data sample x i,t ∈ R d and makes the prediction f (x i,t ; θ t ) where θ t denotes the parameter of the model at time step t. Note that generalization to the scenario where at each time step, each client receives a dataset instead of a single data sample is straightforward. After making prediction, client i, ∀i ∈ [N ] observes label y i,t . Then client i, ∀i ∈ [N ] computes the loss of its prediction L(f (x i,t ; θ t ), y i,t ) where L(•, •) denotes the loss function. After computing the loss, clients send their updates to the server (e.g., by sending the gradient loss ∇L(f (x i,t ; θ t ), y i,t )) and the server aggregates information from clients to update θ t to θ t+1 to be used by clients to make predictions at time step t + 1. For ease of presentation, it is assumed that all clients can send their updates to the server every time step. Generalizing the results for the cases where only a fraction of clients can send their updates is straightforward. Furthermore, it is assumed that the label y i,t for any i and t is determined by the environment through a process unknown to the clients. This implies that the data distribution observed by client i can be nonstationary, and client i, ∀i ∈ [N ] does not know the distribution. Additionally, the data distribution can differ across clients. The goal is to enable clients to collaborate with the server to minimize the cumulative regret of clients over time. The regret of client i at time step t is defined as the difference between the prediction loss L(f (x i,t ; θ t ), y i,t ) and the loss L(f (x i,t ; θ * ), y i,t ) corresponding to the model with the optimal parameter θ * . Therefore, the average cumulative regret of clients up to time horizon T is defined asEQUATIONwhere θ * denotes the optimal model parameter in hindsight and can be expressed asEQUATIONOne common way to solve the problem is that clients employ online gradient descent to update models locally and the server exploits federated averaging [31, 37] to update the global model.At each time step t, client i obtains the locally updated model parameter ψ i,t+1 as follows:EQUATIONwhere η is the learning rate. Aggregating locally updated model parameters, the server obtains the updated global model parameter for time step t + 1 asEQUATIONThis continues up until the time horizon T . This paper examines regret under some or all of the following assumptions: A1. The loss function L(f (x; θ), y) is convex with respect to θ. A2. The gradient of the loss is bounded as ∥∇L(f (x; θ), y)∥ ≤ G. A3. For any x, θ, the loss is bounded as 0 ≤ L(f (x; θ), y) ≤ 1.The following theorem specifies the regret bound for federated learning employing online gradient descent under A1 and A2. Theorem 1. Employing online gradient descent, the following cumulative regret upper bound is guaranteed for federated learning under assumptions A1 and A2:EQUATIONProof of Theorem 1 can be found in Appendix A. According to Theorem 1, choosing η = O( 1/T ), the cumulative regret in ( 5) is bounded from above as O( √ T ). If the time horizon T is unknown, the doubling trick technique (see e.g., [2, 22] ) can be effectively used to set the learning rate to maintain theoretical guarantees. Consider the case where each client learns the model locally using online gradient descent without collaborating with other clients and the server. Let ϕ i,t denote the local model parameter learned by client i at time step t, employing online gradient descent locally. The regret of client i in this case is equivalent to federated learning regret where there is only one client. Therefore, substituting N = 1 and θ 0 = 0 in (27) of Appendix A, for the cumulative regret of client i with respect to any θ, we getEQUATIONAveraging ( 6) over all clients and substituting θ with θ * in (6), we obtainEQUATIONComparing ( 5) with (7) , it can be inferred that under assumptions A1 and A2, federated learning does not achieve tighter regret bound than the case where each client independently learns its local model. Hence, from a theoretical standpoint, it remains uncertain whether collaboration in federated learning yields any improvement over local online model training. Intuitively, collaboration in federated learning may prove advantageous when there exists similarity among data samples observed by clients over time. However, in cases where data distribution is heterogeneous and such similarities are lacking, employing online local training may yield superior results for a client. Given the lack of prior information on data distribution in online scenarios, each client can independently assess over time whether utilizing the model learned through federated learning for predictions is beneficial.Furthermore, according to assumption A1, the theoretical guarantees obtained in ( 5) and ( 7) hold if the loss is convex with respect to the model parameter. However, in the case of non-convex models, such as neural networks, these theoretical guarantees are not applicable. Non-convex models, like neural networks, may encounter the forgetting process [51, 45] , where the model tends to overfit to recently observed data samples. Consequently, it may not be feasible to derive a single model parameter using online gradient descent that achieves sublinear regret with respect to the best model in hindsight. The subsequent section introduces a novel algorithm aimed at assisting clients in addressing such scenarios. 
personalized federated learning methods: 
 The current section introduces personalized federated learning algorithms where each client dynamically learns the prediction performance of both the models trained through federated learning over time and the locally learned model.At each time step, each client constructs an ensemble model comprising the federated model and its locally learned model to make a prediction. Let ϕ i,t be the model parameter locally learned by client i such that at each time step t, client i updates ϕ i,t via gradient descent asEQUATIONFurthermore, let clients and the server collaborate to update the federated model parameter θ t as outlined in ( 3) and ( 4). At time step t, client i makes the prediction for x i,t using its personalized ensemble model f i,t (•) expressed asEQUATIONwhere α i,t and β i,t represent the weights assigned by client i to the federated and local models, respectively, indicating the credibility of predictions from each model. After making predictions and observing the label y i,t , client i computes the loss of predictions from the federated and local models, updating the weights α i,t and β i,t using the multiplicative update rule as follows:EQUATIONEQUATIONwhere η c is a learning rate. Client i initializes α i,1 = 1 and β i,1 = 1. The proposed algorithm is personalized since according to (9) , each client constructs its own ensemble model to perform prediction. The personalized regret of client i is defined asC i,T = T t=1 L(f i,t (x i,t ), y i,t ) - T t=1 L(f (x i,t ; ϕ * i ), y i,t) where ϕ * i denotes the best hindsight model parameter for client i which can be expressed as ϕ * i = arg min ϕ T t=1 L(f (x i,t ; ϕ i ), y i,t ). The following theorem establishes the personalized regret upper bound for client i as well as global regret of all clients with respect to the best model parameter in hindsight. Theorem 2. Under assumptions A1-A3, employing the ensemble model in (9) for online prediction, the global regret of all clients is bounded from above asEQUATIONwhile client i achieves the following personalized regret upper bound:EQUATION)Proof of Theorem 2 is given in Appendix B. According to (11) and (12) in Theorem 2, settingη = O(1/ √ T ), η c = O(1/ √ T )both the global regret for all clients and the personalized regret of each client i achieve a regret bound of O( √ T ). This demonstrates that while the ensemble model in (9) ensures personalized regret guarantees for each client with respect to its best model in hindsight, it also enables clients to leverage federated learning, thus enjoying sublinear regret in comparison to the best global model in hindsight.Comparison with Federated and Local Models. The main advantage of using the ensemble model instead of the federated model lies in Theorem 2, where it is shown that the ensemble model can attain the global regret guarantee provided by the federated model while employing the federated model, achieving the personalized regret guarantee in (12) is not feasible. However, according to ( 6) and (7) , using the online local training, each client achieves sublinear regret with respect to its best model while all clients achieve sublinear regret with respect to the best global model in hindsight. This efficacy of local training stems from clients adapting the model to their individual data. In contrast, in federated learning, the model is trained on all data samples across clients, potentially leading to higher generalizability compared to its local counterpart. If there are similarities in the distribution of data samples among clients over time, the federated model is anticipated to achieve greater accuracy in online prediction. However, due to the lack of available information regarding the relationships between data samples observed by clients before online prediction, these advantages may not be reflected in theoretical bounds. The proposed method constructs an ensemble to harness the advantages of both federated and local models for online prediction, as evidenced by Theorem 2. Experimental results in Subsection in 5.1 confirm that the ensemble model achieves superior performance compared to both local and federated models.Regret guarantees for the ensemble method, as outlined in Theorem 2, are contingent upon the model's convexity. However, if the model is non-convex, achieving such guarantees may not be feasible. Particularly, non-convex models such as neural networks are susceptible to the forgetting process [51, 45] , wherein applying online gradient descent may lead to overfitting to recently observed data samples. This section introduces a novel algorithm that allows clients to make online predictions using non-convex models while simultaneously collaborating to fine-tune the model. The scenario assumes the existence of a pre-trained model, and the objective for clients is two-fold: to make real-time predictions and to refine the model for alignment with their preferences. This situation may arise, for instance, in fine-tuning large foundation models to tailor them to client preferences.Let the server and clients collaborate to fine-tune the non-convex model f (•; •). At each time step t, client i updates the model on the batch of recently observed samples with size b asEQUATIONThen the server aggregates locally updated parameters and updates the federated model parameter as in (4). Furthermore, each clients learns its own local model by fine-tuning the pre-trained model locally via online gradient descent as in ( 8) on the batch of b recently observed samples. While online gradient descent methods are well-known for their efficiency in handling dynamic environments, employing the update rule of ( 13) for non-convex models may lead to overfitting to recently observed batches. To mitigate potential forgetting, the server saves the federated model parameters every n time step, where n is an integer hyperparameter.Let D t represent the set of model parameters stored by the server at time step t. At time step τ = (j -1)n + 1, the server adds θ τ to D τ meaning that D τ +1 = D τ ∪ {θ τ }. Let ρ j denotes the j-th model parameter in D t . It can be concluded that ρ j = θ (j-1)n+1 . The server continues saving model parameters every n time steps until time step U ≤ T . Client i assigns a weight w ij,t to the j-th model in D t , which assesses the credibility of the predictions given by the model parameter ρ j . Client i initializes w ij,1 = 1 Input: Model f (•; •), batch size b, n, U , D i,1 = ∅. 2: for t = 1, . . . , T do 3:The server transmits θ t to all clients.4:for all i ∈ [N ], client i do 5:Performs model selection given D t according to Algorithm 1 to obtain M i,t .6:Makes prediction fi,t (x i,t ) as in (15) using chosen model set M i,t .Upon receiving y i,t , updates α i,t , β i,t , γ i,t , δ i,t and {w ij,t } |Dt| j=1 as in ( 10), ( 16) and ( 17).Updates the local model asϕ i,t+1 = ϕ i,t -η b t τ =t-b ∇L(f (x i,t ; ϕ i,τ), y i,t ).Updates the federated model asψ i,t+1 = θ t -η b t τ =t-b ∇L(f (x i,τ ; θ t ), y i,τ ).10:Sends ψ i,t+1 to the server.11:end for 12:if t ≤ U and t mod n = 0 then 13:The server updates D t as D t+1 = D t ∪ {θ t }.14:end if 15:The server updates θ t as θ t+1 = 1 N N i=1 ψ i,t+1 . 16: end for client i downloads all models in M i,t from the server. Upon receiving the models, client i constructs an ensemble model fi,t (•) asEQUATIONClient i makes the prediction for x i,t asEQUATIONwhere f i,t (x i,t ) is the ensemble of local and federated models as defined in (9) . Furthermore, γ i,t and δ i,t are weights assigned by client i to ensemble models f i,t (•) and fi,t (•), respectively. Upon observing the label y i,t after prediction, client i updates weights γ i,t and δ i,t asEQUATIONEQUATIONFurthermore, α i,t and β i,t which are used to construct f i,t (x i,t ) in ( 9) are updated as in (10) . The weight w ij,t is updated using the importance sampling loss asEQUATIONwhere 1 j∈Mi,t denotes the indicator function and is 1 if j ∈ M i,t . Moreover, q ij,t is the probability that client i selects the j-th model in D t to be in M i,t and can be expressed asq ij,t = 1 -(1 -p ij,t ) Mwhere p ij,t is defined in step 3 of Algorithm 1. The proposed algorithm, named Federated Learning with Personalized Online Ensemble (Fed-POE) is summarized in Algorithm 2. It is useful to note that the proposed method in Subsection 4.1 is a special case of Fed-POE by setting M = 0 and b = 1.Efficiency of Fed-POE. To perform model selection using Fed-POE, clients do not need to store all model parameters in D t for all t ∈ [T ]. Instead, the server, which has higher storage capacity than the clients, stores all model parameters, and clients download a subset of at most M model parameters.The hyperparameter M can be chosen such that clients can handle the memory and computational requirements of making predictions with the selected subset of models. Let C F denote the number of computations required to fine-tune the model f , and let C I represent the number of computations required to make an inference with model f . Assume that the complexity of model selection in Algorithm 1 is negligible compared to fine-tuning and making inferences with model f . According to Algorithm 2, each client performs 2C F + (M + 2)C I computations per time step. Therefore, the computational complexity of Fed-POE for each client is O(C F + M C I ). Beyond memory and computational considerations, selecting a subset of models from D t helps clients improve their prediction accuracy. Specifically, using the model weights {w ij,t } at time step t, client i selects models that perform better on its data while pruning those with lower performance. Let h j (x i,t ) = f (x i,t ; ρ j ) denote the model associated with the j-th model parameter in D t . Furthermore, h loc (x i,t ) = f (x i,t ; ϕ i,t ) and h fed (x i,t ) = f (x i,t ; θ t ) represent the local and federated models, respectively, . Let the set of models H be defined asH := {h j | ∀j : 1 ≤ j ≤ |D T |} ∪ {h loc , h fed }.This set H includes all models that can be employed by each client using Fed-POE. The best model in hindsight h * and the best model in hindsight for client i, h * i are defined asEQUATIONEQUATION)The following theorem provides the regret upper bound of Fed-POE. Theorem 3. Under assumption A3, employing Fed-POE in Algorithm 2, the expected global regret of all clients is bounded from above asEQUATIONwhile client i achieves the following expected personalized regret upper bound:EQUATION)The expectation is taken with respect to randomization in model selection.The proof of Theorem 3 can be found in Appendix C. According to (19) and (20) , settingη c = O(1/ √ T ), n = O( √ T ) and U = O( √ T ), both the personalized and global regrets of clients achieve sublinear regret of O( √ T ). Since clients construct their ensemble models using a timevarying subset of models, employing existing model selection and ensemble learning approaches [13, 15, 39, 46, 20, 41, 21] may not ensure the sublinear regrets stated in Theorem 3. However, by using the proposed Algorithm 1, Fed-POE guarantees sublinear regret bounds while allowing clients the flexibility to select time-varying and personalized subsets of models for their ensembles. 
experiments: 
 The present section studies the performance of Fed-POE in Algorithm 2 compared to other baselines. Experiments are conducted on both image classification and regression tasks. The performance of federated learning is examined in both convex and non-convex cases. Codes are available at https://github.com/pouyamghari/Fed-POE.The performance of the proposed Fed-POE is evaluated on online regression tasks. For these tasks, clients and the server collaborate to train a random feature kernel-based model, which is known to be convex [25, 48, 18] . Details about the random feature kernel-based model used in the experiments can be found in Appendix D. The performance of Fed-POE is compared with a baseline called Local, where clients train their models locally without participating in federated learning. Additionally, Fed-POE is compared to personalized federated learning baselines Ditto [30] and Fed-Rep [8] , the online federated learning baseline Fed-OMD [37] , and online federated kernel learning baselines eM-KOFL [26] and POF-MKL [19] . Mean square error (MSE) is used as the metric to evaluate the performance of algorithms on regression task. MSE for client i can be expressed as MSE i = 1 T T t=1 (ŷ ij,t -y i,t ) 2 where ŷij,t denotes the prediction of client i at time step t. The performance of algorithms are tested on two regression datasets Air [58] and WEC [40] . Air dataset is a time-series dataset that each data sample contains air quality features and the goal is to predict the concentration of contamination in the air. Each sample in WEC dataset contains features of different wave energy converters and the goal is to predict power output. Data samples are distributed non-i.i.d among 400 clients. Time horizon T for both datasets is 250. More information about datasets and distributed data among clients is presented in Appendix D.Table 1 : Average MSE (×10 -3 ) and its standard deviation (×10 -3 ) across clients for online regression on Air and WEC datasets. 1 shows that the performance of Fed-POE relative to other baselines is more consistent across different datasets.The performance of the proposed Fed-POE on an image classification task is compared with Local, Ditto [30] , Fed-Rep [8] , Fed-OMD [37] , Fed-ALA [57] , and Fed-DS [35] . Fed-ALA [57] is a personalized federated learning model suitable for deep neural networks, while Fed-DS [35] is a federated learning algorithm designed to handle data streams. Experiments are conducted on CIFAR-10 [28] and Fashion MNIST (FMNIST) [55] datasets. CIFAR-10 and FMNIST contain 60, 000 and 70, 000 images. For both CIFAR-10 and FMNIST, a CNN with a VGG architecture [49] , consisting of two blocks, is pre-trained on a subset of training samples from each dataset. The training datasets are biased towards class 0. More details about training the CNNs can be found in Appendix D. For both the CIFAR-10 and FMNIST datasets, 10, 000 test samples are sequentially received by clients. There are 20 clients in total, and the data samples are distributed non-i.i.d. among them. For CIFAR-10, each client is biased towards one specific class, with 55% of the samples belonging to that class and 5% of the samples belonging to each of the other 9 classes. For FMNIST, each client is biased towards two classes, and the distribution of samples is time-variant. More details about the data distribution among clients and experimental setup can be found in Appendix D. At each time step, all algorithms employ batch of size 10 for model update. The learning rates for all algorithms are set to η = 0.01/T and η c = 1/ √ T . The server stores models every n = 20 time steps. The metric to evaluate the performance of algorithms is the accuracy. The accuracy for client i is defined as Accuracy i = 1 T T t=1 1 ŷi,t=yi,t where ŷi,t denotes the label predicted by client i at time step t. that Fed-POE efficiently leverages the advantages of both federated and local models. To analyze the effect of the number of models M and batch size b on the Fed-POE performance, experiments are conducted on the CIFAR-10 dataset, varying the batch size b and the number of models M selected by each client to construct the ensemble model. As observed in Table 3 , the batch size b = 1 results in the worst accuracy, mainly due to the forgetting process where models overfit to the most recently observed data. However, increasing the batch size from b = 10 or b = 20 to b = 30 does not significantly improve the accuracy. Larger batch sizes may lead the model to perform better on older data, as the model is trained on older data over more iterations. Therefore, this study concludes that a moderate batch size is optimal, considering that increasing the batch size also increases computational complexity. Table 4 in Appendix D presents the accuracy of Fed-POE on both the CIFAR-10 and FMNIST datasets with varying values of M . The table shows that employing the saved models by the server in D t improves performance, as setting M = 0 results in the worst accuracy. Moreover, it can be observed that increasing M does not necessarily lead to further accuracy improvement. This aligns with the intuition behind selecting a subset of models rather than using all models. 
conclusions: 
 This paper proposed Fed-POE, a personalized federated learning algorithm designed for online prediction and model fine-tuning. Fed-POE constructs an ensemble using local models and federated models stored by the server periodically over time. Theoretical analysis demonstrated that Fed-POE achieves sublinear regret. Experimental results revealed that the relative performance of local models compared to federated models depends on the dataset, making the decision between local model training and federated learning challenging. However, experimental results also show that Fed-POE consistently outperforms both local and federated models across all datasets. This indicates that Fed-POE effectively leverages the advantages of both local and federated models. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We believe that the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In Section 5 and Appendix D, we properly cited the original owners of assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This section provides the proof of Theorem 1. The proof follows similar steps to those in [37] , and it is included here for the sake of completeness and to make the paper self-contained.According to (3) and ( 4), for any θ it can be written thatEQUATIONDue to convexity of L(•, •) it can be concluded thatEQUATION)Combining ( 21) with ( 22), we getEQUATIONUsing assumption A2 and Arithmetic Mean Geometric Mean (AM-GM) inequality it can be written thatEQUATION)Combining ( 23) with (24) , we arrive atEQUATIONDividing both sides by 2η N yieldsEQUATIONSumming ( 26) over time, we obtainEQUATIONPlugging in θ = θ * and θ 0 = 0 into (27) and considering the fact that ∥θ T +1 -θ∥ 2 ≥ 0, we findEQUATION)which proves the Theorem.According to (10), we can writeEQUATIONUsing the inequality e -x ≤ 1 -x + 1 2 x 2 , ∀x ≥ 0, from (29) we arrive atEQUATION)Taking the logarithm from both sides of (30) and using the inequality 1 + x ≤ e x , we getEQUATION)Considering the assumption that 0 ≤ L(f (x; θ), y) ≤ 1, ∀x, θ, (31) can be relaxed toEQUATIONSumming (32) over time, we obtainEQUATION)According to Hölder's inequality, for any positive real numbers p and q satisfying 1 p + 1 q = 1, the following inequality holds:EQUATIONTo meet the condition 1 p + 1 q = 1, it is necessary that p ≥ 1 and q ≥ 1. Consequently, (34) can be modified to:EQUATION)Considering the fact that α i,1 = β i,1 = 1, based on (35) we can writeEQUATIONAccording to the update rule in ( 29), ( 36) is equivalent toEQUATIONCombining (33) with (37) we arrive atEQUATION)Due to the convexity of L(•, •), we can writeEQUATIONCombining (38) with (39) we getEQUATIONSubstituting p = ∞ and q = 1 in (40), we obtainEQUATIONSince Fed-POE updates ϕ i,t locally using online gradient descent as outlined in (8), according to (6) , for any ϕ we can writeEQUATION)Substituting ϕ * i in (42) along with combining (41) with (42) , we can conclude thatEQUATIONwhich proves the personalized regret upper bound of Fed-POE in (12) . Furthermore, plunging in p = 1 and q = ∞ into (40), we getEQUATION)Summing ( 44) over all clients, we obtainEQUATION)Combining ( 45) with (28) , we arrive atEQUATIONwhich proves the Theorem.According to assumption A3 that 0 ≤ L(f (x; θ), y) ≤ 1, it can be written thatEQUATION)Let ℓ ij,t denote the importance sampling loss estimate, which is expressed asEQUATIONLet the total number of model parameters stored by the server after time step U is D and W i,t = D j=1 w ij,t . For any t > U , considering ( 17), we can writeEQUATION)Employing the inequality e -x ≤ 1 -x + 1 2 x 2 , ∀x ≥ 0, from ( 49) we obtainEQUATION)Taking the logarithm from both sides of (50) and using the inequality 1 + x ≤ e x , we arrive atEQUATIONSumming ( 51) over time, we getEQUATION)Considering the fact that the weights {w ij,t } D j=1 are initialized as w ij,1 = 1, ∀j ∈ [D], it can concluded that W i,U ≤ D. Theefore, for any j ∈ [D], the left hand side of ( 52) is bounded from below asEQUATIONCombining ( 53) with ( 52) yieldsEQUATIONThe expected values of ℓ ij,t and ℓ 2 ij,t given observed losses up until time step t can be obtained asEQUATIONTaking the expectation from (54) , we arrive atEQUATIONSinceq ij,t = 1 -(1 -p ij,t ) M = p ij,t (1 + (1 -p ij,t ) + (1 -p ij,t ) 2 + . . . + (1 -p ij,t ) M -1), it can be concluded that q ij,t ≥ p ij,t . Therefore, ( 56) can be relaxed toEQUATIONAccording to model selection procedure adopted by Fed-POE presented in Algorithm 1, client i chooses a subset of models by sampling them in M rounds with replacement. Let a ij,t ≥ 0 denote the number of times that the model j in D t is chosen by client i at time step t. The number of different situations for selected subset of models M i,t is equal to the number of solutions for the linear equationEQUATION)Let A denote the set of all possible solutions for ( 58) such that if a ∈ A where a = [a 1 , . . . , a D ], a 1 , . . . , a D satisfies (58) . Therefore, for the expected loss of the ensemble fi,t (x i,t ) in ( 14), we can writeEQUATIONwhere f (k) i,t (x i,t ) denote the k-th possible ensemble model generated by client i using Fed-POE. Using the Jensen inequality and convexity of the loss function, we can relax (59) toEQUATIONwhereM (k) i,t and W (k)i,t are the k-th possible model subset and weight summations, respectively. Rearranging the right hand side of (60), we can writeEQUATIONwhere B j is the set of all possible solutions for the linear equation in ( 58) condition on a ij,t ≥ 1.Since for any k, we have w ij,t ≤ W (k) i,t , (61) can be relaxed toEQUATION) Since |Bj | k=1 D m=1 p b m,k im,tincludes all possibilities in B j , we can conclude that|Bj | k=1 D m=1 p b m,k im,t = 1.Combining this with (62), we obtainEQUATION)Combining ( 63) with (57) , we arrive atEQUATION)Since 0 ≤ L(f (x; θ), y) ≤ 1, it can be written thatEQUATION)Combining ( 65) with (64), we getEQUATION)Since fi,t (•) similar to f i,t (•) is the ensemble of two models, following the same derivation steps from ( 29) to ( 41) by substituting fi,t (x i,t ), f i,t (x i,t ) and fi,t (x i,t ) with f i,t (x i,t ), f (x i,t ; θ t ) and f (x i,t ; ϕ i,t ), respectively, we can conclude thatEQUATIONEQUATION)Taking the expectation from both sides of (67b) with respect to randomization in model selection along with combining (67b) with (66) yieldsEQUATIONFurthermore, combining (67a) with ( 41) and ( 44) and taking the expectation with respect to model selection randomization, we obtainEQUATIONEQUATIONRecall that h j (•) associated with the j-th model parameter in D t be defined as h j (x i,t ) = f (x i,t ; ρ j ) while h loc (•) and h fed (•) correspond to the local and federated models, respectively, defined as h loc (x i,t ) = f (x i,t ; ϕ i,t ) and h fed (x i,t ) = f (x i,t ; θ t ). Also recall that H := {h j | ∀j : 1 ≤ j ≤ |D T |} ∪ {h loc , h fed }. Comparing the right hand sides of (69a) and (69b) with that of (68) and considering the fact that D ≥ 2, for any h ∈ H we can writeEQUATIONBy substituting h(•) with h * i (•) as defined in (18b) and considering the fact that D ≤ U/n, we obtain the personalized regret upper bound for client i as shown in (20) . Moreover, taking the average of (70) across clients and substituting h(•) with h * (•), we arrive atEQUATIONwhich proves the theorem.This section presents supplementary experimental results and details about experimental setup. All experiments were carried out using Intel(R) Core(TM) i7-10510U CPU @ 1.80 GHz 2.30 GHz processor with a 64-bit Windows operating system. To implement Fed-OMD for both regression and image classification, we used the ℓ 2 -norm as a regularizer function for mirror descent. For implementing Ditto for both regression and image classification, we set the regularization factor λ to 1. In the case of image classification using Fed-Rep, clients locally fine-tune the last two layers of the CNN model, while the rest of the network is used as the global backbone to generate representations. Furthermore, Fed-POE is compatible with any federated learning method and can utilize any federated algorithm. For CIFAR-10, Fed-POE uses Fed-OMD, while for FMNIST, Fed-POE uses Fed-Rep. To fine-tune the CNN model, Fed-POE and all baselines use the SGD optimizer and the cross-entropy loss function.Table 4 presents the accuracy of clients for image classification using Fed-POE with varying values of M . As can be seen for both CIFAR-10 and FMNIST, when M > 0, the accuracy is higher than in the case where M = 0. The case where M = 0 corresponds to clients not using models stored by the server in their ensemble. Therefore, these results show that constructing the ensemble using previous federated model parameters stored by the server improves the accuracy of Fed-POE. This indicates the effectiveness of the model selection procedure of Fed-POE presented in Algorithm 1. Additionally, these results show that increasing M does not necessarily lead to further accuracy improvement. This implies the effectiveness of Fed-POE's model selection in pruning model parameters from the ensemble that have relatively lower accuracy. Figure 1 illustrates the average cumulative global regret of clients over time using Fed-POE and all other baselines. As depicted, Fed-POE achieves sublinear regret for both CIFAR-10 and WEC datasets. This corroborates the theoretical results in Theorems 2 and 3.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: This paper studies the problem of personalized federated learning. While personalized federated learning, in general, has societal impact, we do not foresee any specific societal impact resulting from our work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper proposes a personalized federated learning algorithm, Fed-POE, which is designed for adaptive prediction and model fine-tuning in dynamic environments, based on the provocative claim that federated learning is not always better than locally-trained models. It is established on some sound theoretical analysis, and also evaluated to show robust performance in experiments. In the final version, we hope that the authors can improve the presentation as suggested by some reviewers, and incorporate more helpful details in the rebuttal into the main paper.

Reviews: review: paper_summary: paper_summary: This paper introduces Fed-POE, a novel personalized federated learning algorithm tailored for online prediction and model fine-tuning. Fed-POE creates an ensemble by integrating local models with those periodically contributed by the server over time. Theoretical analysis confirms that Fed-POE attains sublinear regret. Empirical results demonstrate that Fed-POE consistently surpasses the performance of both local and federated models across all evaluated datasets, which indicates that Fed-POE effectively leverages the advantages of both local and federated models.
strength_weakness: strength_weakness: My primary concern is that the assertion the proposed algorithm can effectively harness the combined advantages of federated and local models is not clearly demonstrated within the theoretical bounds. The paper presents two principal theoretical results: Theorem 2 provides the regret upper bound for the proposed algorithm in convex scenarios, while Theorem 3 addresses non-convex cases. Both theorems establish sublinear regret bounds that are consistent with those for federated learning using a straightforward online gradient descent approach.  I recommend enhancing the clarity of the proposed method's advantages in the theorems by incorporating assumptions about the data distributions.
questions: questions: See weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces an interesting perspective about the role of ensembles of models in federated learning. The provocative claim is the fact that federated learning is not always better than locally-trained models. This is contextualized in the field of not IID data and time-varying data generating processes. To address this issue the paper introduces from the theoretical point of view how to quantify the regret in federated and locally trained models. In addition it includes an analysis of non-convex models by managing an history of models. The overall impression about the paper is positive even if some points could have been better explored (in particular the part related to not IID data that is somehow the core of the paper).
strength_weakness: strength_weakness: - The federated models somehow includes the locally-trained model. I would have appreciated a further analysis about the fact that the two "sides" of the average model are related each other. 
- The setting in which eta and eta_c scales with T prevents adaptation in the long run (which is somehow the core of the paper). How to deal with that?
- Federated learning typically takes also into account the complexity of the learning phase (i.e., the amount of info to be transmitted, e.g., the models). This is not quantified here. And this could be also a weak point in the fed-poe algorithm.
questions: questions: See Weaknesses box
limitations: limitations: See Weaknesses box
paper_summary: paper_summary: This paper proposes a novel personalized federated learning algorithm, Fed-POE, which is designed for adaptive prediction and model fine-tuning in dynamic environments. It addresses the challenge of real-time predictions on streaming data by constructing a personalized model that combines a locally fine-tuned model with multiple federated models. Theoretical analysis and experiments on real datasets demonstrate its effectiveness in achieving sublinear regret bounds and improved online prediction accuracy.
strength_weakness: strength_weakness: 1. Although the presented method is novel, it is simply a combination of the previous personalized federated learning approaches as well as ensemble learning and provides comparably little conceptual originality.  The contribution's main novelty seems to be that integrating results from prior models would be beneficial in mitigating catastrophic forgetting in online federated learning.

2. Experimental results show that the improvement in the accuracy of Fed-POE compared to other methods is not significant, but ensemble learning inevitably increases the computational overhead increase. The paper needs to analyze whether this trade-off is reasonable.

3. The paper needs more experiments to prove the effectiveness of the method, for example, for real-time predictions, the size of the old data replay is crucial, and the authors should design experiments to analyze the effect of batch size b on the experimental results. This paper also needs experiment results on the accuracy over the time step.
questions: questions: 1. The method in this paper does not significantly improve accuracy and even has a larger standard deviation. Can you give me more reasons to support your methods？

2. The method is designed with two parts to mitigate catastrophic forgetting (old data replay and integration of multiple old models), The complex model updating process is unreasonable for real-time prediction. Can you design more ablation experiments to analyze these two parts？
limitations: limitations: see the weaknesses and questions
paper_summary: paper_summary: This paper introduces a personalized federated learning algorithm to address the challenges of real-time predictions in non-stationary environments. Clients fine-tune models online, combining their locally fine-tuned models with multiple federated models learned over time. This approach ensures efficient adaptation to evolving data streams, with theoretical analysis and experiments on real datasets demonstrating its effectiveness.
strength_weakness: strength_weakness: 1. Contributions are suggested to list by items for clear summaries.
2. The baselines in Table 1 are all before the 2022 year, more latest related methods published in 2023 should be compared.
3. Fed-POE has limited improvements on Air and FMNIST datasets.
4. The process of combining locally fine-tuned models with multiple federated models may introduce significant computational overhead for clients, especially those with limited resources.
5. As the number of clients increases, managing and integrating multiple personalized models can become complex, posing scalability challenges for the proposed algorithm.
questions: questions: No
limitations: limitations: Yes

Comments: comment: Thank you very much for your feedback. We would be happy to address any further concerns or questions you may have.
Thanks for the authors' detailed feedback. In light of these explanations, I will revise my score accordingly.