Towards Federated Learning on Time-Evolving Heterogeneous Data

1 INTRODUCTION . Federated Learning ( FL ) has recently emerged as a critical distributed machine learning paradigm to preserve user/client privacy . Clients engaged in the training process of FL only communicate their local model parameters , rather than their private local data , with the central server . As the workhorse algorithm in FL , FedAvg ( McMahan et al. , 2017 ) performs multiple local stochastic gradient descent ( SGD ) updates on the available clients before communicating with the server . Despite its success , FedAvg suffers from the large heterogeneity ( non-iid-ness ) in the data presented on the different clients , causing drift in each client ’ s updates and resulting in slow and unstable convergence ( Karimireddy et al. , 2020b ) . To address this issue , a new line of study has been suggested lately that either simulates the distribution of the whole dataset using preassigned weights of clients ( Wang et al. , 2020 ; Reisizadeh et al. , 2020 ; Mohri et al. , 2019 ; Li et al. , 2020a ) or adopts variance reduction methods ( Karimireddy et al. , 2020b ; a ; Das et al. , 2020 ; Haddadpour et al. , 2021 ) . However , the FL formulation in these approaches always assumes a fixed data distribution among clients throughout all training rounds , while in practice this assumption does not always hold because of the complex , uncontrolled , and unpredictable client behaviors . Local datasets , for example , often vary over time , and intermittent clients can join or depart during training without prior notice . The difficulty in addressing the time-varying heterogeneity of FL lies in the stateless nature of the local datasets , which includes the unpredictable future datasets and the impossibility of retaining all prior local datasets . To this end , we first provide a novel Continual Federated Learning ( CFL ) formulation and then propose a unified CFL framework as our solution . This framework encompasses a variety of design choices for approximating the local objective functions in the previous training rounds , with the difference between actual and estimated local object functions described as information loss for further analysis . To capture the time-varying data heterogeneity in the CFL framework , we expand the theoretical assumption on the client drift—which has been extensively utilized recently in Karimireddy et al . ( 2020b ) ; Khaled et al . ( 2020 ) ; Li et al . ( 2020b ) —to include both client drift and time drift . This allows us to quantify the difference between local client objective function and global objective function for the considered time-varying heterogeneous clients . We provide convergence rates for our unified CFL framework in conjunction with information loss and new models of both client and time drifts . Our analysis reveals a faster convergence of CFL framework than FedAvg , with the benefit dependent on approximation quality . The rate of FedAvg obtained from our framework is consistent with previous work , while a similarly simplified rate on Continual Learning ( CL ) ( French , 1999 ; Kirkpatrick et al. , 2017 ) for the convex and strongly convex cases is novel . Finally , in extensive empirical results , we demonstrate that CFL methods—stemmed from the CFL framework with different approximation techniques—significantly outperform the SOTA FL competitors on various realistic datasets . These numerical observations corroborate our theoretical findings . Contribution . We summarize our key contributions below . • We present a unified framework , termed Continual Federated Learning ( CFL ) , together with a novel client and time drift modeling approach , to capture complex FL scenarios involving timevarying heterogeneous data . This is the first theoretical study , to our knowledge , that describes the time-varying nature of FL . • We provide rigorous convergence analysis for the CFL methods . Our theoretical analysis explains the faster and more stabilized optimization of the CFL methods over that of FedAvg , and conjecture their variance reduction effect . In addition , we provide tight convergence rates for standalone CL methods on convex and strongly-convex problems : to the best of our knowledge , we are the first to provide such guarantees for CL on SGD . • We examine several approximation techniques for CFL framework : the insights therein offer a valuable practical guideline . We demonstrate the efficacy and necessity of CFL methods over the SOTA FL baselines across a range of time-varying heterogeneous data scenarios and datasets . 2 RELATED WORK . Federated Learning . FedAvg ( McMahan et al. , 2017 ; Lin et al. , 2020b ) is the de facto standard FL algorithm , in which multiple local SGD steps are executed on the available clients to alleviate the communication bottleneck . While communication efficient , heterogeneity , such as system heterogeneity ( Li et al. , 2018 ; Wang et al. , 2020 ; Mitra et al. , 2021 ; Diao et al. , 2021 ) and statistical/objective heterogeneity ( Li et al. , 2018 ; Wang et al. , 2020 ; Mitra et al. , 2021 ; Lin et al. , 2020a ; Karimireddy et al. , 2020b ; a ) , results in inconsistent optimization objectives and drifted clients models , impeding federated optimization considerably . A line of work has been proposed to address the heterogeneity in FL . FedProx ( Li et al. , 2018 ) adds the regularization term on the distance of local and global models when performing local training—similar formulations can be found in other recent FL works ( Hanzely & Richtárik , 2020 ; Dinh et al. , 2020 ; Li et al. , 2021 ) for various purposes . To address the issue of objective heterogeneity e.g . caused by heterogeneous data , works like SCAFFOLD ( Karimireddy et al. , 2020b ; Mitra et al. , 2021 ) introduce the idea of variance reduction on the client local update steps . FedNova ( Wang et al. , 2020 ) further proposes a general framework for unifying FedAvg and FedProx , and argues that while averaging , local updates should be normalized to minimize heterogeneity induced by different number of local update steps . However , most of these prior works focus on the fixed heterogeneity across clients and throughout the entire optimization procedure ; we instead consider the novel scenario with time-varying data heterogeneity . The theoretical study on the convergence of FedAvg can date back to the parallel SGD analysis on the identical functions Zinkevich et al . ( 2010 ) and recently is improved by Stich ( 2019 ) ; Stich & Karimireddy ( 2020 ) ; Patel & Dieuleveut ( 2019 ) ; Khaled et al . ( 2020 ) ; Woodworth et al . ( 2020b ) . For the analysis of heterogeneous data , Li et al . ( 2020b ) first give the convergence rate of FedAvg on non-iid datasets with random selection , assuming that the client optimum are -close . Woodworth et al . ( 2020a ) ; Khaled et al . ( 2020 ) give tighter convergence rates under the assumption of bounded gradient drift . All above works give a O ( 1/T ) convergence rate for convex local objective functions . More recently , Karimireddy et al . ( 2020b ) ; Koloskova et al . ( 2020 ) give the convergence analysis of local SGD for non-convex objective functions under bounded gradient noise assumptions , and obtain a O ( 1/ √ T ) convergence rate . Our theoretical analysis framework covers more challenging timevarying data heterogeneity in FL , which has not been considered in the community yet—our rate can be simplified to the standard FL scenario , matching the tight analysis in Karimireddy et al . ( 2020b ) . Continual Learning . Continual learning , also known as incremental learning or lifelong learning , aims to learn from ( time-varying ) sequential data while avoiding the problem of catastrophic forgetting ( French , 1999 ; Kirkpatrick et al. , 2017 ) . There exists a large amount of works , from the perspectives of regularization ( Kirkpatrick et al. , 2017 ; Li & Hoiem , 2017 ; Zenke et al. , 2017 ) , experience replay ( Castro et al. , 2018 ; Rebuffi et al. , 2017 ) , and dynamic architectures ( Maltoni & Lomonaco , 2019 ; Rusu et al. , 2016 ) . In this paper , we examine both regularization and experience replay based methods , and compare their empirical performance in depth . Despite the empirical success , the theoretical analysis of CL is limited : only a very recent preprint ( Yin et al. , 2020b ) provides a viewpoint of regularization-based continual learning , and only for the single worker scenario . In this work , we provide tight convergence analysis for both CL and CFL on SGD . Continual Federated Learning . To our knowledge , the scenario of CFL was originally described in Bui et al . ( 2018 ) in order to federated train Bayesian Neural Network and continually learn for Gaussian Process models—it is orthogonal to our optimization aspect in this paper . FedCurv ( Shoham et al. , 2019 ) blends EWC ( Kirkpatrick et al. , 2017 ) regularization with FedAvg , and empirically shows a faster convergence . FedWeIT ( Yoon et al. , 2021 ) extends the regularization idea , with a focus on selective inter-client knowledge transfer for task-adaptive parameters . In a very recent parallel ( and empirical ) work , CDA-FedAvg ( Casado et al. , 2021 ) uses a locally maintained long-term data samples , with asynchronous communication ; however , this design choice lacks sufficient numerical results to justify the performance gain ( even over FedAvg ) and it can not be applied to more general scenarios . Our theoretically sound CFL framework covers the regularization component of FedCurv and the core set part of CDA-FedAvg , and is orthogonal to the neural architecture manipulation idea in FedWeIT . 3 CONTINUAL FEDERATED LEARNING FRAMEWORK . 3.1 FORMULATION . Conventional FL Formulation . The standard FL typically considers a sum-structured distributed optimization problem as below : f ? = minω∈Rd [ f ( ω ) : = ∑N i=1 pifi ( ω ) ] , ( 1 ) where the objective function f ( ω ) : Rd → R is the weighted sum of the local objective functions fi ( ω ) : = EDi [ Fi ( ω ) ] of N nodes/clients , and pi is the weight of client i . In practice , it may be infeasible to select all clients each round , especially for cross-device setup ( McMahan et al. , 2017 ; Kairouz et al. , 2019 ) . The standard FedAvg then randomly selects S clients to receive the model parameters from the server ( S ≤ N ) in each communication round and performs K local SGD update steps in the form of ωt , i , k = ωt , i , k−1 − ηl ( ∇fi ( ωt , i , k−1 ) + νt , i , k−1 ) , with local step-size ηl and gradient noise νt , i , k−1 . The selected clients then communicate the updates ∆ωt , i = ωt , i , K − ωt with the server for the model aggregation : ωt+1 = ωt − ηgS ∑S i=1 ∆ωt , i . Continual FL Formulation . Despite its wide usage , the standard FL formulation given in Equation ( 1 ) can not properly reflect actual time-varying scenarios , such as local datasets changing over time or intermittent clients joining or leaving during training . To address this problem , we propose the Continual Federated Learning ( CFL ) formulation : f ? = minω∈Rd [ f ( ω ) : = ∑T t=1 ∑ i∈St pt , ift , i ( ω ) ] , ( CFL ) where ft , i ( ω ) represents the local objective function of client i at time t. St is a subset of clients sampled from all clients set Ω , where |St| = S. For the time-varying scenarios , client i could have different local objective functions ft , i ( ω ) due to the changing local datasets on different t .

Summary: This paper proposes to study time evolving heterogeneous data, and proposes Continual Federated Learning (CFL) to address this problem. Their analysis achieves this by introducing time-drift to capture data heterogeneity across time. Convergence results are presented followed by numerical results on time-varying and heterogeneous settings.