Towards Federated Learning on Time-Evolving Heterogeneous Data

1 INTRODUCTION . Federated Learning ( FL ) has recently emerged as a critical distributed machine learning paradigm to preserve user/client privacy . Clients engaged in the training process of FL only communicate their local model parameters , rather than their private local data , with the central server . As the workhorse algorithm in FL , FedAvg ( McMahan et al. , 2017 ) performs multiple local stochastic gradient descent ( SGD ) updates on the available clients before communicating with the server . Despite its success , FedAvg suffers from the large heterogeneity ( non-iid-ness ) in the data presented on the different clients , causing drift in each client ’ s updates and resulting in slow and unstable convergence ( Karimireddy et al. , 2020b ) . To address this issue , a new line of study has been suggested lately that either simulates the distribution of the whole dataset using preassigned weights of clients ( Wang et al. , 2020 ; Reisizadeh et al. , 2020 ; Mohri et al. , 2019 ; Li et al. , 2020a ) or adopts variance reduction methods ( Karimireddy et al. , 2020b ; a ; Das et al. , 2020 ; Haddadpour et al. , 2021 ) . However , the FL formulation in these approaches always assumes a fixed data distribution among clients throughout all training rounds , while in practice this assumption does not always hold because of the complex , uncontrolled , and unpredictable client behaviors . Local datasets , for example , often vary over time , and intermittent clients can join or depart during training without prior notice . The difficulty in addressing the time-varying heterogeneity of FL lies in the stateless nature of the local datasets , which includes the unpredictable future datasets and the impossibility of retaining all prior local datasets . To this end , we first provide a novel Continual Federated Learning ( CFL ) formulation and then propose a unified CFL framework as our solution . This framework encompasses a variety of design choices for approximating the local objective functions in the previous training rounds , with the difference between actual and estimated local object functions described as information loss for further analysis . To capture the time-varying data heterogeneity in the CFL framework , we expand the theoretical assumption on the client drift—which has been extensively utilized recently in Karimireddy et al . ( 2020b ) ; Khaled et al . ( 2020 ) ; Li et al . ( 2020b ) —to include both client drift and time drift . This allows us to quantify the difference between local client objective function and global objective function for the considered time-varying heterogeneous clients . We provide convergence rates for our unified CFL framework in conjunction with information loss and new models of both client and time drifts . Our analysis reveals a faster convergence of CFL framework than FedAvg , with the benefit dependent on approximation quality . The rate of FedAvg obtained from our framework is consistent with previous work , while a similarly simplified rate on Continual Learning ( CL ) ( French , 1999 ; Kirkpatrick et al. , 2017 ) for the convex and strongly convex cases is novel . Finally , in extensive empirical results , we demonstrate that CFL methods—stemmed from the CFL framework with different approximation techniques—significantly outperform the SOTA FL competitors on various realistic datasets . These numerical observations corroborate our theoretical findings . Contribution . We summarize our key contributions below . • We present a unified framework , termed Continual Federated Learning ( CFL ) , together with a novel client and time drift modeling approach , to capture complex FL scenarios involving timevarying heterogeneous data . This is the first theoretical study , to our knowledge , that describes the time-varying nature of FL . • We provide rigorous convergence analysis for the CFL methods . Our theoretical analysis explains the faster and more stabilized optimization of the CFL methods over that of FedAvg , and conjecture their variance reduction effect . In addition , we provide tight convergence rates for standalone CL methods on convex and strongly-convex problems : to the best of our knowledge , we are the first to provide such guarantees for CL on SGD . • We examine several approximation techniques for CFL framework : the insights therein offer a valuable practical guideline . We demonstrate the efficacy and necessity of CFL methods over the SOTA FL baselines across a range of time-varying heterogeneous data scenarios and datasets . 2 RELATED WORK . Federated Learning . FedAvg ( McMahan et al. , 2017 ; Lin et al. , 2020b ) is the de facto standard FL algorithm , in which multiple local SGD steps are executed on the available clients to alleviate the communication bottleneck . While communication efficient , heterogeneity , such as system heterogeneity ( Li et al. , 2018 ; Wang et al. , 2020 ; Mitra et al. , 2021 ; Diao et al. , 2021 ) and statistical/objective heterogeneity ( Li et al. , 2018 ; Wang et al. , 2020 ; Mitra et al. , 2021 ; Lin et al. , 2020a ; Karimireddy et al. , 2020b ; a ) , results in inconsistent optimization objectives and drifted clients models , impeding federated optimization considerably . A line of work has been proposed to address the heterogeneity in FL . FedProx ( Li et al. , 2018 ) adds the regularization term on the distance of local and global models when performing local training—similar formulations can be found in other recent FL works ( Hanzely & Richtárik , 2020 ; Dinh et al. , 2020 ; Li et al. , 2021 ) for various purposes . To address the issue of objective heterogeneity e.g . caused by heterogeneous data , works like SCAFFOLD ( Karimireddy et al. , 2020b ; Mitra et al. , 2021 ) introduce the idea of variance reduction on the client local update steps . FedNova ( Wang et al. , 2020 ) further proposes a general framework for unifying FedAvg and FedProx , and argues that while averaging , local updates should be normalized to minimize heterogeneity induced by different number of local update steps . However , most of these prior works focus on the fixed heterogeneity across clients and throughout the entire optimization procedure ; we instead consider the novel scenario with time-varying data heterogeneity . The theoretical study on the convergence of FedAvg can date back to the parallel SGD analysis on the identical functions Zinkevich et al . ( 2010 ) and recently is improved by Stich ( 2019 ) ; Stich & Karimireddy ( 2020 ) ; Patel & Dieuleveut ( 2019 ) ; Khaled et al . ( 2020 ) ; Woodworth et al . ( 2020b ) . For the analysis of heterogeneous data , Li et al . ( 2020b ) first give the convergence rate of FedAvg on non-iid datasets with random selection , assuming that the client optimum are -close . Woodworth et al . ( 2020a ) ; Khaled et al . ( 2020 ) give tighter convergence rates under the assumption of bounded gradient drift . All above works give a O ( 1/T ) convergence rate for convex local objective functions . More recently , Karimireddy et al . ( 2020b ) ; Koloskova et al . ( 2020 ) give the convergence analysis of local SGD for non-convex objective functions under bounded gradient noise assumptions , and obtain a O ( 1/ √ T ) convergence rate . Our theoretical analysis framework covers more challenging timevarying data heterogeneity in FL , which has not been considered in the community yet—our rate can be simplified to the standard FL scenario , matching the tight analysis in Karimireddy et al . ( 2020b ) . Continual Learning . Continual learning , also known as incremental learning or lifelong learning , aims to learn from ( time-varying ) sequential data while avoiding the problem of catastrophic forgetting ( French , 1999 ; Kirkpatrick et al. , 2017 ) . There exists a large amount of works , from the perspectives of regularization ( Kirkpatrick et al. , 2017 ; Li & Hoiem , 2017 ; Zenke et al. , 2017 ) , experience replay ( Castro et al. , 2018 ; Rebuffi et al. , 2017 ) , and dynamic architectures ( Maltoni & Lomonaco , 2019 ; Rusu et al. , 2016 ) . In this paper , we examine both regularization and experience replay based methods , and compare their empirical performance in depth . Despite the empirical success , the theoretical analysis of CL is limited : only a very recent preprint ( Yin et al. , 2020b ) provides a viewpoint of regularization-based continual learning , and only for the single worker scenario . In this work , we provide tight convergence analysis for both CL and CFL on SGD . Continual Federated Learning . To our knowledge , the scenario of CFL was originally described in Bui et al . ( 2018 ) in order to federated train Bayesian Neural Network and continually learn for Gaussian Process models—it is orthogonal to our optimization aspect in this paper . FedCurv ( Shoham et al. , 2019 ) blends EWC ( Kirkpatrick et al. , 2017 ) regularization with FedAvg , and empirically shows a faster convergence . FedWeIT ( Yoon et al. , 2021 ) extends the regularization idea , with a focus on selective inter-client knowledge transfer for task-adaptive parameters . In a very recent parallel ( and empirical ) work , CDA-FedAvg ( Casado et al. , 2021 ) uses a locally maintained long-term data samples , with asynchronous communication ; however , this design choice lacks sufficient numerical results to justify the performance gain ( even over FedAvg ) and it can not be applied to more general scenarios . Our theoretically sound CFL framework covers the regularization component of FedCurv and the core set part of CDA-FedAvg , and is orthogonal to the neural architecture manipulation idea in FedWeIT . 3 CONTINUAL FEDERATED LEARNING FRAMEWORK . 3.1 FORMULATION . Conventional FL Formulation . The standard FL typically considers a sum-structured distributed optimization problem as below : f ? = minω∈Rd [ f ( ω ) : = ∑N i=1 pifi ( ω ) ] , ( 1 ) where the objective function f ( ω ) : Rd → R is the weighted sum of the local objective functions fi ( ω ) : = EDi [ Fi ( ω ) ] of N nodes/clients , and pi is the weight of client i . In practice , it may be infeasible to select all clients each round , especially for cross-device setup ( McMahan et al. , 2017 ; Kairouz et al. , 2019 ) . The standard FedAvg then randomly selects S clients to receive the model parameters from the server ( S ≤ N ) in each communication round and performs K local SGD update steps in the form of ωt , i , k = ωt , i , k−1 − ηl ( ∇fi ( ωt , i , k−1 ) + νt , i , k−1 ) , with local step-size ηl and gradient noise νt , i , k−1 . The selected clients then communicate the updates ∆ωt , i = ωt , i , K − ωt with the server for the model aggregation : ωt+1 = ωt − ηgS ∑S i=1 ∆ωt , i . Continual FL Formulation . Despite its wide usage , the standard FL formulation given in Equation ( 1 ) can not properly reflect actual time-varying scenarios , such as local datasets changing over time or intermittent clients joining or leaving during training . To address this problem , we propose the Continual Federated Learning ( CFL ) formulation : f ? = minω∈Rd [ f ( ω ) : = ∑T t=1 ∑ i∈St pt , ift , i ( ω ) ] , ( CFL ) where ft , i ( ω ) represents the local objective function of client i at time t. St is a subset of clients sampled from all clients set Ω , where |St| = S. For the time-varying scenarios , client i could have different local objective functions ft , i ( ω ) due to the changing local datasets on different t .

Summary: The paper presents continual federated learning (CFL) to address the issue that data at clients in FL may vary over time. The main objective of CFL is to optimize the time-aggregated objective across multiple clients. While historical objective functions are usually not available, the paper considers a surrogate objective that approximates the original historical objective. A theoretical analysis is presented that takes into account the approximation error (referred to as information loss in the paper) and the time drift in local objective functions. Experiments were also conducted that show core set methods to perform well.