Anarchic Federated Learning

1 INTRODUCTION . Federated Learning ( FL ) has recently emerged as an important distributed learning framework that leverages numerous workers to collaboratively learn a joint model ( Li et al. , 2019a ; Yang et al. , 2019 ; Kairouz et al. , 2019 ) . Since its inception , FL algorithms have become increasingly powerful and have been able to handle various heterogeneity in data , network environments , worker computing capabilities , etc . Moreover , most of the prevailing FL algorithms ( e.g. , FedAvg ( McMahan et al. , 2016 ) and its variants ( Li et al. , 2018 ; Zhang et al. , 2020b ; Karimireddy et al. , 2020b ; a ; Acar et al. , 2021 ) ) enjoy so-called “ linear speedup effect , ” i.e. , the convergence time of an FL algorithm decreases linearly as the number of workers increases ( Stich , 2018 ; Yu et al. , 2019 ; Wang & Joshi , 2018 ; Khaled et al. , 2019 ; Karimireddy et al. , 2020b ; Yang et al. , 2021 ; Qu et al. , 2020 ) . To achieve these salient features , most of the existing FL algorithms have adopted a server-centric approach , i.e. , the worker behaviors are tightly “ dictated ” by the server . For example , the server in these FL algorithms can i ) determine either all or a subset of workers to participate in each round of FL update ; ii ) fully control the timing for synchronization and whether to accept/reject information sent from the workers ; iii ) precisely specify the algorithmic operations ( e.g. , the number of local steps performed at each worker before communicating with the server ) , etc . Despite achieving strong performance guarantees , such a server-centric approach introduces several limitations . Specifically , these server-centric FL algorithms often implicitly rely on the following assumptions : ( 1 ) each worker is available for training upon the server ’ s request and throughout a complete round ; ( 2 ) all participating workers are willing to execute the same number of local updates and communicate with the server in a synchronous manner following a common clock . Unfortunately , in edge networks where many FL systems are deployed , these assumptions are restrictive or even problematic due to the following reasons . First , many requested edge devices on the worker side may not be available in each round because of , e.g. , communication errors or battery outages . Second , the use of synchronous communication and an identical number of local updates across all workers ignores the fact that worker devices in edge-based FL systems are heterogeneous in computation and communication capabilities . As a result , stragglers ( i.e. , slow workers ) could significantly slow down the training process . To mitigate the straggler effect , various robust FL algorithms have been developed . For example , the server in FedAvg ( McMahan et al. , 2016 ) can simply ignore and drop the information from the stragglers to speedup learning . However , this may lead to other problems such as wasted computation/energy ( Wang et al. , 2019 ) , slower convergence ( Li et al. , 2018 ) , or biased/unfair uses of worker data ( Kairouz et al. , 2019 ) . Moreover , the synchronous nature of the server-centric approaches implies many networking problems ( e.g. , interference between workers , periodic traffic spikes , high complexity in maintaining a network-wide common clock ) . The above limitations of the current server-centric FL approaches motivate us to propose a new paradigm in FL , which we call Anarchic Federated Learning ( AFL ) . In stark contrast to servercentric FL , workers in AFL are completely free of the “ dictation ” from the server . Specifically , each worker has complete freedom to choose when and how long to participate in FL without following any control signals from the server . As a result , the information fed back from workers is inherently asynchronous . Also , each worker can independently determine the number of local update steps to perform in each round based on its current local situation ( e.g. , battery level , communication channels , privacy concerns ) . In other words , the amount of local computation at each worker is time-varying , device-dependent , and fully controlled by the worker itself . Clearly , AFL has a much lower server-worker coordination complexity and avoids the aforementioned pitfalls in server-centric FL approaches . However , AFL also introduces significant challenges in algorithmic design on the server-side because the server needs to work much harder to handle the chaotic worker behaviors in AFL ( e.g. , asynchrony , spatial and temporal heterogeneity in computing ) . Toward this end , several fundamental questions naturally arise : 1 ) Is it possible to design algorithms that converge under AFL ? 2 ) If the answer to the previous question is yes , how fast could the algorithms converge ? 3 ) Can the new AFL-based algorithms still achieve the desired “ linear speedup effect ? ” In this paper , we answer the above fundamental questions of AFL affirmatively . Our main contributions and key results are summarized as follows : • We propose a new FL paradigm called Anarchic Federated Learning ( AFL ) , where the workers are allowed to engage in training at will and choose the number of local update steps based on their own time-varying situations ( computing resources , energy levels , etc. ) . This loose worker-server coupling significantly simplifies the implementations and renders AFL particularly suitable for FL deployments in edge computing environments . For any AFL algorithms under general worker information arrival processes and non-i.i.d . data across workers , we first establish a fundamental convergence error lower bound that depends on the data heterogeneity in the AFL system . Then , we propose two Anarchic Federated Averaging ( AFA ) algorithms with two-sided learning rates for two classes of FL problems ( cross-device and cross-silo ) ( Kairouz et al. , 2019 ; Wang et al. , 2021 ) . • For AFL in the cross-device ( CD ) setting , our AFA-CD algorithm converges to an error ball whose size matches the fundamental lower bound , with an O ( 1/ √ mT ) convergence rate where m is the number of collected workers in each round of update and T is the total number of rounds . We note that this convergence rate retains the highly desirable “ linear speedup effect ” under AFL.1 Moreover , under the special case with uniform workers ’ participation ( equivalent to uniform workers sampling in conventional FL ( Li et al. , 2019c ; Karimireddy et al. , 2020b ; a ; Acar et al. , 2021 ) ) , AFA-CD can further converge to a stationary point ( i.e. , a singleton ) at a convergence rate that matches the state-of-the-art of conventional distributed and federated learning . • For AFL in the cross-silo ( CS ) setting , our proposed AFA-CS algorithm achieves an enhanced convergence rate of O ( 1/ √ MT ) by leveraging historical feedback and variance reduction tech- niques , where M is the total number of workers . This suggests that , not only can “ linear speedup ” be achieved under AFL-CS , the speedup factor also depends on the total number of workers M instead of the number of collected workers m in each round ( M > m ) . To our knowledge , this result is new in the FL literature . • We validate the proposed algorithms with extensive experiments on CV and NLP tasks and further explore the effect of the asynchrony and local step number in AFL . We also numerically show that our AFL is a general algorithmic framework in the sense that various advanced FL techniques ( e.g. , FedProx ( Li et al. , 2018 ) and SCAFFOLD ( Karimireddy et al. , 2020b ) ) can be integrated as the optimizers in our AFA framework to further enhance the AFL performance . 1To attain -accuracy , it takes O ( 1/ 2 ) steps for an algorithm with an O ( 1/ √ T ) convergence rate , while needing O ( 1/m 2 ) steps for another algorithm with an O ( 1/ √ mT ) convergence rate ( the hidden constant in Big-O is the same ) . In this sense , O ( 1/ √ mT ) implies a linear speedup with respect to the number of workers . The rest of the paper is organized as follows . In Section 2 , we review related work . In Section 3 , we introduce AFL and our AFA algorithms , which are followed by their convergence analysis in Section 4 . We present the numerical results in Section 5 and conclude the work in Section 6 . 2 RELATED WORK . Server-Centric Federated Learning Algorithms : To date , one of the prevailing FL algorithms is Federated Averaging ( FedAvg ) , which was first proposed in ( McMahan et al. , 2016 ) as a heuristic to improve communication efficiency and data privacy for FL . Since then , there have been substantial follow-ups of FedAvg that focus on non-i.i.d . ( heterogeneous ) data ( see , e.g. , FedProx ( Li et al. , 2018 ) , FedPD ( Zhang et al. , 2020b ) , SCAFFOLD ( Karimireddy et al. , 2020b ) , FedNova ( Wang et al. , 2020 ) , FedDyn ( Acar et al. , 2021 ) , and MIME ( Karimireddy et al. , 2020a ) ) , which are closely related to our work . The main idea for these algorithms is to control the “ model drift ” ( due to heterogeneous datasets and the use of multiple local update steps on the worker side of FedAvg ) . While these algorithms achieved various degrees of success in dealing with data heterogeneity , they are all server-centric synchronous algorithms that are not easy to implement in edge-based FL due to straggler issues ( see discussions in Section 1 ) . Federated Learning with Flexible Worker Participation : Recently , some attempts have been made to alleviate the strict requirements on worker ’ s participation , such as allowing different local steps ( Ruan et al. , 2021 ; Wang et al. , 2020 ) and asynchronous FL ( Avdiukhin & Kasiviswanathan , 2021 ; Xie et al. , 2019 ) . However , most of these works either lack theoretical performance guarantees or require strong assumptions . For example , Ruan et al . ( 2021 ) assumed strongly convex loss function and bounded aggregation coefficient ; Avdiukhin & Kasiviswanathan ( 2021 ) assumed bounded gradients and same computation time per iteration for all workers . Our AFL paradigm considered in this paper is more general and subsumes all the above settings as special cases . We note , however , that AFL differs from conventional FL with flexible worker participation in that the worker ’ s participation in AFL and its local optimization process are completely determined by the workers , and not by the sampling requests from the server . This is more practical since it allows workers to participate in FL under drastically different situations in network , charging/idle cycles , etc . Due to the complex couplings between various sources of randomness and multiple layers of heterogeneity in spatial and temporal domains in AFL , the training algorithm design for AFL and its theoretical analysis is far from a straightforward combination of existing FL techniques for flexible worker participation . Asynchronous Distributed Optimization : The asynchrony in AFL also shares some similarity with asynchronous distributed optimization . The basic idea of asynchronous distributed optimization is to forgo the common clock in the system to lower the system implementation complexity in distributed optimization . However , due to extra noise introduced by asynchrony , it is highly non-trivial to establish the convergence performance of asynchronous distributed optimization algorithms . To address this challenge , asynchronous distributed optimization has been studied extensively in the machine learning and optimization literature ( see , e.g. , Lian et al . ( 2018 ) ; Niu et al . ( 2011 ) ; Agarwal & Duchi ( 2012 ) ; Paine et al . ( 2013 ) ; Xie et al . ( 2019 ) ; Zhang et al . ( 2020a ) and references therein ) . We note that the AFL paradigm considered in this paper is more general and subsumes asynchronous distributed optimization as a special case . To see this , note that in addition to the asynchronous updates at the server , the workers in AFL could further have different numbers of local update steps . Moreover , the workers may not even need to be work-conserving ( i.e. , workers could be idle between rounds of updates ) . As a result , the convergence analysis of AFL is much more challenging .

Summary: This paper analyses a variant of generalized federated averaging ([Wang et al.](https://arxiv.org/abs/2107.06917)) with partial worker participation and asynchrony in the stateful (i.e., the worker specific data can be saved on the server) and stateless settings, which characterize cross-silo and cross-device federated learning ([Kairouz et al.](https://arxiv.org/abs/1912.04977?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%253A+arxiv%252FQSXk+%2528ExcitingAds%2521+cs+updates+on+arXiv.org%2529)) respectively. Specifically for each global update, the server uses fresh gradients from $m$ out of $M$ machines, where each machine can make some local SGD updates starting from a stale global iterate (i.e., the gradients are delayed). This setting is termed *anarchic federated learning* (AFl) when the workers can have an arbitrary number of local steps and gradient delays.  The authors first provide a lower bound for convergence to a first-order stationary point in the AFL setting. Then they upper bound the convergence of their algorithms in the stateful and stateless setting when the local updates and delay are bounded. Under some regimes, and assumptions on the worker sampling and delay distribution, a linear convergence speed-up can be shown w.r.t. the number of machines ($m$ for the stateless setting and $M$ for the stateful setting). Some experiments are provided to measure the effect of the anarchic worker behavior.