Anarchic Federated Learning

1 INTRODUCTION . Federated Learning ( FL ) has recently emerged as an important distributed learning framework that leverages numerous workers to collaboratively learn a joint model ( Li et al. , 2019a ; Yang et al. , 2019 ; Kairouz et al. , 2019 ) . Since its inception , FL algorithms have become increasingly powerful and have been able to handle various heterogeneity in data , network environments , worker computing capabilities , etc . Moreover , most of the prevailing FL algorithms ( e.g. , FedAvg ( McMahan et al. , 2016 ) and its variants ( Li et al. , 2018 ; Zhang et al. , 2020b ; Karimireddy et al. , 2020b ; a ; Acar et al. , 2021 ) ) enjoy so-called “ linear speedup effect , ” i.e. , the convergence time of an FL algorithm decreases linearly as the number of workers increases ( Stich , 2018 ; Yu et al. , 2019 ; Wang & Joshi , 2018 ; Khaled et al. , 2019 ; Karimireddy et al. , 2020b ; Yang et al. , 2021 ; Qu et al. , 2020 ) . To achieve these salient features , most of the existing FL algorithms have adopted a server-centric approach , i.e. , the worker behaviors are tightly “ dictated ” by the server . For example , the server in these FL algorithms can i ) determine either all or a subset of workers to participate in each round of FL update ; ii ) fully control the timing for synchronization and whether to accept/reject information sent from the workers ; iii ) precisely specify the algorithmic operations ( e.g. , the number of local steps performed at each worker before communicating with the server ) , etc . Despite achieving strong performance guarantees , such a server-centric approach introduces several limitations . Specifically , these server-centric FL algorithms often implicitly rely on the following assumptions : ( 1 ) each worker is available for training upon the server ’ s request and throughout a complete round ; ( 2 ) all participating workers are willing to execute the same number of local updates and communicate with the server in a synchronous manner following a common clock . Unfortunately , in edge networks where many FL systems are deployed , these assumptions are restrictive or even problematic due to the following reasons . First , many requested edge devices on the worker side may not be available in each round because of , e.g. , communication errors or battery outages . Second , the use of synchronous communication and an identical number of local updates across all workers ignores the fact that worker devices in edge-based FL systems are heterogeneous in computation and communication capabilities . As a result , stragglers ( i.e. , slow workers ) could significantly slow down the training process . To mitigate the straggler effect , various robust FL algorithms have been developed . For example , the server in FedAvg ( McMahan et al. , 2016 ) can simply ignore and drop the information from the stragglers to speedup learning . However , this may lead to other problems such as wasted computation/energy ( Wang et al. , 2019 ) , slower convergence ( Li et al. , 2018 ) , or biased/unfair uses of worker data ( Kairouz et al. , 2019 ) . Moreover , the synchronous nature of the server-centric approaches implies many networking problems ( e.g. , interference between workers , periodic traffic spikes , high complexity in maintaining a network-wide common clock ) . The above limitations of the current server-centric FL approaches motivate us to propose a new paradigm in FL , which we call Anarchic Federated Learning ( AFL ) . In stark contrast to servercentric FL , workers in AFL are completely free of the “ dictation ” from the server . Specifically , each worker has complete freedom to choose when and how long to participate in FL without following any control signals from the server . As a result , the information fed back from workers is inherently asynchronous . Also , each worker can independently determine the number of local update steps to perform in each round based on its current local situation ( e.g. , battery level , communication channels , privacy concerns ) . In other words , the amount of local computation at each worker is time-varying , device-dependent , and fully controlled by the worker itself . Clearly , AFL has a much lower server-worker coordination complexity and avoids the aforementioned pitfalls in server-centric FL approaches . However , AFL also introduces significant challenges in algorithmic design on the server-side because the server needs to work much harder to handle the chaotic worker behaviors in AFL ( e.g. , asynchrony , spatial and temporal heterogeneity in computing ) . Toward this end , several fundamental questions naturally arise : 1 ) Is it possible to design algorithms that converge under AFL ? 2 ) If the answer to the previous question is yes , how fast could the algorithms converge ? 3 ) Can the new AFL-based algorithms still achieve the desired “ linear speedup effect ? ” In this paper , we answer the above fundamental questions of AFL affirmatively . Our main contributions and key results are summarized as follows : • We propose a new FL paradigm called Anarchic Federated Learning ( AFL ) , where the workers are allowed to engage in training at will and choose the number of local update steps based on their own time-varying situations ( computing resources , energy levels , etc. ) . This loose worker-server coupling significantly simplifies the implementations and renders AFL particularly suitable for FL deployments in edge computing environments . For any AFL algorithms under general worker information arrival processes and non-i.i.d . data across workers , we first establish a fundamental convergence error lower bound that depends on the data heterogeneity in the AFL system . Then , we propose two Anarchic Federated Averaging ( AFA ) algorithms with two-sided learning rates for two classes of FL problems ( cross-device and cross-silo ) ( Kairouz et al. , 2019 ; Wang et al. , 2021 ) . • For AFL in the cross-device ( CD ) setting , our AFA-CD algorithm converges to an error ball whose size matches the fundamental lower bound , with an O ( 1/ √ mT ) convergence rate where m is the number of collected workers in each round of update and T is the total number of rounds . We note that this convergence rate retains the highly desirable “ linear speedup effect ” under AFL.1 Moreover , under the special case with uniform workers ’ participation ( equivalent to uniform workers sampling in conventional FL ( Li et al. , 2019c ; Karimireddy et al. , 2020b ; a ; Acar et al. , 2021 ) ) , AFA-CD can further converge to a stationary point ( i.e. , a singleton ) at a convergence rate that matches the state-of-the-art of conventional distributed and federated learning . • For AFL in the cross-silo ( CS ) setting , our proposed AFA-CS algorithm achieves an enhanced convergence rate of O ( 1/ √ MT ) by leveraging historical feedback and variance reduction tech- niques , where M is the total number of workers . This suggests that , not only can “ linear speedup ” be achieved under AFL-CS , the speedup factor also depends on the total number of workers M instead of the number of collected workers m in each round ( M > m ) . To our knowledge , this result is new in the FL literature . • We validate the proposed algorithms with extensive experiments on CV and NLP tasks and further explore the effect of the asynchrony and local step number in AFL . We also numerically show that our AFL is a general algorithmic framework in the sense that various advanced FL techniques ( e.g. , FedProx ( Li et al. , 2018 ) and SCAFFOLD ( Karimireddy et al. , 2020b ) ) can be integrated as the optimizers in our AFA framework to further enhance the AFL performance . 1To attain -accuracy , it takes O ( 1/ 2 ) steps for an algorithm with an O ( 1/ √ T ) convergence rate , while needing O ( 1/m 2 ) steps for another algorithm with an O ( 1/ √ mT ) convergence rate ( the hidden constant in Big-O is the same ) . In this sense , O ( 1/ √ mT ) implies a linear speedup with respect to the number of workers . The rest of the paper is organized as follows . In Section 2 , we review related work . In Section 3 , we introduce AFL and our AFA algorithms , which are followed by their convergence analysis in Section 4 . We present the numerical results in Section 5 and conclude the work in Section 6 . 2 RELATED WORK . Server-Centric Federated Learning Algorithms : To date , one of the prevailing FL algorithms is Federated Averaging ( FedAvg ) , which was first proposed in ( McMahan et al. , 2016 ) as a heuristic to improve communication efficiency and data privacy for FL . Since then , there have been substantial follow-ups of FedAvg that focus on non-i.i.d . ( heterogeneous ) data ( see , e.g. , FedProx ( Li et al. , 2018 ) , FedPD ( Zhang et al. , 2020b ) , SCAFFOLD ( Karimireddy et al. , 2020b ) , FedNova ( Wang et al. , 2020 ) , FedDyn ( Acar et al. , 2021 ) , and MIME ( Karimireddy et al. , 2020a ) ) , which are closely related to our work . The main idea for these algorithms is to control the “ model drift ” ( due to heterogeneous datasets and the use of multiple local update steps on the worker side of FedAvg ) . While these algorithms achieved various degrees of success in dealing with data heterogeneity , they are all server-centric synchronous algorithms that are not easy to implement in edge-based FL due to straggler issues ( see discussions in Section 1 ) . Federated Learning with Flexible Worker Participation : Recently , some attempts have been made to alleviate the strict requirements on worker ’ s participation , such as allowing different local steps ( Ruan et al. , 2021 ; Wang et al. , 2020 ) and asynchronous FL ( Avdiukhin & Kasiviswanathan , 2021 ; Xie et al. , 2019 ) . However , most of these works either lack theoretical performance guarantees or require strong assumptions . For example , Ruan et al . ( 2021 ) assumed strongly convex loss function and bounded aggregation coefficient ; Avdiukhin & Kasiviswanathan ( 2021 ) assumed bounded gradients and same computation time per iteration for all workers . Our AFL paradigm considered in this paper is more general and subsumes all the above settings as special cases . We note , however , that AFL differs from conventional FL with flexible worker participation in that the worker ’ s participation in AFL and its local optimization process are completely determined by the workers , and not by the sampling requests from the server . This is more practical since it allows workers to participate in FL under drastically different situations in network , charging/idle cycles , etc . Due to the complex couplings between various sources of randomness and multiple layers of heterogeneity in spatial and temporal domains in AFL , the training algorithm design for AFL and its theoretical analysis is far from a straightforward combination of existing FL techniques for flexible worker participation . Asynchronous Distributed Optimization : The asynchrony in AFL also shares some similarity with asynchronous distributed optimization . The basic idea of asynchronous distributed optimization is to forgo the common clock in the system to lower the system implementation complexity in distributed optimization . However , due to extra noise introduced by asynchrony , it is highly non-trivial to establish the convergence performance of asynchronous distributed optimization algorithms . To address this challenge , asynchronous distributed optimization has been studied extensively in the machine learning and optimization literature ( see , e.g. , Lian et al . ( 2018 ) ; Niu et al . ( 2011 ) ; Agarwal & Duchi ( 2012 ) ; Paine et al . ( 2013 ) ; Xie et al . ( 2019 ) ; Zhang et al . ( 2020a ) and references therein ) . We note that the AFL paradigm considered in this paper is more general and subsumes asynchronous distributed optimization as a special case . To see this , note that in addition to the asynchronous updates at the server , the workers in AFL could further have different numbers of local update steps . Moreover , the workers may not even need to be work-conserving ( i.e. , workers could be idle between rounds of updates ) . As a result , the convergence analysis of AFL is much more challenging .

Summary: This paper proposes a general FL framework, called anarchic federated learning (AFL), that allows voluntary client participation of clients, with individual update steps and delay. Algorithms for both cross-device and cross-silo are presented. Convergence upper bounds and a lower bound are derived. Experimental results are reported to demonstrate the effectiveness of the proposed algorithm.