Fast Federated Learning in the Presence of Arbitrary Device Unavailability

1 Introduction . Federated learning is a machine learning setting in which a central server coordinates with a large number of devices to collectively train a shared model [ 28 , 34 , 20 , 33 , 25 , 26 ] . Practical advantages of this training scheme are mainly twofold . First , each device keeps the private data locally and hence preserves its data privacy . Second , federated learning can make use of idle computing resources and lower computation costs . Although federated learning successfully scales up with data sizes and accelerates training via more affordable computing power [ 43 , 38 , 36 ] , the collaborative setup leads to new challenges due to large variations among individual computing devices . Our work aims to formulate and investigate the impact of device variations on FL from an optimization perspective . In FL , a device can differ from its peers in multiple aspects [ 17 , 25 ] . First , the data distribution and local task can be different among devices . To address the data variation , non-i.i.d . objective models were proposed and analyzed by [ 26 , 18 , 19 , 42 , 25 ] . We follow this line of work and formulate our optimization objective as a sum of stochastic functions on individual devices ( See Eqn . ( 1 ) ) . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A second variation among devices is caused by different computing and communication speeds . One natural way to formulate the variation in computation speeds is to allow asynchronous updates and model the updates as delayed responses . Lots of novel research has studied the problem with different delay models , e.g. , [ 35 , 4 , 27 , 11 , 45 , 1 , 5 , 13 ] . However , the delayed setup assumes that all devices make roughly the same number of ( delayed ) responses in the end . This behavior may deviate largely from the FL practice , where each device , e.g. , personal cell phones , can have very different active duration when participating in the FL training , and hence make different numbers of responses . For this reason , our work aims to address this third discrepancy among devices caused by individual availability patterns . The third device heterogeneity caused by different availability patterns is less studied in optimization for federated learning problems . In this model , instead of making a delayed response , devices can abort the training halfway , e.g. , due to battery level , incoming calls , etc , and fail to return their responses upon the central server ’ s requests [ 28 , 7 , 17 ] . To handle missing responses , researchers propose algorithms where the central server may collect responses from only a fraction of the devices and make updates [ 18 , 28 , 42 , 25 , 26 , 17 , 30 , 14 ] . Previous works on collecting responses from a fraction of devices can be divided into two categories . When the response distribution is known , one could collect only the fastest responses and re-weight according to their response probability [ 17 , 26 , 30 ] . This model can be restrictive , as in practice , the exact distribution may not be available and may evolve . Another line of work assumes that the server can arbitrarily decide and sample a set of devices to collect responses accordingly in every communication round [ 18 , 28 , 42 , 25 , 14 ] . This model does not require knowing the response possibility . However , the response time can be very long if the selected subset contains unavailable devices . In this work , we address the above limitations by studying federated learning in the presence of arbitrary device unavailability . Within this practical setup , we propose an algorithm that automatically accommodates for the underlying unavailability and allows patterns of the device unavailability to be non-stationary and even adversarial . Furthermore , our algorithm can achieve optimal convergence rates in the presence of device inactivity and automatically reduce to best-known rates if all devices are active . Our contributions are summarized as follows . • We investigate the federated learning problem with a practical formulation of device participation , which does not require each device to be online according to an ( either known or unknown ) distribution . • We propose the Memory-augmented Impatient Federated Averaging ( MIFA ) algorithm that is agnostic to the availability pattern . It efficiently avoids excessive latency induced by inactive devices , successfully exploits the information about the descent direction in stale and noisy gradients , and corrects the gradient bias using the memorized latest updates . • We prove that MIFA achieves minimax optimal convergence rates O ( τ̄T+1 NKT ) for smooth , strongly convex functions , and O ( √ ν̄+1 NKT ) for smooth , non-convex functions , and es- tablish matching lower bounds . Here , N , K and T stand for the number of devices , local updates and communication rounds respectively . τ̄T and ν̄ characterize how actively devices participate in training ( see formal definitions in Sections 3 , 5 and 6 ) . MIFA also achieves optimal convergence rates in the ideal case when all devices are active . • We provide an explicit characterization of the improvement over baseline algorithms through a case study and empirically verify our results on real-world datasets . 2 Related work . Federated learning . Federated Averaging ( FedAvg ) was first proposed in [ 28 ] . [ 26 , 19 , 18 , 42 ] provided convergence analysis for FedAvg on non-i.i.d . data and quantified how data heterogeneity degrades the convergence rate . Several variants of FedAvg were designed to deal with data heterogeneity . FedProx [ 25 ] adds a proximal term to local objective functions , while FSVRG [ 21 ] and SCAFFOLD [ 18 ] employ variance reduction techniques . One line of work focused on variations in computation capabilities among devices [ 39 , 31 , 40 ] . These models assume that responses are delayed but not missing . To address the missing response , some work assumes that the server can actively sample a subset of devices to respond [ 18 , 28 , 42 , 25 , 14 ] or that the pattern of device availability is known [ 26 , 17 , 10 , 30 ] . These results do not generalize to adversarial inactive patterns . [ 32 ] discussed the impact of device inactivity on convergence but their proposed algorithm diverges if there exists an inactive device in each round of communication . However , our setup allows adversarial patterns under certain non-distributional assumptions ( see Section 5 ) while our proposed algorithm still achieves convergence . Asynchronous distributed optimization . Our work is related to literature in the field of traditional asynchronous distributed optimization in that our proposed algorithm uses stale gradients . The problem setup for asynchronous distributed algorithms can be divided into two categories [ 13 ] . One is the shared-data ( i.i.d . ) setting , where all workers can access the whole dataset . In this setting , the local gradient is an unbiased estimator of the global gradient [ 35 , 4 , 27 , 11 , 45 , 1 ] . In contrast , we assume each worker has non-i.i.d . data , and hence the local stochastic gradient can not be viewed as an unbiased estimator of the global gradient . The other less studied setting in distributed optimization is the distributed-data setting ( non-i.i.d . ) , where data are partitioned among workers . Specifically , [ 5 ] proposed an asynchronous incremental aggregated gradient algorithm that uses buffered gradients to update the global model . Unlike our setup , this algorithm evaluates full local gradients , performs only one local step , and was analyzed under the bounded delay assumption . [ 13 ] models the delay as stochastic and assumes that the server has knowledge of the distribution , but our formulation is distribution-free . [ 6 ] allows workers to perform multiple local steps and communicate with the server at different times , but the authors assume that all workers are available and compute at the same rate . Comparison with an independent work . While preparing the manuscript , we were unaware of an independent work [ 41 ] that investigated the same setup and proposed a similar algorithm called FedLaAvg . Their main theorem established the convergence rate of O ( √ νmax N0.5T ( G 2 + σ2 ) ) for smooth and non-convex problems , where G2 is the uniform upper bound for the squared norm of stochastic gradients and νmax is the maximum number of inactive rounds . In comparison , we prove the minimax optimal rate ofO ( √ ν̄ NKT σ 2 ) without the bounded gradient assumption , also improving νmax to ν̄ . Furthermore , our result achieves a linear speedup in N and K. Apart from non-convex functions , we also derive the minimax optimal rates for strongly convex smooth functions under the mild assumption that allows for arbitrary and unbounded number of inactive rounds . Both of our results achieve linear speedups in terms of N and K , and automatically recover the best-known rates of FedAvg when all devices are active . We also show that our proposed algorithm achieves acceleration over unbiased baseline algorithms in the presence of stragglers . 3 Problem Setup . We consider optimizing the following problem in a Federated Learning setting : min w∈Rd f ( w ) : = 1N ∑N i=1 fi ( w ) : = 1 N ∑N i=1 Eξi [ fi ( w , ξi ) ] , ( 1 ) where w is the optimization variable , e.g. , parameters of a machine learning model , N is the number of participating devices , fi is the local loss function on device i , and ξi describes the randomness in local data distribution . In the ideal federated learning setup ( see Figure 1 ( a ) ) , all devices return responses within similar time , and hence the central server collects all the local updates . In this case , the computation cost is usually measured by the number of local stochastic oracle evaluations , which is proportional to the number of rounds . In a delayed FL setup ( see Figure 1 ( b ) ) , devices are always active upon the central server ’ s request but may return responses with a delay . Here , all devices return almost the same number of responses in the long term . As we discussed , the above setups do not depict a real-world scenario in which a device can have a longer inactive duration than active duration . In such cases , the communication interval is much longer than the local computation time required for each update , and each device generates an unequal number of responses [ 28 , 17 ] . This motivates our setup in Figure 1 ( c ) . In our proposed setup , we use t to index the global communication rounds . We say a device participates or is active at round t if it can complete the computation task and send back the update at the end of round t. We define A ( t ) as the set of all active devices at round t. Notice that we make no assumptions on the distribution of the participation patterns of devices and allow them to be arbitrary . Directly applying FedAvg to the proposed setup can be problematic due to the existence of inactive devices . To accommodate for inactive devices , we discuss three natural variants of FedAvg and their limitations . The detailed algorithms can be found in Appendix A . • Biased FedAvg . At each communication round , the global model is updated with a direct average of local updates from the active devices . This naive approach induces bias when data distribution and response patterns vary among devices . • FedAvg with device sampling . The server selects a subset of S devices randomly without replacement , and then waits until all devices in the subset S respond . This is how original FedAvg [ 28 ] addresses device unavailability . Note that over T communication rounds , the global model is updated less than T times due to waiting . This approach is prone to stragglers and we refer the readers to Section 5.1 for a detailed discussion . • FedAvg with importance sampling [ 26 , 13 , 17 ] . The local updates from the active devices are weighted by the reciprocal of the participation probabilities to avoid bias . This approach is only applicable when the response of each device is i.i.d . over rounds and it requires the knowledge of participation probabilities .

Summary: This work studies federated learning setup where clients might participate and drop out from the training arbitrarily. Broadly speaking, this is partial participation setup for FL, where there is no direct control on device availability. For such a setup, this paper designs MIFA algorithm, which stores latest gradients from devices and does not wait for strugglers during the training. The method is analyzed for strongly convex and non-convex cases with an additional (weak) assumption on device participation.