Fast Federated Learning in the Presence of Arbitrary Device Unavailability

1 Introduction . Federated learning is a machine learning setting in which a central server coordinates with a large number of devices to collectively train a shared model [ 28 , 34 , 20 , 33 , 25 , 26 ] . Practical advantages of this training scheme are mainly twofold . First , each device keeps the private data locally and hence preserves its data privacy . Second , federated learning can make use of idle computing resources and lower computation costs . Although federated learning successfully scales up with data sizes and accelerates training via more affordable computing power [ 43 , 38 , 36 ] , the collaborative setup leads to new challenges due to large variations among individual computing devices . Our work aims to formulate and investigate the impact of device variations on FL from an optimization perspective . In FL , a device can differ from its peers in multiple aspects [ 17 , 25 ] . First , the data distribution and local task can be different among devices . To address the data variation , non-i.i.d . objective models were proposed and analyzed by [ 26 , 18 , 19 , 42 , 25 ] . We follow this line of work and formulate our optimization objective as a sum of stochastic functions on individual devices ( See Eqn . ( 1 ) ) . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A second variation among devices is caused by different computing and communication speeds . One natural way to formulate the variation in computation speeds is to allow asynchronous updates and model the updates as delayed responses . Lots of novel research has studied the problem with different delay models , e.g. , [ 35 , 4 , 27 , 11 , 45 , 1 , 5 , 13 ] . However , the delayed setup assumes that all devices make roughly the same number of ( delayed ) responses in the end . This behavior may deviate largely from the FL practice , where each device , e.g. , personal cell phones , can have very different active duration when participating in the FL training , and hence make different numbers of responses . For this reason , our work aims to address this third discrepancy among devices caused by individual availability patterns . The third device heterogeneity caused by different availability patterns is less studied in optimization for federated learning problems . In this model , instead of making a delayed response , devices can abort the training halfway , e.g. , due to battery level , incoming calls , etc , and fail to return their responses upon the central server ’ s requests [ 28 , 7 , 17 ] . To handle missing responses , researchers propose algorithms where the central server may collect responses from only a fraction of the devices and make updates [ 18 , 28 , 42 , 25 , 26 , 17 , 30 , 14 ] . Previous works on collecting responses from a fraction of devices can be divided into two categories . When the response distribution is known , one could collect only the fastest responses and re-weight according to their response probability [ 17 , 26 , 30 ] . This model can be restrictive , as in practice , the exact distribution may not be available and may evolve . Another line of work assumes that the server can arbitrarily decide and sample a set of devices to collect responses accordingly in every communication round [ 18 , 28 , 42 , 25 , 14 ] . This model does not require knowing the response possibility . However , the response time can be very long if the selected subset contains unavailable devices . In this work , we address the above limitations by studying federated learning in the presence of arbitrary device unavailability . Within this practical setup , we propose an algorithm that automatically accommodates for the underlying unavailability and allows patterns of the device unavailability to be non-stationary and even adversarial . Furthermore , our algorithm can achieve optimal convergence rates in the presence of device inactivity and automatically reduce to best-known rates if all devices are active . Our contributions are summarized as follows . • We investigate the federated learning problem with a practical formulation of device participation , which does not require each device to be online according to an ( either known or unknown ) distribution . • We propose the Memory-augmented Impatient Federated Averaging ( MIFA ) algorithm that is agnostic to the availability pattern . It efficiently avoids excessive latency induced by inactive devices , successfully exploits the information about the descent direction in stale and noisy gradients , and corrects the gradient bias using the memorized latest updates . • We prove that MIFA achieves minimax optimal convergence rates O ( τ̄T+1 NKT ) for smooth , strongly convex functions , and O ( √ ν̄+1 NKT ) for smooth , non-convex functions , and es- tablish matching lower bounds . Here , N , K and T stand for the number of devices , local updates and communication rounds respectively . τ̄T and ν̄ characterize how actively devices participate in training ( see formal definitions in Sections 3 , 5 and 6 ) . MIFA also achieves optimal convergence rates in the ideal case when all devices are active . • We provide an explicit characterization of the improvement over baseline algorithms through a case study and empirically verify our results on real-world datasets . 2 Related work . Federated learning . Federated Averaging ( FedAvg ) was first proposed in [ 28 ] . [ 26 , 19 , 18 , 42 ] provided convergence analysis for FedAvg on non-i.i.d . data and quantified how data heterogeneity degrades the convergence rate . Several variants of FedAvg were designed to deal with data heterogeneity . FedProx [ 25 ] adds a proximal term to local objective functions , while FSVRG [ 21 ] and SCAFFOLD [ 18 ] employ variance reduction techniques . One line of work focused on variations in computation capabilities among devices [ 39 , 31 , 40 ] . These models assume that responses are delayed but not missing . To address the missing response , some work assumes that the server can actively sample a subset of devices to respond [ 18 , 28 , 42 , 25 , 14 ] or that the pattern of device availability is known [ 26 , 17 , 10 , 30 ] . These results do not generalize to adversarial inactive patterns . [ 32 ] discussed the impact of device inactivity on convergence but their proposed algorithm diverges if there exists an inactive device in each round of communication . However , our setup allows adversarial patterns under certain non-distributional assumptions ( see Section 5 ) while our proposed algorithm still achieves convergence . Asynchronous distributed optimization . Our work is related to literature in the field of traditional asynchronous distributed optimization in that our proposed algorithm uses stale gradients . The problem setup for asynchronous distributed algorithms can be divided into two categories [ 13 ] . One is the shared-data ( i.i.d . ) setting , where all workers can access the whole dataset . In this setting , the local gradient is an unbiased estimator of the global gradient [ 35 , 4 , 27 , 11 , 45 , 1 ] . In contrast , we assume each worker has non-i.i.d . data , and hence the local stochastic gradient can not be viewed as an unbiased estimator of the global gradient . The other less studied setting in distributed optimization is the distributed-data setting ( non-i.i.d . ) , where data are partitioned among workers . Specifically , [ 5 ] proposed an asynchronous incremental aggregated gradient algorithm that uses buffered gradients to update the global model . Unlike our setup , this algorithm evaluates full local gradients , performs only one local step , and was analyzed under the bounded delay assumption . [ 13 ] models the delay as stochastic and assumes that the server has knowledge of the distribution , but our formulation is distribution-free . [ 6 ] allows workers to perform multiple local steps and communicate with the server at different times , but the authors assume that all workers are available and compute at the same rate . Comparison with an independent work . While preparing the manuscript , we were unaware of an independent work [ 41 ] that investigated the same setup and proposed a similar algorithm called FedLaAvg . Their main theorem established the convergence rate of O ( √ νmax N0.5T ( G 2 + σ2 ) ) for smooth and non-convex problems , where G2 is the uniform upper bound for the squared norm of stochastic gradients and νmax is the maximum number of inactive rounds . In comparison , we prove the minimax optimal rate ofO ( √ ν̄ NKT σ 2 ) without the bounded gradient assumption , also improving νmax to ν̄ . Furthermore , our result achieves a linear speedup in N and K. Apart from non-convex functions , we also derive the minimax optimal rates for strongly convex smooth functions under the mild assumption that allows for arbitrary and unbounded number of inactive rounds . Both of our results achieve linear speedups in terms of N and K , and automatically recover the best-known rates of FedAvg when all devices are active . We also show that our proposed algorithm achieves acceleration over unbiased baseline algorithms in the presence of stragglers . 3 Problem Setup . We consider optimizing the following problem in a Federated Learning setting : min w∈Rd f ( w ) : = 1N ∑N i=1 fi ( w ) : = 1 N ∑N i=1 Eξi [ fi ( w , ξi ) ] , ( 1 ) where w is the optimization variable , e.g. , parameters of a machine learning model , N is the number of participating devices , fi is the local loss function on device i , and ξi describes the randomness in local data distribution . In the ideal federated learning setup ( see Figure 1 ( a ) ) , all devices return responses within similar time , and hence the central server collects all the local updates . In this case , the computation cost is usually measured by the number of local stochastic oracle evaluations , which is proportional to the number of rounds . In a delayed FL setup ( see Figure 1 ( b ) ) , devices are always active upon the central server ’ s request but may return responses with a delay . Here , all devices return almost the same number of responses in the long term . As we discussed , the above setups do not depict a real-world scenario in which a device can have a longer inactive duration than active duration . In such cases , the communication interval is much longer than the local computation time required for each update , and each device generates an unequal number of responses [ 28 , 17 ] . This motivates our setup in Figure 1 ( c ) . In our proposed setup , we use t to index the global communication rounds . We say a device participates or is active at round t if it can complete the computation task and send back the update at the end of round t. We define A ( t ) as the set of all active devices at round t. Notice that we make no assumptions on the distribution of the participation patterns of devices and allow them to be arbitrary . Directly applying FedAvg to the proposed setup can be problematic due to the existence of inactive devices . To accommodate for inactive devices , we discuss three natural variants of FedAvg and their limitations . The detailed algorithms can be found in Appendix A . • Biased FedAvg . At each communication round , the global model is updated with a direct average of local updates from the active devices . This naive approach induces bias when data distribution and response patterns vary among devices . • FedAvg with device sampling . The server selects a subset of S devices randomly without replacement , and then waits until all devices in the subset S respond . This is how original FedAvg [ 28 ] addresses device unavailability . Note that over T communication rounds , the global model is updated less than T times due to waiting . This approach is prone to stragglers and we refer the readers to Section 5.1 for a detailed discussion . • FedAvg with importance sampling [ 26 , 13 , 17 ] . The local updates from the active devices are weighted by the reciprocal of the participation probabilities to avoid bias . This approach is only applicable when the response of each device is i.i.d . over rounds and it requires the knowledge of participation probabilities .

Summary: The paper tries to overcome the challenge of federated learning that devices may drop out of the training process arbitrarily ( which doesn't follow any known or unknown distribution). This paper proposes the Memory-augmented Impatient Federated Averaging (MIFA) algorithm and provides its theoretical analysis under arbitrary device unavailability. Experiments have been done to show the advantages of the proposed method.