Zeno++: Robust Fully Asynchronous SGD

1 INTRODUCTION . Synchronous training and asynchronous training are the two most common paradigms of distributed machine learning . On the one hand , synchronous training requires , periodically , the global updates at the server to be blocked until all the workers respond . In contrast , for asynchronous training , the server updates the global model immediately after a worker responds . Theoretical and experimental analysis ( Dutta et al. , 2018 ) suggests that synchronous training is more stable with less noise , but can also be slowed down by the global barrier across all the workers . Asynchronous training is generally faster , but needs to address instability and noisiness due to staleness . In this paper , we focus on asynchronous training . We study the security of distributed asynchronous Stochastic Gradient Descent ( SGD ) in a centralized worker-server architecture , also known as the Parameter Server ( or PS ) architecture . In the PS architecture , there are server nodes and worker nodes . When combined with an asynchronous SGD approach , each worker pulls the global model from the servers , estimates the gradients using the local portion of the training data , then sends the gradient estimates to the servers . The servers update the model as soon as a new gradient is received from any worker . The security of machine learning has gained increasing attention in recent years . In particular , tolerance to Byzantine failures ( Blanchard et al. , 2017 ; Chen et al. , 2017 ; Yin et al. , 2018 ; Feng et al. , 2014 ; Su and Vaidya , 2016a ; b ; Xie et al. , 2018b ; Alistarh et al. , 2018 ; Cao and Lai , 2018 ) has become an important topic in the distributed machine learning literature . Byzantine failures are well-studied for the distributed systems ( Lamport et al. , 1982a ) . However , in distributed machine learning , Byzantine failures have unique properties . In brief , the goal of Byzantine workers is to prevent convergence of the model training . By construction , Byzantine failures ( Lamport et al. , 1982b ) assume the worst case , i.e. , the Byzantine workers can behave arbitrarily . Such failures may be caused by a variety of reasons including but not limited to : hardware/software bugs , vulnerable communication channels , poisoned datasets , or malicious attackers . To make things worse , groups of Byzantine workers can collude , potentially resulting in more harmful attacks . It is also clear that as the worst case , Byzantine failures generalize benign failures such as hardware or software errors . Unlike previous work ( Damaskinos et al. , 2018 ) , we tackle Byzantine tolerance in a more general scenario . The Byzantine tolerance of asynchronous SGD is challenging in this case because of : • Asynchrony . The lack of synchrony incurs additional noise for the stochastic gradients . Such noise makes it more difficult to distinguish the Byzantine gradients from the benign ones , especially as Byzantine behavior may exacerbate staleness . • Unpredictable successive updates . The lack of synchronous scheduling makes it possible for the server to receive updates from Byzantine workers successively . Thus , even in the standard scenario , where less than half of the workers are Byzantine , the server can be suffocated by successive Byzantine gradients . • Unbounded number of Byzantine workers . For the Byzantine tolerance in fully asynchronous training , the assumption of a bounded number of Byzantine workers is meaningless . In Byzantinetolerant synchronous training ( Blanchard et al. , 2017 ; Chen et al. , 2017 ; Yin et al. , 2018 ; Xie et al. , 2018b ; a ) , the servers can compare the candidate gradients with each other , and utilize the majority assumption to filter out the harmful gradients , or use robust aggregation to bound the error . However , such strategies are infeasible in asynchronous training , since there is nothing to compare to or aggregate . Aggregating the successive gradients is also meaningless since the successive gradients could all be pushed by the same Byzantine worker . Furthermore , although most of the previous work ( Blanchard et al. , 2017 ; Chen et al. , 2017 ; Yin et al. , 2018 ; Feng et al. , 2014 ; Su and Vaidya , 2016a ; b ; Alistarh et al. , 2018 ; Cao and Lai , 2018 ) assumes a majority of honest workers , this requirement is not guaranteed to be satisfied in practice . The key idea of our approach is to estimate the descent of the loss value after the candidate gradient is applied to the model parameters , based on the Byzantine-tolerant synchronous SGD algorithm , Zeno ( Xie et al. , 2018b ) . Intuitively , if the loss value decreases , the candidate gradient is likely to result in optimization progress . For computational efficiency , we also propose a lazy update . To the best of our knowledge , this paper is the first to theoretically and empirically study Byzantinetolerant fully asynchronous SGD with anonymous workers , and potentially an unbounded number of Byzantine workers . In summary , our contributions are : • We propose Zeno++ , a new approach for Byzantine-tolerant fully asynchronous SGD with anonymous workers . • We show that Zeno++ tolerates Byzantine workers without any limit on either the staleness or the number of Byzantine workers . • We prove the convergence of Zeno++ for non-convex problems . • Experimental results validate that 1 ) existing algorithms may fail in practical scenarios , and 2 ) Zeno++ gracefully handles such cases . 2 RELATED WORK . Most of the existing Byzantine-tolerant SGD algorithms focus on synchronous training . Chen et al . ( 2017 ) ; Su and Vaidya ( 2016a ; b ) ; Yin et al . ( 2018 ) ; Xie et al . ( 2018a ) use robust statistics ( Huber , 2011 ) including the geometric median , coordinate-wise median , and trimmed mean as Byzantinetolerant aggregation rules . Blanchard et al . ( 2017 ) ; Mhamdi et al . ( 2018 ) propose Krum and its variants , which select the candidates with minimal local sum of Euclidean distances . Alistarh et al . ( 2018 ) utilize historical information to identify harmful gradients . Chen et al . ( 2018 ) use coding theory and majority voting to recover correct gradients . Most of these synchronous algorithms assume that most of the workers are non-Byzantine . However , in practice , there are no guarantees that the number of Byzantine workers can be controlled . Xie et al . ( 2018b ) ; Cao and Lai ( 2018 ) propose synchronous SGD algorithms for an unbounded number of Byzantine workers . Recent years have witnessed an increasing number of large-scale machine learning algorithms , including asynchronous SGD ( Zinkevich et al. , 2009 ; Lian et al. , 2018 ; Zheng et al. , 2017 ; Zhou et al. , 2018 ) . Damaskinos et al . ( 2018 ) proposed Kardam , which to our knowledge is the only prior work to address Byzantine-tolerant asynchronous training . Kardam utilizes the Lipschitzness of the gradients to filter out outliers . However , Kardam assumes a threat model much weaker than ours . The major differences in the threat model are listed as follows : • Verification of worker identity . Unlike Kardam , we do not require verifying the identities of the workers when the server receives gradients . Kardam uses the so-called empirical Lipschitz coefficient , to test the benignity of the gradient sent by a specific worker . Such a mechanism keeps the record of the empirical Lipschitz coefficient of each worker . Thus , whenever a gradient is received , the Kardam server must be able to identify the identity/index of the worker . However , since Byzantine workers can behave arbitrarily , they can fake their identities/indices when sending gradients to the servers . Thus , Kardam assumes a threat model much weaker than the traditional Byzantine failure/threat model . Note that for synchronous training , the server can partially counter the index spoofing attack by simply filtering out all the gradients with duplicated indices . However , such an approach is infeasible for asynchronous training . • Bounded staleness of workers/limit of successive gradients . Unlike Kardam , we do not require bounded staleness of the workers . Kardam requires that the number of gradients successively received from a single worker is bounded above . To be more specific , on the server , any sequence of successively received gradients of length 2q + 1 must contain at least q + 1 gradients from honest workers . However , in real-world asynchronous training , such an assumption is very difficult to satisfy . • A majority of honest workers . Unlike Kardam , we do not require a majority of honest workers . Kardam requires that the number of Byzantine workers is less than one-third of the total number of workers – much stronger restriction than the standard setting that allows for the number of Byzantine workers to be up to 50 % of the total number of workers . Zeno++ further extends this guarantee to allow for not only 50 % , but also a majority of Byzantine workers . 3 MODEL . We consider the following optimization problem : minx∈Rd F ( x ) , where F ( x ) = 1 m ∑ i∈ [ m ] Ezi∼Dif ( x ; zi ) , for ∀i ∈ [ m ] , zi is sampled from the local data Di on the ith device . We solve this problem in a distributed manner with m workers . Each worker trains the model on local data . In each iteration , the ith worker will sample n independent data points from the dataset Di , and compute the gradient of the local empirical loss Fi ( x ) = 1n ∑n j=1 f ( x ; zi , j ) , ∀i ∈ [ m ] , where zi , j ∼ Di is the jth sampled data on the ith worker . When there are no Byzantine failures , the servers update the model whenever a new gradient is received : xt+1 = xt − γtgτ , gτ = 1 n ∑ j∈ [ n ] ∇f ( xτ ; zi , j ) , τ ≤ t , i ∈ [ m ] . When there are Byzantine failures , gτ can be replaced by arbitrary value ( Damaskinos et al. , 2018 ) . Formally , we define the threat model as follows . Definition 1 . ( Threat Model ) . When the server receives a gradient estimator g̃τ , it is either correct or Byzantine . If sent by a Byzantine worker , g̃τ is assigned arbitrary value . If sent by an honest worker , the correct gradient is 1n ∑n j=1∇f ( xτ ; zi , j ) , τ ≤ t , i ∈ [ m ] . Thus , we have g̃τ = { arbitrary value , if sent by a Byzantine worker , 1 n ∑n j=1∇f ( xτ ; zi , j ) , τ ≤ t , i ∈ [ m ] , otherwise . We assume that q out of m workers are Byzantine , where q ≤ m. Furthermore , the indices of Byzantine workers can change across different iterations . 4 METHODOLOGY . In this section , we introduce Zeno++ , a Byzantine-tolerant asynchronous SGD algorithm based on inner-product validation . Zeno++ is a computationally efficient version of its prototype : Zeno+ .

Summary: This paper addresses security of distributed optimization algorithm under Byzantine failures. These failures usually prevent convergence of training neural network models. Focusing on the asynchronous SGD algorithm implemented with a parameter-server, the authors propose to use stochastic line search ideas to detect whether the gradients are good descent directions or not. It applies to a general scenario including repeated and unbounded Byzantine failures.