Collaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)

1 Introduction . The distributed nature of data , the prohibitive cost of data transfers and the privacy concerns all call for collaborative machine learning . The idea consists for each machine to keep its data locally and to “ simply ” exchange with other machines what it learned so far . If all machines correctly communicate and execute the algorithms assigned to them , collaborative learning is rather easy . It can be achieved through the standard workhorse optimization algorithm : stochastic gradient descent ( SGD ) [ 37 ] , which can be effectively distributed through averaging [ 26 ] . ∗Authors are listed alphabetically . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . But in a practical distributed setting , hardware components may crash , software can be buggy , communications can be slowed down , data can be corrupted and machines can be hacked . Besides , large-scale machine learning systems are trained on user-generated data , which may be crafted maliciously . For example , recommendation algorithms have such a large influence on social medias that there are huge incentives from industries and governments to fabricate data that bias the learning algorithms and increase the visibility of some contents over others [ 6 , 33 ] . In the parlance of distributed computing , “ nodes '' can be Byzantine [ 28 ] , i.e. , they can behave arbitrarily maliciously , to confuse the system . Given that machine learning ( ML ) is now used in many critical applications ( e.g. , driving , medication , content moderation ) , its ability to tolerate Byzantine behavior is of paramount importance . In this paper , we precisely define and address , for the first time , the problem of collaborative learning in a fully decentralized , Byzantine , heterogeneous and asynchronous environment with non-convex loss functions . We consider n nodes , which may be machines or different accounts on a social media . Each node has its own local data , drawn from data distributions that may greatly vary across nodes . The nodes seek to collectively learn from each other , without however exchanging their data . None of the nodes is trusted , and any f < n nodes can be Byzantine . Contributions . We first precisely formulate the collaborative learning problem . Then , we give our main contribution : an equivalence between collaborative learning and a new more abstract problem we call averaging agreement . More precisely , we provide two reductions : from collaborative learning to averaging agreement and from averaging agreement to collaborative learning . We prove that both reductions essentially preserve the correctness guarantees on the output . The former reduction is the most challenging one to design and to prove correct . First , to update nodes ’ models , we use averaging agreement to aggregate nodes ’ stochastic gradients . Then , to avoid model drift , we regularly “ contract ” the nodes ’ models using averaging agreement . To prove correctness , we bound the diameter of honest nodes ’ models , and we analyze the effective gradient [ 12 ] . We then carefully select a halting iteration , for which correctness can be guaranteed . Our tight reduction allows to derive both impossibility results and optimal algorithms for collaborative learning by studying the “ simpler ” averaging agreement problem . We prove lower bounds on the correctness and Byzantine resilience that any averaging agreement algorithm can achieve , which implies the same lower bounds for collaborative learning . We then propose two optimal algorithms for averaging agreement . Our first algorithm is asymptotically optimal with respect to correctness , up to a multiplicative constant , when nearly all nodes are honest . Our second algorithm achieves optimal Byzantine resilience . Each of these algorithms induces an optimal collaborative learning protocol . While our algorithms apply in a very general setting , they can easily be tweaked for more specific settings with additional assumptions , such as the presence of a trusted parameter server [ 30 ] , the assumption of homogeneous ( i.e . i.i.d . ) local data or synchrony ( Section 3.3 and Section 6 ) . We implemented and evaluated our algorithms in a distributed environment with 3 ResNet models [ 24 ] . More specifically , we present their throughput overhead when compared to a non–robust collaborative learning approach with both i.i.d . and non–i.i.d . data ( i.e . we highlight the cost of heterogeneity ) . Essentially , we show that our first algorithm is more lightweight with a slowdown of at most 1.7X in the i.i.d . case and almost the triple in the non–i.i.d . case . Our second algorithm adds slightly more than an order of magnitude overhead : here the non-i.i.d . slowdown is twice the i.i.d . one . Related work : Byzantine learning . Several techniques have recently been proposed for Byzantine distributed learning , where different workers collaborate through a central parameter server [ 30 ] to minimize the average of their loss functions [ 26 ] . In each round , the server sends its model parameters to the workers which use their local data to compute gradients . Krum and Multi-Krum [ 4 ] use a distance–based scheme to eliminate Byzantine inputs and average the remaining ones . Median-based aggregation alternatives were also considered [ 38 ] . Bulyan [ 13 ] uses a meta–algorithm against a strong adversary that can fool the aforementioned aggregation rules in high–dimensional spaces . Coding schemes were used in Draco [ 8 ] and Detox [ 35 ] . In [ 2 ] , quorums of workers enable to reach an information theoretical learning optimum , assuming however a strong convex loss function . Kardam [ 9 ] uses filters to tolerate Byzantine workers in an asynchronous setting . All these approaches assume a central trusted ( parameter server ) machine . The few decentralized approaches that removed this single point of failure , restricted however the problem to ( a ) homogeneous data distribution , ( b ) convex functions , and/or ( c ) a weak ( non– Byzantine ) adversary . MOZI [ 22 ] combines a distance–based aggregation rule with a performance– based filtering technique , assuming that adversaries send models with high loss values , restricting thereby the arbitrary nature of a Byzantine agent that can craft poisoned models whose losses are small only with respect to the honest nodes ’ incomplete loss functions . The technique is also inapplicable to heterogeneous learning , where nodes can have a biased loss function compared to the average of all loss functions2 . BRIDGE [ 39 ] and ByRDiE [ 40 ] consider gradient descent ( GD ) and coordinate descent ( CD ) optimizations , respectively . Both rely on trimmed–mean to achieve Byzantine resilience assuming a synchronous environment and strongly convex loss functions3 with homogeneous data distribution ( i.i.d. ) . In addition , none of their optimization methods is stochastic : at each step , each node is supposed to compute the gradient on its entire local data set . ByzSGD [ 12 ] starts from the classical model of several workers and one server , which is then replicated for Byzantine resilience . It is assumed that up to 1/3 of the server replicas and up to 1/3 of the workers can be Byzantine , which is stronger than what we assume in the present paper where nodes play both roles and tolerate any subset of 1/3 Byzantine nodes . More importantly , ByzSGD assumes that all communication patterns between honest servers eventually hold with probability 1 ; we make no such assumption here . Additionally , our present paper is more general , considering heterogeneous data distributions , as opposed to [ 12 ] . Furthermore , heterogeneity naturally calls for personalized collaborative learning [ 14 , 23 , 10 , 15 ] , where nodes aim to learn local models , but still leverage collaborations to improve their local models . Interestingly , our general scheme encompasses personalized collaborative learning . Maybe more importantly , our reduction to averaging agreement yields new more precise bounds that improve upon all the results listed above . These are we believe of interest , even in more centralized , homogeneous , synchronous and convex settings . In particular , our reduction can easily be adapted to settings where parameter servers and workers with local data are different entities , as in [ 12 ] . Related work : Agreement . A major challenge in collaborative learning is to guarantee `` some '' agreement between nodes about the appropriate parameters to consider . Especially in non-convex settings , this is critical as , otherwise , the gradients computed by a node may be completely irrelevant for another node . The agreement could be achieved using the traditional consensus abstraction [ 28 ] . Yet , consensus is impossible in asynchronous environments [ 18 ] and when it is possible ( with partial synchrony ) , its usage is expensive and would be prohibitive in the context of modern ML models , with a dimension d in the order of billions . In fact , and as we show in this paper , consensus is unnecessary . An alternative candidate abstraction is approximate agreement . This is a weak form of consensus introduced in [ 11 ] where honest nodes converge to values that are close to each other , while remaining in the convex hull of the values proposed by honest nodes . In the one-dimensional case , optimal convergence rate has been achieved in both synchronous [ 16 ] and asynchronous environments [ 17 ] , while optimal asynchronous Byzantine tolerance was attained by [ 1 ] . The multi-dimensional version was addressed by [ 31 ] , requiring however nd local computations in each round , and assuming n > f ( d+ 2 ) . This is clearly impractical in the context of modern ML . By leveraging some distributed computing techniques [ 36 , 1 ] , we prove that collaborative learning can be reduced to averaging agreement , which is even weaker than approximate agreement . This enables us to bring down the requirement on the number of honest nodes from n > f ( d + 2 ) to n > 3f , and only require linear computation time in d. Structure . The rest of the paper is organized as follows . In Section 2 , we precisely define the problems we aim to solve . Section 3 states our main result , namely , the equivalence between collaborative learning and averaging agreement . Section 4 describes our two solutions to averaging agreement , and proves their optimality . Section 5 reports on our empirical evaluation and highlight important takeaways . Finally , Section 6 concludes . The full proofs are provided in the supplementary material , as well as the optimized algorithm for homogeneous local data . 2Besides , MOZI , focusing on convex optimization , assumes that eventually , models on honest nodes do not drift among each others , which may not hold for Byzantine nodes could influence the honest models to drift away from each other [ 3 ] . 3Convexity greatly helps , as the average of good models will necessarily be a good model . This is no longer the case in non-convex optimization , which includes the widely used neural network framework . 2 Model and Problem Definitions . 2.1 Distributed computing assumptions . We consider a standard distributed computing model with a set [ n ] = { 1 , . . . , n } of nodes , out of which h are honest and f = n− h are Byzantine . For presentation simplicity , we assume that the first h nodes are honest . But crucially , no honest node knows which h− 1 other nodes are honest . The f Byzantine nodes know each other , can collude , and subsequently know who the h remaining honest nodes are . Essentially , we assume a single adversary that controls all the Byzantine nodes . These nodes can send arbitrary messages , and they can send different messages to different nodes . In the terminology of distributed computing , the adversary is omniscient but not omnipotent . Such an adversary has access to all learning and deployment information , including the learning objective , the employed algorithm , as well as the dataset . We consider a general asynchronous setting [ 5 ] : the adversary can delay messages to honest nodes : no bound on communication delays or relative speeds is assumed . We denote BYZ the algorithm adopted by the adversary . Yet , we assume that the adversary is not able to delay all messages indefinitely [ 7 ] . Besides , the adversary is not able to alter the messages from the honest nodes , which can authenticate the source of a message to prevent spoofing and Sybil attacks . Also for presentation simplicity , we assume that processes communicate in a round-based manner [ 5 ] . In each round , every honest node broadcasts a message ( labelled with the round number ) and waits until it successfully gathers messages from at most q ≤ h other nodes ( labelled with the correct round number ) , before performing some local computation and moving to the next round . Even though the network is asynchronous , each round is guaranteed to eventually terminate for all honest nodes , as the h honest nodes ’ messages will all be eventually delivered . Evidently , however , some of them may be delivered after the node receives q messages ( including Byzantine nodes ’ ) . Such messages will fail to be taken into account . Our learning algorithm will then rely on main rounds ( denoted t in Section 3 ) , each of which is decomposed into sub-rounds that run averaging agreements .

Summary: The paper considers collaborative learning in an (asynchronous) decentralized setup in the presence of Byzantine nodes. The authors define two problems: Byzantine-robust collaborative learning and Byzantine-robust ‘averaging agreement’. The main contribution is to show an equivalence between these two problems. Then, the paper proposes two algorithms for averaging agreement, which yield Byzantine-robust collaborative learning solutions. Finally, the paper shows that when the number of Byzantine nodes is larger than 1/3, then no algorithm can achieve Byzantine averaging agreement.