Asynchronous Decentralized Online Learning

1 Introduction . Decentralized online learning has recently received increasing research attention due to its capability of processing large-scale streaming data in decentralized computational architectures ( e.g. , mobile networks and CPU/GPU clusters ) . Many different algorithms have been proposed [ 31 , 32 , 22 , 15 , 28 , 16 , 24 ] and successfully applied in various application scenarios , such as recommendation systems [ 29 ] , energy management [ 5 ] and cooperative navigation [ 21 ] . Most of the existing decentralized online algorithms are investigated under the synchronous setting , where all the learners are restricted to communicating their updated information ( e.g. , model parameters , gradients , dual variables ) with each other at the same pace . However , this synchronization mechanism incurs the straggler problem , which can markedly deteriorate the efficiency of the algorithms [ 30 , 17 , 3 ] . In more detail , at each iteration , the fast learners have to wait for the slow learners , i.e. , the stragglers , to finish their update and communication . Consequently , their speeds will be dragged down by those of the stragglers , which may dramatically reduce the algorithms ’ overall efficiency as the speeds of different learners in a decentralized system usually vary widely owing to varying computing powers , storage capacities , network bandwidths , etc . As is well acknowledged , designing the asynchronous counterpart of a synchronous algorithm is the most common way to resolve the straggler problem . However , compared to the well-studied ∗Equal contributions . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . centralized online [ 9 , 18 ] and decentralized stochastic [ 17 , 12 , 3 , 23 ] settings , how to do asynchronous learning in the decentralized online setting remains largely unexplored . The major obstacle is its intrinsic hardness . In asynchronous decentralized online learning , each learner has five basic types of actions , namely prediction , weight update , message sending , message receiving , and model averaging . Due to asynchronization , each learner executes its own actions independently . Hence the orders of the actions of different learners are intermixed together , which results in very complex interaction dynamics . By far , even no formal framework exists that can accurately describe the whole process of asynchronous decentralized online learning , not to mention algorithm design or theoretical analysis . In this paper , we conduct a systematic study of asynchronous decentralized online learning . To begin with , we formulate the framework of Asynchronous Decentralized Online Convex Optimization ( AD-OCO ) . We first specify the action sequence of each learner using its own indices , then propose a novel event indexing system , which delicately characterizes the complete orders of actions of all learners by mapping them into a single time axis . To the best of our knowledge , this is the first formal framework that gives a complete characterization of the whole interaction dynamics of asynchronous decentralized online learning . Based on the proposed AD-OCO framework , we develop the Asynchronous Decentralized Online Gradient-Push ( AD-OGP ) algorithm . AD-OGP consists of two key modules . The first module is a gossip-based communication scheme , in which each learner is only allowed to communicate with its immediate neighbors . In general , there are two major types of gossiping strategies in decentralized algorithms , namely symmetric gossiping [ 13 , 17 ] and asymmetric gossiping [ 19 , 3 ] . In asymmetric gossiping , the gossiping sender does not need to wait for the response message from the gossiping receiver after it sends out its message . Hence asymmetric gossiping does not require any local synchronization among learners , which is needed in symmetric gossiping . This makes asymmetric gossiping more suitable for asynchronous online learning . In particular , we adopt the push-sum strategy [ 14 , 19 ] , which is the most common way to realize asymmetric gossiping . The second module is a local update scheme , which builds upon online gradient descent ( OGD ) [ 33 ] . When combining push-sum with OGD , if we directly apply the standard projection operation , the resulting algorithm crashes as it may generate predictions outside of the feasible region . To resolve this problem , we devise a novel weighted projection operation , which makes its local update compatible with push-sum . Moreover , in AD-OGP , we allow each learner to perform model averaging as long as it is unoccupied and its receiving buffer is not empty , which effectively reduces the communication delays . We emphasize that the above innovations , namely weighted projection and instantaneous model averaging , are two special designs when tailoring asynchronization to the online setting . We further conduct the first regret analysis of asynchronous decentralized online learning . In particular , we extend the graph augmentation technique [ 10 ] to the online setting to handle message delays , then disentangle the complex effects of predictions , weight update , and model averaging via our proposed event indexing system . We derive a non-trivial regret bound of AD-OGP , which captures the effects of the network structures , the levels of processing delays , and the levels of communication delays . Our bound is in the same order O ( √ T ) as that of its synchronous counterpart [ 27 ] . In addition , we make an extra contribution to the convergence analysis of push-sum . Specifically , we reduce a factor of √ n ( n is the dimension of model parameters ) in the convergence rate of push-sum compared to that in [ 2 ] , and our final bound is now independent of n. Experimental results on two large-scale real-world datasets show that AD-OGP runs substantially faster than its synchronous counterpart , with nearly negligible loss in performance . We further design elaborate experiments to verify the effectiveness of the two innovations of asymmetric gossiping and instantaneous model averaging in AD-OGP . Specifically , with either innovation removed or weakened , the reduced algorithm will be ineffective in some specific settings . The theoretical results of AD-OGP ’ s regret bound on different graphs and delay levels are also well confirmed . 2 Asynchronous Decentralized Online Convex Optimization . In this section , we formally describe the AD-OCO framework . Our framework is defined over an undirected graph G = ( V , E ) with vertex set V = { 1 , . . . , m } and edge set E ⊂ V × V . Each node i ∈ V represents an individual learner which maintains a local model wi ∈ K , where K is a convex compact decision set . Each learner i can only communicate with its immediate neighbors N ( i ) = { j ∈ V | ( i , j ) ∈ E } . We first illustrate the whole process of AD-OCO . As the process is very complex , we divide its description into two parts : ( i ) Individual view , which specifies the action sequence of each individual learner ; ( ii ) System view , which characterizes the interaction dynamics among different learners . Individual View Each learner i ∈ V is selected adversarially by the environment to make predictions at certain time points . We index the prediction rounds of learner i as τ = 1 , . . . , Ni . At each round τ , learner i predicts with its most recent model wτi . Then the environment reveals a convex loss function fτi : K → R , and learner i suffers the loss fτi ( wτi ) . When receiving the feedback , learner i begins to calculate the gradient ∇fτi ( wτi ) . After finishing the calculation , it uses the gradient to update its model wi ← U ( wi ; ∇fτi ( wτi ) ) , where U is the local update scheme . Then it selects a subset of neighbors , and sends a copy of its updated model to each of them . Besides sending out its own model copies , learner i will also receive its neighbors ’ model copies at certain time points . These copies will be stored in its receiving buffer Bi . Whenever learner i is not occupied with gradient calculation or local update , and its buffer is not empty , it can average its model with the model copies stored in the buffer , i.e. , wi ← A ( wi ; Bi ) , where A is the model averaging scheme . System View Due to asynchronization , each learner executes its actions independently . Hence the orders of actions of all learners are intermixed . To precisely characterize the interaction dynamics among learners , we need to specify the complete orders of the actions of all learners by mapping them into a single time axis . To this end , we propose a novel event indexing system . We view either a prediction or a local update of any learner as an event , and use a virtual counter t to index these events , which will increase by one each time some learner makes a prediction or completes a local update . Let T = ∑ i∈V Ni be the total number of predictions of all learners , then the total number of local updates is also T . For each event t ∈ { 1 , . . . , 2T } , we use δt ∈ { 0 , 1 } to distinguish the event type : ( i ) δt = 0 if it is a prediction event , and ( ii ) δt = 1 if it is a local update event . We further use it to denote the learner that executes the event t , and ft to denote the associated feedback ; then there exists some τ ∈ { 1 , . . . , Nit } such that ft = fτit . In particular , if event t is a local update event ( i.e. , δt = 1 ) , we denote lt as the index of the associated prediction event1 , and dg ( t ) = t− lt as the gap between these two indices . Since any prediction must occur before its corresponding local update , here we have t ≥ 2 , lt < t and dg ( t ) > 0 . Let card ( A ) denote the cardinality of set A . Now we can introduce a quantity of ft called the processing delay dp ( t ) = card { lt < s < t | δs = 1 } , which measures the number of local updates that occur between the prediction and the local update associated with ft . Note that our definition of processing delay is in line with those in other asynchronous frameworks for stochastic optimization [ 17 , 3 ] . Recall that , after any local update event t , the learner it that executes this event will send out its model copy to a subset St of its neighbors N ( it ) . We focus on the copy sent to any neighbor j ∈ St. We assume such copy is used to average learner j ’ s model between events rtj and ( rtj + 1 ) for some rtj ≥ t. For the message sent from learner it to learner j after event t , we introduce a quantity termed the message delay dmitj ( t ) = rtj − t , which measures the number of events that occur while this copy is in transmission or learner j ’ s receiving buffer . Note that in the asynchronous stochastic setting [ 3 , 23 ] , the message delay is measured by solely counting the number of local updates . In the online setting , we also count the predictions because whether model averaging occurs before or after a prediction will affect the gradient calculation , which further affects the corresponding weight update . With the above event indexing system , we can now characterize the evolution of model wi . For each learner i , we denote wi ( t ) as its most recent model before event t , and ui ( t ) as its model immediately after event t. We first give the detailed characterization of local updates . Specifically , for each local update event t , it is executed by learner it . The associated gradient was computed w.r.t . learner it ’ s model wit ( lt ) at prediction event lt . So the local update on it can be characterized as uit ( t ) = U ( wit ( t ) ; ∇flt ( wit ( lt ) ) . The models of other learners do not change at it ’ s local update event , so uj ( t ) = wj ( t ) for j ̸= it . We then give a full description of model averaging . Recall that , in our indexing system , model averaging operation is not regarded as an event . Hence we need to specify it using the indices of its nearby events . Concretely , for each learner i , we consider its model averaging during the interval 1Note that in our indexing system , when δt = 1 , the feedback and its associated learner can be represented by the index of either the prediction event lt or the local update event t , i.e. , flt = ft and ilt = it . Protocol 1 Asynchronous Decentralized Online Convex Optimization ( AD-OCO ) 1 : Input : Time horizon T , convex set K , local update scheme U , and model averaging scheme A . 2 : Initialize : Local models wj ( 1 ) ∈ K , ∀j ∈ V . 3 : for t = 1 , . . . , 2T do 4 : if δt = 0 then // for a prediction event 5 : A learner it ∈ V is selected by the environment adversarially to predict with wit ( t ) . 6 : Learner it receives a convex function ft from environment , and suffers the loss ft ( wit ( t ) ) . 7 : Learner it starts to compute the gradient∇ft ( wit ( t ) ) . 8 : else // for a local update event 9 : Learner it performs a local update : uit ( t ) = U ( wit ( t ) ; ∇flt ( wit ( lt ) ) . 10 : Learner it selects a subset St of its neighbors N ( it ) and sends uit ( t ) to each j ∈ St. 11 : end if 12 : for i ∈ V do 13 : Learner i performs model averaging using model copies B1i ( t ) , . . . , Bsi ( t ) if it is unoccupied : wi ( t+ 1 ) = { ui ( t ) , s = 0 ; A ( · · · ( A ( A ( ui ( t ) , B1i ( t ) ) , B2i ( t ) ) · · · ) , Bsi ( t ) ) , s ≥ 1 . 14 : end for 15 : end for between event t and t + 1 . If it is not occupied in this interval , we assume it performs s model averaging operations for some s ≥ 0 , each time using copies in B1i ( t ) , . . . , Bsi ( t ) . Recall that learner i ’ s models at the beginning and the end of such interval are ui ( t ) and wi ( t+ 1 ) respectively . Hence the overall effect of model averaging on learner i between event t and t+ 1 can be expressed as wi ( t+ 1 ) = { ui ( t ) , s = 0 ; A ( · · · ( A ( A ( ui ( t ) , B1i ( t ) ) , B2i ( t ) ) · · · ) , Bsi ( t ) ) , s ≥ 1 . Regret Definition In AD-OCO , the regret w.r.t . an arbitrary reference learner j ∈ V is defined as Regretj = ∑ t∈PT ft ( wj ( t ) ) − ∑ t∈PT ft ( w ∗ ) , where PT = { t ∈ { 1 , . . . , 2T } | δt = 0 } denotes the set of all prediction events , and w∗ ∈ argminw∈K ∑ t∈PT ft ( w ) denotes the best fixed model in hindsight . The goal of the learners is to make a sequence of predictions { wi ( t ) } i∈V , t∈PT , so that the regret w.r.t . any reference learner j is sublinear in T , i.e. , limT→∞ Regretj/T = 0 . Note that our regret is in line with the regret defined in the synchronous setting [ 27 , 31 ] , where all the losses { ft | t ∈ PT } are evaluated by a fixed reference learner j .

Summary: This paper proposes an asynchronous decentralized learning protocol. The motivation is to remove the need for computation nodes to synchronize broadcasting their local models (e.g., to a server), which improves efficiency and brings the setting closer to practical scenarios. The main contributions include (1) an event indexing system; (2) an projection scheme to ensure weight aggregation does not exit the feasibility region; (3) an instantaneous averaging scheme that aggregates neighboring models as soon as each node goes idle and (4) a regret bound analysis.