Asynchronous Decentralized Online Learning

1 Introduction . Decentralized online learning has recently received increasing research attention due to its capability of processing large-scale streaming data in decentralized computational architectures ( e.g. , mobile networks and CPU/GPU clusters ) . Many different algorithms have been proposed [ 31 , 32 , 22 , 15 , 28 , 16 , 24 ] and successfully applied in various application scenarios , such as recommendation systems [ 29 ] , energy management [ 5 ] and cooperative navigation [ 21 ] . Most of the existing decentralized online algorithms are investigated under the synchronous setting , where all the learners are restricted to communicating their updated information ( e.g. , model parameters , gradients , dual variables ) with each other at the same pace . However , this synchronization mechanism incurs the straggler problem , which can markedly deteriorate the efficiency of the algorithms [ 30 , 17 , 3 ] . In more detail , at each iteration , the fast learners have to wait for the slow learners , i.e. , the stragglers , to finish their update and communication . Consequently , their speeds will be dragged down by those of the stragglers , which may dramatically reduce the algorithms ’ overall efficiency as the speeds of different learners in a decentralized system usually vary widely owing to varying computing powers , storage capacities , network bandwidths , etc . As is well acknowledged , designing the asynchronous counterpart of a synchronous algorithm is the most common way to resolve the straggler problem . However , compared to the well-studied ∗Equal contributions . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . centralized online [ 9 , 18 ] and decentralized stochastic [ 17 , 12 , 3 , 23 ] settings , how to do asynchronous learning in the decentralized online setting remains largely unexplored . The major obstacle is its intrinsic hardness . In asynchronous decentralized online learning , each learner has five basic types of actions , namely prediction , weight update , message sending , message receiving , and model averaging . Due to asynchronization , each learner executes its own actions independently . Hence the orders of the actions of different learners are intermixed together , which results in very complex interaction dynamics . By far , even no formal framework exists that can accurately describe the whole process of asynchronous decentralized online learning , not to mention algorithm design or theoretical analysis . In this paper , we conduct a systematic study of asynchronous decentralized online learning . To begin with , we formulate the framework of Asynchronous Decentralized Online Convex Optimization ( AD-OCO ) . We first specify the action sequence of each learner using its own indices , then propose a novel event indexing system , which delicately characterizes the complete orders of actions of all learners by mapping them into a single time axis . To the best of our knowledge , this is the first formal framework that gives a complete characterization of the whole interaction dynamics of asynchronous decentralized online learning . Based on the proposed AD-OCO framework , we develop the Asynchronous Decentralized Online Gradient-Push ( AD-OGP ) algorithm . AD-OGP consists of two key modules . The first module is a gossip-based communication scheme , in which each learner is only allowed to communicate with its immediate neighbors . In general , there are two major types of gossiping strategies in decentralized algorithms , namely symmetric gossiping [ 13 , 17 ] and asymmetric gossiping [ 19 , 3 ] . In asymmetric gossiping , the gossiping sender does not need to wait for the response message from the gossiping receiver after it sends out its message . Hence asymmetric gossiping does not require any local synchronization among learners , which is needed in symmetric gossiping . This makes asymmetric gossiping more suitable for asynchronous online learning . In particular , we adopt the push-sum strategy [ 14 , 19 ] , which is the most common way to realize asymmetric gossiping . The second module is a local update scheme , which builds upon online gradient descent ( OGD ) [ 33 ] . When combining push-sum with OGD , if we directly apply the standard projection operation , the resulting algorithm crashes as it may generate predictions outside of the feasible region . To resolve this problem , we devise a novel weighted projection operation , which makes its local update compatible with push-sum . Moreover , in AD-OGP , we allow each learner to perform model averaging as long as it is unoccupied and its receiving buffer is not empty , which effectively reduces the communication delays . We emphasize that the above innovations , namely weighted projection and instantaneous model averaging , are two special designs when tailoring asynchronization to the online setting . We further conduct the first regret analysis of asynchronous decentralized online learning . In particular , we extend the graph augmentation technique [ 10 ] to the online setting to handle message delays , then disentangle the complex effects of predictions , weight update , and model averaging via our proposed event indexing system . We derive a non-trivial regret bound of AD-OGP , which captures the effects of the network structures , the levels of processing delays , and the levels of communication delays . Our bound is in the same order O ( √ T ) as that of its synchronous counterpart [ 27 ] . In addition , we make an extra contribution to the convergence analysis of push-sum . Specifically , we reduce a factor of √ n ( n is the dimension of model parameters ) in the convergence rate of push-sum compared to that in [ 2 ] , and our final bound is now independent of n. Experimental results on two large-scale real-world datasets show that AD-OGP runs substantially faster than its synchronous counterpart , with nearly negligible loss in performance . We further design elaborate experiments to verify the effectiveness of the two innovations of asymmetric gossiping and instantaneous model averaging in AD-OGP . Specifically , with either innovation removed or weakened , the reduced algorithm will be ineffective in some specific settings . The theoretical results of AD-OGP ’ s regret bound on different graphs and delay levels are also well confirmed . 2 Asynchronous Decentralized Online Convex Optimization . In this section , we formally describe the AD-OCO framework . Our framework is defined over an undirected graph G = ( V , E ) with vertex set V = { 1 , . . . , m } and edge set E ⊂ V × V . Each node i ∈ V represents an individual learner which maintains a local model wi ∈ K , where K is a convex compact decision set . Each learner i can only communicate with its immediate neighbors N ( i ) = { j ∈ V | ( i , j ) ∈ E } . We first illustrate the whole process of AD-OCO . As the process is very complex , we divide its description into two parts : ( i ) Individual view , which specifies the action sequence of each individual learner ; ( ii ) System view , which characterizes the interaction dynamics among different learners . Individual View Each learner i ∈ V is selected adversarially by the environment to make predictions at certain time points . We index the prediction rounds of learner i as τ = 1 , . . . , Ni . At each round τ , learner i predicts with its most recent model wτi . Then the environment reveals a convex loss function fτi : K → R , and learner i suffers the loss fτi ( wτi ) . When receiving the feedback , learner i begins to calculate the gradient ∇fτi ( wτi ) . After finishing the calculation , it uses the gradient to update its model wi ← U ( wi ; ∇fτi ( wτi ) ) , where U is the local update scheme . Then it selects a subset of neighbors , and sends a copy of its updated model to each of them . Besides sending out its own model copies , learner i will also receive its neighbors ’ model copies at certain time points . These copies will be stored in its receiving buffer Bi . Whenever learner i is not occupied with gradient calculation or local update , and its buffer is not empty , it can average its model with the model copies stored in the buffer , i.e. , wi ← A ( wi ; Bi ) , where A is the model averaging scheme . System View Due to asynchronization , each learner executes its actions independently . Hence the orders of actions of all learners are intermixed . To precisely characterize the interaction dynamics among learners , we need to specify the complete orders of the actions of all learners by mapping them into a single time axis . To this end , we propose a novel event indexing system . We view either a prediction or a local update of any learner as an event , and use a virtual counter t to index these events , which will increase by one each time some learner makes a prediction or completes a local update . Let T = ∑ i∈V Ni be the total number of predictions of all learners , then the total number of local updates is also T . For each event t ∈ { 1 , . . . , 2T } , we use δt ∈ { 0 , 1 } to distinguish the event type : ( i ) δt = 0 if it is a prediction event , and ( ii ) δt = 1 if it is a local update event . We further use it to denote the learner that executes the event t , and ft to denote the associated feedback ; then there exists some τ ∈ { 1 , . . . , Nit } such that ft = fτit . In particular , if event t is a local update event ( i.e. , δt = 1 ) , we denote lt as the index of the associated prediction event1 , and dg ( t ) = t− lt as the gap between these two indices . Since any prediction must occur before its corresponding local update , here we have t ≥ 2 , lt < t and dg ( t ) > 0 . Let card ( A ) denote the cardinality of set A . Now we can introduce a quantity of ft called the processing delay dp ( t ) = card { lt < s < t | δs = 1 } , which measures the number of local updates that occur between the prediction and the local update associated with ft . Note that our definition of processing delay is in line with those in other asynchronous frameworks for stochastic optimization [ 17 , 3 ] . Recall that , after any local update event t , the learner it that executes this event will send out its model copy to a subset St of its neighbors N ( it ) . We focus on the copy sent to any neighbor j ∈ St. We assume such copy is used to average learner j ’ s model between events rtj and ( rtj + 1 ) for some rtj ≥ t. For the message sent from learner it to learner j after event t , we introduce a quantity termed the message delay dmitj ( t ) = rtj − t , which measures the number of events that occur while this copy is in transmission or learner j ’ s receiving buffer . Note that in the asynchronous stochastic setting [ 3 , 23 ] , the message delay is measured by solely counting the number of local updates . In the online setting , we also count the predictions because whether model averaging occurs before or after a prediction will affect the gradient calculation , which further affects the corresponding weight update . With the above event indexing system , we can now characterize the evolution of model wi . For each learner i , we denote wi ( t ) as its most recent model before event t , and ui ( t ) as its model immediately after event t. We first give the detailed characterization of local updates . Specifically , for each local update event t , it is executed by learner it . The associated gradient was computed w.r.t . learner it ’ s model wit ( lt ) at prediction event lt . So the local update on it can be characterized as uit ( t ) = U ( wit ( t ) ; ∇flt ( wit ( lt ) ) . The models of other learners do not change at it ’ s local update event , so uj ( t ) = wj ( t ) for j ̸= it . We then give a full description of model averaging . Recall that , in our indexing system , model averaging operation is not regarded as an event . Hence we need to specify it using the indices of its nearby events . Concretely , for each learner i , we consider its model averaging during the interval 1Note that in our indexing system , when δt = 1 , the feedback and its associated learner can be represented by the index of either the prediction event lt or the local update event t , i.e. , flt = ft and ilt = it . Protocol 1 Asynchronous Decentralized Online Convex Optimization ( AD-OCO ) 1 : Input : Time horizon T , convex set K , local update scheme U , and model averaging scheme A . 2 : Initialize : Local models wj ( 1 ) ∈ K , ∀j ∈ V . 3 : for t = 1 , . . . , 2T do 4 : if δt = 0 then // for a prediction event 5 : A learner it ∈ V is selected by the environment adversarially to predict with wit ( t ) . 6 : Learner it receives a convex function ft from environment , and suffers the loss ft ( wit ( t ) ) . 7 : Learner it starts to compute the gradient∇ft ( wit ( t ) ) . 8 : else // for a local update event 9 : Learner it performs a local update : uit ( t ) = U ( wit ( t ) ; ∇flt ( wit ( lt ) ) . 10 : Learner it selects a subset St of its neighbors N ( it ) and sends uit ( t ) to each j ∈ St. 11 : end if 12 : for i ∈ V do 13 : Learner i performs model averaging using model copies B1i ( t ) , . . . , Bsi ( t ) if it is unoccupied : wi ( t+ 1 ) = { ui ( t ) , s = 0 ; A ( · · · ( A ( A ( ui ( t ) , B1i ( t ) ) , B2i ( t ) ) · · · ) , Bsi ( t ) ) , s ≥ 1 . 14 : end for 15 : end for between event t and t + 1 . If it is not occupied in this interval , we assume it performs s model averaging operations for some s ≥ 0 , each time using copies in B1i ( t ) , . . . , Bsi ( t ) . Recall that learner i ’ s models at the beginning and the end of such interval are ui ( t ) and wi ( t+ 1 ) respectively . Hence the overall effect of model averaging on learner i between event t and t+ 1 can be expressed as wi ( t+ 1 ) = { ui ( t ) , s = 0 ; A ( · · · ( A ( A ( ui ( t ) , B1i ( t ) ) , B2i ( t ) ) · · · ) , Bsi ( t ) ) , s ≥ 1 . Regret Definition In AD-OCO , the regret w.r.t . an arbitrary reference learner j ∈ V is defined as Regretj = ∑ t∈PT ft ( wj ( t ) ) − ∑ t∈PT ft ( w ∗ ) , where PT = { t ∈ { 1 , . . . , 2T } | δt = 0 } denotes the set of all prediction events , and w∗ ∈ argminw∈K ∑ t∈PT ft ( w ) denotes the best fixed model in hindsight . The goal of the learners is to make a sequence of predictions { wi ( t ) } i∈V , t∈PT , so that the regret w.r.t . any reference learner j is sublinear in T , i.e. , limT→∞ Regretj/T = 0 . Note that our regret is in line with the regret defined in the synchronous setting [ 27 , 31 ] , where all the losses { ft | t ∈ PT } are evaluated by a fixed reference learner j .

Summary: This work considers decentralized online learning in the asynchronous setup. Precisely, the learners do not make predictions at the same time and both the local updates and message transmissions incur delays. To account for this situation, the authors propose to index the time by two types of events: predictions and updates. Under this framework, an asynchronous push-sum algorithm is studied and its regret bound is established. The experiments do not only demonstrate the benefit of asynchronous method but also investigate several design choices of the algorithm.