A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

√ n−γ lnn ) a.s. ; for the 1/n type stepsize , the same is O ( √ n−1 ln lnn ) a.s . These decay rates do not depend on the graph depicting the interactions among the different agents . 1 Introduction . Can a machine train itself in the same way an infant learns to sit up , crawl , and walk ? That is , can a device interact with the environment and figure out the action sequence required to complete a given task ? The study of algorithms that enable such decision-making is what the field of Reinforcement Learning ( RL ) is all about [ 40 ] . In contrast , the mathematics needed to analyze such schemes is what forms the focus in Stochastic Approximation ( SA ) theory [ 2 , 4 ] . More generally , SA refers to an iterative scheme that helps find zeroes or optimal points of a function , for which only noisy evaluations are possible . In this work , we analyze a family of Distributed Stochastic Approximation ( DSA ) algorithms [ 24 ] that is useful in Multi-Agent Reinforcement Learning ( MARL ) [ 22 , 51 ] . In the MARL framework , we have multiple agents or learners that continually engage with a shared environment : the agents pick local actions , and the environment responds by transitioning to a new state and giving each agent a different local reward . Additionally , the agents also gossip about local computations with each other . The goal of the agents is to cooperatively find action policies that maximize the collective rewards obtained over time . Algorithms useful in this endeavor have found empirical success in domains as diverse as gaming [ 28 ] , robotics [ 29 ] , autonomous driving [ 35 ] , communication networks [ 21 ] , power grids [ 34 ] , and economics [ 19 ] . However , theoretical analyses of such schemes are still very minimal , and this is what this paper aims to address . For the purpose of analysis , MARL methods are often viewed as special cases of DSA algorithms . The archetypical form of a DSA scheme withm distributed nodes can be described as follows . Let G 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . be a directed graph representing the connections between these nodes , andW ≡ ( Wij ) ∈ [ 0 , 1 ] m×m a matrix whose ij-th entry denotes the strength of the edge j → i . It is assumed thatW is compatible with G , i.e. , Wij > 0 only if j → i ∈ G. Then , at agent i , the above scheme ( written as row vectors ) has the update rule xn+1 ( i ) = ∑ j∈Ni Wijxn ( j ) + αn [ hi ( xn ) +Mn+1 ( i ) ] , n ≥ 0 , ( 1 ) where xn ∈ Rm×d is the joint estimate of the solution at time n , its j-th row , i.e. , xn ( j ) denotes1 the estimate obtained at agent j , Ni represents the set of in-neighbors of node i in G , αn is the stepsize , hi : Rm×n → Rd is the driving function at agent i , and Mn+1 ( i ) ∈ Rd is the noise in its evaluation at time n. This update rule has two parts : a weighted average of the estimates obtained by gossip and a refinement based on local computations . Clearly , the joint update rule of all the agents is xn+1 =Wxn + αn [ h ( xn ) +Mn+1 ] , ( 2 ) where Mn+1 is the m × d matrix whose i-th row is Mn+1 ( i ) , and h is the function that maps x ∈ Rm×d to the m× d matrix whose i-th row is hi ( x ) . Two important points about the above framework are as follows : i . ) we allow hi to be a function of all of xn and not just of xn ( i ) , as is commonly assumed , and ii . ) the computations at different nodes in the above setup run synchronously on a common clock . Related Work : We now give a summary of relevant theoretical results from the DSA and MARL literature . For ease of discussion , we categorize them into i . ) asymptotic and ii . ) finite-time results . The asymptotic ones mainly concern almost sure ( a.s. ) convergence [ 42 , 3 , 26 , 24 , 15 , 52 , 50 , 39 , 18 ] . The first four papers here provide convergence guarantees for a broad family of nonlinear DSA algorithms . The other articles also do the same , but in context of specific MARL schemes such as distributed Q-learning , distributed actor-critic methods , distributed TD methods , and their offpolicy variations . Two other kinds of asymptotic results also exist in the literature . The first is the CLT shown in [ 27 ] for the average of estimates obtained at different nodes in a generic DSA scheme . The other is the convergence in mean result obtained in [ 48 ] for a distributed policy gradient method . Finite-time literature , in contrast , majorly talks about expectation bounds . Assuming there exists a unique x∗ that solves ∑m i=1 hi ( x ) = 0 , these results describe the rate at which E∥xn − x∗∥ decays with n. Notable contributions for DSA here are [ 49 , 44 ] . Compared to ours , these look at a slightly different setup : the measurement noise at each node has a Markov component in place of a martingale difference term . In this new setup , [ 49 ] shows that , for any sufficiently small but constant stepsize ϵ , the expected error decreases linearly to a ball of radius O ( √ ϵ ln ( 1/ϵ ) ) . On the other hand , [ 44 ] deals with the case where h is additionally non-convex and shows that E∥xn−x∗∥ = O ( n−1/4 √ lnn ) , which is comparable to the best known bound in the centralized setting . Expectation bounds in the MARL framework primarily concern policy evaluation methods [ 10 , 11 , 37 , 5 ] . The first three papers here deal with the distributed TD ( 0 ) method . These show that a result similar to the one in [ 49 ] holds for this method under constant stepsizes . In contrast , when αn is of the 1/n type , it is proven that E∥xn − x∗∥ = O ( 1/ √ n ) . Similar bounds have also been derived in [ 5 ] for two distributed variants of the TDC method . There are also some other works that derive finite-time bounds [ 45 , 9 , 46 , 54 , 13 , 36 , 33 , 53 ] , but we do not discuss them in this paper since the algorithms proposed there do not fit the update rule given in ( 2 ) . The different finite-time results , as also the asymptotic CLT , do provide insights into the rate at which an iterative method converges . However , there are some significant issues with these studies . First , except [ 27 ] , all others require the gossip matrix to be doubly stochastic , at least in the mean . While this assumption simplifies the analysis , it also severely restricts the communication protocol choices . In fact , as pointed out in [ 27 ] , this condition even limits the use of a natural broadcast node , one that transmits its local estimate to all the neighbors without expecting all of them to respond . Second , these works only talk about convergence rates in the expected or the CLT sense . By their very nature , these results do not reveal much about the decay rates along different sample paths . Finally , all current results , including the ones on convergence , only apply to constant or square-summable stepsizes . Nothing is known about the slowly-decaying non-square-summable 1By default , all our vectors are row vectors . We use ′ for conjugate transpose . ones , which are generally preferable since they give similar benefits as constant stepsizes and , often , also guarantee convergence . Note that such issues also plague much of the distributed stochastic optimization literature [ 47 , 38 , 20 , 16 , 32 , 31 ] . Key Contributions : The highlights of this work are as follows . 1 . Law of Iterated Logarithm ( LIL ) : We derive a novel law of iterated logarithm for the DSA scheme given in ( 2 ) . That is , for a suitably defined x∗ , we show that lim sup [ αn ln tn+1 ] −1/2∥xn − x∗∥ ≤ C a.s. on every sample path in the event E ( x∗ ) : = { xn → x∗ } . ( 3 ) Here , C ≥ 0 is some constant2 and tn = ∑n−1 k=0 αk . Also , the norm that we work is the operator norm . In particular , for any x ∈ Cm×d , ∥x∥ : = sup u∈Cm , v∈Cd { |uxv′| : ∥u∥ = ∥v∥ = 1 } . ( 4 ) This result is the first of its kind in the distributed setup . Further , as discussed in Remark 2.3 later , it provides deeper insights about the asymptotic behavior of ( xn ) n≥0 than other existing results , which only discuss convergence rates in the expected or the CLT sense . 2 . Analysis and Gossip Matrix : The above result is obtained via a new approach we develop here for analyzing DSA schemes . Let π ∈ Rm be such that πW = π and let Q : = I− 1′π , ( 5 ) where 1 ∈ Rm denotes the vector of all ones . Then an outline of our approach is that we express xn−x∗ as a sum of 1′π ( xn−x∗ ) andQxn and , thereafter , analyze each summand by treating its update rule as a separate SA scheme . This contrasts the usual approach ( e.g. , [ 27 , 10 , 11 ] ) where the error is split into ( 1′1/m ) ( xn − x∗ ) and ( I− 1′1/m ) xn . In fact , this is the main reason why , unlike other existing results , ours does not require that the gossip matrix be doubly stochastic . 3 . Concentration Inequality and Stepsizes : We also improve upon an existing concentration result ( [ 12 , Corollary 6.4.25 ] ) for a sum of martingale differences ; see Lemma 4.6 . Specifically , by modifying the original proof from [ 12 ] , we show that the result stated there actually holds under a broader set of conditions . The key benefit of this is that , unlike other related results , our LIL result does not require that the stepsize sequence be square-summable . 4 . MARL Application : We use our theory to prove a law of iterated logarithm for the distributed TD ( 0 ) algorithm with linear function approximation . This is the first such result in MARL . Contents : The rest of the paper is structured as follows . In Section 2 , we formally state our main result along with all the assumptions needed . We also pinpoint the new insights that our result provides . In Section 3 , we give a demonstration of how our result can be applied in the MARL setup . In particular , there we talk about the distributed TD ( 0 ) algorithm with linear function approximation and prove that it indeed satisfies all the assumptions of our main result . Section 4 has two parts . In the first part , we state some key intermediate lemmas and then use the same to derive our main result . The latter part , in contrast , focuses on proofs of these intermediate results ; note that we only sketch their proofs here and leave the details to the appendix . Finally , in Section 5 , we conclude with a summary of our findings and discuss some interesting future directions .

Summary: This paper studies asymptotic convergence rates for a family of distributed Stochastic Approximation Schemes in a cooperative multi-agent learning setting (Nonlinear Gossip models). They provide convergence rates which applies almost surely on sample paths, improving over the existing analysis that analyse the expected behavior. In addition they have weaker assumptions on the Gossip matrix and the step sizes (removing square summability).  As a result, they analyze policy evaluation through TD(0) with linear function approximation for the infinite horizon reinforcement learning problem and the convergence rates follow for TD(0).