Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

1 INTRODUCTION . The use of reinforcement learning ( RL ) to obtain policies that describe solutions to a Markov decision process ( MDP ) in which an autonomous agent interacting with an unknown environment aims to optimize its long term reward is now standard Sutton & Barto ( 1998 ) . Multi-agent or distributed reinforcement learning is useful when a team of agents interacts with an unknown environment or system and aims to collaboratively accomplish tasks involving distributed decision-making . Distributed here implies that agents exchange information only with their neighbors according to a certain communication graph . Recently , many distributed algorithms for multi-agent RL have been proposed and analyzed ( Zhang et al. , 2019 ) . The basic result in such works is of the type that if the graph describing the communication among the agents is bi-directional ( and hence can be represented by a doubly stochastic matrix ) , then an algorithm that builds on traditional consensus algorithms converges to a solution in terms of policies to be followed by the agents that optimize the sum of the utility functions of all the agents ; further , both finite and infinite time performance of such algorithms can be characterized ( Doan et al. , 2019 ; Zhang et al. , 2018b ) . This paper aims to relax the assumption of requiring bi-directional communication among agents in a distributed RL algorithm . This assumption is arguably restrictive and will be violated due to reasons such as packet drops or delays , differing privacy constraints among the agents , heterogeneous capabilities among the agents in which some agents may be able to communicate more often or with more power than others , adversarial attacks , or even sophisticated resilient consensus algorithms being used to construct the distributed RL algorithm . A uni-directional communication graph can be represented through a ( possibly time-varying ) stochastic – which may not be doubly stochastic – matrix being used in the algorithm . As we discuss in more detail below , relaxing the assumption of a doubly stochastic matrix to simply a stochastic matrix in the multi-agent and distributed RL algorithms that have been proposed in the literature , however , complicates the proofs of their convergence and finite time performance characterizations . The main result in this paper is to provide a finite time bound on the mean-square error for a multi-agent linear stochastic approximation algorithm in which the agents interact over a time-varying directed graph characterized by a stochastic matrix . This paper , thus , extends the applicability of distributed and multi-agent RL algorithms presented in the literature to situations such as those mentioned above where bidirectional communication at every time step can not be guaranteed . As we shall see , this extension is technically challenging and requires new proof techniques that may be of independent interest . Related Work ( See Appendix B for more related work ) Many distributed reinforcement learning algorithms have now been proposed in the literature . In this setting , each agent can receive information only from its neighbors , and no single agent can solve the problem alone or by ‘ taking the lead ’ . A backbone of almost all distributed RL algorithms proposed in the literature is the consensus-type interaction among the agents , dating back at least to Tsitsiklis ( 1984 ) . Many works have analyzed asymptotic convergence of such RL algorithms using ODE methods ( Zhang & Zavlanos , 2019 ; Zhang et al. , 2018b ; Suttle et al. , 2020 ; Zhang et al. , 2018a ) . This can be viewed as an application of ideas from distributed stochastic approximation ( Kushner & Yin , 1987 ; Stanković et al. , 2010 ; Huang , 2012 ; Stanković & Stanković , 2016 ; Bianchi et al. , 2013 ; Stanković et al. , 2016 ) . Finite-time performance guarantees for distributed RL have also been provided in works , most notably in Doan et al . ( 2019 ; 2021 ) ; Wang et al . ( 2020 ) ; Zhang et al . ( 2021 ) ; Sun et al . ( 2020 ) ; Zeng et al . ( 2020 ) . The assumption that is the central concern of this paper and is made in all the existing finite-time analyses for distributed RL algorithms is that the consensus interaction is characterized by doubly stochastic matrices ( Doan et al. , 2019 ; 2021 ; Wang et al. , 2020 ; Zhang et al. , 2021 ; Sun et al. , 2020 ; Zeng et al. , 2020 ) at every time step , or at least in expectation , i.e. , W1 = 1 and 1 > E ( W ) = 1 > ( Bianchi et al. , 2013 ) . Intuitively , doubly stochastic matrices imply symmetry in the communication graph , which almost always requires bidirectional communication graphs . More formally , the assumption of doubly stochastic matrices is restrictive since distributed construction of a doubly stochastic matrix needs to either invoke algorithms such as the Metropolis algorithm ( Xiao et al. , 2005 ) which requires bi-directional communication of each agent ’ s degree information ; or to utilize an additional distributed algorithm ( Gharesifard & Cortés , 2012 ) which significantly increases the complexity of the whole algorithm design . Doubly stochastic matrices in expectation can be guaranteed via so-called broadcast gossip algorithms which still requires bi-directional communication for convergence ( Bianchi et al. , 2013 ) . In a realistic network , especially with mobile agents such as autonomous vehicles , drones , or robots , uni-directional communication is inevitable due to various reasons such as asymmetric communication and privacy constraints , non-zero communication failure probability between any two agents at any given time , and application of resilient consensus in the presence of adversary attacks ( Vaidya et al. , 2012 ; LeBlanc et al. , 2013 ) , all leading to an interaction among the agents characterized by a stochastic matrix , which may further be timevarying . The problem of design of distributed RL algorithms with time-varying stochastic matrices and characterizing either their asymptotic convergence or finite time analysis remains open . As a step towards solving this problem , we propose a novel distributed stochastic approximation algorithm and provide its convergence analyses when a time-dependent stochastic matrix is being used due to uni-directional communication in a dynamic network . One of the first guarantees to be lost as the assumption of doubly stochastic matrices is removed is that the algorithm converges to a “ policy ” that maximizes the sum of reward functions of all the agents . Instead , the convergence is to a set of policies that optimize a convex combination of the network-wise accumulative reward , with the exact combination depending on the limit product of the infinite sequence of stochastic matrices . Nonetheless , by defining the error as the deviation of the output of the algorithm from the eventual equilibrium point , we derive finite-time bounds on the mean-square error . We consider both the cases with constant and time-varying step sizes . In the important special case where the goal is to optimize the average of the individual accumulative rewards of all the agents , we provide a distributed stochastic approximation algorithm , which builds on the push-sum idea ( Kempe et al. , 2003 ) that has been used to solve distributed averaging problem over strongly connected graphs , and characterize its finite-time performance . Thus , this paper provides the first distributed algorithm that can be applied ( e.g. , in Temporal difference ( TD ) learning , see Appendix D ) to converge to the policy maximizing the team objective of the sum of the individual utility functions over time-varying , unidirectional , communication graphs , and characterizes the finite-time bounds on the mean-square error of the algorithm output from the equilibrium point under appropriate assumptions . Technical Innovation and Contributions There are two main technical challenges in removing the assumption of doubly stochastic matrices being used in the analysis of distributed stochastic approximation algorithms . The first is in the direction of finite-time analysis . For distributed RL algorithms , finite-time performance analysis essentially boils down to two parts , namely bounding the consensus error and bounding the “ single-agent ” mean-square error . For the case when consensus interaction matrices are all doubly stochastic , the consensus error bound can be derived by analyzing the square of the 2-norm of the deviation of the current state of each agent from the average of the states of the agents . With consensus in the presence of doubly stochastic matrices , the average of the states of the agents remains invariant . Thus , it is possible to treat the average value as the state of a fictitious agent to derive the mean-square consensus error bound with respect to the limiting point . More formally , this process relies on two properties of a doubly stochastic matrix W , namely that ( 1 ) 1 > W = 1 > , and ( 2 ) if xt+1 = Wxt , then ‖xt+1 − ( 1 > xt+1 ) 1‖2 ≤ σ2 ( W ) ‖xt − ( 1 > xt ) 1‖2 where σ2 ( W ) denotes the second largest singular value of W ( which is strictly less than one if W is irreducible ) . Even if the doubly stochastic matrix is time-varying ( denoted by Wt ) , property ( 1 ) still holds and property ( 2 ) can be generalized as in Nedić et al . ( 2018 ) . Thus , the square of the 2-norm ‖xt − ( 1 > xt ) 1‖22 is a quadratic Lyapunov function for the average consensus processes . Doubly stochastic matrices in expectation can be treated in the same way by looking at the expectation . This is the core on which all the existing finite-time analyses of distributed RL algorithms are based . However , if each consensus interaction matrix is stochastic , and not necessarily doubly stochastic , the above two properties may not hold . In fact , it is well known that quadratic Lyapunov functions for general consensus processes xt+1 = Stxt , with St being stochastic , do not exist ( Olshevsky & Tsitsiklis , 2008 ) . This breaks down all the existing analyses and provides the first technical challenge that we tackle in this paper . Specifically , we appeal to the idea of quadratic comparison functions for general consensus processes . This was first proposed in Touri ( 2012 ) and makes use of the concept of “ absolute probability sequences ” . We provide a general analysis methodology and results that subsume the existing finite-time analyses for single-timescale distributed linear stochastic approximation and TD learning as special cases ( see Appendix D ) . The second technical challenge arises from the fact that with stochastic matrices , the distributed RL algorithms may not converge to the policies that maximize the average of the utility functions of the agents . To regain this property , we propose a new algorithm that utilizes a push-sum protocol for consensus . However , finite-time analysis for such a push-based distributed algorithm is challenging . Almost all , if not all , the existing push-based distributed optimization works build on the analysis in Nedić & Olshevsky ( 2014 ) ; however , that analysis assumes that a convex combination of the entire history of the states of each agent ( and not merely the current state of the agent ) is being calculated . This assumption no longer holds in our case . To obtain a direct finite-time error bound without this assumption , we propose a new approach to analyze our push-based distributed algorithm by leveraging our consensus-based analyses to establish direct finite-time error bounds for stochastic approximation . Specifically , we tailor an “ absolute probability sequence ” for the pushbased stochastic approximation algorithm and exploit its properties . Such properties have never been found in the existing literature and may be of independent interest for analyzing any push-sum based distributed algorithm . We now list the main contributions of our work . We propose a novel consensus-based distributed linear stochastic approximation algorithm driven by Markovian noise in which each agent evolves according to its local stochastic approximation process and the information from its neighbors . We assume only a ( possibly time-varying ) stochastic matrix being used during the consensus phase , which is a more practical assumption when only unidirectional communication is possible among agents . We establish both convergence guarantees and finite-time bounds on the mean-square error , defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation . The equilibrium point can be an “ uncontrollable ” convex combination of the local equilibria of all the agents in the absence of communication . We consider both the cases of constant and time-varying step-sizes . Our results subsume the existing results on convergence and finite-time analysis of distributed RL algorithms that assume doubly stochastic matrices and bi-directional communication as special cases . In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional , we propose a push-type distributed stochastic approximation algorithm and establish its finite-time performance bound . It is worth emphasizing that it is straightforward to extend our algorithm from the straight average point to any pre-specified convex combination . Since it is well known that TD algorithms can be viewed as a special case of linear stochastic approximation ( Tsitsiklis & Roy , 1997 ) , our distributed linear stochastic approximation algorithms and their finitetime bounds can be applied to TD algorithms in a straight-forward manner ; ( see distributed TD ( λ ) in Appendix D ) . Notation We use Xt to represent that a variable X is time-dependent and t ∈ { 0 , 1 , 2 , . . . } is the discrete time index . The ith entry of a vector x will be denoted by xi and , also , by ( x ) i when convenient . The ijth entry of a matrixAwill be denoted by aij and , also , by ( A ) ij when convenient . We use 1n to denote the vectors in Rn whose entries all equal to 1 ’ s , and I to denote the identity matrix , whose dimension is to be understood from the context . Given a set S with finitely many elements , we use |S| to denote the cardinality of S. We use d·e to denote the ceiling function . A vector is called a stochastic vector if its entries are nonnegative and sum to one . A square nonnegative matrix is called a row stochastic matrix , or simply stochastic matrix , if its row sums all equal one . Similarly , a square nonnegative matrix is called a column stochastic matrix if its column sums all equal one . A square nonnegative matrix is called a doubly stochastic matrix if its row sums and column sums all equal one . The graph of an n × n matrix is a direct graph with n vertices and a directed edge from vertex i to vertex j whenever the ji-th entry of the matrix is nonzero . A directed graph is strongly connected if it has a directed path from any vertex to any other vertex . For a strongly connected graph G , the distance from vertex i to another vertex j is the length of the shortest directed path from i to j ; the longest distance among all ordered pairs of distinct vertices i and j in G is called the diameter of G. The union of two directed graphs , Gp and Gq , with the same vertex set , written Gp ∪Gq , is meant the directed graph with the same vertex set and edge set being the union of the edge set of Gp and Gq . Since this union is a commutative and associative binary operation , the definition extends unambiguously to any finite sequence of directed graphs .

Summary: This paper studies the convergence distributed linear SA under Markovian sampling. Both asymptotic and non-asymptotic convergence rate results are provided. This paper consider the setting in which the agents interact over a time-varying directed graph, which extend previous studies. Overall the results in this paper is solid and has some technical contributions.