Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

1 INTRODUCTION . The use of reinforcement learning ( RL ) to obtain policies that describe solutions to a Markov decision process ( MDP ) in which an autonomous agent interacting with an unknown environment aims to optimize its long term reward is now standard Sutton & Barto ( 1998 ) . Multi-agent or distributed reinforcement learning is useful when a team of agents interacts with an unknown environment or system and aims to collaboratively accomplish tasks involving distributed decision-making . Distributed here implies that agents exchange information only with their neighbors according to a certain communication graph . Recently , many distributed algorithms for multi-agent RL have been proposed and analyzed ( Zhang et al. , 2019 ) . The basic result in such works is of the type that if the graph describing the communication among the agents is bi-directional ( and hence can be represented by a doubly stochastic matrix ) , then an algorithm that builds on traditional consensus algorithms converges to a solution in terms of policies to be followed by the agents that optimize the sum of the utility functions of all the agents ; further , both finite and infinite time performance of such algorithms can be characterized ( Doan et al. , 2019 ; Zhang et al. , 2018b ) . This paper aims to relax the assumption of requiring bi-directional communication among agents in a distributed RL algorithm . This assumption is arguably restrictive and will be violated due to reasons such as packet drops or delays , differing privacy constraints among the agents , heterogeneous capabilities among the agents in which some agents may be able to communicate more often or with more power than others , adversarial attacks , or even sophisticated resilient consensus algorithms being used to construct the distributed RL algorithm . A uni-directional communication graph can be represented through a ( possibly time-varying ) stochastic – which may not be doubly stochastic – matrix being used in the algorithm . As we discuss in more detail below , relaxing the assumption of a doubly stochastic matrix to simply a stochastic matrix in the multi-agent and distributed RL algorithms that have been proposed in the literature , however , complicates the proofs of their convergence and finite time performance characterizations . The main result in this paper is to provide a finite time bound on the mean-square error for a multi-agent linear stochastic approximation algorithm in which the agents interact over a time-varying directed graph characterized by a stochastic matrix . This paper , thus , extends the applicability of distributed and multi-agent RL algorithms presented in the literature to situations such as those mentioned above where bidirectional communication at every time step can not be guaranteed . As we shall see , this extension is technically challenging and requires new proof techniques that may be of independent interest . Related Work ( See Appendix B for more related work ) Many distributed reinforcement learning algorithms have now been proposed in the literature . In this setting , each agent can receive information only from its neighbors , and no single agent can solve the problem alone or by ‘ taking the lead ’ . A backbone of almost all distributed RL algorithms proposed in the literature is the consensus-type interaction among the agents , dating back at least to Tsitsiklis ( 1984 ) . Many works have analyzed asymptotic convergence of such RL algorithms using ODE methods ( Zhang & Zavlanos , 2019 ; Zhang et al. , 2018b ; Suttle et al. , 2020 ; Zhang et al. , 2018a ) . This can be viewed as an application of ideas from distributed stochastic approximation ( Kushner & Yin , 1987 ; Stanković et al. , 2010 ; Huang , 2012 ; Stanković & Stanković , 2016 ; Bianchi et al. , 2013 ; Stanković et al. , 2016 ) . Finite-time performance guarantees for distributed RL have also been provided in works , most notably in Doan et al . ( 2019 ; 2021 ) ; Wang et al . ( 2020 ) ; Zhang et al . ( 2021 ) ; Sun et al . ( 2020 ) ; Zeng et al . ( 2020 ) . The assumption that is the central concern of this paper and is made in all the existing finite-time analyses for distributed RL algorithms is that the consensus interaction is characterized by doubly stochastic matrices ( Doan et al. , 2019 ; 2021 ; Wang et al. , 2020 ; Zhang et al. , 2021 ; Sun et al. , 2020 ; Zeng et al. , 2020 ) at every time step , or at least in expectation , i.e. , W1 = 1 and 1 > E ( W ) = 1 > ( Bianchi et al. , 2013 ) . Intuitively , doubly stochastic matrices imply symmetry in the communication graph , which almost always requires bidirectional communication graphs . More formally , the assumption of doubly stochastic matrices is restrictive since distributed construction of a doubly stochastic matrix needs to either invoke algorithms such as the Metropolis algorithm ( Xiao et al. , 2005 ) which requires bi-directional communication of each agent ’ s degree information ; or to utilize an additional distributed algorithm ( Gharesifard & Cortés , 2012 ) which significantly increases the complexity of the whole algorithm design . Doubly stochastic matrices in expectation can be guaranteed via so-called broadcast gossip algorithms which still requires bi-directional communication for convergence ( Bianchi et al. , 2013 ) . In a realistic network , especially with mobile agents such as autonomous vehicles , drones , or robots , uni-directional communication is inevitable due to various reasons such as asymmetric communication and privacy constraints , non-zero communication failure probability between any two agents at any given time , and application of resilient consensus in the presence of adversary attacks ( Vaidya et al. , 2012 ; LeBlanc et al. , 2013 ) , all leading to an interaction among the agents characterized by a stochastic matrix , which may further be timevarying . The problem of design of distributed RL algorithms with time-varying stochastic matrices and characterizing either their asymptotic convergence or finite time analysis remains open . As a step towards solving this problem , we propose a novel distributed stochastic approximation algorithm and provide its convergence analyses when a time-dependent stochastic matrix is being used due to uni-directional communication in a dynamic network . One of the first guarantees to be lost as the assumption of doubly stochastic matrices is removed is that the algorithm converges to a “ policy ” that maximizes the sum of reward functions of all the agents . Instead , the convergence is to a set of policies that optimize a convex combination of the network-wise accumulative reward , with the exact combination depending on the limit product of the infinite sequence of stochastic matrices . Nonetheless , by defining the error as the deviation of the output of the algorithm from the eventual equilibrium point , we derive finite-time bounds on the mean-square error . We consider both the cases with constant and time-varying step sizes . In the important special case where the goal is to optimize the average of the individual accumulative rewards of all the agents , we provide a distributed stochastic approximation algorithm , which builds on the push-sum idea ( Kempe et al. , 2003 ) that has been used to solve distributed averaging problem over strongly connected graphs , and characterize its finite-time performance . Thus , this paper provides the first distributed algorithm that can be applied ( e.g. , in Temporal difference ( TD ) learning , see Appendix D ) to converge to the policy maximizing the team objective of the sum of the individual utility functions over time-varying , unidirectional , communication graphs , and characterizes the finite-time bounds on the mean-square error of the algorithm output from the equilibrium point under appropriate assumptions . Technical Innovation and Contributions There are two main technical challenges in removing the assumption of doubly stochastic matrices being used in the analysis of distributed stochastic approximation algorithms . The first is in the direction of finite-time analysis . For distributed RL algorithms , finite-time performance analysis essentially boils down to two parts , namely bounding the consensus error and bounding the “ single-agent ” mean-square error . For the case when consensus interaction matrices are all doubly stochastic , the consensus error bound can be derived by analyzing the square of the 2-norm of the deviation of the current state of each agent from the average of the states of the agents . With consensus in the presence of doubly stochastic matrices , the average of the states of the agents remains invariant . Thus , it is possible to treat the average value as the state of a fictitious agent to derive the mean-square consensus error bound with respect to the limiting point . More formally , this process relies on two properties of a doubly stochastic matrix W , namely that ( 1 ) 1 > W = 1 > , and ( 2 ) if xt+1 = Wxt , then ‖xt+1 − ( 1 > xt+1 ) 1‖2 ≤ σ2 ( W ) ‖xt − ( 1 > xt ) 1‖2 where σ2 ( W ) denotes the second largest singular value of W ( which is strictly less than one if W is irreducible ) . Even if the doubly stochastic matrix is time-varying ( denoted by Wt ) , property ( 1 ) still holds and property ( 2 ) can be generalized as in Nedić et al . ( 2018 ) . Thus , the square of the 2-norm ‖xt − ( 1 > xt ) 1‖22 is a quadratic Lyapunov function for the average consensus processes . Doubly stochastic matrices in expectation can be treated in the same way by looking at the expectation . This is the core on which all the existing finite-time analyses of distributed RL algorithms are based . However , if each consensus interaction matrix is stochastic , and not necessarily doubly stochastic , the above two properties may not hold . In fact , it is well known that quadratic Lyapunov functions for general consensus processes xt+1 = Stxt , with St being stochastic , do not exist ( Olshevsky & Tsitsiklis , 2008 ) . This breaks down all the existing analyses and provides the first technical challenge that we tackle in this paper . Specifically , we appeal to the idea of quadratic comparison functions for general consensus processes . This was first proposed in Touri ( 2012 ) and makes use of the concept of “ absolute probability sequences ” . We provide a general analysis methodology and results that subsume the existing finite-time analyses for single-timescale distributed linear stochastic approximation and TD learning as special cases ( see Appendix D ) . The second technical challenge arises from the fact that with stochastic matrices , the distributed RL algorithms may not converge to the policies that maximize the average of the utility functions of the agents . To regain this property , we propose a new algorithm that utilizes a push-sum protocol for consensus . However , finite-time analysis for such a push-based distributed algorithm is challenging . Almost all , if not all , the existing push-based distributed optimization works build on the analysis in Nedić & Olshevsky ( 2014 ) ; however , that analysis assumes that a convex combination of the entire history of the states of each agent ( and not merely the current state of the agent ) is being calculated . This assumption no longer holds in our case . To obtain a direct finite-time error bound without this assumption , we propose a new approach to analyze our push-based distributed algorithm by leveraging our consensus-based analyses to establish direct finite-time error bounds for stochastic approximation . Specifically , we tailor an “ absolute probability sequence ” for the pushbased stochastic approximation algorithm and exploit its properties . Such properties have never been found in the existing literature and may be of independent interest for analyzing any push-sum based distributed algorithm . We now list the main contributions of our work . We propose a novel consensus-based distributed linear stochastic approximation algorithm driven by Markovian noise in which each agent evolves according to its local stochastic approximation process and the information from its neighbors . We assume only a ( possibly time-varying ) stochastic matrix being used during the consensus phase , which is a more practical assumption when only unidirectional communication is possible among agents . We establish both convergence guarantees and finite-time bounds on the mean-square error , defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation . The equilibrium point can be an “ uncontrollable ” convex combination of the local equilibria of all the agents in the absence of communication . We consider both the cases of constant and time-varying step-sizes . Our results subsume the existing results on convergence and finite-time analysis of distributed RL algorithms that assume doubly stochastic matrices and bi-directional communication as special cases . In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional , we propose a push-type distributed stochastic approximation algorithm and establish its finite-time performance bound . It is worth emphasizing that it is straightforward to extend our algorithm from the straight average point to any pre-specified convex combination . Since it is well known that TD algorithms can be viewed as a special case of linear stochastic approximation ( Tsitsiklis & Roy , 1997 ) , our distributed linear stochastic approximation algorithms and their finitetime bounds can be applied to TD algorithms in a straight-forward manner ; ( see distributed TD ( λ ) in Appendix D ) . Notation We use Xt to represent that a variable X is time-dependent and t ∈ { 0 , 1 , 2 , . . . } is the discrete time index . The ith entry of a vector x will be denoted by xi and , also , by ( x ) i when convenient . The ijth entry of a matrixAwill be denoted by aij and , also , by ( A ) ij when convenient . We use 1n to denote the vectors in Rn whose entries all equal to 1 ’ s , and I to denote the identity matrix , whose dimension is to be understood from the context . Given a set S with finitely many elements , we use |S| to denote the cardinality of S. We use d·e to denote the ceiling function . A vector is called a stochastic vector if its entries are nonnegative and sum to one . A square nonnegative matrix is called a row stochastic matrix , or simply stochastic matrix , if its row sums all equal one . Similarly , a square nonnegative matrix is called a column stochastic matrix if its column sums all equal one . A square nonnegative matrix is called a doubly stochastic matrix if its row sums and column sums all equal one . The graph of an n × n matrix is a direct graph with n vertices and a directed edge from vertex i to vertex j whenever the ji-th entry of the matrix is nonzero . A directed graph is strongly connected if it has a directed path from any vertex to any other vertex . For a strongly connected graph G , the distance from vertex i to another vertex j is the length of the shortest directed path from i to j ; the longest distance among all ordered pairs of distinct vertices i and j in G is called the diameter of G. The union of two directed graphs , Gp and Gq , with the same vertex set , written Gp ∪Gq , is meant the directed graph with the same vertex set and edge set being the union of the edge set of Gp and Gq . Since this union is a commutative and associative binary operation , the definition extends unambiguously to any finite sequence of directed graphs .

Summary: This paper addresses a consensus problem in stochastic approximation, which has application to policy iteration in reinforcement learning. Specifically, the authors address the case where the interaction graph between agents is stochastic, but not doubly stochastic. Under certain assumptions, the authors prove convergence of their proposed algorithm in a certain sense.