Multi-Agent Reinforcement Learning in Stochastic Networked Systems

1 Introduction . Multi-Agent Reinforcement Learning ( MARL ) has achieved impressive performance in a wide array of applications including multi-player game play [ 42 , 31 ] , multi-robot systems [ 13 ] , and autonomous driving [ 25 ] . In comparison to single-agent reinforcement learning ( RL ) , MARL poses many challenges , chief of which is scalability [ 57 ] . Even if each agent ’ s local state/action spaces are small , the size of the global state/action space can be large , potentially exponentially large in the number of agents , which renders many RL algorithms such as Q-learning not applicable . A promising approach for addressing the scalability challenge that has received attention in recent years is to exploit application-specific structures , e.g. , [ 18 , 35 , 38 ] . A particularly important example of such a structure is a networked structure , e.g. , applications in multi-agent networked systems such as social networks [ 7 , 27 ] , communication networks [ 60 , 51 ] , queueing networks [ 34 ] , and smart transportation networks [ 59 ] . In these networked systems , it is often possible to exploit static , local dependency structures [ 16 , 17 , 1 , 32 ] , e.g. , the fact that agents only interact with a fixed set of neighboring agents throughout the game . This sort of dependency structure often leads to scalable , distributed algorithms for optimization and control [ 16 , 1 , 32 ] , and has proven effective for designing scalable and distributed MARL algorithms , e.g . [ 35 , 38 ] . This work was supported by NSF grants CNS-2106403 and NGSDI-2105648 , with additional support from Amazon AWS , PIMCO , and the Resnick Sustainability Insitute . Yiheng Lin was supported by Kortschak Scholars program . The work of Longbo Huang was supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grants 2020AAA0108400 and 2020AAA0108403 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , many real-world networked systems have inherently time-varying , non-local dependencies . For example , in the context of wireless networks , each node can send packets to other nodes within a fixed transmission range . However , the interference range , in which other nodes can interfere the transmission , can be larger than the transmission range [ 53 ] . As a result , due to potential collisions , the local reward of each node not only depends on its own local state/action , but also depends on the actions of other nodes within the interference range , which may be more than one-hop away . In addition , a node may be able to observe other nodes ’ local states before picking its local action [ 33 ] . Things become even more complex when mobility and stochastic network conditions are considered . These lead to dependencies that are both stochastic and non-local . Although one can always fix and localize the dependence model , this leads to considerably reduced performance . Beyond wireless networks , similar stochastic and non-local dependencies exists in epidemics [ 30 ] , social networks [ 7 , 27 ] , and smart transportation networks [ 59 ] . A challenging open question in MARL is to understand how to obtain algorithms that are scalable in settings where the dependencies are stochastic and non-local . Prior work considers exclusively static and local dependencies , e.g. , [ 35 , 38 ] . It is clear that hardness results apply when the dependencies are too general [ 24 ] . Further , results in the static , local setting to this point rely on the concept of exponential decay [ 35 , 16 ] , meaning the agents ’ impact on each other decays exponentially in their graph distance . This property relies on the fact that the dependencies are purely local and static , and it is not clear whether it can still be exploited when the interactions are more general . This motivates an important open question : Is it possible to design scalable algorithms for stochastic , non-local networked MARL ? Contributions . In this paper , we introduce a class of stochastic , non-local dependency structures where every agent is allowed to depend on a random subset of agents . In this context , we propose and analyze a Scalable Actor Critic ( SAC ) algorithm that provably learns a near-optimal local policy in a scalable manner ( Theorem 2.5 ) . This result represents the first provably scalable method for stochastic networked MARL . Key to our approach is that the class of dependencies we consider leads to a µ-decay property ( Definition 2.1 ) . This property generalizes the exponential decay property underlying recent results such as [ 35 , 16 ] , which does not apply to stochastic non-local dependencies , and enables the design of an efficient and scalable algorithm for settings with stochastic , non-local dependencies . Our analysis of the algorithm reveals an important trade-off : as deeper interactions appear more frequently , the “ information ” can spread more quickly from one part of the network to another , which leads to the efficiency of the proposed method to degrade . This is to be expected , as when the agents are allowed to interact globally , the problem becomes a single-agent tabular Q-learning problem with an exponentially large state space , which is known to be intractable since the sample complexity is polynomial in the size of the state/action space [ 12 , 24 ] . The key technical result underlying our analysis of the Scalable Actor Critic algorithm is a finite-time analysis of a general stochastic approximation scheme featuring infinite-norm contraction and state aggregation ( Theorem 3.1 ) . We apply this result to networked MARL using the local neighborhood of each agent to provide state aggregation ( SA ) . This result also applies beyond MARL . Specifically , we show that it yields finite-time bounds on Temporal Difference ( TD ) /Q learning with state aggregation ( Theorem 3.2 ) . To the best of our knowledge the resulting bound is the first finite-time bound on asynchronous Q-learning with state aggregation . Additionally , it yields a novel analysis for TDlearning with state aggregation ( the first error bound in the infinity norm ) that sheds new insight into how the error depends on the quality of state abstraction . These two results are important contributions in their own right . Due to space constraints , we discuss asynchronous Q-learning with state aggregation in Appendix D.4 . Related literature . The prior work that is most related to our paper is [ 38 ] , which also studies MARL in a networked setting . The key difference is that we allow the dependency structure among agents to be non-local and stochastic , while [ 38 ] requires the dependency structure to be local and static . The generality of setting means techniques from [ 38 ] do not apply and adds considerable complexity to the proof in two aspects . First , instead of analyzing the algorithm directly like [ 38 ] , we derive a finite-time error bound for TD learning with state aggregation ( Section 3.1 and 3.2 ) , and then establish its connection with the algorithm ( Section 2.3 ) . Second , we need a more general decay property ( Definition 2.1 ) than the exponential one used in [ 38 ] . Defining and establishing this general decay property for the non-local and stochastic setting is highly non-trivial ( Section 2.1 ) . More broadly , MARL has received considerable attention in recent years , see [ 57 ] for a survey . The line of work most relevant to the current paper focuses on cooperative MARL . In the cooperative setting , each agent can decide its local actions but share a common global state with other agents . The objective is to maximize a global reward by working cooperatively . Notable examples of this approach include [ 6 , 10 ] and the references therein . In contrast , we study a situation where each agent has its own state that it acts upon . Despite the differences , like our situation , cooperative MARL problems still face scalability issues since the joint-action space is exponentially large . A variety of methods have been proposed to deal with this , including independent learners [ 8 , 29 ] , where each agent employs a single-agent RL policy . Function approximation is another approach that can significantly reduce the space/computational complexity . One can use linear functions [ 58 ] or neural networks [ 28 ] in the approximation . A limitation of these approaches is the lack of theoretical guarantees on the approximation error . In contrast , our technique not only reduces the space/computational complexity significantly , but also has theoretical guarantees on the performance loss in settings with stochastic and non-local dependencies . The mean-field approach [ 45 , 56 , 19 ] provides another way to address the scalability issue , but under very different settings compared to ours . Specifically , the mean-field approach typically assumes homogeneous agents with identical local state/action space and policies , and each agent depends on other agents through their population or “ mean ” behavior . In contrast , our approach considers a local-interaction model , where there is an underlying graph and each agent depends on neighboring agents in the graph . Further , our approach allows heterogeneous agents , which means that the local state/action spaces and policies can differ among the agents . Another related line of work uses centralized training with decentralized execution , e.g. , [ 28 , 15 ] , where there is a centralized coordinator that can communicate with all the agents and keep track of their experiences and policies . In contrast , our work only requires distributed training , where we constrain the scale of communication in training within the -hop neighborhood of each agent . More broadly , this paper contributes to a growing literature that uses exponential decay to derive scalable algorithms for learning in networked systems . The specific form of exponential decay that we generalize is related to the idea of “ correlation decay ” studied in [ 16 , 17 ] , though their focus is on solving static combinatorial optimization problems whereas ours is on learning policies in dynamic environments . Most related to the current paper is [ 38 ] , which shows an exponential decay property in a restricted networked MARL model with purely local dependencies . In contrast , we show a more general µ-decay property holds for a general form of stochastic , non-local dependencies . The technical work in this paper contributes to the analysis of stochastic approximation ( SA ) , which has received considerable attention over the past decade [ 54 , 44 , 11 , 55 ] . Our work is most related to [ 37 ] , which uses an asynchronous nonlinear SA to study the finite-time convergence rate for asynchronous Q-learning on a single trajectory . Beyond [ 37 ] , there are many other works that use SA schemes to study TD learning and Q-learning , e.g . [ 44 , 52 , 20 ] . The finite-time error bound for TD learning with state aggregation in our work is most related to the asymptotic convergence limit given in [ 49 ] and the application of SA scheme to asynchronous Q-learning in [ 37 ] . Beyond these papers , other related work in the broader area of RL with state aggregation includes [ 26 , 23 , 22 , 9 , 43 ] . We add to this literature with a novel finite-time convergence bound for a general SA with state aggregation . This result , in turn , yields the first finite-time error bound in the infinity norm for both TD learning with state aggregation and Q-learning with state aggregation .

Summary: This paper first study the finite time analysis of linear SA with Markov noise, then apply the result to study TD with state aggregation. This paper further study the convegence of MARL based on exp-decay assumption.  The result is not strong and the author did not make sufficient comparison with the state-of-the-art. Specifically, the rate of linear SA/TD is not compared with the state-of-the-art in terms of conditional number dependence. The rate of AC algorithm seems to be very slow but the author did not make justification for that.