Multi-Agent Reinforcement Learning in Stochastic Networked Systems

1 Introduction . Multi-Agent Reinforcement Learning ( MARL ) has achieved impressive performance in a wide array of applications including multi-player game play [ 42 , 31 ] , multi-robot systems [ 13 ] , and autonomous driving [ 25 ] . In comparison to single-agent reinforcement learning ( RL ) , MARL poses many challenges , chief of which is scalability [ 57 ] . Even if each agent ’ s local state/action spaces are small , the size of the global state/action space can be large , potentially exponentially large in the number of agents , which renders many RL algorithms such as Q-learning not applicable . A promising approach for addressing the scalability challenge that has received attention in recent years is to exploit application-specific structures , e.g. , [ 18 , 35 , 38 ] . A particularly important example of such a structure is a networked structure , e.g. , applications in multi-agent networked systems such as social networks [ 7 , 27 ] , communication networks [ 60 , 51 ] , queueing networks [ 34 ] , and smart transportation networks [ 59 ] . In these networked systems , it is often possible to exploit static , local dependency structures [ 16 , 17 , 1 , 32 ] , e.g. , the fact that agents only interact with a fixed set of neighboring agents throughout the game . This sort of dependency structure often leads to scalable , distributed algorithms for optimization and control [ 16 , 1 , 32 ] , and has proven effective for designing scalable and distributed MARL algorithms , e.g . [ 35 , 38 ] . This work was supported by NSF grants CNS-2106403 and NGSDI-2105648 , with additional support from Amazon AWS , PIMCO , and the Resnick Sustainability Insitute . Yiheng Lin was supported by Kortschak Scholars program . The work of Longbo Huang was supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grants 2020AAA0108400 and 2020AAA0108403 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , many real-world networked systems have inherently time-varying , non-local dependencies . For example , in the context of wireless networks , each node can send packets to other nodes within a fixed transmission range . However , the interference range , in which other nodes can interfere the transmission , can be larger than the transmission range [ 53 ] . As a result , due to potential collisions , the local reward of each node not only depends on its own local state/action , but also depends on the actions of other nodes within the interference range , which may be more than one-hop away . In addition , a node may be able to observe other nodes ’ local states before picking its local action [ 33 ] . Things become even more complex when mobility and stochastic network conditions are considered . These lead to dependencies that are both stochastic and non-local . Although one can always fix and localize the dependence model , this leads to considerably reduced performance . Beyond wireless networks , similar stochastic and non-local dependencies exists in epidemics [ 30 ] , social networks [ 7 , 27 ] , and smart transportation networks [ 59 ] . A challenging open question in MARL is to understand how to obtain algorithms that are scalable in settings where the dependencies are stochastic and non-local . Prior work considers exclusively static and local dependencies , e.g. , [ 35 , 38 ] . It is clear that hardness results apply when the dependencies are too general [ 24 ] . Further , results in the static , local setting to this point rely on the concept of exponential decay [ 35 , 16 ] , meaning the agents ’ impact on each other decays exponentially in their graph distance . This property relies on the fact that the dependencies are purely local and static , and it is not clear whether it can still be exploited when the interactions are more general . This motivates an important open question : Is it possible to design scalable algorithms for stochastic , non-local networked MARL ? Contributions . In this paper , we introduce a class of stochastic , non-local dependency structures where every agent is allowed to depend on a random subset of agents . In this context , we propose and analyze a Scalable Actor Critic ( SAC ) algorithm that provably learns a near-optimal local policy in a scalable manner ( Theorem 2.5 ) . This result represents the first provably scalable method for stochastic networked MARL . Key to our approach is that the class of dependencies we consider leads to a µ-decay property ( Definition 2.1 ) . This property generalizes the exponential decay property underlying recent results such as [ 35 , 16 ] , which does not apply to stochastic non-local dependencies , and enables the design of an efficient and scalable algorithm for settings with stochastic , non-local dependencies . Our analysis of the algorithm reveals an important trade-off : as deeper interactions appear more frequently , the “ information ” can spread more quickly from one part of the network to another , which leads to the efficiency of the proposed method to degrade . This is to be expected , as when the agents are allowed to interact globally , the problem becomes a single-agent tabular Q-learning problem with an exponentially large state space , which is known to be intractable since the sample complexity is polynomial in the size of the state/action space [ 12 , 24 ] . The key technical result underlying our analysis of the Scalable Actor Critic algorithm is a finite-time analysis of a general stochastic approximation scheme featuring infinite-norm contraction and state aggregation ( Theorem 3.1 ) . We apply this result to networked MARL using the local neighborhood of each agent to provide state aggregation ( SA ) . This result also applies beyond MARL . Specifically , we show that it yields finite-time bounds on Temporal Difference ( TD ) /Q learning with state aggregation ( Theorem 3.2 ) . To the best of our knowledge the resulting bound is the first finite-time bound on asynchronous Q-learning with state aggregation . Additionally , it yields a novel analysis for TDlearning with state aggregation ( the first error bound in the infinity norm ) that sheds new insight into how the error depends on the quality of state abstraction . These two results are important contributions in their own right . Due to space constraints , we discuss asynchronous Q-learning with state aggregation in Appendix D.4 . Related literature . The prior work that is most related to our paper is [ 38 ] , which also studies MARL in a networked setting . The key difference is that we allow the dependency structure among agents to be non-local and stochastic , while [ 38 ] requires the dependency structure to be local and static . The generality of setting means techniques from [ 38 ] do not apply and adds considerable complexity to the proof in two aspects . First , instead of analyzing the algorithm directly like [ 38 ] , we derive a finite-time error bound for TD learning with state aggregation ( Section 3.1 and 3.2 ) , and then establish its connection with the algorithm ( Section 2.3 ) . Second , we need a more general decay property ( Definition 2.1 ) than the exponential one used in [ 38 ] . Defining and establishing this general decay property for the non-local and stochastic setting is highly non-trivial ( Section 2.1 ) . More broadly , MARL has received considerable attention in recent years , see [ 57 ] for a survey . The line of work most relevant to the current paper focuses on cooperative MARL . In the cooperative setting , each agent can decide its local actions but share a common global state with other agents . The objective is to maximize a global reward by working cooperatively . Notable examples of this approach include [ 6 , 10 ] and the references therein . In contrast , we study a situation where each agent has its own state that it acts upon . Despite the differences , like our situation , cooperative MARL problems still face scalability issues since the joint-action space is exponentially large . A variety of methods have been proposed to deal with this , including independent learners [ 8 , 29 ] , where each agent employs a single-agent RL policy . Function approximation is another approach that can significantly reduce the space/computational complexity . One can use linear functions [ 58 ] or neural networks [ 28 ] in the approximation . A limitation of these approaches is the lack of theoretical guarantees on the approximation error . In contrast , our technique not only reduces the space/computational complexity significantly , but also has theoretical guarantees on the performance loss in settings with stochastic and non-local dependencies . The mean-field approach [ 45 , 56 , 19 ] provides another way to address the scalability issue , but under very different settings compared to ours . Specifically , the mean-field approach typically assumes homogeneous agents with identical local state/action space and policies , and each agent depends on other agents through their population or “ mean ” behavior . In contrast , our approach considers a local-interaction model , where there is an underlying graph and each agent depends on neighboring agents in the graph . Further , our approach allows heterogeneous agents , which means that the local state/action spaces and policies can differ among the agents . Another related line of work uses centralized training with decentralized execution , e.g. , [ 28 , 15 ] , where there is a centralized coordinator that can communicate with all the agents and keep track of their experiences and policies . In contrast , our work only requires distributed training , where we constrain the scale of communication in training within the -hop neighborhood of each agent . More broadly , this paper contributes to a growing literature that uses exponential decay to derive scalable algorithms for learning in networked systems . The specific form of exponential decay that we generalize is related to the idea of “ correlation decay ” studied in [ 16 , 17 ] , though their focus is on solving static combinatorial optimization problems whereas ours is on learning policies in dynamic environments . Most related to the current paper is [ 38 ] , which shows an exponential decay property in a restricted networked MARL model with purely local dependencies . In contrast , we show a more general µ-decay property holds for a general form of stochastic , non-local dependencies . The technical work in this paper contributes to the analysis of stochastic approximation ( SA ) , which has received considerable attention over the past decade [ 54 , 44 , 11 , 55 ] . Our work is most related to [ 37 ] , which uses an asynchronous nonlinear SA to study the finite-time convergence rate for asynchronous Q-learning on a single trajectory . Beyond [ 37 ] , there are many other works that use SA schemes to study TD learning and Q-learning , e.g . [ 44 , 52 , 20 ] . The finite-time error bound for TD learning with state aggregation in our work is most related to the asymptotic convergence limit given in [ 49 ] and the application of SA scheme to asynchronous Q-learning in [ 37 ] . Beyond these papers , other related work in the broader area of RL with state aggregation includes [ 26 , 23 , 22 , 9 , 43 ] . We add to this literature with a novel finite-time convergence bound for a general SA with state aggregation . This result , in turn , yields the first finite-time error bound in the infinity norm for both TD learning with state aggregation and Q-learning with state aggregation .

Summary: The paper proposes a Scalable Actor-Critic algorithm for a stochastic network of agents. First of all, the authors introduce a new finite-time analysis of stochastic approximation. Then they show how stochastic approximation can be used for state aggregation. Finally they proposed the main algorithms and the theoretical results.