Reinforcement Learning under a Multi-agent Predictive State Representation Model: Method and Theory

1 INTRODUCTION . Real-world multi-agent systems are considered partially observable since agents do not have a complete perception of the environment states . The predictive state representation ( PSR ) ( Littman et al. , 2001 ) is a representation of the dynamic system state by a vector of predictions of tests conditioned on a history . The tests are sequences of actions and observations , which are true if and only if all the observations occur , given that all the actions taken and the histories are sequences of actions and observations recorded before . It records a sequence of actions and observations and focuses on observable quantities rather than the system ’ s dynamics . Therefore , compared with learning the hidden-state-based partially observable Markov decision process ( POMDP ) Kaelbling et al . ( 1998 ) models from observation data , learning PSR should be easier and less prone to local minimal problems ( Singh et al. , 2012 ) . In addition , in the cases where the moment-method is used to learn a model for controlled systems that incorporates actions , PSR is considered as a more general setting to reconstruct the structure of the dynamic system than POMDP ( Azizzadenesheli et al. , 2016 ; Hamilton , 2014 ) . Furthermore , ( Littman et al. , 2001 ; James & Singh , 2004 ; Singh et al. , 2012 ) showed that PSR can offer an alternative more compact state representation for POMDP models . Overall , PSR is considered a generalization of POMDP in terms of representation power and the development of learning algorithms . Due to the above benefits of PSR , it has been used as the basis of some model-based single-agent reinforcement learning ( James & Singh , 2004 ; Boots et al. , 2011 ; Hamilton et al. , 2014 ; Hefny et al. , 2018a ) . Moreover , the predictive states that act as sufficient statistics for the states of multiagent dynamic systems could potentially help multi-agent decision-making in the complicated , partially observable domains . However , there is no method for developing multi-agent predictive state representation ( MAPSR ) for multi-agent reinforcement learning ( MARL ) . Due to this reason , we focused on building an algorithm for learning the optimal policies by representing the multi-agent dynamic systems with the MAPSR . An existing algorithm by ( Chen et al. , 2020 ) provides a spectral framework for learning MAPSR using a tensor , with its dimensions representing the agents and joint history . A tensor decomposition is implemented to learn the model parameters . This type of algorithm avoids explicit estimation of the latent state and can get an unbiased estimation if an infinite amount of data is available . However , it is not easy to incorporate prior information such as sparsity or structure because each new source of information leads to new moments and a different and more complex set of equations to solve . ( Hefny et al. , 2015 ) represent the distributions in PSR via observable statistics and learn PSR by supervised learning , alleviating the above problem . Later , ( Hefny et al. , 2018a ) extend this method to develop a new one called predictive state controlled model that can represent the system model where the agents can affect the environment . Due to the trend of deep RL , ( Hefny et al. , 2018b ) developed an end-to-end policy learning method by using the states represented by a predictive state controlled model as the belief state to fit the policy function to train RL . Both the PSR and RL parameters are updated by using a combined loss objective . The PSR part maintains a sufficient representation of distributions under the partially observable environment , the deep RL algorithm brings efficiency for learning . The environment of MARL can be partially observable and always non-stationary ( one agent learning while the other agents ’ actions can influence the environment ) ( Bowling & Veloso , 2002 ) . Predictive state controlled model can model the prediction of systems states affected by actions ; if we design a state representation model for MARL environment that can predict the future observations of agents upon exerting future actions of other agents and conditioned on past histories , then the non-stationary environment caused by other agents learning can be modeled . Due to the above reason , we seek to develop a MARL-based PSR model and learning algorithm , which we call MAPSRL . It could potentially ameliorate both the problem of partially observable observations and non-stationarity by other agents ’ interaction for multi-agent decision making . The paper ’ s first goal is to develop a PSR model for MARL environment . In MARL settings , the distribution of observed observations is dependent on its actions and relevant to other agents and histories . So far , many works ( Ryu et al. , 2020 ; Liu et al. , 2020b ; Niu et al. , 2021 ) that take other agents ’ actions into consideration have been shown to achieve competitive performance . In order to consider the interactive relationship under the MAPSR , we introduce a dynamic interaction graph to characterize the “ neighborhoods ” of agents . Such a graph topology , which is often easy to obtain nowadays , enables us to encode the adjacency relationship of agents into our MAPSR model . We build our model under three common graphs , namely static complete graphs , static non-complete graphs , and dynamic graphs , covering as many real-world scenarios as possible . Under the existence of such a dynamic interaction graph , we build predictions of its observations by considering its neighbor ’ s actions for every agent . We call this primitive interactive predictions . Then an agent ’ s PSR will be formulated as a linear combination of those primitive interactive predictions . Due to the decentralized formulation , our model allows implementing the learning of the MAPSR on a single agent level while maintaining other agents ’ interactions through the encoding of the interactive graph topology . As set as our second goal , we analytically upper bound theL2-norm difference between the underlying true and learned predictive state representations , providing theoretical guarantees for the performance of our learning algorithm under the MAPSR model in all three graph settings . Our paper ’ s third goal is to develop an algorithm that integrates the MAPSR model to the MARL . We develop an online algorithm that simultaneously learns the MAPSR , and the agents ’ policies end-toend with a fully differentiable neural network structure . Our algorithm provides a new model-based MARL , and it has the flexibility to be extended to most of the offline MARL algorithms . We focus on the continuous action space , which is more practical and common to real-world applications such as robotic control . Finally , we test our algorithm through systematic numerical experiments on MAMujoCo robotic learning experiments ( de Witt et al. , 2020 ) and multi-agent particle learning environments ( Ryu et al. , 2020 ) , and compare our proposed method against two baselines as detailed in Section 6 . The results demonstrate the efficiency of our method . 2 RELATED WORKS . Partially Observable Environment . Real-world agents often experience situations that the observed signals are aliased and do not fully determine their state in the system . This is particularly true for multiple agents environments where agents have partial observability due to limited communication ( Oliehoek & Amato , 2016 ) . In accommodation to the partially observable environment , POMDP has been adopted by ( Kaelbling et al. , 1998 ) , and the algorithms ( Kaelbling et al. , 1998 ; Cassandra , 1998 ; Thrun , 1999 ; Pineau et al. , 2003 ; Poupart & Vlassis , 2008 ; Platt Jr et al. , 2010 ) for determining an optimal policy have shifted to using the probability distributions ( belief state ) over the state space instead of exact state space . In general , they have high complexity or suffer from local optima . Moreover , the most common POMDP policy learning assumes the agent has access to a priori knowledge of the system . The access to such prior knowledge has a premise that the agent has considerable domain knowledge ( Kaelbling et al. , 1998 ) . However , it is expected that the real-world agents learn the system model and thus a planning policy further without knowledge of the domain . Overview of PSR . ( Littman et al. , 2001 ; James & Singh , 2004 ) introduced the PSR over an expressive and robust framework for modeling dynamical systems and defined PSR as a representation of state by using a vector of predictions of fully observable quantities ( tests ) conditioned on past events ( histories ) . A predictive model is constructed directly from execution traces in the PSR framework , utilizing minimal prior information about the domain . The PSR paradigm subsumes POMDP as a special case ( Littman et al. , 2001 ) . PSR is considered much more compact than POMDP ( Aberdeen et al. , 2007 ) . The spectral learning method has been proved to show success for learning the PSR ( Boots et al. , 2011 ; Jiang et al. , 2018 ) . There are other classes of dynamical system learning algorithms that are based on likelihood-based optimization or sampling approaches ( Frigola et al. , 2013 ) , but they are prone to poor local optima . The spectral learning represents the estimated state by sufficient statistics of future observations and estimates the model parameters by method of moments . However , this line of algorithms is hard to incorporate prior information ( Hefny et al. , 2015 ) . Thus , ( Hefny et al. , 2015 ; 2018a ) introduce the supervised learning method to learn PSR and proves its convergence . Although many works study PSR in discrete action space ( Hsu et al. , 2012 ; Siddiqi et al. , 2010 ; Boots et al. , 2011 ) , ( Boots et al. , 2013 ) proposes Hilbert space embedding ( HSE ) -PSR to deal with continuous actions . ( Hefny et al. , 2018a ) uses an approximation of HSE-PSR by Random Fourier transform ( RFF ) and built a more principled generalization of PSR to deal with high dimensions . However , all of these studies aim for the single agent scenario . PSR and RL . The predictive states estimated by the PSR are considered as states in a fully observable Markov Decision Process so that the value function is learned on these states . This line of work has been done in the single-agent environment ( Boots & Gordon , 2010 ; Boots et al. , 2011 ; Hamilton et al. , 2014 ; Venkatraman et al. , 2017 ; Hefny et al. , 2018b ) . Especially , ( Hefny et al. , 2018b ) proposes the recurrent predictive state in the RNN network . Moreover , the learning PSR and policy functions are connected with the end-to-end training . MAPSR and MARL . MAPSR model is formulated by ( Chen et al. , 2020 ) and the model parameters are learnt by tensor decomposition . This formulation is aligned with the spectral algorithm , so it also has the same issue as the single-agent case . MARL algorithms have been developed to learn the Markov Game environment ( Lyu & Amato , 2020 ; Son et al. , 2019 ; Zhang et al. , 2019 ; Rashid et al. , 2018 ; Foerster et al. , 2018 ; Lowe et al. , 2017 ) . The partially observable states are pervasive in this environment , bringing non-stationarity to the learning algorithm ( Bowling & Veloso , 2002 ) so that the learning needs to consider the non-stationarity . Recent works propose more sophisticated deep MARL algorithms for multi-agent problems under the paradigm of centralized training with decentralized execution ( Zhou et al. , 2020 ; Sunehag et al. , 2017 ; Lowe et al. , 2017 ; Foerster et al. , 2018 ; Rashid et al. , 2018 ) . Other than their methods , our approach also accounts for the importance of macroscopic measures of underline systems . Also , providing a predictive state as a belief state rather than the partially observable observations can further help the agent learn . Like the single-agent case , we hope our method can further improve these algorithms .

Summary: The authors present a framework and method in which predictive state representations for multiple agents simultatneously acting and interacting within an environment. This is presented in a general way, where predictive states are Hilbert space operators which when applied to sequences of observations and actions appropriately predict the predictions of these state representations. The key advance in this work is to apply this existing PSR framework to networks of agents, with 3 types of agent network: static complete graph (all agents affect all others experience); static non-complete graph (only some agents affect one another); and dynamic non-complete graph (agents affect one another in a time varying way). A number of theoretical results are presented, including PAC bounds for the approximators in the framework.  The authors then present two closely related methods to learn policies alongside these multi agent PSRs in an online way. The first MAPSRL-1 is akin to the independent actor critic (Foerster et al., 2017) and thus may suffer from the apparent non-stationarity of the environment from the perspective of any one agent. MAPSRL-2 addresses this by incorporating the PSR information from other agents into the policy gradient update.  The paper presents a series of experiments based on environments encoded in the  OpenAI Gym MAMujoco system. These are environments presented in previous papers and there are a broad selection of these.