Reinforcement Learning under a Multi-agent Predictive State Representation Model: Method and Theory

1 INTRODUCTION . Real-world multi-agent systems are considered partially observable since agents do not have a complete perception of the environment states . The predictive state representation ( PSR ) ( Littman et al. , 2001 ) is a representation of the dynamic system state by a vector of predictions of tests conditioned on a history . The tests are sequences of actions and observations , which are true if and only if all the observations occur , given that all the actions taken and the histories are sequences of actions and observations recorded before . It records a sequence of actions and observations and focuses on observable quantities rather than the system ’ s dynamics . Therefore , compared with learning the hidden-state-based partially observable Markov decision process ( POMDP ) Kaelbling et al . ( 1998 ) models from observation data , learning PSR should be easier and less prone to local minimal problems ( Singh et al. , 2012 ) . In addition , in the cases where the moment-method is used to learn a model for controlled systems that incorporates actions , PSR is considered as a more general setting to reconstruct the structure of the dynamic system than POMDP ( Azizzadenesheli et al. , 2016 ; Hamilton , 2014 ) . Furthermore , ( Littman et al. , 2001 ; James & Singh , 2004 ; Singh et al. , 2012 ) showed that PSR can offer an alternative more compact state representation for POMDP models . Overall , PSR is considered a generalization of POMDP in terms of representation power and the development of learning algorithms . Due to the above benefits of PSR , it has been used as the basis of some model-based single-agent reinforcement learning ( James & Singh , 2004 ; Boots et al. , 2011 ; Hamilton et al. , 2014 ; Hefny et al. , 2018a ) . Moreover , the predictive states that act as sufficient statistics for the states of multiagent dynamic systems could potentially help multi-agent decision-making in the complicated , partially observable domains . However , there is no method for developing multi-agent predictive state representation ( MAPSR ) for multi-agent reinforcement learning ( MARL ) . Due to this reason , we focused on building an algorithm for learning the optimal policies by representing the multi-agent dynamic systems with the MAPSR . An existing algorithm by ( Chen et al. , 2020 ) provides a spectral framework for learning MAPSR using a tensor , with its dimensions representing the agents and joint history . A tensor decomposition is implemented to learn the model parameters . This type of algorithm avoids explicit estimation of the latent state and can get an unbiased estimation if an infinite amount of data is available . However , it is not easy to incorporate prior information such as sparsity or structure because each new source of information leads to new moments and a different and more complex set of equations to solve . ( Hefny et al. , 2015 ) represent the distributions in PSR via observable statistics and learn PSR by supervised learning , alleviating the above problem . Later , ( Hefny et al. , 2018a ) extend this method to develop a new one called predictive state controlled model that can represent the system model where the agents can affect the environment . Due to the trend of deep RL , ( Hefny et al. , 2018b ) developed an end-to-end policy learning method by using the states represented by a predictive state controlled model as the belief state to fit the policy function to train RL . Both the PSR and RL parameters are updated by using a combined loss objective . The PSR part maintains a sufficient representation of distributions under the partially observable environment , the deep RL algorithm brings efficiency for learning . The environment of MARL can be partially observable and always non-stationary ( one agent learning while the other agents ’ actions can influence the environment ) ( Bowling & Veloso , 2002 ) . Predictive state controlled model can model the prediction of systems states affected by actions ; if we design a state representation model for MARL environment that can predict the future observations of agents upon exerting future actions of other agents and conditioned on past histories , then the non-stationary environment caused by other agents learning can be modeled . Due to the above reason , we seek to develop a MARL-based PSR model and learning algorithm , which we call MAPSRL . It could potentially ameliorate both the problem of partially observable observations and non-stationarity by other agents ’ interaction for multi-agent decision making . The paper ’ s first goal is to develop a PSR model for MARL environment . In MARL settings , the distribution of observed observations is dependent on its actions and relevant to other agents and histories . So far , many works ( Ryu et al. , 2020 ; Liu et al. , 2020b ; Niu et al. , 2021 ) that take other agents ’ actions into consideration have been shown to achieve competitive performance . In order to consider the interactive relationship under the MAPSR , we introduce a dynamic interaction graph to characterize the “ neighborhoods ” of agents . Such a graph topology , which is often easy to obtain nowadays , enables us to encode the adjacency relationship of agents into our MAPSR model . We build our model under three common graphs , namely static complete graphs , static non-complete graphs , and dynamic graphs , covering as many real-world scenarios as possible . Under the existence of such a dynamic interaction graph , we build predictions of its observations by considering its neighbor ’ s actions for every agent . We call this primitive interactive predictions . Then an agent ’ s PSR will be formulated as a linear combination of those primitive interactive predictions . Due to the decentralized formulation , our model allows implementing the learning of the MAPSR on a single agent level while maintaining other agents ’ interactions through the encoding of the interactive graph topology . As set as our second goal , we analytically upper bound theL2-norm difference between the underlying true and learned predictive state representations , providing theoretical guarantees for the performance of our learning algorithm under the MAPSR model in all three graph settings . Our paper ’ s third goal is to develop an algorithm that integrates the MAPSR model to the MARL . We develop an online algorithm that simultaneously learns the MAPSR , and the agents ’ policies end-toend with a fully differentiable neural network structure . Our algorithm provides a new model-based MARL , and it has the flexibility to be extended to most of the offline MARL algorithms . We focus on the continuous action space , which is more practical and common to real-world applications such as robotic control . Finally , we test our algorithm through systematic numerical experiments on MAMujoCo robotic learning experiments ( de Witt et al. , 2020 ) and multi-agent particle learning environments ( Ryu et al. , 2020 ) , and compare our proposed method against two baselines as detailed in Section 6 . The results demonstrate the efficiency of our method . 2 RELATED WORKS . Partially Observable Environment . Real-world agents often experience situations that the observed signals are aliased and do not fully determine their state in the system . This is particularly true for multiple agents environments where agents have partial observability due to limited communication ( Oliehoek & Amato , 2016 ) . In accommodation to the partially observable environment , POMDP has been adopted by ( Kaelbling et al. , 1998 ) , and the algorithms ( Kaelbling et al. , 1998 ; Cassandra , 1998 ; Thrun , 1999 ; Pineau et al. , 2003 ; Poupart & Vlassis , 2008 ; Platt Jr et al. , 2010 ) for determining an optimal policy have shifted to using the probability distributions ( belief state ) over the state space instead of exact state space . In general , they have high complexity or suffer from local optima . Moreover , the most common POMDP policy learning assumes the agent has access to a priori knowledge of the system . The access to such prior knowledge has a premise that the agent has considerable domain knowledge ( Kaelbling et al. , 1998 ) . However , it is expected that the real-world agents learn the system model and thus a planning policy further without knowledge of the domain . Overview of PSR . ( Littman et al. , 2001 ; James & Singh , 2004 ) introduced the PSR over an expressive and robust framework for modeling dynamical systems and defined PSR as a representation of state by using a vector of predictions of fully observable quantities ( tests ) conditioned on past events ( histories ) . A predictive model is constructed directly from execution traces in the PSR framework , utilizing minimal prior information about the domain . The PSR paradigm subsumes POMDP as a special case ( Littman et al. , 2001 ) . PSR is considered much more compact than POMDP ( Aberdeen et al. , 2007 ) . The spectral learning method has been proved to show success for learning the PSR ( Boots et al. , 2011 ; Jiang et al. , 2018 ) . There are other classes of dynamical system learning algorithms that are based on likelihood-based optimization or sampling approaches ( Frigola et al. , 2013 ) , but they are prone to poor local optima . The spectral learning represents the estimated state by sufficient statistics of future observations and estimates the model parameters by method of moments . However , this line of algorithms is hard to incorporate prior information ( Hefny et al. , 2015 ) . Thus , ( Hefny et al. , 2015 ; 2018a ) introduce the supervised learning method to learn PSR and proves its convergence . Although many works study PSR in discrete action space ( Hsu et al. , 2012 ; Siddiqi et al. , 2010 ; Boots et al. , 2011 ) , ( Boots et al. , 2013 ) proposes Hilbert space embedding ( HSE ) -PSR to deal with continuous actions . ( Hefny et al. , 2018a ) uses an approximation of HSE-PSR by Random Fourier transform ( RFF ) and built a more principled generalization of PSR to deal with high dimensions . However , all of these studies aim for the single agent scenario . PSR and RL . The predictive states estimated by the PSR are considered as states in a fully observable Markov Decision Process so that the value function is learned on these states . This line of work has been done in the single-agent environment ( Boots & Gordon , 2010 ; Boots et al. , 2011 ; Hamilton et al. , 2014 ; Venkatraman et al. , 2017 ; Hefny et al. , 2018b ) . Especially , ( Hefny et al. , 2018b ) proposes the recurrent predictive state in the RNN network . Moreover , the learning PSR and policy functions are connected with the end-to-end training . MAPSR and MARL . MAPSR model is formulated by ( Chen et al. , 2020 ) and the model parameters are learnt by tensor decomposition . This formulation is aligned with the spectral algorithm , so it also has the same issue as the single-agent case . MARL algorithms have been developed to learn the Markov Game environment ( Lyu & Amato , 2020 ; Son et al. , 2019 ; Zhang et al. , 2019 ; Rashid et al. , 2018 ; Foerster et al. , 2018 ; Lowe et al. , 2017 ) . The partially observable states are pervasive in this environment , bringing non-stationarity to the learning algorithm ( Bowling & Veloso , 2002 ) so that the learning needs to consider the non-stationarity . Recent works propose more sophisticated deep MARL algorithms for multi-agent problems under the paradigm of centralized training with decentralized execution ( Zhou et al. , 2020 ; Sunehag et al. , 2017 ; Lowe et al. , 2017 ; Foerster et al. , 2018 ; Rashid et al. , 2018 ) . Other than their methods , our approach also accounts for the importance of macroscopic measures of underline systems . Also , providing a predictive state as a belief state rather than the partially observable observations can further help the agent learn . Like the single-agent case , we hope our method can further improve these algorithms .

Summary: The paper introduces a predictive state representation (PSR)-based MARL framework. The framework uses a graph representation to model the interactions between agents. Performance bounds are given for the learned predictive state representation.  With the MAPSR, individual policies are trained by replacing the partial observation o with PSR Q. The experiments results show the advantage of the proposed method compared with the baselines experimented in the paper.