Surprise Minimizing Multi-Agent Learning with Energy-based Models

sites.google.com/view/surprise-web/ 1 INTRODUCTION . The rise of RL has led to an increasing interest in the study of multi-agent systems ( Lowe et al. , 2017 ; Vinyals et al. , 2019 ) , commonly known as Multi-Agent Reinforcement Learning ( MARL ) . In the case of partially observable settings , MARL enables the learning of policies with centralised training and decentralised control ( Kraemer & Banerjee , 2016 ) . This has proven to be useful for exploiting value-based methods which motivate collaboration across large number of agents . But how do agents behave in the presence of sudden environmental changes ? Consider the problem of autonomous driving wherein a driver ( agent ) autonomously operates a vehicle in real-time . The driver learns to optimize the reward function by maintaining constant speed and covering more distance in different traffic conditions . Whenever the vehicle approaches an obstacle , the driver acts to avoid it by utilizing the brake and directional steering commands . However , due to the fast-paced dynamics of the environment , say fast-moving traffic , the agent may abruptly encounter an obstacle ( a person running across the street ) which may result in a collision . Irrespective of the optimal action ( pushing of brakes ) executed by the agent , the vehicle may fail to evade the collision as a result of the abrupt temporal change . The above arises as a consequence of surprise , which is defined as a statistical measure of uncertainty . Surprise minimization ( Berseth et al. , 2019 ) is a recent phenomenon observed in the case of singleagent RL methods which deals with environments consisting of rapidly changing states . In the case of model-based RL ( Kaiser et al. , 2019 ) , surprise minimization is used as an effective planning tool in the agent ’ s model ( Berseth et al. , 2019 ) whereas in the case of model-free RL , surprise minimization is witnessed as an intrinsic motivation ( Achiam & Sastry , 2017 ; Macedo et al. , 2004 ) or generalization problem ( Chen , 2020 ) . On the other hand , MARL does not account for surprise across agents as a result of which agents remain unaware of drastic changes in the environment ( Macedo & Cardoso , 2005 ) . Thus , surprise minimization in the case of multi-agent settings requires attention from a critical standpoint . A potential pathway to treat surprising states may be realized in light of free-energy minimization . The free-energy principle depicts convergence to local niches and provides a general recipe for cognitive stability among agents . Through this lens , we unify surprise with free-energy in the multi-agent setting . We construct a temporal EBM which represents an estimate of surprise agents may face in the environment . All agents jointly minimize this estimate utilizing temporal difference learning upon their value functions and the EBM . Our formulation of free-energy minimization is theoretically akin to minimizing the entropy in conjugate gradient space . This insight provides a suitable convergence result towards minimum surprising states ( or niches ) of the agent state distributions . In an empirical study of multi-agent tasks which present significant collaboration bottlenecks and fast-paced dynamics , we validate our theoretical claims and motivate the practical usage of EBMs in MARL . 2 RELATED WORK . Surprise Minimization : Despite the recent success of value-based methods ( Mnih et al. , 2016 ; Hessel et al. , 2017 ) RL agents suffer from spurious state spaces and encounter sudden changes in trajectories . Quantitatively , surprise has been studied as a measure of deviation ( Berseth et al. , 2019 ; Chen , 2020 ) among states encountered by the agent during its interaction with the environment . While exploring ( Burda et al. , 2019 ; Thrun , 1992 ) the environment , agents tend to have higher deviation among states which is gradually reduced by gaining a significant understanding of state-action transitions . In the case of model-based RL , agents can leverage spurious experiences ( Berseth et al. , 2019 ) and plan effectively for future steps . On the other hand , in the case of model-free RL , surprise results in sample-inefficient learning ( Achiam & Sastry , 2017 ) . This is primarily addressed by making use of rigorous exploration strategies ( Stadie et al. , 2015 ; Lee et al. , 2019 ) . High-dimensional exploration further requires extrinsic feature engineering ( Kulkarni et al. , 2016 ) and meta models ( Gupta et al. , 2018 ) . A suitable way to tackle high-dimensional dynamics is by utilizing surprise as a penalty on the reward ( Chen , 2020 ) . This leads to improved generalization for single-agent interactions ( Ren et al. , 2005 ) . Our proposed approach is orthogonal to the aforesaid methods . Energy-based Models : EBMs have been successfully implemented in single-agent RL methods ( O ’ Donoghue et al. , 2016 ; Haarnoja et al. , 2017 ) . These typically make use of Boltzmann distributions to approximate policies ( Levine & Abbeel , 2014 ) . Such a formulation results in the minimization of free energy within the agent . While policy approximation depicts promise in the case of unknown dynamics , inference methods ( Toussaint , 2009 ) play a key role in optimizing goal-oriented behavior . A second type of usage of EBMs follows the maximization of entropy ( Ziebart et al. , 2008 ) . The maximum entropy framework ( Haarnoja et al. , 2018b ) highlighted in Soft Q-Learning ( SQL ) ( Haarnoja et al. , 2017 ) allows the agent to obey a policy which maximizes its reward and entropy concurrently . Maximization of agent ’ s entropy results in diverse and adaptive behaviors ( Ziebart , 2010 ) which may be difficult to accomplish using standard exploration techniques ( Burda et al. , 2019 ; Thrun , 1992 ) . The maximum entropy framework is akin to approximate inference in the case of policy gradient methods ( Schulman et al. , 2017 ) . Such a connection between likelihood ratio gradient techniques and energy-based formulations leads to diverse and robust policies ( Haarnoja , 2018 ) and their hierarchical extensions ( Haarnoja et al. , 2018a ) which preserve the lower levels of hierarchies . In the case of MARL , EBMs have witnessed limited applicability as a result of the increasing number of agents and complexity within each agent ( Buşoniu et al. , 2010 ) . While the probabilistic framework is readily transferable to opponent-aware multi-agent systems ( Wen et al. , 2019 ) , cooperative settings consisting of coordination between agents require a firm formulation of energy which is scalable in the number of agents ( Grau-Moya et al. , 2018 ) and accounts for environments consisting of spurious states ( Wei et al. , 2018 ) . Our theoretical formulation is motivated by these methods in literature . 3 PRELIMINARIES . 3.1 MULTI-AGENT LEARNING . We review the cooperative MARL setup . The problem is modeled as a Dec-Partially Observable Markov Decision Process ( POMDP ) ( Oliehoek & Amato , 2016 ) defined by the tuple ( S , A , r , N , P , Z , O , γ ) where the state space S and action space A are discrete , r : S × A → [ rmin , rmax ] presents the reward observed by agents a ∈ N where N is the set of all agents , P : S × S ×A → [ 0 , ∞ ) presents the unknown transition model consisting of the transition probability to the next state s′ ∈ S given the current state s ∈ S and joint action u ∈ A ( a combination of each agent ’ s action ua ∈ Aa ) at time step t and γ is the discount factor . We consider a partially observable setting in which each agent n draws individual observations z ∈ Z according to the observation function O ( s , u ) : S × A → Z . We consider a joint policy πθ ( u|s ) as a function of model parameters θ . Standard RL defines the agent ’ s objective to maximize the expected discounted reward Eπθ [ ∑T t=0 γ tr ( st , ut ) ] as a function of the parameters θ . The joint action-value function for agents is represented as Q ( u , s ; θ ) = Eπθ [ ∑T t=1 γ tr ( s , u ) |s = st , u = ut ] which is the expected sum of payoffs obtained in state s upon performing action u by following the policy πθ . We denote the optimal policy πθ∗ ( shorthand π∗ ) such that Q ( u , s ; θ∗ ) ≥ Q ( u , s ; θ ) ∀s ∈ S , u ∈ A . In the case of multiple agents , the joint optimal policy can be expressed as the Nash Equilibrium ( Nash , 1950 ) of the Stochastic Markov Game as π∗ = ( π1 , ∗ , π2 , ∗ , ... πN , ∗ ) such that Q ( ua , s ; θ∗ ) ≥ Q ( ua , s ; θ ) ∀s ∈ S , u ∈ A , a ∈ N . Q-Learning is an off-policy , model-free algorithm suitable for continuous and episodic tasks . The algorithm uses semi-gradient descent to minimize the Temporal Difference ( TD ) error in Equation 1 . L ( θ ) = E s , u , s′∼R [ ( r + γmax u′∈A Q ( u′ , s′ ; θ− ) −Q ( u , s ; θ ) ) 2 ] ( 1 ) where y = r + γmax u′∈A Q ( u′ , s′ ; θ− ) is the TD target consisting of θ− as the target parameters andR denotes the replay buffer . 3.2 ENERGY-BASED MODELS . EBMs ( LeCun et al. , 2006 ; 2007 ) have been successfully applied in the field of machine learning ( Teh et al. , 2003 ) and probabilistic inference ( MacKay , 2002 ) . A typical EBM E formulates the equilibrium probabilities ( Sallans & Hinton , 2004 ) P ( v , h ) = exp ( −E ( v , h ) ) ∑ v̂ , ĥ [ exp ( −E ( v̂ , ĥ ) ) ] via a Boltzmann distribution ( Levine & Abbeel , 2014 ) where v and h are the values of the visible and hidden variables and v̂ and ĥ are all the possible configurations of the visible and hidden variables respectively . The probability distribution over all the visible variables can be obtained by summing over all possible configurations of the hidden variables . This is mathematically expressed in Equation 2 . P ( v ) = ∑ h exp ( −E ( v , h ) ) ∑ v̂ , ĥ exp ( −E ( v̂ , ĥ ) ) ( 2 ) Here , E ( v , h ) is called the equilibrium free energy which is the minimum of the variational free energy and ∑ v̂ , ĥ exp ( −E ( v̂ , ĥ ) ) is the partition function . 4 ENERGY-BASED SURPRISE MINIMIZATION . We begin by constructing surprise minimization as an energy-based problem in the temporal setting . The motivation behind an energy-based formulation stems from rapidly changing states as an undesired niche among agents in the case of partially-observed settings . To steer agents away from this niche , we further construct a method which incorporates the theoretical aspect of the study .

Summary: This paper introduced a suprise term in optimizing the policies (action-value functions in Q-learning) for solving the non-stationary challenge due to the rapid changes from the environment for MARL scenarios. This work not only proposed the concept of suprising value in the context of MARL, but also gave an operator (i.e. a contration) that can make the sequence of suprising values converges to a fixed point. The general sketch of the proofs is correct. Also, the authors showed that the convex conjugate of the suprising value operator is analogous to the minimization of the uncertainty among agents. By incorporating the suprising value into the Q-learning algorithm, the term led by the ratio between target and behaviour suprising value (suprising ratio) can also be interpreted as an intrinsic reward. Moreover, the authors also compare the proposed surprising minimization objective with the Soft Q-learning objective.