LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning 

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) has emerged as a powerful tool to enable autonomous agents to solve various tasks such as ride-sharing ( Zhou , Luo , et al. , 2020 ) and swarm robotics ( Hüttenrauch et al. , 2017 ; Mguni et al. , 2018 ) . In multi-agent systems ( MAS ) , maximising system performance often requires agents to coordinate during exploration and learn coordinated joint actions ( Matignon et al. , 2012 ) . However , in many MAS , the reward signal provided by the environment is not sufficient to guide the agents towards coordinated behaviour ( Matignon et al. , 2012 ) . Consequently , relying on solely the individual rewards received by the agents may not lead to optimal outcomes ( Mguni et al. , 2019 ) . This problem is worsened by the fact that MAS can have many stable points some of which lead to arbitrarily bad outcomes ( Roughgarden & Tardos , 2007 ) . As in single agent RL , in MARL inefficient exploration can dramatically decrease sample efficiency . In MAS , a major challenge is how to overcome sample inefficiency from poorly coordinated exploration . Unlike single agent RL , in MARL , the collective of agents is typically required to coordinate its exploration to find their optimal joint policies1 . A second issue is that in many MAS settings of interest , such as video games and physical tasks , rich informative signals of the agents ’ joint performance are not readily available ( Hosu & Rebedea , 2016 ) . For example , in StarCraft Micromanagement ( Samvelyan et al. , 2019 ) , the sparse reward alone ( win , lose ) gives insufficient information to guide agents toward their optimal joint policy . Consequently , MARL requires large numbers of samples producing a great need for MARL methods that can solve such problems efficiently . To aid coordinated learning , algorithms such as QMIX ( Rashid et al. , 2018 ) , MAVEN ( Mahajan et al. , 2019 ) and COMA ( Foerster et al. , 2018 ) , so-called centralised critic and decentralised execution ( CT-DE ) methods use a centralised critic whose role is to estimate the agents ’ expected returns . The critic makes use of all available information generated by the system , specifically the global state and the joint action . To enable effective CT-DE , it is critical that the joint greedy action 1In single agent RL exploration issues can be mitigated by adjusting exploration rates or the policy variance ( Tijsma et al. , 2016 ) . However it has been shown that the same is not possible in MARL ( Mahajan et al. , 2019 ) . should be equivalent to the collection of individual greedy actions of agents , which is called the IGM ( Individual-Global-Max ) principle ( Son et al. , 2019 ) . CT-DE methods are however , prone to convergence to suboptimal joint policies ( Wang , Ren , Han , et al. , 2020 ) . In particular , existing value factorisations , e.g . QMIX and VDN ( Sunehag et al. , 2017 ) , can not ensure an exact guarantee of IGM consistency ( Wang , Ren , Liu , et al. , 2020 ) . Moreover , CT-DE methods such as QMIX require a monotonicity condition which is violated in scenarios where multiple agents must coordinate but are penalised if only a subset of them do so ( see Exp . 2 , Sec . 6.1 ) . To tackle these issues , in this paper we introduce a new MARL framework , LIGS that constructs intrinsic rewards online which guide MARL learners towards their optimal joint policy . LIGS involves an adaptive intrinsic reward agent , Generator that selects intrinsic rewards to add according to the history of visited states and joint actions performed by the agents . Generator adaptively guides the agents ’ exploration and behaviour towards coordination and maximal joint performance . A pivotal feature of LIGS is the novel combination of RL and switching controls ( Bayraktar & Egami , 2010 ; Mguni , 2018 ) which enables it to determine the best set of states to learn to add intrinsic rewards while disregarding less useful states . This enables Generator to quickly learn how to set intrinsic rewards that guide the agents during their learning process . Moreover , the intrinsic rewards added by Generator can significantly deviate from the environment rewards . This enables LIGS to both promote complex joint exploration patterns and decompose difficult tasks . Despite this flexibility , special features within LIGS ensure the underlying optimal policies are preserved so that the agents learn to solve the task at hand . Overall , LIGS has several advantages : • LIGS has the freedom to introduce rewards that vastly deviate from the environment rewards . With this , LIGS promotes coordinated exploration ( i.e . visiting unplayed state-joint actions ) among the agents enabling them to find joint policies that maximise the system rewards and generates intrinsic rewards to aid solving sparse reward MAS . • LIGS selects which best states to add intrinsic rewards adaptively in response to the agents ’ behaviour while the agents learn leading to an efficient learning process . • LIGS ’ s intrinsic rewards preserve the agents ’ optimal joint policy and ensure that the total environment return is ( weakly ) increased . To enable the framework to perform successfully , we overcome several challenges : i ) Firstly , constructing an intrinsic reward can change the underlying problem leading to the agents solving irrelevant tasks ( Mannion et al. , 2017 ) . We resolve this by endowing the intrinsic reward function with special form which both allows a rich spread of intrinsic rewards while preserving the optimal policy . ii ) Secondly , introducing intrinsic reward functions can worsen the agents ’ performance ( S. Devlin & Kudenko , 2011 ) and doing so while training can lead to convergence issues . We prove LIGS leads to better performing policies and that LIGS ’ s learning process converges and preserves the MARL learners ’ convergence properties . iii ) Lastly , adding an agent Generator with its own goal leads to a Markov game ( MG ) with N + 1 agents ( Fudenberg & Tirole , 1991 ) . Tractable methods for solving MGs are extremely rare with convergence only in special cases ( Yang & Wang , 2020 ) . Nevertheless , using a special set of features in LIGS ’ s design , we prove LIGS converges to a solution in which it learns an intrinsic reward function that improves the agents ’ performance . 2 RELATED WORK . Multi-agent exploration methods seek to promote coordinated exploration among MARL learners . Maven et . al ( Mahajan et al. , 2019 ) propose a hybridisation of value and policy-based methods that uses mutual information to learn a diverse set of behaviours between agents . Though this approach promotes coordinated exploration , it does not encourage exploration of novel states . Other approaches to promote exploration in MARL while assuming aspects of the environment are known in advance and agents can perform perfect communication between themselves ( Viseras et al. , 2016 ) . Similarly , to promote coordinated exploration in partially observable settings , ( Pesce & Montana , 2020 ) propose end-to-end learning of a communication protocol through a memory device . In general , exploration-based methods provide no performance guarantees nor do they ensure the optimal policy ( of the underlying dec-MDP ) is preserved . Moreover , many employ heuristics that naively reward exploration to unvisited states without consideration of the environment reward . This can lead to spurious objectives being maximised . Reward shaping ( Harutyunyan et al. , 2015 ) ( RS ) is a technique which aims to alleviate the problem of sparse and uninformative rewards by supplementing the agent ’ s reward with a prefixed term F . In ( Ng et al. , 1999 ) it was established that adding a shaping reward function of the form F ( st+1 , st ) = γφ ( st+1 ) −φ ( st ) preserves the optimal policy and in some cases can aid learning . RS has been extended to MAS ( S. Devlin & Kudenko , 2011 , 2016 ; S. Devlin et al. , 2011 ; S. M. Devlin & Kudenko , 2012 ; Mannion et al. , 2018 ; Sadeghlou et al. , 2014 ) in which it is used to promote convergence to efficient social welfare outcomes . Poor choices of F in a MAS can slow the learning process and can induce convergence to poor system performance ( S. Devlin & Kudenko , 2011 ) . In MARL , the question of which shaping function to use remains unaddressed . Typically , RS algorithms rely on hand-crafted shaping reward functions that are constructed using domain knowledge , contrary to the goal of autonomous learning ( S. Devlin & Kudenko , 2011 ) . As we later describe LIGS , which successfully learns an instrinsic reward function F , uses a similar form as PBRS however , F is now augmented to include the actions of another RL agent to learn the intrinsic rewards online . In ( Du et al. , 2019 ) an approach towards learning intrinsic rewards is proposed in which a parameterised intrinsic reward is learned using a bilevel approach through a centralised critic . Loosely related are single-agent methods ( Dilokthanakul et al. , 2019 ; Kulkarni et al. , 2016 ; Pathak et al. , 2017 ; Zheng et al. , 2018 ) which , in general , introduce heuristic terms to generate intrinsic rewards . Within these categories , closest to our work is the intrinsic reward approach in ( Du et al. , 2019 ) . There , the agents ’ policies and intrinsic rewards are learned with a bilevel approach . In contrast , LIGS performs these operations concurrently leading to a faster , more efficient procedure . A crucial point of distinction is that in LIGS , the intrinsic rewards are constructed by an RL agent ( Generator ) with its own reward function . Consequently , LIGS can generate complex patterns of intrinsic rewards , encourage joint exploration . Additionally , LIGS learns intrinsic rewards only at relevant states , this confers high computational efficiency . Lastly , unlike exploration-based methods e.g. , ( Mahajan et al. , 2019 ) , LIGS ensures preservation of the agents ’ joint optimal policy for the task . 3 PRELIMINARIES . A fully cooperative MAS is modelled by a decentralised-Markov decision process ( dec-MDP ) ( Yang & Wang , 2020 ) . A dec-MDP is an augmented MDP involving a set of N ≥ 2 agents denoted by N that independently decide actions to take which they do so simultaneously over many rounds . Formally , a dec-MDP is a tuple M = 〈N , S , ( Ai ) i∈N , P , R , γ〉 where S is the finite set of states , Ai is an action set for agent i ∈ N and R : S ×A→ P ( D ) is the reward function that all agents jointly seek to maximise where D is a compact subset of R and lastly , P : S ×A×S → [ 0 , 1 ] is the probability function describing the system dynamics where A : = ×Ni=1Ai . Each agent i ∈ N uses a Markov policy πi : S ×Ai → [ 0 , 1 ] to select its actions . At each time t ∈ 0 , 1 , . . . , the system is in state st ∈ S and each agent i ∈ N takes an action ait ∈ Ai . The joint action at = ( a1t , . . . , aNt ) ∈ A produces an immediate reward ri ∼ R ( st , at ) for agent i ∈ N and influences the next-state transition which is chosen according to P . The goal of each agent i is to maximise its expected returns measured by its value function vπ i , π−i ( s ) = Eπi , π−i [ ∑∞ t=0 γ tR ( st , at ) ] , where Πi is a compact Markov policy space and −i denotes the tuple of agents excluding agent i. Intrinsic rewards can strongly induce more efficient learning ( and can promote convergence to higher performing policies ) ( S. Devlin & Kudenko , 2011 ) . We tackle the problem of how to learn intrinsic rewards produced by a function F ? that leads to the agents learning policies that jointly maximise the system performance ( through coordinated learning ) . Naturally , the problem of learning F ? can be approached by optimising the input θ ∈ Rm of a parameteric function : F̂ ( s , a ; θ ) ; that is , finding θ ? ∈ Rm for which F ? ( s , a ) = F̂ ( s , a , θ ? ) given an intrinsic reward F̂ where s is the state and a is the agents ’ joint action . Determining this function is a significant challenge since poor choices can hinder learning and the concurrency of multiple learning processes presents potential convergence issues in a system already populated by multiple learners ( Zinkevich et al. , 2006 ) . Additionally , we require that the method preserves the optimal joint policy and the underlying dec-MDP . Note that using an optimisation procedure to find θ ? directly does not make use of information generated by intermediate state-joint-action-reward tuples of the RL problem which can guide the optimisation .

Summary: This paper introduces a learned centralized exploration reward for multi-agent settings. The exploration reward is factorized into an on/off gate (dubbed ‘switching control’) and a scale function. Some mathematical derivations are included (sketched in the main text, with details in the appendices) to provide theoretical guarantees on how the exploration reward changes the solutions the training procedure might find. Evaluations on gridworld environments that target specific difficulties of multi-agent exploration show promising results. Some maps from the SMAC benchmark are also included, and again show good results.