LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning 

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) has emerged as a powerful tool to enable autonomous agents to solve various tasks such as ride-sharing ( Zhou , Luo , et al. , 2020 ) and swarm robotics ( Hüttenrauch et al. , 2017 ; Mguni et al. , 2018 ) . In multi-agent systems ( MAS ) , maximising system performance often requires agents to coordinate during exploration and learn coordinated joint actions ( Matignon et al. , 2012 ) . However , in many MAS , the reward signal provided by the environment is not sufficient to guide the agents towards coordinated behaviour ( Matignon et al. , 2012 ) . Consequently , relying on solely the individual rewards received by the agents may not lead to optimal outcomes ( Mguni et al. , 2019 ) . This problem is worsened by the fact that MAS can have many stable points some of which lead to arbitrarily bad outcomes ( Roughgarden & Tardos , 2007 ) . As in single agent RL , in MARL inefficient exploration can dramatically decrease sample efficiency . In MAS , a major challenge is how to overcome sample inefficiency from poorly coordinated exploration . Unlike single agent RL , in MARL , the collective of agents is typically required to coordinate its exploration to find their optimal joint policies1 . A second issue is that in many MAS settings of interest , such as video games and physical tasks , rich informative signals of the agents ’ joint performance are not readily available ( Hosu & Rebedea , 2016 ) . For example , in StarCraft Micromanagement ( Samvelyan et al. , 2019 ) , the sparse reward alone ( win , lose ) gives insufficient information to guide agents toward their optimal joint policy . Consequently , MARL requires large numbers of samples producing a great need for MARL methods that can solve such problems efficiently . To aid coordinated learning , algorithms such as QMIX ( Rashid et al. , 2018 ) , MAVEN ( Mahajan et al. , 2019 ) and COMA ( Foerster et al. , 2018 ) , so-called centralised critic and decentralised execution ( CT-DE ) methods use a centralised critic whose role is to estimate the agents ’ expected returns . The critic makes use of all available information generated by the system , specifically the global state and the joint action . To enable effective CT-DE , it is critical that the joint greedy action 1In single agent RL exploration issues can be mitigated by adjusting exploration rates or the policy variance ( Tijsma et al. , 2016 ) . However it has been shown that the same is not possible in MARL ( Mahajan et al. , 2019 ) . should be equivalent to the collection of individual greedy actions of agents , which is called the IGM ( Individual-Global-Max ) principle ( Son et al. , 2019 ) . CT-DE methods are however , prone to convergence to suboptimal joint policies ( Wang , Ren , Han , et al. , 2020 ) . In particular , existing value factorisations , e.g . QMIX and VDN ( Sunehag et al. , 2017 ) , can not ensure an exact guarantee of IGM consistency ( Wang , Ren , Liu , et al. , 2020 ) . Moreover , CT-DE methods such as QMIX require a monotonicity condition which is violated in scenarios where multiple agents must coordinate but are penalised if only a subset of them do so ( see Exp . 2 , Sec . 6.1 ) . To tackle these issues , in this paper we introduce a new MARL framework , LIGS that constructs intrinsic rewards online which guide MARL learners towards their optimal joint policy . LIGS involves an adaptive intrinsic reward agent , Generator that selects intrinsic rewards to add according to the history of visited states and joint actions performed by the agents . Generator adaptively guides the agents ’ exploration and behaviour towards coordination and maximal joint performance . A pivotal feature of LIGS is the novel combination of RL and switching controls ( Bayraktar & Egami , 2010 ; Mguni , 2018 ) which enables it to determine the best set of states to learn to add intrinsic rewards while disregarding less useful states . This enables Generator to quickly learn how to set intrinsic rewards that guide the agents during their learning process . Moreover , the intrinsic rewards added by Generator can significantly deviate from the environment rewards . This enables LIGS to both promote complex joint exploration patterns and decompose difficult tasks . Despite this flexibility , special features within LIGS ensure the underlying optimal policies are preserved so that the agents learn to solve the task at hand . Overall , LIGS has several advantages : • LIGS has the freedom to introduce rewards that vastly deviate from the environment rewards . With this , LIGS promotes coordinated exploration ( i.e . visiting unplayed state-joint actions ) among the agents enabling them to find joint policies that maximise the system rewards and generates intrinsic rewards to aid solving sparse reward MAS . • LIGS selects which best states to add intrinsic rewards adaptively in response to the agents ’ behaviour while the agents learn leading to an efficient learning process . • LIGS ’ s intrinsic rewards preserve the agents ’ optimal joint policy and ensure that the total environment return is ( weakly ) increased . To enable the framework to perform successfully , we overcome several challenges : i ) Firstly , constructing an intrinsic reward can change the underlying problem leading to the agents solving irrelevant tasks ( Mannion et al. , 2017 ) . We resolve this by endowing the intrinsic reward function with special form which both allows a rich spread of intrinsic rewards while preserving the optimal policy . ii ) Secondly , introducing intrinsic reward functions can worsen the agents ’ performance ( S. Devlin & Kudenko , 2011 ) and doing so while training can lead to convergence issues . We prove LIGS leads to better performing policies and that LIGS ’ s learning process converges and preserves the MARL learners ’ convergence properties . iii ) Lastly , adding an agent Generator with its own goal leads to a Markov game ( MG ) with N + 1 agents ( Fudenberg & Tirole , 1991 ) . Tractable methods for solving MGs are extremely rare with convergence only in special cases ( Yang & Wang , 2020 ) . Nevertheless , using a special set of features in LIGS ’ s design , we prove LIGS converges to a solution in which it learns an intrinsic reward function that improves the agents ’ performance . 2 RELATED WORK . Multi-agent exploration methods seek to promote coordinated exploration among MARL learners . Maven et . al ( Mahajan et al. , 2019 ) propose a hybridisation of value and policy-based methods that uses mutual information to learn a diverse set of behaviours between agents . Though this approach promotes coordinated exploration , it does not encourage exploration of novel states . Other approaches to promote exploration in MARL while assuming aspects of the environment are known in advance and agents can perform perfect communication between themselves ( Viseras et al. , 2016 ) . Similarly , to promote coordinated exploration in partially observable settings , ( Pesce & Montana , 2020 ) propose end-to-end learning of a communication protocol through a memory device . In general , exploration-based methods provide no performance guarantees nor do they ensure the optimal policy ( of the underlying dec-MDP ) is preserved . Moreover , many employ heuristics that naively reward exploration to unvisited states without consideration of the environment reward . This can lead to spurious objectives being maximised . Reward shaping ( Harutyunyan et al. , 2015 ) ( RS ) is a technique which aims to alleviate the problem of sparse and uninformative rewards by supplementing the agent ’ s reward with a prefixed term F . In ( Ng et al. , 1999 ) it was established that adding a shaping reward function of the form F ( st+1 , st ) = γφ ( st+1 ) −φ ( st ) preserves the optimal policy and in some cases can aid learning . RS has been extended to MAS ( S. Devlin & Kudenko , 2011 , 2016 ; S. Devlin et al. , 2011 ; S. M. Devlin & Kudenko , 2012 ; Mannion et al. , 2018 ; Sadeghlou et al. , 2014 ) in which it is used to promote convergence to efficient social welfare outcomes . Poor choices of F in a MAS can slow the learning process and can induce convergence to poor system performance ( S. Devlin & Kudenko , 2011 ) . In MARL , the question of which shaping function to use remains unaddressed . Typically , RS algorithms rely on hand-crafted shaping reward functions that are constructed using domain knowledge , contrary to the goal of autonomous learning ( S. Devlin & Kudenko , 2011 ) . As we later describe LIGS , which successfully learns an instrinsic reward function F , uses a similar form as PBRS however , F is now augmented to include the actions of another RL agent to learn the intrinsic rewards online . In ( Du et al. , 2019 ) an approach towards learning intrinsic rewards is proposed in which a parameterised intrinsic reward is learned using a bilevel approach through a centralised critic . Loosely related are single-agent methods ( Dilokthanakul et al. , 2019 ; Kulkarni et al. , 2016 ; Pathak et al. , 2017 ; Zheng et al. , 2018 ) which , in general , introduce heuristic terms to generate intrinsic rewards . Within these categories , closest to our work is the intrinsic reward approach in ( Du et al. , 2019 ) . There , the agents ’ policies and intrinsic rewards are learned with a bilevel approach . In contrast , LIGS performs these operations concurrently leading to a faster , more efficient procedure . A crucial point of distinction is that in LIGS , the intrinsic rewards are constructed by an RL agent ( Generator ) with its own reward function . Consequently , LIGS can generate complex patterns of intrinsic rewards , encourage joint exploration . Additionally , LIGS learns intrinsic rewards only at relevant states , this confers high computational efficiency . Lastly , unlike exploration-based methods e.g. , ( Mahajan et al. , 2019 ) , LIGS ensures preservation of the agents ’ joint optimal policy for the task . 3 PRELIMINARIES . A fully cooperative MAS is modelled by a decentralised-Markov decision process ( dec-MDP ) ( Yang & Wang , 2020 ) . A dec-MDP is an augmented MDP involving a set of N ≥ 2 agents denoted by N that independently decide actions to take which they do so simultaneously over many rounds . Formally , a dec-MDP is a tuple M = 〈N , S , ( Ai ) i∈N , P , R , γ〉 where S is the finite set of states , Ai is an action set for agent i ∈ N and R : S ×A→ P ( D ) is the reward function that all agents jointly seek to maximise where D is a compact subset of R and lastly , P : S ×A×S → [ 0 , 1 ] is the probability function describing the system dynamics where A : = ×Ni=1Ai . Each agent i ∈ N uses a Markov policy πi : S ×Ai → [ 0 , 1 ] to select its actions . At each time t ∈ 0 , 1 , . . . , the system is in state st ∈ S and each agent i ∈ N takes an action ait ∈ Ai . The joint action at = ( a1t , . . . , aNt ) ∈ A produces an immediate reward ri ∼ R ( st , at ) for agent i ∈ N and influences the next-state transition which is chosen according to P . The goal of each agent i is to maximise its expected returns measured by its value function vπ i , π−i ( s ) = Eπi , π−i [ ∑∞ t=0 γ tR ( st , at ) ] , where Πi is a compact Markov policy space and −i denotes the tuple of agents excluding agent i. Intrinsic rewards can strongly induce more efficient learning ( and can promote convergence to higher performing policies ) ( S. Devlin & Kudenko , 2011 ) . We tackle the problem of how to learn intrinsic rewards produced by a function F ? that leads to the agents learning policies that jointly maximise the system performance ( through coordinated learning ) . Naturally , the problem of learning F ? can be approached by optimising the input θ ∈ Rm of a parameteric function : F̂ ( s , a ; θ ) ; that is , finding θ ? ∈ Rm for which F ? ( s , a ) = F̂ ( s , a , θ ? ) given an intrinsic reward F̂ where s is the state and a is the agents ’ joint action . Determining this function is a significant challenge since poor choices can hinder learning and the concurrency of multiple learning processes presents potential convergence issues in a system already populated by multiple learners ( Zinkevich et al. , 2006 ) . Additionally , we require that the method preserves the optimal joint policy and the underlying dec-MDP . Note that using an optimisation procedure to find θ ? directly does not make use of information generated by intermediate state-joint-action-reward tuples of the RL problem which can guide the optimisation .

Summary: The paper focuses on learning intrinsic rewards for multi-agent reinforcement learning, which is an important problem. Different from previous works on this topic, the authors propose to train an agent with a learnable gating function that incentives other agents. Theoretical analysis and empirical evaluation are provided to prove the effectiveness of the proposed method.