Learning to Shape Rewards using a Game of Two Partners

1 INTRODUCTION . Reinforcement learning ( RL ) offers the potential for autonomous agents to learn complex behaviours without the need for human intervention [ 39 ] . Despite the notable success of RL in a variety domains [ 8 , 31 , 35 , 24 ] , enabling RL algorithms to learn successfully in numerous real-world tasks remains a challenge [ 41 ] . A key obstacle to the success of RL algorithms is the requirement of a rich reward signal that can guide the agent towards an optimal policy [ 7 ] . In many settings of interest such as physical tasks and video games , rich informative signals of the agent ’ s performance are not readily available [ 14 ] . For example , in the video game Super Mario [ 33 ] , the agent must perform sequences of hundreds of actions while receiving no rewards for it to successfully complete its task . In this setting , the sparse reward provides infrequent feedback of the agent ’ s performance . This leads to RL algorithms requiring large numbers of samples ( and high expense ) for solving problems [ 14 ] . Consequently , there is great need for RL techniques that solve these problems efficiently . Various biological systems have mechanisms that produce rewards for activities that present little or no direct biological utility . Such mechanisms , which have been fashioned over millions of years are designed to provide intrinsic rewards to promote behaviours that improve chances of outcomes with biological utility [ 1 ] . In RL , reward shaping ( RS ) is a tool to introduce intrinsic rewards known as shaping rewards that supplement environment reward signals . These rewards can encourage exploration and insert structural knowledge in the absence of informative environment rewards which can improve learning outcomes [ 10 ] . In general however , RS algorithms assume hand-crafted and domainspecific shaping functions whose construction is typically highly labour intensive . This runs contrary to the aim of autonomous learning . Moreover , poor choices of shaping rewards can worsen the agent ’ s performance [ 9 ] . To resolve these issues , a useful shaping reward must be obtained autonomously . Inspired by naturally occurring systems , we tackle the problem of sparse and uninformative rewards by developing a framework that autonomously constructs shaping rewards during learning . Our framework , ROSA , works by introducing an additional RL agent , Shaper , that adaptively learns to construct shaping rewards by observing Controller ( the agent whose goal is to solve the environment task ) while Controller learns to solve its task . This generates tailored shaping rewards without the need for domain knowledge or manual engineering . The shaping rewards supplement the environment reward and promote effective learning , our framework therefore addresses the key challenges in RS . The resulting framework is a two-player nonzero-sum Markov game ( MG ) [ 34 ] – an extension of Markov decision process ( MDP ) that involves two independent learners with distinct objectives . In our framework , the two agents that have distinct learning agendas , cooperate to achieve Controller ’ s objective . This MG formulation confers various advantages : 1 ) The shaping-reward function is constructed fully autonomously . The game also ensures the shaping reward improves Controller ’ s performance unlike RS methods that can lower performance . 2 ) By learning the shaping-reward function while Controller learns its optimal policy , Shaper learns to adaptively facilitate Controller ’ s learning and improve Controller ’ s performance . 3 ) Both learning processes converge so Controller learns the optimal value function for its task . 4 ) ROSA learns subgoals [ 28 ] to decompose complex tasks and promote complex exploration patterns . An integral component of ROSA is a novel combination of RL and switching controls [ 2 , 22 ] . This enables Controller to quickly determine useful states to learn to add and calibrate shaping rewards ( i.e . the states in which adding shaping rewards improve Controller ’ s performance ) and disregard others . This is in contrast with an RL controller ( i.e . Controller ) which must learn its best actions at every state . This leads to Shaper quickly finding shaping rewards that guide Controller ’ s learning process toward optimal trajectories ( and away from suboptimal trajectories , c.f . Experiment 1 ) . For our two-player framework to succeed we have to overcome several obstacles . Solving MGs involves finding a stable point in which each player responds optimally to the actions of the other . In our MG , this stable point describes a pair of policies for which Shaper introduces a shaping reward that improves performance and , with that , Controller executes an optimal policy for the task . Tractable methods for solving MGs are rare with convergence of MG methods being seldom guaranteed except in a few special cases [ 42 ] . Nevertheless , using special features in the design of our game , we prove the existence of a stable point solution of our MG. We then prove the convergence of our learning method to the solution of the game and show that the solution coincides with the solution of the Controller ’ s problem . This ensures Shaper learns a shaping-reward function that improves Controller ’ s performance and that Controller learns the optimal value function for the task . 2 RELATED WORK . Reward Shaping ( RS ) adds a shaping function F to supplement the agent ’ s reward to boost learning . RS however has some critical limitations . First , RS does not offer a means of finding F . Second , poor choices of F can worsen the agent ’ s performance [ 9 ] . Last , adding shaping rewards can change the underlying problem therefore generating policies that are completely irrelevant to the task [ 20 ] . In [ 27 ] it was established that potential-based reward shaping ( PBRS ) which adds a shaping function of the form F ( st+1 , st ) = γφ ( st+1 ) − φ ( st ) preserves the optimal policy of the problem . Recent variants of PBRS include potential-based advice which defines F over the state-action space [ 13 ] and approaches that include time-varying shaping functions [ 11 ] . Although the last issue can be addressed using potential-based reward shaping ( PBRS ) [ 27 ] , the first two issues remain . To avoid manual engineering of F , useful shaping rewards must be obtained autonomously . Towards this [ 45 ] introduce an RS method that adds a shaping-reward function prior which fits a distribution from data obtained over many tasks . Recently , [ 16 ] use a bilevel technique to learn a scalar coefficient for an already-given shaping-reward function . Nevertheless , constructing F while training can produce convergence issues since the reward function now changes during training . [ 17 ] . Moreover , while F is being learned the reward can be corrupted by inappropriate signals that hinder learning . Curiosity based reward shaping aims to encourage the agent to explore states by rewarding the agent for novel state visitations using exploration heuristics . One approach is to use state visitation counts [ 29 ] . More elaborate approaches such as [ 6 ] introduce a measure of state novelty using the prediction error of features of the visited states from a random network . [ 30 ] use the prediction error of the next state from a learned dynamics model and [ 15 ] maximise the information gain about the agent ’ s belief of the system dynamics . In general , these methods provide no performance guarantees nor do they ensure the optimal policy ( of the underlying MDP ) is preserved . Moreover , they naively reward exploration to unvisited states without consideration of the environment reward . This can lead to spurious objectives being maximised ( see Experiment 3 in §6 ) . Within these two categories , closest to our work are bilevel approaches for learning the shaping function [ 16 , 36 ] . Unlike [ 16 ] which requires a useful shaping reward to begin with , ROSA constructs a shaping reward function from scratch leading to a fully autonomous method . Moreover , in [ 16 , 36 ] , the agent ’ s policy and shaping rewards are learned with consecutive updates . In contrast , ROSA performs these operations concurrently leading to a faster , more efficient procedure . Also in contrast to [ 16 , 36 ] , ROSA learns shaping rewards only at relevant states , this confers high computational efficiency ( see Experiment 2 , §6 ) ) . As we describe , ROSA , which successfully learns the shapingreward function F , uses a similar form as PBRS . However in ROSA , F is augmented to include the actions of another RL agent to learn the shaping rewards online . Lastly , unlike curiosity-based methods e.g. , [ 6 , 30 ] , our method preserves the agent ’ s optimal policy for the task ( see Experiment 3 , §6 ) and introduces intrinsic rewards that promote complex learning behaviour ( see Experiment 1 , §6 ) . 3 PRELIMINARIES & NOTATIONS . In RL , an agent sequentially selects actions to maximise its expected returns . The underlying problem is typically formalised as a MDP 〈S , A , P , R , γ〉 where S is the set of states , A is the discrete set of actions , P : S ×A×S → [ 0 , 1 ] is a transition probability function describing the system ’ s dynamics , R : S ×A → R is the reward function measuring the agent ’ s performance , and the factor γ ∈ [ 0 , 1 ) specifies the degree to which the agent ’ s rewards are discounted over time [ 39 ] . At time t the system is in state st ∈ S and the agent must choose an action at ∈ A which transitions the system to a new state st+1 ∼ P ( ·|st , at ) and produces a reward R ( st , at ) . A policy π : S × A → [ 0 , 1 ] is a probability distribution over state-action pairs where π ( a|s ) represents the probability of selecting action a ∈ A in state s ∈ S. The goal of an RL agent is to find a policy π ? ∈ Π that maximises its expected returns given by the value function : vπ ( s ) = E [ ∑∞ t=0 γ tR ( st , at ) |at ∼ π ( ·|st ) ] where Π is the agent ’ s policy set . We denote this MDP by M. A two-player Markov game ( MG ) is an augmented MDP involving two agent that simultaneously take actions over many rounds [ 34 ] . In the classical MG framework , each agent ’ s rewards and the system dynamics are now influenced by the actions of both agents . Therefore , each agent i ∈ { 1 , 2 } has its reward function Ri : S ×A1 ×A2 → R and action set Ai and its goal is to maximise its own expected returns . The system dynamics , now influenced by both agents , are described by a transition probability P : S × A1 × A2 × S → [ 0 , 1 ] . As we discuss in the next section , ROSA induces a specific MG in which the dynamics are influenced by only Controller . In RS the question of which φ to insert has not been addressed . Moreover it has been shown that poor choices of φ hinder learning [ 9 ] . Consequently , in general RS methods rely on hand-crafted shaping-reward functions that are constructed using domain knowledge ( whenever available ) . In the absence of a useful shaping-reward function F , the challenge is to learn a shaping-reward function that leads to more efficient learning while preserving the optimal policy . Naturally , we can formalise the problem of learning such an F by constructing F as a parametric function of θ ∈ Rm : F̂ ( st+1 , st ; θ ) : = γφ̂ ( st+1 , θ ) − φ̂ ( st , θ ) . Now the problem is to find θ ? ∈ Rm for φ ( s ) = φ̂ ( s , θ ? ) such that F ( st+1 , st ) = F̂ ( st+1 , st ; θ ? ) , i.e. , we aim to find θ ? that yields a useful shaping-reward function . Determining this function is a significant challenge ; poor choices can hinder the learning process , moreover attempting to learn the shaping-function while learning the RL agent ’ s policy presents convergence issues given the two concurrent learning processes [ 44 ] . Another issue is that using an optimisation procedure to find θ ? directly does not make use of information generated by intermediate state-action-reward tuples of the RL problem which can help to guide the optimisation .

Summary: The authors propose ROSA, a reward shaping method that trains a separate “Shaper” policy to learn how to generate reward bonuses. The problem is formulated as a two-player Markov game, in which the environment dynamics are only affected by the primary, “Controller” policy, but the Shaper and Controller policies each optimize their own rewards. The Controller’s reward is the normal reward plus a potential-based reward that depends on the previous two states and the previous two Shaper actions. The shapers’ reward is a combination of the Controller’s reward, a penalty for switching often, and a count-based exploration bonus. The shaper’s rewards are also automatically gated by a switching function, which is hard-coded to switch with higher probability whenever a curiosity-metric increases (in this case using the RND exploration metric). The authors extend the standard reward shaping result to show that including the actions of the Shaper in the potential-based reward shaping term maintains policy invariance. Then, the authors empirically demonstrate on a grid-world environment that ROSA provides useful reward shaping that guides the agent towards the goal and ignores irrelevant parts of the state space. The authors also compare to and find that ROSA outperforms ICM, RND, PPO, and LIRPG on Gravitar, and gets similar performance on Solaris and Super Mario as some of these prior works.