Learning to Dynamically Select Between Reward Shaping Signals

1 INTRODUCTION . Although numerous successes have been reported in reinforcement learning ( RL ) , it still suffers from several drawbacks that prevent it from performing to expectation in many real-life situations . One critical limitation is the sample complexity . In order to arrive at an acceptable solution , RL requires an enormous amount of experience ( i.e. , data ) before useful behaviors are learned . Reward shaping is one approach that seeks to address this problem , providing additional feedback in the form of shaping rewards to allow an RL agent to learn faster . Moreover , shaping rewards that follow a potential form preserve guarantees that optimal solutions will be found despite the altered feedback ( Ng et al. , 1999 ) . However , until recently , most reward shaping signals and functions have been hand-engineered . This task is notoriously difficult , as even slight incorrectness can lead to local optima that do not solve the present problem ( Randløv & Alstrøm , 1998 ) . Automatic reward shaping eliminates the difficulty of shaping reward signal and function design by learning the shaping reward function that in turn enables optimal learning of the policy . Automatic reward shaping in itself is an extremely difficult problem to solve . In order to simplify the problem , we break down the idea into two sub-problems : ( 1 ) learning shaping reward signals , and ( 2 ) learning how to exploit shaping reward signals to provide an appropriate shaping reward at each state of the learning process . This paper focuses on the latter task , i.e. , ( 2 ) , which we refer to as ” automatic reward adaptation ” . Problem Definition : Given a set of shaping reward signals ~φ = φ1 , . . . , φn , learn to adapt these signals automatically to produce a single shaping reward F ( s , a , s′ ) : ~φ→ R for each state s , action a , next state s′ tuple in the learning process . The full reward for any transition in the RL problem is R ( s , a , s′ ) = r ( s , a , s′ ) + F ( s , a , s′ ) , where r is the original reward of the RL problem before shaping . Our proposed approach to automatic reward adaptation is to learn to dynamically select the right shaping reward signal φi from ~φ at each transition encountered in learning . In addition , our method learns using minimal infrastructure and with value-based gradients . We avoid the use of models and additional approximate value functions ( which previous approaches to automatic reward shaping typically rely on ) and perform updates using already-present values as feedback and with minimal additional computation . The proposed ideas have been verified through experiments in a variety of environments using different shaping reward paradigms . The basis of our shaping signal selection approach is rooted in how humans seem to react in realistic decision-making situations . Given a set of basic reward signals , say ” comfort ” and ” selfpreservation ” , humans have the uncanny ability to identify when and how much one should listen to any given signal depending on the current situation . For example , in a lane keeping task , if we were near the edge of a lane with no cars around us , we would simply listen to the ” comfort ” reward signal telling us to move closer to the center of the lane . However , if there was another car moving into the same lane at the same time , we would instead follow the ” self-preservation ” reward signal dictating that we stay as reasonably far away from other cars as possible . Both signals lead to correct performance but are applicable in entirely different situations , provide different information , and , most importantly , induce different behaviors . While we recognize that explicit selection risks not using available information in other shaping reward signals , we argue that it provides a guaranteed improvement over only the environment reward . In this sense , selection does not hinder the RL agent in learning even though it is incomplete relative to optimal reward shaping , which is extremely difficult to design . This work parallels the area of multi-objective RL ( MORL ) , which investigates how to perform RL in the presence of multiple rewards ( Roijers et al. , 2013 ) . However , there are a couple differences between our idea and previous work in automatic reward shaping and MORL . First , much research in automatic reward shaping focuses on the first sub-problem ( 1 ) described above , i.e. , learning the parameters of some shaping reward signal ( usually only one ) . However , realistic problems often involve a number of ( possibly conflicting ) goals or signals that can not be trivially summarized as a single signal or function . Our work focuses on the second sub-problem ( 2 ) and investigates if there are better ways to access provided shaping reward signals for more effective reward shaping . Second , while MORL also handles environments with multiple feedback signals , it aims to solve a different problem overall . MORL attempts to learn solutions that best optimize the multiple rewards presented to it , but our idea attempts to best exploit multiple rewards in a way that solves the single objective present in the problem . Furthermore , MORL typically learns a linear combination of signals as the final reward shaping function . While effective in many cases , a learned fixed combination does not always suit each individual state in learning , especially if the environment is dynamic . That being said , we do not discount the potential of combination in automatic reward adaptation.1 Our goal is simply to consider another approach with promising flexibility . 2 RELATED WORK . The fundamental theory behind reward shaping regarding its optimality-preserving guarantees was documented in Ng et al . ( 1999 ) . Since then , the guarantees of potential-based reward shaping have been expanded to time-varying potential options by Harutyunyan et al . ( 2015 ) , which more naturally reflects realistic problems and the concept of automatic reward shaping . One of the initial works in this area demonstrated impressive results but the shaping reward function was heavily engineered ( Laud & DeJong , 2002 ) . More recent research has made progress towards greater autonomy . The mechanisms driving automatic reward shaping typically fall into a few categories , with one of the originals involving the use of abstractions to learn values on a simpler version of the problem before using these values as shaping rewards ( Marthi , 2007 ) . Other projects have followed up this idea using both model-free and model-based RL methods ( Grześ & Kudenko , 2010 ) as well as with the estimated modeling of the reward function itself ( Marom & Rosman , 2018 ) . One particular work directly bootstraps the model-based learning in R-max into a shaping reward function ( Asmuth et al. , 2008 ) . Yet another builds a graph of the environment before using subgoals as a means of defining shaping rewards which adjust as the graph is updated ( Marashi et al. , 2012 ) . Credit assignment is another form of automatic reward shaping . It injects information into previous experiences such that learning is enhanced when replay occurs . Song & Jin ( 2011 ) implemented this by identifying critical states and using these landmarks as sub-rewards to make learning easier . De Villiers & 1We will study flexible combination of individual shaping signals in our future work . Sabatta ( 2020 ) directly augmented the replay buffer with propagated rewards to make the reward surface more dense and accelerate learning . Zheng et al . ( 2018 ) integrated learning of an intrinsic motivation signal along with the agent ’ s value and policy learning . While representative of automatic reward shaping , they only consider the single intrinsic motivation feedback signal when performing reward shaping . Adjacent to our focus is the area of multi-objective RL ( MORL ) . Brys et al . ( 2014 ) explored some connections between multi-objective frameworks and reward shaping which Brys et al . ( 2017 ) later expanded on by directly characterizing shaping reward signals as multiple objectives and applying MORL to successfully solve such problems . Van Seijen et al . ( 2017 ) implemented a form of automatic reward adaptation by learning a separate value function for each reward signal and using a value estimated from each reward signal ’ s associated value to update the policy . Fu et al . ( 2019 ) addressed the presence of multiple goals ( in the form of rewards ) by explicitly calculating the best weighting for each reward in a linear combination after some amount of experience . Tajmajer ( 2018 ) similarly used a linear combination to combine multiple rewards but simultaneously learned a set of decision values that acted as weights for each reward in the final feedback computation . As mentioned previously , our work differs from these methods primarily in that our method does not use linear combination , uses less learning infrastructure , and aims to control reward shaping ( not the agent itself in the pursuit of multiple goals ) . Instead of learning the weights for rewards in solving a problem with multiple objectives , we dynamically select between shaping reward signals to use in learning at any given point . While inspired by concepts in MORL , this selection technique is , to our knowledge , new . 3 BACKGROUND . 3.1 REINFORCEMENT LEARNING . RL is a form of machine learning that specializes in learning to solve problems that involve unknown environments and require sequences of decisions . Such problems are often characterized as Markov Decision Processes ( MDPs ) , and described by a tuple M = ( S , A , τ , R , γ ) . S represents the set of states , A is the set of possible actions , τ is the transition probability function τ : S x A → S , and R is the reward function R : S x A x S → R. γ is a parameter that describes the discount factor . MDPs operate generally through the following repeating set of steps : a state is observed , an action is taken which transitions the agent into another state while a reward is given by the environment , then the transition and reward are used to update the policy and the value function ( if used ) . The goal of any given agent attempting to solve an MDP is to generate a policy through interactions with the environment that maps states to actions , π : S → A . This policy is optimized towards the objective of maximizing the expected sum of discounted rewards G = Eπ [ ∑ γr | S = s , A = a ] , or return . Q-learning is a well-established algorithm that follows the Q-value , or action-value , form of this objective by attempting to find the action-value function Q ( s , a ) = Eπ [ G | s , a ] ( Kröse , 1995 ) . Updates take the following form : Q ( s , a ) = Q ( s , a ) + α ( r + γmax a Q ( s′ , a ) −Q ( s , a ) ) ( 1 ) with α as the learning rate ( Sutton & Barto , 2018 ) . Typically , some form of encouraged exploration is implemented , most often by specifying a probability ( often decaying over time ) with which the agent must select a random action instead of the greedy one .

Summary: The idea of potential-based reward shaping (PBRS) is to improve the performance of learning agents by incorporating additional domain knowledge into their reward function (making it dense), while maintaining the same asymptotic convergence guarantees. However, one aspect of PBRS, that is introduced in this work, is how to select among multiple shaping signals in an adaptive manner that is also dependent on the current context, in a way that can overall improve the performance of the agent. A method is proposed  here that learns to select among two or more shaping functions based on experience, represented as the difference between the TD error (or surprise) and the shaped reward signal, the conjecture being that signals that are closer to the TD error can lead to the best improvement in the agent's performance. The signal with the smallest difference is used as a self-supervised label to train a classifier that selects the signal. Experiments show that the method is at par with other similar approaches while being simpler to implement.