Learning to Dynamically Select Between Reward Shaping Signals
Authors: Alexander Politowicz
Bing Liu
Venue:  (2021)
Keywords: selection
automatic
reward
shaping
reinforcement learning

Abstract: Reinforcement learning (RL) algorithms often have the limitation of sample complexity. Previous research has shown that the reliance on large amounts of experience can be mitigated through the presence of additional feedback. Automatic reward shaping is one approach to solving this problem, using automatic identification and modulation of shaping reward signals that are more informative about how agents should behave in any given scenario to learn and adapt faster. However, automatic reward shaping is still very challenging. To better study it, we break it down into two separate sub-problems: learning shaping reward signals in an application and learning how the signals can be adaptively used to provide a single reward feedback in the RL learning process. This paper focuses on the latter sub-problem. Unlike existing research, which tries to learn one shaping reward function from shaping signals, the proposed method learns to dynamically select the right reward signal to apply at each state, which is considerably more flexible. We further show that using an online strategy that seeks to match the learned shaping feedback with optimal value differences can lead to effective reward shaping and accelerated learning. The proposed ideas are verified through experiments in a variety of environments using different shaping reward paradigms.

Under review as a conference paper at ICLR 2021
LEARNING TO DYNAMICALLY SELECT BETWEEN RE- WARD SHAPING SIGNALS
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) algorithms often have the limitation of sample com- plexity. Previous research has shown that the reliance on large amounts of expe- rience can be mitigated through the presence of additional feedback. Automatic reward shaping one approach to solving this problem, using automatic identifi- cation and modulation of shaping reward signals that are more informative about how agents should behave in any given scenario to learn and adapt faster. How- ever, automatic reward shaping is still very challenging. To better study it, we break it down into two separate sub-problems: learning shaping reward signals application and learning how the signals can be adaptively used to provide single reward feedback in the RL learning process. This paper focuses on the lat- ter sub-problem. Unlike existing research, which tries to learn one shaping reward function from shaping signals, the proposed method learns to dynamically select the right reward signal to apply at each state, which is considerably more flexible. We further show that using an online strategy that seeks to match the learned shap- ing feedback with optimal value differences can lead to effective reward shaping and accelerated learning. The proposed ideas are verified through experiments in a variety of environments using different shaping reward paradigms.
INTRODUCTION
Although numerous successes have been reported in reinforcement learning (RL), it still suffers from several drawbacks that prevent it from performing to expectation in many real-life situations. One critical limitation is the sample complexity. In order to arrive at acceptable solution, RI. requires an enormous amount of experience (i.e., data) before useful behaviors are learned. Reward shaping one approach that seeks to address this problem, providing additional feedback in the form of shaping rewards to allow an RL agent to learn faster. Moreover, shaping rewards that follow potential form preserve guarantees that optimal solutions will be found despite the altered feedback (Ng et al. 1999). However, until recently, most reward shaping signals and functions have been hand-engineered. This task is notoriously difficult, as even slight incorrectness can lead to local optima that do not solve the present problem (Randløv & Alstrøm 1998). Automatic reward shaping eliminates the difficulty of shaping reward signal and function design by learning the shaping reward function that in turn enables optimal learning of the policy.
Automatic reward shaping in itself is an extremely difficult problem to solve. In order to simplify the problem, we break down the idea into two sub-problems: (I) learning shaping reward signals, and (2) learning how to exploit shaping reward signals to provide an appropriate shaping reward at each state of the learning process. This paper focuses the latter task, i.e., (2), which we refer to as "automatic reward adaptation".
Given a set of shaping reward signals 3 0 21,...., On learn to adapt these signals automatically to produce single shaping reward F(s, s') : R for each state action a, next state tuple in the learning process. The full reward for any transition the RL problem is R(s, s') r(s, a, s') + as.,a,s)) where is the original reward of the RL problem before shaping.
Problem Definition:
Our proposed approach automatic reward adaptation is to learn to dynamically select the right shaping reward signal 6; from o at each transition encountered in learning. In addition, our method

---

Under review as a conference paper at ICLR 2021
learns using minimal infrastructure and with value-based gradients. We avoid the use of models and additional approximate value functions (which previous approaches to automatic reward shaping typically rely on) and perform updates using already-present present values feedback and with minimal additional computation. The proposed ideas have been verified through experiments in a variety of environments using different shaping reward paradigms.
The basis of our shaping signal selection approach is rooted in how humans seem to react in re- alistic decision-making situations. Given set of basic reward signals, say "comfort" and "self- preservation", humans have the uncanny ability to identify when and how much one should listen to any given signal depending on the current situation. For example, in lane keeping task, if we were near the edge of a lane with no cars around us, we would simply listen to the "comfort" re- ward signal telling us to move closer to the center of the lane. However, if there was another car moving into the same lane at the same time, we would instead follow the seeff-rreservation" reward signal dictating that we stay as reasonably far away from other cars as possible. Both signals lead to correct performance but are applicable in entirely different situations, provide different information, and, most importantly, induce different behaviors. While we recognize that explicit selection risks not using available information in other shaping reward signals, argue that it provides a guaran- teed improvement over only the environment reward. In this sense, selection does not hinder the RL agent in learning even though it is incomplete relative optimal reward shaping, which is extremely difficult to design.
This work parallels the area of multi-objective RL (MORL), which investigates how to perform RL in the presence of multiple rewards (Roijers et al., 2013). However, there are a couple differences between our idea and previous work in automatic reward shaping and MORL. First, much research in automatic reward shaping focuses on the first sub-problem (1) described above, i.e., learning the parameters of some shaping reward signal (usually only one). However, realistic problems often involve number of (possibly conflicting) goals or signals that cannot be trivially summarized as a single signal or function. Our work focuses on the second sub-problem (2) and investigates there are better ways to access provided shaping reward signals for more effective reward shaping. Second, while MORL also handles envirouments with multiple feedback signals, it aims to solve a different problem overall. MORL attempts to learn solutions that best optimize the multiple rewards presented to it, but our idea attempts to best exploit multiple rewards in a way that solves the single objective present in the problem. Furthermore, MORL typically learns linear combination of sig- nals as the final reward shaping function. While effective in many cases, a learned fixed combination does not always suit each individual state in learning, especially if the environment is dynamic. That being said, we do not discount the potential of combination in automatic reward adaptation] I Our goal is simply consider another approach with promising flexibility.
2 RELATED WORK
The fundamental theory behind reward shaping regarding its pptimaliit-preseevvvi guarantees was documented in Ng et al. (1999). Since then, the guarantees of potential- based reward shaping have been expanded time-varying potential options by Harutyunyan et al. (2015), which more naturally reflects realistic problems and the concept of automatic reward shaping. One of the initial works in this area demonstrated impressive results but the shaping reward function was heavily engineered (Laud & DeJong 2002). More recent research has made progress towards greater autonomy. The mechanisms driving automatic reward shaping typically fall into a few categories, with one of the originals involving the use of abstractions to learn values simpler version of the problem before using these values as shaping rewards (Marthi 2007). Other projects have followed up this idea using both model-free and model-based RL methods (Grzes & Kudenko 2010) well as with the estimated modeling of the reward function itself (Marom & Rosman] 2018). One particular work directly bootstraps the model-based learning in R-max into shaping reward function (Asmuth al. 2008). Yet another builds a graph of the environment before using subgoals as means of defining shaping rewards which adjust as the graph is updated (Marashi et al., 2012). Credit assignment is another form of automatic reward shaping. It injects information into previous experiences such that learning is enhanced when replay occurs. [Song & Jin (201T) implemented this by identifying critical states and using these landmarks as sub-rewards to make learning easier. De Villiers &
We will study flexihle combination of individual shaping signals in our future work.
2

---

Under review as a conference paper at ICLR 2021
Sabatta (2020) directly augmented the replay buffer with propagated rewards to make the reward surface more dense and accelerate learning. Zheng et al. (2018) integrated learning of an intrinsic motivation signal along with the agent's value and policy learning. While representative of automatic reward shaping, they only consider the single intrinsic motivation feedback signal when performing reward shaping.
Adjacent to our focus the area of multi-objective RL (MORL). Brys et al. (2014) explored some connections between multi-objective frameworks and reward shaping which Brys al. (2017) later expanded on by directly characterizing shaping reward signals as multiple objectives and applying MORL to successfully solve such problems. Van Seijen et al. (2017) implemented form of au- tomatic reward adaptation by learning separate value function for each reward signal and using value estimated from each reward signal's 's associated value to update the policy. Fu et al. (2019) addressed the presence of multiple goals (in the form of rewards) by explicitly calculating the best weighting for each reward in linear combination after some amount of experience. [ajmajer (2018) similarly used a linear combination to combine multiple rewards but simultaneously learned a set of decision values that acted as weights for each reward in the final feedback computation. As men- tioned previously, our work differs from these methods primarily in that our method does not linear combination, uses less learning infrastructure, and aims to control reward shaping (not the agent itself in the pursuit of multiple goals). Instead of learning the weights for rewards in solving a problem with multiple objectives, we dynamically select between shaping reward signals to use in learning at any given point. While inspired by concepts in MORL, this selection technique is, to our knowledge, new
3 BACKGROUND
3.1 REINFORCEMENT LEARNING
RL is a form of machine learning that specializes learning to solve problems that involve unknown environments and require sequences of decisions. Such problems are often characterized as Markov Decision Processes (MDPs), and described by a tuple M = (S, A, т, R, 7). represents the set of states, A is the set of possible actions, is the transition probability function is the reward function : S x A x S - R. ? is a parameter that describes the discount factor. MDPs operate generally through the following repeating set of steps: state is observed, an action is taken which transitions the agent into another state while a reward is given by the environment, then the transition and reward used update the policy and the value function (if used).
S I A - S, and
The goal of any given agent attempting to solve an MDP is to generate a policy through interactions with the environment that maps states actions, S + A. This policy is optimized towards the objective of maximizing the expected sum of discounted rewards G Ez 6> 7r | SS= 8, A = a]. or return. Q-learning is a well-established algorithm that follows the Q-value, or action-value, form of this objective by attempting to find the action-value function Q(s a) = E [G s, a] (Kröse 1995). Updates take the following form:
Q(s,a) = Q(+,a) aa(r 7 max Q( -((s,a))
(1)
Typically, some form of encouraged exploration is implemented, most often by specifying probability (often decaying over time) with which the agent must select a random action instead of the greedy one.
with a as the learning rate (Sutton& Barto 2018
3.2 REWARD SHAPING
In the above update equation (Eq. 1), the reward specified is the only feedback provided by the environment. Reward shaping seeks to amplify this feedback by providing more information, often referred to the shaping reward F. The inclusion of this term intuitively leads to the augmented update equation:
Q(s.a) = Q(s, a) + a(r F(s.a,8)) + max ((s, -ss,)
Note that is reliant on the current state, the taken action, and the next state. The shaping reward can be formulated such that it has guarantees of convergence the original MDP's optimal policy
3

---

Under review as a conference paper at ICLR 2021
if the shaping reward function follows the potential shaping paradigm. A shaping reward F is in potential-base form if it takes the difference of any shaping reward signal o, accounting for the discount factor:
F(s,a, s') = 70(s') o(s)
(3)
Of note is the fact that the optimal shaping reward function is exactly the optimal value function (gg
al. 1999
4 METHODS
4.1 AUTOMATIC REWARD ADAPTATION
Before we describe our method of representing reward signals, we argue for an intuitive and gen- eral shaping reward format. Assume that agent encounters set of additional shaping reward signals 6. These signals should imitate human instincts (e.g. "comfort" or "self-preservation") and thus be able to be easily formulated for many problems. This assumption is justified because an agent attempting to solve an RL problem must be provided with a manually designed feedback at some level, whether that be in the form of the environment reward function or through set of shaping reward signals. Additionally, this assumption enables improvement because it allows for the specification of intuitive rewards that domain experts can more easily design.
According to Ng et al. (1999), the optimal form of any shaping reward signal is exactly the opti- mal value function. Thus, learning an optimal shaping reward function is equivalent to finding the parameters 77 that give shaping rewards closest to the optimal values. The goal then is for all shap- ing reward signals oh to attempt to approximate at least a component of the optimal value function, and for the potential-based difference in these shaping reward signals r0(s') ((s) to attempt to approximate the shape of the optimal value surface (i.e. the difference in optimal values). How- ever, optimal values, and of course the optimal value function, are unknown in any RL problem (and indeed are the motivation for RL). Options for estimating optimal values include receiving ex- plicit knowledge from a (typically external) expert or through experience. Assuming the presence of an expert in realistic applications is unreliable, and thus the most robust option is to attempt to learn the shaping reward function through experience. The following selection method, parame- terized by n, is updated according to these tenets. Adjustments are made towards minimizing the difference between potential-based shaping reward signals and the difference in approximated val- ues (r + y max, Q(s', a) - Q(s. a)) (s") - 6, (s)). The convergence guarantees of RL given by general policy iteration (Sutton 2000) hold in this augmented learning procedure given that the shaping reward signals do not override the optimal values. This has been previously shown with respect to reward shaping by Brys et al. (2017) and can be achieved by scaling of the shaping reward signal differences.
4.2 SELECTION
Selection in the context of automatic reward adaptation describes a procedure in which some mecha- nism, henceforth referred to as the "selector", makes an exclusive decision regarding which shaping reward signal to listen to. A selector can be represented as a function approximator that receives as input the current state and the taken action a. The output is softmax set of probabilities that describes which portion of the current shaping reward signals to use in characterizing the shaping reward value F.
F(s, a, s') Sof toft) (75(6)) x(s))
Shaping reward signals are again used in the potential-based form. The processing of these signals through the selector does not invalidate the optimal policy guarantees, seen in this equivalence:
F F(s. a, 1Softmar(0) T (7ō(5') x(s))
= 8- (-0(')))
(5)
(1 (1softmazz()== 7 ((s))
= - (s') 30(s)
4

---

Under review as a conference paper at ICLR 2021
The selector is updated according the previously-deserioe principle of differences in shaping reward signals upproximating the differences in the optimal value function across corresponding transitions. Each signal's 's distance from the estimated value difference is calculated using (r + max, Q(s', a) (s, a)) (s') (s())) In addition to the value distance, an augmenting factor can be included better guide the selector. Note that the magnitude of a difference in shaping reward signals (s' (s(s) represents some measure of importance in the associated shaping reward signal. This factor can used directly in the update to adjust the selector towards signals that potentially earry more significant information. Overall, the augmented distance takes the following
form:
I(r + maxa Q( 0)) - Q(s, (70,(5') ((0(5)) di(s)))
(10:(5)) - (.(5) +)
The term is small constant to avoid divide-by-zero errors. The shaping reward signal with the smallest distance is considered to be the best decision and set as the target for the selector update. Categorical cross-entropy is then applied the loss function between the selector output probabilities and the target vector. Algorithm I outlines a basic implementation of selection in automatic reward adaptation. Note that this implementation of selection is inherently dynamic
Algorithm 1:
Automatic Reward Adaptation RI. Selection
Initialize: policy и, Q-value function Q, and selector Σ while not converged do
Sample action from policy
Take action a, observe reward and new state for experience (s, a, r, s') in replay buffer do Sample selector softmax probabilities from Σ
F(s, a, 8') =1sooftmee((;= E. (70(8') x(s))
(ss Q(s, a) + a(r F(s, a, s') + max, Q(s', -(s,a)) Q(s,a))
diffs: 1(+t max, Q(s (7))))))))))))))))--))))))))))))
((()(())
10
argmin(d:ffaa==i
Σ - c
12
end
13
end
determines which shaping reward signal to prioritize based the current state and action. As an augmentative component within the agent's learning infrastructure, the selector constantly adjusts to experienced transitions and provides adaptable shaping feedback online.
EXPERIMENTS
The selection paradigm was tested on a variety baselines provided by MuloCo (Todorov et al,, 2012) and OpenAI Gym (Brockman al., 2016). Reacher was used from MuJoCo and required the agent to maintain the tip of robotic arm close to some goal point in 2D plane. From Gym, selected for their diversity in reward functions. In CartPole and Pendulum, the agent must balance a linkage in the vertical up position. MountainCar challenging task that requires an agent to control car with the goal of reaching the top of slope.
CartPole, Pendulum, and MountainCar
Shaping reward signals were defined for each of the environments such way that a variety of different shaping potential surfaces could be tested. For CartPole, each signal rewarded the agent for keeping the corresponding state component, either cart position or pole angle, near zero. For Pendulum, the first shaping reward signal rewarded the agent purely for how close the pole was to vertical up. The second shaping reward signal accounted for both the pole angle and velocity and was the accurate signal. The MountainCar environment shaping reward signals tested similarly for distinction capabilities in the selector. The first shaping reward signal rewarded horizontal position of the car and suboptimal. The second shaping reward signal rewarded for combination of
5

---

Under review as a conference paper at ICLR 2021
Table 1: Experimental environments and shaping reward functions
Environment
Shaping Functions
Φ1 10
Gym CartPole
@2 = cart_position
@1 cos(polo (pole_anggle)
Gym Pendulum
Φ pole velocityl cos(pole aangle)
Φ1 car -psiitioo
Gym MountainCar
2 car velocityl 10 + car position
distance distance S 1.5
Φ1
0
distance > 1.5
MuJoCo Reacher
0
distance S 1.5
Φ
distance
distance > 1.5
car horizontal position and velocity and was effective at leading the agent to the solution. For the Reacher environment, each shaping reward signal provided information leading the goal point but in different regions of the state space. The selector was required to learn when to listen to each signal. Further mathematical details described in Table 1.
For Reacher, Pendulum, and MountainCar, DDPG with decaying noisy actions for exploration was used for its continuous control capabilities and competitiveness in many classic RL problems (Lil- lierap et al., (016)_ A version DQN with no target network and epsilon-greedh exploration was used for CartPole (Mnih al., 2015). Neural networks were used for all function approximation implementations. For the sake of space, hyperparameters for each environment are documented in the Appendix. To account for randomness, seeds were used to control random parameters in the neural networks as well as in the exploration noise.
Performance was measured as the number episodes until the averaged total reward over the past 50 episodes was maximized. Implementations of un-augmented algorithms were included basic comparison standard learning (denoted as the "base" agents). In addition, a competitive compar- ison was included in the form of agent that learned from direct linear combination of provided shaping reward signals (denoted as the "shaped" agents). Experiments were run five times, with the experiment showing the best performance taken and presented in the figures.
Comparisons were made to the recent automatic reward adaptation method described in Fu et al. (2019), which explicitly calculated best weightings of shaping reward signals. Their tests were performed in two similar environments, CartPole and MountainCar, and we applied our selection technique to their shaping reward signals in each problem.
All experiments were run on one of two setups. The first was a Windows 10 Home laptop contain- ing 16 GB of memory, Intel i7-9750H CPU, and an NVIDIA GeForce GTX 1650 graphics card. The other laptop ran Ubuntu 16.04 and contained 8 GB of memory, an Intel i7-48 19MQ CPU, and an NVIDIA GeForce 840M graphics card. All experiments were run on the GPUs when possible. Experiment programs were written in Python 3.6 and heavily utilized the following libraries: Ten- sorFlow 2.1.0, gym 0.15.4, mujoco-py 1.0. Code implementing the ideas detailed in this work can be found at https://github.comm/
6 DISCUSSION
Visualization of the convergence speeds for all tests are shown in Figure In all environments, the selection agent outperformed the base agent and typically matched the shaped agent's performance closely. each environment, the selection agent was able to identify which shaping reward signal(s) to listen to and use those signals to reinforce correct behaviors. This can be particularly clearly seen in the Reacher and MountainCar environments. the shaping reward signals in each region leading the agent to the goal point. In MountainCar, the selector correctly identified the optimal shaping reward signal (Φ2) and ignored the poor shaping reward signal (.1). The base agent was unable to find the solutions either of these problems and
Reacher, the selection agent correctly identified

---

Under review as a conference paper at ICLR 2021
CartPole
MountainCar
200
80
Averaged Reward
150
60
40
100
20
50
0
-20
base
500
1000
1500
2000
50
100
150
200
shaped
Pendulum
Reacher
-25
select
-250
-50
Averaged Reward
-500
-75
-750
-1000
100
1250
-25
-1500
-150
-1750
150
2500
5000
7500
10000
Episodes
Episodes
Figure 1: Performance results of the selection method against base and shaped agents
got stuck in local optimum of not moving all. The CartPole environment shows that selection is able to maintain optimal solutions despite perturbations in learned behaviors. The selection agent was able to quickly re-converge the solution each time fluctuation occurred. This also under- lines the dynamic nature of our selection method for automatic reward adaptation. In response to poor behaviors encountered during learning, the selector was able re-discover the correct shaping reward signals and re-learn the optimal solution. The Pendulum environment was quickly solved by all three agents. It acts as baseline check for the selection method and demonstrates that selection does not lead the agent to other solutions with suboptimal performance.
The Fu versions of CartPole and MountainCar used slightly different shaping reward signals. For CartPole, they used signals for cart position and pole angle that were similar to ours except for small value scaling differences. For MountainCar, they defined signals for cart position, velocity, and height. Each signal rewarded transitions leading higher corresponding values. It should be noted that the Fu version of MountainCar used different reward function that gave constant small negative reward for all time steps. However, the goal and optimal policy remain the same for this version of the problem. For the comparison experiments, we applied our selection method in the Fu versions of each environment and using their defined shaping reward signals. Effectiveness of our method and Fu's method in accelerating learning are shown in Figure 2. Convergence speeds are more explicitly details in Table 2. In this case, convergence defined the earliest episode in which the maximum averaged reward was observed.
Table 2: Episodes until convergence for each automatic reward adaptation method (value of verged reward in parentheses)
Selection
Fu
CartPole
602 (1000.0)
849 (995.31)
MountainCar
862 (-117.27)
964 (-111.62)

---

Under review as a conference paper at ICLR 2021
CartPole
MountainCar
1000
-200
800
Averaged Reward
400
600
comparison
select
400
800
200
-1000
0
200
400
600
800
1000
0
200
400
600
800
1000
Episades
Episodes
Figure 2: Comparisons in performance between our selection idea and the automatic reward adap- tation method presented in Fu et al. (2019)
When applied to these versions of the problems, the selector was still able to identify the correct shaping reward signals to listen to. The selection agent converged the correct policy both the CartPole and MountainCar problems, often faster than the Fu agent. In CartPole, selection arrived at a better solution than Fu earlier in learning. In MountainCar, selection learned a policy that per- formed slightly worse than that learned by Fu's method, but discovered it carlier. Later in learning, the selection agent maintained competitive performance despite lacking the extra time to compute optimal usage of available shaping reward signals. After reaching convergence, the selection agent's performance remained close to that of the Fu agent. While there was some fluctuation in the pol- icy, the selection method was consistently able to adjust the agent back towards the optimal policy. Overall, our version of automatic reward adaptation competed respectably even when learning with less infrastructure.
7 CONCLUSION
Automatic reward adaptation a newly considered component of automatic reward shaping that fo- cuses on accelerating reinforcement learning through the dynamic processing of multiple available shaping reward signals. Design was focused on adjusting the selector towards better approximat- ing changes in the optimal value function. By following the learned value function, the selected shaping reward function could more quickly provide information that pushed the agent towards the optimal solution. Furthermore, through the use of these value-based updates, the selector is free of excess external interaction and minimizes offline computation. These properties potentially promote our implementation of automatic reward adaptation step toward RL. that is more applicable to realistic problems.
There are many paths forward from this point. First and foremost the other component auto- matic reward shaping: automatic learning of reward signals themselves. While there has been work investigating how to approach this idea, much research remains to be done regarding how these signals and should be characterized, how they should be bounded (if all), and how we generalize the learning procedures to different domains, just to name a few paths. addition, there many more ways of selection (and combination) that can be explored that have not been analyzed in this work. We intend to continue pursuing automatic reward shaping ideas in future work.
REFERENCES
John Asmuth, Michael L. Littman, and Robert Zinkov. Potential- based shaping in model-based reinforcement learning. Proceedings of the National Conference on Artificial Intelligence, 2: 604-609, 2008.
8

---

Under review as a conference paper at ICLR 2021
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. pp. 1-4, 2016. URL httt:://axii..ttt///////t/ owg. 1606 01540
Tim Brys, Anna Harutyunyan, Peter Vranex, Matthew E. Taylor, Daniel Kudenko, and Ann Nowe. Multi-objectivization of reinforcement learning problems by reward shaping. Proceedings of the International Joint Conference on Neural Networks, pp. 2315-2322, 2014. doi: 10.1109/IICN..
2014.6889732.
Tim Brys, Anna Harutyunyan, Peter Vrancx, Ann Nowé, and Matthew E. Taylor. Multi- objectivization and ensembles of shapings reinforcement learning. Neurocomputing, 263:48- 59, 2017. ISSN 18728286. doi: wwwpmm...ecomm.......2..... 10.1016 (tw.o//...ecco... 2017 02.....2.96
http://://dx dei. org/
Byron De Villiers and Deon Sabatta. Hindsight reward shaping in deep reinforcement learning. 2020 International SAUPEC/Robbeeci/PPRSSS Conference, SAUPPCCRRbbecc//RRAA 2020, 2020.
doi: 10.1 AAW.SAAAAARRR.C.PPP.AA44........
Zhao Yang Fu, De Chuan Zhan, Xin Chun Li, and Yi Xing Lu. Automatic successive reinforce- ment learning with multiple auxiliary rewards. IJCAI International Joint Conference on Artificial Intelligence, Augussug: 2342, 2019. ISSN 10450823. doi: 00.24963/jccai22219/344
Marek Grzeš and Daniel Kudenko. Online learning of shaping rewards in reinforcement learning-
Neural Networks, 23(4):541-550, 2010. ISSN 08936080. doi:
URL http:t:::.xx..i..ooooo /tt.////////////////../.o/.././/
Anna Harutyunyan, Sam Devlin, Peter Vranex, and Ann Nowé. Expressing arbitrary reward func- tions as potential-based advice. Proceedings of the National Conference on Artificial Intelligence, 4:2652-2658, 2015.
Ben J.A. Kröse. Learning from delayed rewards. Robotics and Autonomous Systems, 15(4):233-235, 1995. ISSN 09218890. doi: 10.
Adam Laud and Gerald Delong. Reinforcement Learning and Shaping: Encouraging Intended Behaviors. Icml, pp. 355-362, 2002. URL http:// /tcplp///bp..../ uni.uniiii uni-tries de/db/cont
1cml http:/ccm220....0 html LaudD02
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. 4th Inter- national Conference on Learning Representations, ICLR 2016 Conference Track Proceedings, 2016.
Maryam Marashi, Alireza Khalilian, and Mohammad Ebrahim Shiri. Automatic reward shaping in Reinforcement I Learning using graph analysis. 2012 2nd International eConference on Com- doi: 10. 2011
puter and Knowledge Engineering, ICCKE 2012. pp. 111-116, 2012. 6395362.
Ofir Marom and Benjamin Rosman. Belief reward shaping in reinforcement learning. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 3762-3769, 2018.
Bhaskara Marthi. Automatic shaping and decomposition of reward functions. ACM International
Conference Proceeding Series, 227:601 -008000166666000000 2007. doi: 10.1145/1273496.12777 1273572.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 -333, feb 2015. ISSN 0028-0836 doi: 10. (038/nature142336 htR.http://ss> doi /tgg 1038/ manarree 4236http: www. nature. com/ http:///aaatteeeee/aaaeeee//////
A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. ICMI., 99:278-287, 1999.

---

Under review as a conference paper at ICLR 2021
Jette Randlóv and P Alstrøm. Learning to Drive Bicycle using Reinforcement Learning and Shap- ing. Proceedings of the International Conference on Machine Learning (ICML), pp. 463 463-471, 1998.
Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013. ISSN 10769757. doi: 10. 1...663/jiii.987.
Jiong Song and Zhao Jin. Implementing autonomous shaping by critical states. Proceedings 2011 6th IEEE Joint International Information Technology and Artificial Intelligence Conference, ITAIC 2011, 1(2010):26--79, 2011. doi: 10. 109/ITAIC.201 .6030203.
Richard Sutton. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 2000.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Tomasz Tajmajer. Modular multi-objective deep reinforcement learning with decision values. Pro- ceedings of the 2018 Federated Conference on Computer Science and Information Systems, Fed- CSIS 2018, 15:85-93, 2018. ISSN 2300-5963. doi: 10. 15439/20188F211.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. IEEE International Conference Intelligent Robots and Systems, pp. 5026-5033, 2012. ISSN 21530858. doi: 10. 109/000S. 201.20022226661
Harm Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang. Hybrid reward architecture for reinforcement learning. Advances in Neural Information Processing Systems, 2017-Deccem(ume):53999 -4039, 2017. ISSN 10495258.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances Neural Information Processing Systems, D008-Deeem(NeurIS):44444 4654, 2018. ISSN 10495258.
A
APPENDIX
Table 3: Performance Hyperparameters for each Environment
CartPole
Pendulum
MountainCar
Reacher
Critic/DQN Learning Rate
0.001
0.002
0.002
0.002
Actor Learning Rate
N/A
0.001
0.001
0.001
Selector Learning Rate
0.002
0.01
0.003
0.003
Target Network Update Rate
N/A
0.005
0.005
0.005
Discount Factor
0.95
0.99
0.99
0.95
Exploration Factor
0.9
0.2
0.9
0.2
Exploration Factor Decay Rate
0.95
N/A
N/A
N/A
Batch Size
64
64
64
64
Replay Memory Size
50000
50000
50000
50000
Optimizer
Adam
Adam
Adam
Adam
DDPG used the Constein-Uhlenbon process for exploration noise generation. Thus, the Exploration Factor was actually the standard deviation for the process instead of the standard epsilon factor used in, for example, DQN, For the same reason, the Exploration Factor Decay Rate for DDPG was listed N/A.
10


Reviews:
[{"review_id": "rZogOl43Zd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present an exploration of learning reward signals and how various reward signals can be adaptively used to provide a single reward feedback channel for a reinforcement learning process. First, they provide a solid introduction and related work research on multi-objective reinforcement learning. I would urge the authors not to present statements about work that they will continue to work on or work towards in the future. These forward-looking statements do not serve the purpose of the current work. \n\nA great deal of real estate in this paper is committed to background information on related work reinforcement learning and reward shaping which does not serve this paper. Indeed it is not until section 4.2 Methods: Selection when the author's finally present details on new methods and experimental design.\n\nThe results and figures presented are unclear and I would urge the authors to reconsider the information that they would present in terms of mean and measure of central tendency. It is hard to compare the significance of their results against their baseline methods. \n\nThis paper feels incomplete to me, and I would like to see this research continued toward a more full submission. Finally, I do not feel as though this paper is suitable for the general audience at ICLR. Perhaps there would be audiences more interested in the specific application and innovations in reward shaping at other workshops.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jKKdFnvRcFd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of review:\nIntruiguing idea, but the core of it is not properly exposed, and the empirical methodology has some flaws.\n\nDescription:\nThis paper seeks to leverage the advantage conferred by reward shaping signals in accelerating RL learning, by adaptively selecting from give shaping signals.\n\nThe main novelty of the work is in the use of a selector to adaptively choose which reward shaping signal to use.\n\nStrengths:\n-\tReward shaping is a proven method to accelerate learning. It is worth exploring more flexible and adaptive mechanisms for reward shaping.\n-\tThe proposed method is simple to implement and understand, and presumably to monitor during training.\n\nWeaknesses:\n-\tThe question of how the shaping signals are selected / designed is not addressed directly.  When I started reading the paper, I expected this to come from transfer (from similar domains), but in the experiments it seems to be hand-designed.\n-\tThe specific mechanism to adjust the selector is obscure.  Parsing through algorithm 1, what is the index “i”?   In addition to a selector, do you assume (line 1 of Alg.1) that you also receive a finite set of shaping signals?  I have a rough idea, but it is not clear from the paper.\n-\tWhat are the properties of this selection mechanism, in terms of convergence of the algorithm?  Does it converge? And under what conditions on the selector would the solution converge to Q()?\n-\tThe empirical results as presented do not meet the standards for good scientific reporting.  There is no standard error on any of the results.  The number of seeds is not reported.  Therefore, I do not have confidence in the results.  Furthermore it’s not clear in Fig.1 that the selected shaping signal is any better than the shaped reward.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Rrv-voHeO1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to select the best reward shaping potential signal out of multiple available shaping potentials. The main idea seems to be to select the shaping signal that minimizes the inverse of the difference of potentials between the next state and the current state. The experiments show the proposed approach works better than other baselines. \n\n- How is the proposed approach different from selecting the potential function that locally offers the highest increase in potential? \nThe proposed augmented distance that decides the selector is = (TD-error - potential-difference)/potential-difference = TD-error/potential-difference. Since we are sampling from experience replay, the s' is fixed which means TD-error is fixed across all potential functions. So the only quantity that changes is potential-difference_i for i in all potential functions. Is it not the same as picking the shaping reward that is the greatest? \nWhat if one of the potential signals is not conducive to learning the original Q. Would that not slow down learning in this case? \n\n- I believe the main weakness of the paper is the experiment sections. The main baseline that the paper uses is a linear combination of multiple potential functions. It would have been much preferable to see the performance of various potential functions separately (not as a linear combination). That would have shown how if the proposed approach is able to select the best potential function out of the available potential functions. \nTable 1 does not say how the proposed shaping functions are potential functions?\nEach experiment could add one more potential function that is negative of the best potential function. This way the effect of the proposed approach could be highlighted, showing how quickly/slowly the proposed approach was able to ignore the negative of good potential function since its not really helping with the learning. \nThe experiments currently are missing confidence intervals / std. error bars. \n\n- The paper is quite unclear. It is not clear what the paper means by optimal value difference, the use of phi is unclear. In the project definition, it is called shaping reward signal. It is only later that the reader realizes that phi is a potential function. \nSo the paper assumes that various potential functions are available as shaping advice. Potential functions are generally defined over the whole state-action pairs. How is the proposed approach applicable when the state-action space is large? What if it is not possible to define an exact potential function over the large state-action space.\nThe paper provides a literature review of multi-objective RL but it is not clear how it is connected to MORL. At certain point in the paper it seemed that the problem of selecting among various shaping potentials will be cast as a MORL but that does not seem to be the case. \nSome minor notation mistakes: In Background: R: SxAxS . I believe the paper meant to use \\times  and not $x$.\nThe variable could be introduced before it is used in equation 5. Only later in the pseudocode, the reader realizes what c is. \nThe above mistakes make it quite difficult to understand the complete picture presented by the paper. \n\n\n\n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "427_VNtsblH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe idea of potential-based reward shaping (PBRS) is to improve the performance of learning agents by incorporating additional domain knowledge into their reward function (making it dense), while maintaining the same asymptotic convergence guarantees. However, one aspect of PBRS, that is introduced in this work, is how to select among multiple shaping signals in an adaptive manner that is also dependent on the current context, in a way that can overall improve the performance of the agent. A method is proposed  here that learns to select among two or more shaping functions based on experience, represented as the difference between the TD error (or surprise) and the shaped reward signal, the conjecture being that signals that are closer to the TD error can lead to the best improvement in the agent's performance. The signal with the smallest difference is used as a self-supervised label to train a classifier that selects the signal. Experiments show that the method is at par with other similar approaches while being simpler to implement. \n\nPros:\n- the problem setting is one of high practical significance, since we may often be presented with various rewards but not know how to use them, and an adaptive approach for selecting between them on the basis of data seems logical\n- the approach seems quite versatile and easy to implement (if my understanding of the workflow is correct), and appears to be compatible with most standard RL algorithms that use a notion of Q-values\n- in principle, as the selector is running entirely online, could be applied in non-stationary RL setting (more comments below). an interesting extension could indeed be had in future work as the authors suggest.\n\nConceptual issues:\n- my main concern with the paper is the claim that the proposed update (6) when used in the context (5) maintains policy invariance. More specifically, the softmax probability over signals, c, depends on the current action (and in fact the history of data) through (6). However, this dependency is not shown in (5). While existing work validates the action-dependence of reward shaping signals, it requires that the potential function be applied in a time consistent and look-ahead manner, and requires the use of the biased-greedy policy (this being because action dependence shifts the Q(s,a) values non-trivially by phi(s, a) and requires it to be corrected). I do not immediately see how the action dependence is handled in a time consistent manner as predicated in the previous work (Wiewiora, 2003). I do admit that in practice, these issues can often be difficult to deal with, particularly with the use of experience replay and an episodic setting. It would be highly beneficial to elaborate on how you were able to address these theoretical issues to maintain policy guarantees? Even if the c is only dependent on state, I believe it should be applied in a time consistent manner to avoid instability, by caching the c from the last iteration and applying it to each phi individually (this would be the dynamic reward shaping work of (Devlin, 2012)). \n- the link between the importance of the shaping signal and the difference in Q values as defined by the TD error (6) is not well discussed. One of the fundamental drawbacks of this approach is that it could be scale sensitive with respect to phi; for instance, let's say that the correct phi_i were scaled by 1000, then why would (6) be able to select the right signal in this case? it seems the TD error would remain constant but the right term in the numerator would increase, making the error larger and forcing it towards the maximal value it could take. \n- the authors suggest that the method should work when the signals provided are adversarial in nature, but the experimental setup does not include adversarial examples, as all the potentials seem useful in different contexts. for example in cartpole, phi1 could be useful when the pole is tilting, while phi2 could be useful when the cart is about to leave the frame. a missing detail is how robust the method is to adversarial signals. Since the TD error is quite noisy, how to avoid the situation in which the signal may be similar to the TD error but the signal is adversarial or not useful? \n- Reinforcing the previous point, can some further intuitive (or even better, theoretical) derivation be provided from first principle to validate why (6) is the right way to select signals? This would strengthen the technical aspects of the paper that could be quite improved at this stage.\n- while the authors point out that the approach of selecting signals based on the primary value function is new, a search reveals some existing work (e.g. Gimelfarb et al., 2018) that used values to select signals, which appears superficially similar to this work. How would the proposed method compare to a non-context specific approach such as this that seems to also use values when comparing signals?  \n\nExperimental details:\n- the authors mention the comparison to a shaped agent. is this the same as using the learned c as a weighting over signals, rather than picking the best signal from the selector output? if so, the results for the weighting appear more stable than the selection of a single signal, so why do the authors not use this weighted combination rather than picking one as the contribution? the selection of a single signal seems less robust to learning errors than a weighting, since it may be brittle and cause instability when the single best expert (chosen by arg max) must suddenly switch. \n- the experiments do not reveal the true benefit of selecting different signals in a state-dependent way, which seems to be the main contribution. for example, how do the results compare to a single signal? also, for the lower dimensional tasks (cart pole, mountain car), it would be really helpful and interesting to illustrate which signal is chosen and how this selection process is changing over time and situation. Given that the signals are chosen so that one should be clearly optimal in different situations, this will validate the claim that adapting the signal to different situations is useful and working as intended.\n- the comparison to Fu et al, 201`9 is rather inconclusive and incomplete. the results on these problems do not indicate that the proposed method advances the state of the art. The authors do mention that the method is easier to implement and run. Does this mean that the expected or discount reward is not the right measure to use or does not reveal the full story? Is computation time, or memory use, a more suitable measure for comparison? If so, this could be one selling point that will strongly benefit the paper.\n- the proposed method seems to have some instability in some situations. while reporting the best case is useful, it does not reveal the full story. I think the paper could be made clearer by reporting the average or median performance with confidence interval bars, and discussing how learning stability is affected by these methods.\n\nOther details:\n- the authors use c twice, one for the underflow correction and one for the softmax\n\nOverall, I think this paper is quite interesting, well motivated and could be of significance to the broad RL community. However, I would like to see more discussion, and given this method is largely empirical, a more thorough analysis of where the method works and what metrics are most suitable to evaluate it.\n\nReferences:\n- Devlin, Sam Michael, and Daniel Kudenko. \"Dynamic potential-based reward shaping.\" Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems. IFAAMAS, 2012.\n- Gimelfarb, Michael, Scott Sanner, and Chi-Guhn Lee. \"Reinforcement learning with multiple experts: A bayesian model combination approach.\" Advances in Neural Information Processing Systems. 2018.\n- Wiewiora, Eric, Garrison W. Cottrell, and Charles Elkan. \"Principled methods for advising reinforcement learning agents.\" Proceedings of the 20th International Conference on Machine Learning (ICML-03). 2003.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to reward shaping in RL centred on the question of how to select between different shaping signals. As such this is an interesting research direction that could make important contributions in the area.
Generally the reviewers felt that the paper is too preliminary in its current form. There were several questions raised around problems with the technical formulation. It was also felt that the experiments could be more rigourous to fully validate the claims of the paper.
Decision: Reject