Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards
Authors: Rati Devidze
Parameswaran Kamalaruban
Adish Singla
Venue: NeurIPS 2022 Accept (2022)
Keywords: reward shaping
intrinsic rewards
reinforcement learning
sparse-reward environments

Abstract: We study the problem of reward shaping to accelerate the training process of a reinforcement learning agent. Existing works have considered a number of different reward shaping formulations; however, they either require external domain knowledge or fail in environments with extremely sparse rewards. In this paper, we propose a novel framework, Exploration-Guided Reward Shaping (ExploRS), that operates in a fully self-supervised manner and can accelerate an agent's learning even in sparse-reward environments. The key idea of ExploRS is to learn an intrinsic reward function in combination with exploration-based bonuses to maximize the agent's utility w.r.t. extrinsic rewards. We theoretically showcase the usefulness of our reward shaping framework in a special family of MDPs. Experimental results on several environments with sparse/noisy reward signals demonstrate the effectiveness of ExploRS.

Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards
Rati Devidze' Srevidze@mpi-sws.
Parameswaran Kamalaruban'
Adish Singla' adishs @ mpi-sws.org
kparameswaran Sattrranttuiing...u
Max Planck Institute for Software Systems (MPI-SWS), Suarbrucken, Germany
"The Alan Turing Institute, London, UK
Abstract
We study the problem of reward shaping to accelerate the training process of a reinforcement learning agent. Existing works have considered a number of different reward shaping formulations; however, they either require external domain knowledge fail in environments with extremely sparse rewards. In this paper, we propose a novel framework, Exploration- Guided Reward Shaping (EXPLORS), that operates in fully sulf-supe manner and can accelerate agent's learning even in sparse-reward environments. The key idea of EXPLORS is to learn intrinsic reward function in combination with asporattion-bsed bonuses to maximize the agent's 's utility w.r.t. extrinsic rewards. We theoretically showease the usefulness of our reward shaping framework in special family of MDPs. Experimental results on several environments with sparse/noisy reward signals demonstrate the effectiveness of EXPLORS.
Introduction
Training reinforcement learning (RL) agents in environments with extremely sparse distracting rewards is challenging. Existing works have studied several approaches to design informative rewards that speed up the agent's convergence [1-7]. One well-studied line of work is aatential-based reward shaping, where a potential function is specified by an expert or obtained via transfer learning techniques (see [3, 8-17]). Another popular approach is to learn rewards via Inverse-RL. using expert demonstrations [18]. Alternatively, one could also consider a manual specification of rewards, e.g., using distance- based metrics [19]. However, these reward design techniques typically rely high-quality domain knowledge and may fail in practice. In fact, the RL agents can easily exploit poorly designed rewards and get stuck in local optima. This naturally leads the fundamental question of how to do online reward shaping without relying on expert domain knowledge. More concretely, can we design informative rewards that will accelerate the agent's training process by leveraging experience gained online during the agent's 's training lifetime itself? [20-24]
To tackle this question, recent works [24-26] have explored fully self-supervised learning of paramet- ric intrinsic rewards that can improve the performance of RL. agents. In particular, these methods alternate between intrinsic reward parameter learning and the agent's policy optimization w.r.t. the learned reward. For instance, Learning Intrinsic Rewards for Policy Gradient (LIRPG) technique [25] updates the intrinsic reward parameters maximize the extrinsic rewards received by the policy from the environment. Self-supervised Online Reward Shaping (SORS) technique [26] infers an reward inference algorithm, TREX [27]. However, these fully self-supervised reward shaping techniques might fail to produce meaningful agent behavior in environments with extremely sparse rewards (called hard-exploration domains) as they lack an
intrinsic reward using classification-ball
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

explicit explorative component. Intuitively. these techniques will not be able to make updates to parameters of their intrinsic reward functions, without receiving a non zero extrinsic reward signal.
In a parallel line of work, several techniques have been proposed to specifically tackle the challenges of extreme sparsity and exploration. One such line of work is to add more stochasticity in the agent's behavior (e.g., [28-30]); however such techniques typically succeed in tasks with already well-shaped rewards. Another important line of work, relevant to our proposed framework, is bonus- driven exploration techniques for tackling hard-exploration domains these techniques augment the extrinsic rewards with additional intrinsic bonus signals to encourage extra exploration [31]. popular category of intrinsic bonuses is count-based bonuses that encourage RL agents to experience infrequently visited states [32-34]. Another category of intrinsic bonuses is providing rewards for improving the agent's 's knowledge about the environment [35-40]. However, simply relying on these bonus-driven signals can mislead the agent towards sub-optimal or bad behaviors - for instance, in hoisy-distraccive domains such as the "noisy TV" problem [41], unpredictable random or noisy outputs would attract the agent's attention forever.
An important research question that we seek to address is: How can we design an online intrinsic reward function, without any domain knowledge, that can speed up the agen''s 's learning process even in environments with extremely sparse rewards and noisy distractions? To this end, we propose a novel framework, Exploration- Guided Reward Shaping (EXPLORS). that learns an intrinsic reward function in combination with exploration-baut bonuses to maximize the agent's 's utility. EXPLORS operates in fully self-supervised manner, and alternates between reward learning and policy optimization. Our main results and contributions are:
I. We propose a novel reward shaping framework, EXPLORS, that operates in a fully self- supervised manner and can accelcrate an agent's learning even in sparse-reward environments. (Section 3.1).
II. We derive intuitive meta-gradients for updating the intrinsic reward component of EXPLORS that enables our framework to be broadly applicable to any RL agent and not only policy- gradient based agents (Sections 3.2 and 3.3).
III. We theoretically showcase the usefulness of our reward shaping framework in accelerating an agent's 's learning in a special family of chain environments (Section 3.4).
IV. We empirically demonstrate the effecttivnness of EXPLORS on several environments with sparse and noisy reward signals (Section 4).
2 Problem Setup
In Section 2.1, we present a general framework of online reward shaping technique for RL agents. In Section 2.2, we discuss the limitations of existing reward shaping techniques.
2.1 General Framework of Online Reward Shaping
Preliminaries. An environment is defined as a Markov Decision Process (MDP) M == (S,A,,, Po R), where the state and action spaces are denoted by S and A respectively. T S S x A + [0, 1] captures the state transition dynamics, i.e., T(s' | 8,a) denotes the probability of landing in state s' by taking action a from state s. 7 is the discounting factor, and Po is the initial state distribution. The reward function is given by R : S x A + [-Rmax, Rmaa]] for some Rmax > O. We denote the true underlying extrinsic reward function by It and the designed reward function by R. We denote a stochastic policy π S + A (A) as mapping from state to a probability distribution over actions, and a deterministic policy π S - as a mapping from a state to an action. For any trajectory { = {(St, (z)) (t)) t we define its cumulative return w.r.t.
reward function R as [E, R) == > 7² R(sr, a.). Then, the expected cumulative return (value) of policy x w.r.t. R is defined as [[r, R) : E [J, R) Po. T, x). where so Po(-), a, ~ T(-1s;), and St+1 ~ (-|,,, a.). The learner seeks to find a policy that has maximum value w.r.t. the extrinsic reward function R, i.e., max, [TT, R).
GGithub repo: https://giiuub..om/mmhhht tewcceeeeemcciiiiiieeiiiinnnnn teaching http:/eeeriii222eeeeeeeerrtiaeie guided guided-reward -haaing....

---

Online reward shaping. A general framework of online reward shaping for RL agents is given in Algorithm 1. A natural objective here is to design informative rewards R2 at each round k so that the resulting final policy тк performs better (i.c., has high value w.r.t. R) compared to the corresponding policy obtained via the standard training with R& R. Note that we consider single lifetime Algorithm Online Reward Shaping
Input: Extrinsic reward R. and RL algorithm
2: Initialization: To. Ro
3: for k = 1, 2....... K do
4: update policy my <- T1 Rk-1)
5: update reward Rş using RR-1 and TTk
6: Output: *
training setting for an RL agent on a single task, i.e., there is no resetting of the policy between rounds.
2.2 Existing Techniques and Issues
popular technique for reward shaping is potential-based reward shaping (PBRS) which guarantees that any optimal policy induced by the designed reward function is also optimal under the extrinsic reward function [3]. However, for PBRS to be effective in accelerating the training process an RL agent, we need to have access to good potential functions based on expert domain knowledge [42]. The focus of our work is on designing fully self-supervised reward shaping techniques. Below, provide discussion of existing techniques that do not require any expert guidance domain knowledge, and also discuss their limitations.
Reward shaping based on exploration bonuses. In the bonus-drivens exploration framework [32- 34], a count-based intrinsic bonus B& (s) is given to the agent to encourage exploration. The bonus (x) measures the "novelty" of state given the history of all transitions up to round k. The authors in [34] extend the classic exploration methods with count-based intrinsic bonuses [43-46] to sigh-iimensional, continuous state spaces. However, these "exploration-only reward shaping techniques do not appropriately combine the successful extrinsic reward signals received from the environment. When there are distractive zones in the state space, these methods will keep on exploring the state space even after obtaining extrinsic reward signals.
Fully self-supervised reward shaping: LIRPG [25]. Learning Intrinsic Rewards for Pol- icy Gradient (LIRPG) technique [25] considers parametric reward function of the form RLIRPG a) = R(s, a) + [s, a). and learns the parameter & of the intrinsic reward function Rş in fully self-supervised manner. LIRPG alternates between learning the intrinsic reward parameter and the agent's policy optimization w.r.t. the learned reward RLIRPG At round k, for fixed Ti. LIRPG updates the parameter ΦR to du by considering the effect such a change would have on the expected cumulative return (w.r.t. Rt) of the learner through the change in the policy TT. i.e., update ő using the gradient [LJ[(L(TT. RIIRPG),, R)) d In order to develop update rule for Φ. LIRPG considers policy gradient style learning algorithm L with parametric policies { : E Rdo }. More concretely, for a parameter 6% at round s.t. TTg == Te.,, the learner's policy update depends on ø as L(TT, LIIPP) = To(a), where (Ø)) = 02
VoJ[ RIJRPG) Based on this learner update, the LIRPG update for the intrinsic reward parameters, round k. is based on the following meta-gradients: Φb Φ + "
dx where 7/ is the learning rate. We note that the LIRPG technique could fail in environments with extremely sparse rewards as the agent may not receive non-zero extrinsic reward signal needed to update the parameter
(V,(())) (e(o) R)),
Moreover, the LIRPG technique is applicable only to policy-gras -radiiet based RL agents.
Fully self-supervised reward shaping: SORS [26]. Self-supervised Online Reward Shaping (SORS) technique [26] considers reward function of the form ÁSORS (s, a) infers the parameter & using a classification-bart unlike T-REX that requires rankings over the trajectories as input, SORS uses the extrinsic reward R as self-supervised learning signal to rank the trajectories generated by the agent during training. By de-
(s,s. and reward inference algorithm, T-REX [27]. However,
3

---

sign, SORS only enforces the relative pairwise ordering over the trajectories w.r.t R when training R and ignores the scale of the returns associated with trajectories w.r.t. R. This makes training a policy challenging when the environment has noisy or distractive reward signals. Further, similar to LIRPG, the SORS technique could fail in environments with extremely sparse rewards as the agent may not obtain any trajectories with non-zero extrinsic reward signal needed to update the parameter Ø.
In this paper, we seek to develop an online reward shaping technique that can accelerate the agent's training process in environments with extremely sparse and distractive rewards, without any expert domain knowledge. As discussed above, techniques that rely only on intrinsic bonuses [32-34] could mislead the agent towards sub-optimal behaviors in noisy-distractive domains. Similarly, the fully self-supervised reward shaping techniques (LIRPG and SORS) might be ineffective in environments with extremely sparse rewards. We overcome these limitations by designing a novel reward shaping framework that appropriately balances exploration (via an intrinsic bonus component) and exploitation (via an intrinsic reward component) of extrinsic reward signals.
3 Exploration- Reward Shaping
In Sections 3.1, and 3.3, we propose an celloration-guiderr reward shaping framework, EXPLORS, accelerate an RI. agent's 's training process. In Section 3.4, we theoretically showcase the usefulness of our framework in a chain environment.
3.1 Our Reward Formulation
We consider the following parametric reward function for EXPLORS (see Algorithm 1):
REXPLORS(s, = R(s,a)+ + SSSLERS a) + BEXPLOB (s),
(I)
where @ E IR and w E Rdd Here, RSELERS corresponds to the intrinsic rewards in self-supervieed
reward shaping techniques, and BEXMLBB corresponds to the intrinsic bonuses in exploration-one reward shaping techniques. At round is of Algorithm 1, RE (s, is designed with parameters (dk-1, Wk-1). Then, given updated policy TR, update the parameters (Øx-1, wk-1) to (øx, wx).
Notation. For the remainder of this section, we drop the superscripts (EXPLORS, SELFRS, and EXPLOB) when referring to the reward functions in Eq. (1). In the subscript of the expectations E, let (als) mean a ~ x--ss), μrt (s, a) mean s ~ d=, a ~ x(11s), and (s) mean s d". Further, we use shorthand notation ur.a and 3 to refer μ Th (s, a) and n² (s), respectively.
Intrinsic reward Rg. We model the intrinsic reward RS using any parameterized function. At round k for fixed Te and Wk-1, we update the parameter Øx-1 to dg by considering the effect such a change would have on the the expected cumulative return w.r.t. Rt through the change in the policy Te [24, 25]. In particular,
update é using the gradient [VTJ( R), R),R) R)] where
R(s,a) a) = Ti(s,a) a) + (s a) Bwz_((s), (s). However, when considering with neural policies, it is challenging directly analyze the impact of ó in the policy тg. Since goal is to design reward shaping technique that is applicable to any RL agent, we consider a simple surrogate learning algorithm L for our analysis. In particular, we consider L with parametric policies (TO 0 E R&do that does single-step vanilla policy gradient update with Q-values computed using h-depth planning. We map the policy Ta to parameter 0₂ E Rdo and define:
[(Ok. Rt) 0% a [VO[(TO, R)] 0x + Eyt. [ log log [o([s)]
TeN (s.00)
where a is the learning rate and Q R.
(s) = E Dn 7d R 00) = 5,00 a, n.z is the
hedepth Q-value w.r.t. R.. Then, we update @ using the following bi-level optimization:
arg max
J Tt)
(PI.U)
subject to 0()) + L(O,. R)
(PL.L)

---

where R(s.a)= := R(s, a) + (s) + (s). In the above bi-level formulation, I with h-depth planning for small values of essentially requires designing more informative intrinsic rewards to benefit the agent's 8 training process [24].
Intrinsic bonus B..- Given state abstraction v : S X (with |. = du we maintain the visitation count of the abstracted states in w. i.c., w[x] corresponds to the visitation counts of the states {s S vi(s) = 2². This allows us implicitly maintain pseudo-counts Nw of visiting states E S. In particular, we set Nw = Broo
2
+ w[(b(s)] for some Bmax À > O. Then, we define the We update wo based on the rollouts in round [32-34].
intrinsic bonus as follows: Bw (s) (N)
3.2 Derivation of Gradient Updates for Rs
In this subsection, we first obtain high-level meta-gradient updates for Rs similar to LIRPG [25]. Then, we derive intuitive meta-gradient updates that would allow EXPLORS to be compatible with any RL agent.
High-level gradient updates for Rg. We solve the bi-level optimization problem (PI.U)(PI.L) of the intrinsic reward component an iterative manner using the gradient updates that we derive below. At round k; for fixed Ty and WR-1- we update the parameter Φ to Φr as follows:
OR Φ + 17 V 18) O2
Φk-1 +
V Val(o)- Po(o) RD)
Φk-1 + VoM())) [oJ[ 10] 02
(T)
(2)
where 17 is the learning rate, the equality in (a) is due to chain rule, and the approximation in (6) made by assuming smoothness condition of [VoJ R)] R)) [VeJ[ R) S
e. (10()x1)) Bell2 for some e > 0. By using the meta- meta-gradient derivations in [47-49], we write the R. (s,a)) [s, [v log no (a|s) 0. where R(s a) = R(s,a) + (R a) + Bwn. (8). By using the policy gradient theorem [50], we write the term ② as follows: V (Tp, R)] 0. Ent. [VologTo(c|s)) 2 (s, a) The above gradient update of dx, involving the terms ( and (2) resembles the LIRPG [25] update. However, both the terms ① and require computing the gradient of the policy, i.e., V log To (als))) This requirement makes the above update applicable only for policy-gradient based agents. Below, we derive intuitive simplifications of the above terms, I and 2. that would enable our technique to be applicable to any RL agent, and not only volicy-gradiens based agents.
term ! as follows: [V)0(0) a EEpt. [Vo
Intuitive gradient updates for Rş. In order to obtain intuitive forms of the terms ! and ②, consider further simplifications to the surrogate learning algorithm T, introduced in Section 3.1. In particular, for our analysis and derivation, we let use tabular representation E R|SH-A and
vss E S,a E A. We define softmax policy given by Toy (a(s)) = exp (0(s,a))
R.A a) ==
exp exp ((s,b)) R.A as) v (s) and AT (s,a) = 0 Th (s, R.
vTt (s). Based on this, the following R
proposition provides intuitive gradient updates for Rg--
Proposition 1. For the simplified surrogate learning algorithm L. with h-depth planning, the gradient term | (())) [oo R)), in Eq. (2) takes the following form: o.
(8, a B.py paaa (s,
[V R.
For the special case of h = 1, the gradient term further simplifies the following form:
a - EE uba A To (s, (o a) - Ex, (6) (Ro Φ2-1
5

---

Compared to Eq. (2), the intuitive gradient update term in the above proposition does not require computing the policy gradient Va log (als). This allows us develop an update rule for intrinsic reward parameter that is applicable to any RI. agent. In particular, given the current policy (possibly without any differentiable parameterization), we simplify Eq. (2) and propose the following gradient update rule for parameter Φ;
0k Φk-1 + r/' Eu [ A (s.
[ (R (b.)))))) (R) b]])
(3)
where 1/' = 7) a. Note that the above gradient update only requires black-box access to the policy s in the form of trajectory rollouts as in the SORS technique [26].
3.3 Empirical Updates and Practical Aspects
In this subsection, we present a concrete pseudocode for training an RL agent with EXPLORS reward shaping technique. Algorithm 2 provides a sketch of the overall training process, interleaving the agent's training with EXPLORS. The sketch presented in Algorithm is adapted from the training process proposed for the SORS technique [26]. Further, consider rollouts where each round corresponds to a single rollout, instead of environment steps, as in SORS. Below, we discuss the empirical updates for intrinsic reward and bonus components of Eq. (1).
Empirical updates for intrinsic reward Rg. We translate the final expectation- based update of Φx in Eq. (3) to its empirical counterpart using the rollout data DD collected by executing the current policy Ty (or recent policies) in the MDP M. At any round k, let D contain collection of trajectories " {<)) where ç² (sh, ag. si aj ................................ ... syg). For given trajectory ç and time index f, we denote a partial trajectory as E = (s), "........ sH). Based on this notation, we empirically update the parameter & as follows:
Σ Emldd(b)) (J(C), Rt) (st)) (V Th (o), ai)
Φk + Φ + mh
02-1"
(4)
where we absorb the normalization factors into Th ignore the term 10 (st), and set ox-1 = =[V> (Rolst, at) (ua()) [Ro(s 6x-1
V Rn (s), at)
Similar to LIRPG [25],
we also maintain a critic ViR,7-,()) approximate V (-) in Eq. (4). We update the parameters of
the critic, Φu-1 to Tx. using the same rollout data 'D and learning rate n. In Algorithm 2, hyper- parameters N,, and N₂ control the frequency of updates for the intrinsic reward Rg and policy म, respectively. For stability reasons, we update the policy more frequently compared to the intrinsic reward, i.e., N₂ < N,. We provide full implementation details in Section 4 and appendices.
Empirical updates for intrinsic bonus Bu, We update Bw based the history of all the states visited up to round k. Similar to #Exploration [34], we use the count-based intrinsic bonuses with a state abstraction v S + Xz.. We maintain the visitation count of the abstracted states in w. For each rollout go, we update the parameter of the intrinsic bonus follows:
wk [x]
w/ [xr +
Σ 1 (x)())) StECK
#}. Vo E X...
Similar to the existing count- exploration techniques [32-34], we use lookahead step when incorporating the bonus term (see line 5 in Algorithm 2). In our implementation, we update the intrinsic bonus at a more fine-graing ane-grained level, i.c., we update B. at each environment step within each round & directly, instead of waiting for the rollout to finish. However, for clear presentation in Algorithm 2, we write the B. update at the level of round k not at the level of environment step t. We provide full implementation details in Section 4 and appendices.
3.4 Theoretical Analysis
In this subsection, we theoretically showcase the usefulness of our exploration-uuides reward shaping framework in accelerating an agent's learning in chain environment with extremely-spars rewards and distractive zones in the state space. Our analysis considers a stylized learning setting with simplified versions of different reward shaping techniques.

---

Algorithm 2 RL Training with EXPLORS
1: Inputs and hyperparameters: RL algorithm L; first-in-firt-oou buffer D with size Dmax: abstraction vx; learning rates ) (ng J: bonus parameters Bmax A; update rates Nr, Na
2: Initialization: Initialize the parameters for intrinsic reward and its critic (øo, Φo), parameters for intrinsic bonus wo, and the policy To
3: for k= 1, 2....... K do
// pooiiy update
4:
if 1%%N, 0 then
5:
Define reward RR-1(s,a,$)) R(s,a) Roz (s, B
6:
Obtain updated policy mh f- ((TT-11 Rx-1) using the latest rollouts in
7:
else
8:
Keep previous policy TH f Tk-1
// data collection
9:
Rollout the policy my in the MDP to obtain a trajectory ş = (st ab; st ....... .
10:
Store gl the buffer D.adá(("): if the buffer ' is full, remove the oldest trajectory intrinsic reward update if R%N,, = 0 then
11:
12:
Obtain updated reward parameter Φ from Φk-1 1 as in Eq. (4) using D and learning rate mi
13:
Obtain updated critic parameter Φy from @ using D and learning rate ni
14:
else
15:
Keep previous parameters Φa <- Φk-1 and Φx +- Φ
// intrinsic bonus update
16:
Update wi as in Eq. (5) using the states visited in the trajectory şh
17:
Define bonus Bug (s) /Nu, (8)
where Nw. (8) = Ban wx [x(((bs))
18:
Output: Policy тK
Chain environment. We consider chain environment M (S,A,T,Po. Y, R) of length M1 n2 1. Let the state space be S {-r 1,, F1,..., J. and the action space be A = -). We always start in the state To. i.e., the initial state distribution is
Po(oo) 1. The transition dynamics is deterministic and given as follows: 1(H++1/4,
for n22 i n4 1001-1,,77 [I +- = for -(12- 1) < S n1, T'(teriinal [nn, 1, mg- ( = The reward function is defined as follows: R(2;, +) = 0 for -02 < < m T[(=,; -) = 1, and TE(x; +-) = 0 for - n22 S i m4. We consider an infinite horizon setting with discounted returns, i.c., H
and f(terminalp [tttemmnallrr.n))++)))))
and 7
Learning algorithm and reward shaping techniques. For our theoretical analysis, we consider a stylized learning setting with a TD-style RL algorithm L and simplified versions of different reward shaping techniques; details are provided in appendices. We analyze the total number time steps required for L to learn optimal policy in the chain environment under four different settings: (i) Case L(SELFRS = 0, Exp LOB = 0) is default setting without any shaping; (ii) Case L(SELFRS EXPLCB 1) uses only the intrinsic bonuses; (iii) Case L(SELFRS
1, ExPE.O 0) uses only the intrinsic rewards; (iv) Case T(S LFRS = EXPT GR = 1) combines intrinsic bonuses with intrinsic rewards. The following theorem compares these four settings and showcases the usefulness of our framework, i.c., Case L(SELFRS
ExP LOB 1) proof is provided in appendices.
Theorem 1. Consider the chain environment M and the algorithm L. defined above. Let cost( ( Exp LoB)) denote the total number time steps required for L(SELFRS, Exp LOB) learn an optimal policy in M. Then, we have the following (expected) costs for the four settings:
(i) E [coss((SSELERRS 0, Exp LOB = 0)) 2 2 1:
(ii cosi(L(SELFRS = 0, EXPLOB 1)) mi (ni 112 - 2);
(ii)) [[cos{(L(SLEERS 1,ExPLOB 0)) > 21
(iv) cos((LSSELFRS = 11, 1,EXPLOB = 1)) S ni + R22 +2
The proof and additional details about the learning setting are provided in appendices.
7

---

4 Experimental Evaluation
In this section, we evaluate our reward shaping framework on three environments: CHAIN (Sec- tion 4.1), ROOM (Section 4.2), and LINEK (Section 4.3). CHAIN corresponds to a navigation task in a chain, adapted from the environment used for theoretical analysis in Section 3.4; this is canonical environment used for studying extremely sparse-reward settings [7]. ROOM corresponds to navigation task in a grid-world where the agent has to learn policy to quickly reach the goal location in one of four rooms, starting from an initial location. Even though this environment has small state/action space, it provides very rich and intuitive problem setting to validate different reward shaping techniques. In fact, variants of ROOM have been used extensively in the litera- ture [10, 11, 14, 17, 51-54]-the environment used in our experiments is adapted from [54]. LINEK corresponds to navigation task in a one- mieeddmensiooal space where the agent has to first pick the correct key and then reach the goal. The agent's 's location is represented as a point on line segment. This environment is inspired by variants of navigation tasks in the literature where an agent needs to perform subtasks [3, 54, 55e environment used in our experiments is adapted from [54]. We give an overview of main results here, and provide a more detailed description of the setup and additional implementation details in appendices.
4.1 Evaluation on CHAIN
CHAIN (Figure 1). We represent the chain environment of length ni + 122 + as an MDP with state-space S consisting of an initial location zo (shown as "blue-cirele",, 22 nodes to the right of zo. and m2 nodes to the left of zo. The rightmost node of the chain is the "goal" state (shown "green-star"). In the left part of the chain, there can be a "distractor" state (shown as "green-plus"). The agent can take two actions given by A := ("leftt, "right")
Figure 1: CHAIND / CCAIN*
action takes the agent to the intended neighboring node with probability of - -Paana) for Prand = 0.05. The agent receives a reward of Rmax = for the "right" action at the goal state, Rais for the "left" action at the distractor state, and 0 for all other state-action pairs. There is a discount factor 7 = 0.99 and the environment resets after a horizon of H = n2 steps. We consider two different variants of the chain environment: (i) CHAIN" with (n1 = 20, n2 40, Rais = 0); (ii) CHAIN+ with (n- = 20, n2- = 40, Rais = 0.01). We defer the full environment details to appendices.
Evaluation setup. We conduct our experiments with two different types of RL agents for CHAIN: tabular REINFORCE agent [7] and tabular Q-learning agent [7]. Algorithm 2 provides a sketch of the overall training process, and shows how agent's 's training interleaves with reward shaping techniques.
We compare the performance of the following reward shaping techniques: (i) NORIG == R is default baseline without any shaping: (ii) ASORS' == R
R RORSS is based on the SORS technique [26] (see
Section 2.2):² (iii) RLIRPG is obtained via adapting the LIRPG technique [25] to our training pipeline (see Algorithm 2, Sections 2.2 and 3.2)-note that PIRPG is not applicable to Q-learning agent;" (iv)
REXPLOB ;= R+ + BEXxPLB uses only the intrinsic bonuses; (v) REELERS = R + R uses only the or
intrinsic rewards; (vi) REXPLORS := R + R8 + EEPLLB combines intrinsic bonuses with intrinsic rewards. We provide full details about the implementation and hyperparameters in appendices.
During training, the agent receives rewards based on R and is evaluated based R. Figure 2 shows results for both the variants of CHAIN environment; the reported results are averaged over 20 runs and convergence plots show the mean performance with standard error bars. These results demonstrate the effectiveness of our exploration-guiee comparison to basclines (RORIG ASORS', RLIRPG" REXPLOB AESELERS) Next, we summarize some of our key findings. First, our results show that FEXPLORS outperforms the baselines in both CHAIND and CHAIN+ environments, irrespective of the RL agent (REINFORCE, and Q-learning). Second, the performance of REXPLORS is better than variants which only use either intrinsic bonuses or intrinsic
Results.
reward shaping framework (fEX.WORS), in
In our implementation. we use a variant of the SORS technique which also incorporates the extrinsic reward component R as done in all other techniques in our evaluation setup.
Throougoou the experimental evaluation, we refer to our implementation of the LIRPG technique as RUIRPG instead of ĀLIRPG our implementation of the LIRPG technique not based on computing meta eeta-gradients as in the original work [25]. Instead, we implemented RLIRPG variant of RSELRRS where we set h - do instead of 1 in A *é (s, a) (see Section 3.2). We provide additional implementation details in appendices. R.6
8

---

EXPLORS
SORS' LIRPT'
ExPLORS
SORS
EXPLORS
SORS
EXPLORS
SORS
SeLvRS ExPLOB
SELFRS
Oaue LIRPT
SELFRS
OXIG
ExpLaB
ExPLOR
ExeLoBi
Expected reward
15
Expected reward
Expected reward
10
Expected 10
LO
10
1.5
1.5 Episode (x100)
Episode (x10f)
Episode (x10')
Episode (x110)
(a) CHAIN", REINFORCE (b) CHAIN", REINFORCE (c) CHAIN", Q-lcarning
(d) CHAIN", Q-learning
Figure 2: Results for CHAIN environment. These plots show convergence in performance of the agent w.r.t. training episodes. (a, b) show results for REINFORCE agent on CHAIN" (i.e., CHAIN variant without any distractor state) and CIAIIN* (i.e., CIHAIN variant with a distractor state). (c, d) show results for Q-Jearning agent on CHAIN" and CHAIN". See Section 4.1 for details.
rewards, i.e., REXPLOB or RSELERS this demonstrates the utility of combining these two signals. Third, results in Figures 2b and 2d show that three reward shaping techniques (ASORS', R.IRPG RSELERS) could fail or lead to sub-optimal policies because of the presence of distractor states.
4.2 Evaluation on ROOM
ROOM (Figure 3). This environment is based on the work of [54]; however, we adapted it to have "distractor" state (shown as "green- plus") that provides a small reward. Similar to the two variants of CHAIN, we have two variants of this environment: (i) ROOM has Rais = 0 at the distractor state shown as "green-plus" (equivalently, there no distractor state); (ii) ROOM* has Rais- = 0.01 the distractor state. The environment-specific parameters (including Prand. Rmax. 7) kept same as in Section 4. We defer full details to appendices.
Figure 3: RоOM RROM*
Evaluation setup and results. Our evaluation setup for this environment is exactly same as that used for CHAIN environment (described in Section 4.1); here, consider only the tabular REINFORCE agent. In particular, all the hyperparameters (related to the REINFORCE agent, reward shaping techniques, and training process) the same in Section 4.1. Figures 5a and 5b show the agent's 's performance for environments ROOM and ROOM* (averaged over 20 runs). These results, along with results obtained in Figure 2, further demonstrate the effectiveness and robustness of REXPLORS across different environments comparison baselines.
4.3 Evaluation on LINEK
LINEK (Figure 4), This environment corresponds to navigation task in a one-dimensional space where the agent has to first pick the correct key and then reach the goal. The environment used in our experiments is based on the work of [54]; however, we adapted it have multiple keys (only one being correct) and "distractor" states that provide a small reward at goal locations even without the correct key. The environment comprises of the following main elements: (a) an agent whose current location (shown as "blue-cirele") is a point x in [0. 1]: (b) goal (shown as "green-star" is available in locations on the segment [0.9, 1]: (c) a set of k keys that are available in locations on the segment [0.0, 0.1]. (d) among & keys. only key is correct and the remaining A 1 keys are wrong (i.e.. irrelevant the goal). Moreover, we consider the agent with two different actions related to picking a key: (a) "pickCorrect" makes the agent collect the correct key required at the goal; (b) "pickWrong" makes the agent collect one of the 1 wrong keys, chosen at random. Similar to Sections 4.1 and 4.2, we use two adaptations of the environment: (i) LINEK" with
Figure 4: LLNEK" / LINEK*
(k = 10, Raie = 0): (ii) LINEK* with (k = 10, Rais = 0.01). We defer full details to appendices.
Experimental setup. We conduct our experiments with a neural REINFORCE agent using two- layered neural network architecture (i.e., one fully connected hidden layer with 256 nodes and RELU activation) [7]. Similar to Section 4.1, we compare the performance of six techniques. As a crucial difference, here we neural- network based reward functions for & PLIRPG", RSEURRS, and

---

EXPLORS
EXPLORS
SORS'
EXPLORS SELFRS EXPLOR
EXPLORS
SORS
SELvRS
ORIO LIRPT
SEIPRS
OBao LJRPC
EXPLOR
ExpLBB
LIRP
ExpLoB
LIRPG
Expected reccadd
1.5
Expected reward
Expected reward
8.
10
10
Expected
10
1.5
1.5
2 Episode (x10")
2 Episodo 6x10')
Episade (x10")
Episodo (x10')
(a) ROOM,, REINFORCE (b) ROOM", REINFORCE (c) LINEK", REINFORCE (d) LINEK*, REINFORCE
Figure 5: Results for ROOM and LINEK environments. These plots show convergence in performance of the agent w.r.f. training episodes. (a, b) show results for REINFORCE agent on ROOM" (i.e., ROOM variant without any distractor state) and ROOM* (i.e., ROOM variant with a distractor state).
(c, d) show results for REINFORCE agent LINEK (i.e., LINEK variant without any distractor
state) and LINEK* (i.e., LINEK variant with distractor states). See Sections 4.2 and 4.3 for details.
REXPLORS (see Footnotes 2 and 3). Based on [25, 26], we use the same neural-network architecture for intrinsic reward functions as used for the agent's policy by applying appropriate transformations at the output layer (e.g., instead of using soft-max, use tanh-clipping to get output reward values for actions). We provide full details about the implementation and hyperparameters in appendices.
Results. During training, the agent receives rewards based on R and is evaluated based on R. Figures Sc and 5d show results for both the variants of LINEK environment; the reported results are averaged over 30 runs and convergence plots show the mean performance with standard error bars. These plots showcase the performance of different techniques as we vary Rai E {0.00, 0.01} - this in turn decides whether there are any distractor states that can serve as local minima for the agent. The convergence behavior in Figures 5c and 5d demonstrates the effectiveness of our exploration- guided reward shaping framework (fREXPLORS) ), in comparison to baselines (ROORGG, RSORS' RLIIPGG' REEXLOB RESELPRS). Next, we summarize some of our key findings. First, our results show that REXPLORS outperforms all the baselines in both LINEK and LINEK* environments. Second, results in Figure 5d show that three reward shaping techniques (πSORS' RHIRPG" ? performed worse than RORIG this is because of the presence of distractor states which create local minima for the agent and these shaped functions could further encourage learning a sub-optimal policy. In contrast, REXPLORS combines the benefits of intrinsic rewards (R.ELERR) and intrinsic bonuses (xPLLB)) to speed up agent's 's learning in a robust and efficient manner. Overall, these results demonstrate that our shaping technique /EXPLORS results in efficient learning even when dealing with complex state representations and when learning neural-networt based intrinsic reward functions.
5 Concluding Discussions
We proposed novel reward shaping framework, EXPLORS, that operates in fully self-supervised manner and could accelerate an agent's learning even in sparse-reward environments. Next, we discuss a few limitations of our work and outline future plan to address them. First, the experimental evaluation is conducted on simpler environments to study the performance of techniques w.r.t. the three characteristics of (a) hard exploration, (b) local minima, and (c) "noisy TV" problem. It would be interesting to evaluate different reward design techniques in more complex environments (e.g., with continuous state/action spaces); this would also require designing benchmark environments that sys- tematically capture the above three characteristics. Second, EXPLORS combines the intrinsic rewards and intrinsic bonuses that allows it to overcome the limitations of state-of -fte-of-the-art techniques. It would be interesting to develop more principled ways to combine these two signals. Third, it would be useful to provide rigorous analysis of EXPLORS in terms of convergence speed and stability of an agent.
Acknowledgments
Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute. Funded/Co- funded by the European Union (ERC, TOPS, 101039090). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
10

---

References
[1] Maja J. Mataric. Reward Functions for Accelcrated Learning. ICML, 1994.
[2] Jette Randløv and Prehen Alström. Learning to Drive a Bicycle Using Reinforcement Learning and Shaping. In ICML,, 1998.
[3] Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy Invariance Under Reward Transfor- mations: Theory and Application Reward Shaping. In ICML, 1999.
[4] Adam Laud and Gerald DeJong. The Influence of Reward the Speed of Reinforcement Learning: An Analysis of Shaping. In ICMI,, 2003.
[5] Falcon Z. Dai and Matthew R. Walter. Maximum Expected Hitting Cost of a Markov Decision Process and Informativeness of Rewards. In NeurIPS, 2019.
[6] Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. RUDDER: Return Decomposition for Delayed Rewards, In NeurlPS, 2019.
[7] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[8] Eric Wiewiora. Potential-Based Shaping and Value Initialization are Equivalent. Journal of Artificial Intelligence Research, 19:205-208, 2003.
[9] Eric Wiewiora, Garrison W. Cottrell, and Charles Elkan. Principled Methods for Advising Reinforcement Learning Agents. In ICML, 2003.
[10] John Asmuth, Michael L., Littman, and Robert Zinkov. Potential-based Shaping in Model-based Reinforcement Learning. In AAAI, 2008.
[11] Marek Grzes and Daniel Kudenko. Plan-based Reward Shaping for Reinforcement Learning. In International IEEE Conference on Intelligent Systems, volume 2, pages 10-22, 2008
[12] Sam Devlin and Daniel Kudenko. Dynamic Potential- Reward Shaping. In AAMAS, 2012.
[13] Marek Grzes. Reward Shaping in Episodic Reinforcement Learning. In AAMAS, 2017.
[14] Alper Demir, Erkin Çilden, and Faruk Polat. Landmark Based Reward Shaping in Reinforcement Learning with Hidden States. In AAMAS, 2019.
[15] Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. Using Natural Language for Reward Shaping in Reinforcement Learning. In ISCAI, 2019.
[16] Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward Shaping via Meta-Leaming, CoRR, hts//901.09330.. 2019.
[17] Yuqian Jiang. Suda Bharadwaj. Bo Wu, Rishi Shah, Ufuk Topcu, and Peter Stone. Temporal- Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks. In AAAI, 2021.
[18] Pieter Abbeel and Andrew Y Ng. Apprenticeship Learning via Inverse Reinforcement Learning. In ICML, 2004.
[19] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping Your Distance: Solving Sparse Reward Tasks Using Saff-Balnncing Shaped Rewards. In NeurlPS, 2019.
[20] Satinder P. Singh, Andrew G. Barto, and Nuttapong Chentanez. Intrinsically Motivated Rein- forcement Learning. In NeurlPS, 2004.
[21] Satinder Singh, Richard Lewis, and Andrew G Barto. Where Do Rewards Come From? In CogSci, 2009.
[22] Satinder Singh, Richard L Lewis, Andrew G Barto, and Jonathan Sorg, Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2010.
[23] Jonathan Sorg, Satinder P Singh, and Richard L Lewis. Internal Rewards Mitigate Agent Boundedness. In ICML, 2010.
[24] Jonathan Sorg, Satinder P. Singh, and Richard L. Lewis. Reward Design via Online Gradient Ascent. In NewrlPS, 2010.
[25] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. Learning Intrinsic Rewards for Policy Gradient Methods. In NewrIPS, 2018

---

[26] Farzan Memarian, Wonjoon Goo, Rudolf Lioutikov, Scott Niekum, and Ufuk Topeu. Self- Supervised Online Reward Shaping in Sparse- Reward Environments. In IROS, 2021.
[27] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations. In ICML, 2019.
[28] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-Level Control Through Deep Reinforcement Learning. Nature, 518(7540):529-533 2015.
[29] Timothy P Lillicrap,, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning. CoRR, abs/1500.02971, 2015.
[30] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In ICML, 2015.
[31] Lilian Weng. Exploration Strategies in Deep Reinforcement Learning. Ellaawwengggihhubee 2020.
[32] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying Count-Based Exploration and Intrinsic Motivation. In NeurIPS, 2016.
[33] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-Based Exploration with Neural Density Models. In ICML, 2017.
[34] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning. In NewrIPS, 2017.
[35] Jürgen Schmidhuber. Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-247, 2010.
[36] Pieere-Yves Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic Motivation Systems for Autonomous Mental Development. IEEE Transactions on Evolutionary Computation,
1 11(2):265-266, 2007.
[37] Fierre-Yves Oudeyer and Frederic Kaplan. What is Intrinsic Motivation? A Typology of Computational Approaches. Frontiers Neurorobotics, 1:6, 2009.
(38] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing Exploration in Reinforcement Learning with Deep Predictive Models. CORR,
2015.
[39] Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-Driven Explo- ration by Self-Supervised Prediction. In ICML, 2017.
[40] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational Information Maximizing Exploration. In NeurlPS, 2016.
[41] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by Random Network Distillation. CoRR, abs/1 128944 2018.
[42] Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic- GGiided Reinforcement Learning. In NeurlPS, 2021.
[43] Ronen I Brafman and Moshe Tennenholtz. R-max: A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research, 3:213-231, 2002.
[44] Alexander Strehl and Michael L Littman. An Analysis of Model-Based Interval Estimation for Markov Decision Processes. Journal of Computer and System Sciences, 1309 2008.
[45] J Zico Kolter and Andrew Y Ng. Near-Bayesian Exploration in Polynomial Time. In ICML, 2009.
[46] Jonathan Sorg, Satinder Singh, and Richard I. Lewis. Variance-Basse Based Rewards for Approximate Bayesian Reinforcement Learning. In UAI, 2010.
12

---

[47] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning Learn by Gradient Descent by Gradient Descent. In NeurlPS, 2016.
[48] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-Leammng with Memory-Augmented Neural Networks. In ICMI,, 2016.
[49] Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Leaming Algorithms. CoRR, abs/1803.02999, 2018.
[50] Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In NeurlPS, 1999.
[51] Amy McGovern and Andrew G. Barto. Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density. In /CMI,, 2001.
[52] Özgür Simsek, Alicia P. Wolfe, and Andrew G. Barto. Identifying Useful Subgoals in Rein- forcement Learning by Local Graph Partitioning. In ICML, 2005.
[53] Michael R. James and Satinder P. Singh. Sarsalandmark: An Algorithm for Learning in POMDPs with Landmarks. In AAMAS, 2009.
[54] Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla. Explicable Reward Design for Reinforcement Learning Agents. In NeurIPS, 2021.
[55] Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling Others using Oneself Multi-Agent Reinforcement Learning. In ICML, 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The paper is organized according to the contributions listed the end of the introduction section.
(b) Did you describe the limitations of your work? [Yes] In Section we discuss the limitations of our work and outline a future plan to address these limitations.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This work presents new online reward shaping technique for reinforcement learning agents. Given the algorithmic and empirical nature of our work applied to learning agents, we do not foresee any direct negative societal impacts of our work in the present form.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We confirm that our paper conforms with the ethics review guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] All assumptions are stated either as part of the theorem statements or the proofs.
(b) Did you include complete proofs all theoretical results? [Yes] Complete proofs of all theoretical results are included in appendices.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] The code is provided as a URL in Footnote
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Details are provided appendices.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Error bars included in all the result graphs
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details are provided in appendices.
4. If you are using existing assets (e.g., code, data, models) or curating/relcusing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "vfVPSw1nJ8", "reviewer": "Reviewer_8ukX", "summary": "The paper takes a holistic approach to reward shaping in RL, and has two main contributions: (1) simplified meta-gradients for updating a parameterized intrinsic reward function, which generalizes beyond policy gradients algorithms, and (2) the combination of said intrinsic reward function with an intrinsic exploration bonus. The approach is self-supervised, and does not require expert knowledge. The combination of different intrinsic signals is studied theoretically in a chain environment, and then evaluated empirically on several low-dimensional MDPs, both in tabular settings and with function approximators.", "questions": "**Main comments**\n* The title is not very informative. I would suggest to consider something that clearly establishes the existence of two shaping terms, as well as the generalization beyond policy gradient algorithms.\n* Similarly, the abstract is also vague. I would recommend to include the concrete contributions of the paper, especially mentioning that the authors generalize self-supervised reward shaping techniques to general RL algorithms. Moreover, the sentence on line 8 does not clarify that two different terms are added to the environment's reward signal.\n* Despite generally stating in the abstract that they would 'theoretically showcase the utility of our reward shaping framework', the authors only present a limited theoretical analysis on a particular family of MDPs. Although this is not negative per se, it should be made clear right from the start.\n* Is there a reason why LINRPG is not included in the empirical evaluation on REINFORCE? In would be particularly interesting to see how SELFRS compares against it.\n* Section 3 could further benefit by clarifying that the derivations until line 163 are reported from existing works. Moreover, it should anticipate that the derivation will skip non-trivial steps, which are however reported in the Appendix.\n* On line 190, the paper mentions \"distractive\" rewards, but I fail to see how the rewards in this particular environment are distractive.\n\n**Minor comments**\n* line 20: To the best of my knowledge, transfer learning approaches do not need domain-knowledge, and can rely solely on data. Can this be clarified?\n* line 131: The authors should anticipate that superscripts (e.g. SelfRS) are dropped from this point on, as they do in the Appendix.\n* line 157: Where is this smoothness condition derived from? is it an assumption?\n* line 209: I would suggest rewriting the bound as $2^{n_1} - 1$ for ease of reading.\n* possibly due to my lack of familiarity, I would ask the authors to explain why the equality between the last line on page 17, and the first line on page 18 holds.\n\n**Nitpicks**\n* line 69: The notation for T does not clarify that its output is a probability distribution over the action space (i.e. integrates to 1).\n* line 234: green-star -> blue-star", "limitations": "Authors present a sufficient discussion of limitations in their conclusion.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is well-written and easy to follow, despite the large quantity of preliminary derivations.\n* The integration of both reward shaping terms is an interesting concept, and the analysis in Appendix D sheds nice insights on the synergy of the two.\n* Empirical performance appears strong across the environments considered in the paper, and the evaluation protocol seems robust.\n\n**Weaknesses**\n* In its main body, the paper does not always clarify which derivations are original, and which are reported from previous works (see following comments).\n* A number of separate concerns on the introductory writing and baselines would need to be addressed to further improve the quality of he paper (see following section, main comments). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qvkO4urqOYS", "reviewer": "Reviewer_JLEi", "summary": "The paper studies the problem of learning informative rewards, without domain knowledge or external supervision, that will accelerate learning in extremely sparse reward settings with noisy distractions. To address this problem, the paper proposes Exploration-Guided Reward Shaping (ExpRS) that balances exploration and explotation of environment rewards. The paper derives the update rules for ExpRS in a general setting, not specific to policy gradients. The paper then shows results on three environments comparing to other reward shaping methods. ", "questions": "* As the authors state, [1] compares on the same set of environments. However, [1] uses a exponential scale on the x-axis. This is important because it shows that the method from [1] takes exponentially less experience to learn. Why did this work not use an exponential x-axis scale? \n* In Theorem 1, it seems using an exploration reward contributes more to the cost than the self-supervised reward shaping. Why is this the case?\n* Given LIPRG [2] was mentioned several times throughout the paper, why not compare to LIPRG as well in the REINFORCE experiments?\n* Why not compare to ExpRD from [1] given that work shows results on the same set of environments? \n\n* The derivations in Eq. 1 detract from the description of the algorithm. I recommend moving the derivation to the supplementary and stating the final result in the main paper.\n* The supplementary material also includes the main paper, reformat to only include supplementary material or move the supplementary to the main paper. \n\n[1] Devidze Rati et al, Explicable reward design for reinforcement learning agents. In NeurIPS, 2021.\n\n[2] Zheng Zeyu et al. On learning intrinsic rewards for policy gradient methods. In NeurIPS, 2018.", "limitations": "The paper acknowledges that it would be interesting to extend the evaluation to more complex tasks. The paper also discusses the limitations of not having a rigerous theoretical analysis of the method in terms of convergence speed and stability. Despite this discussion, the paper does not sufficiently analyze limitations of the method. In what settings would ExpRD not perform as well? What are the limitations of learning a more complex reward in ExpRD?\n", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The proposed method is intuitive and to the best of my knowledge novel. The paper discusses benefits of both an intrinsic shaped reward and intrinsic bonus rewards and describes how to combine both into one algorithm. The paper also derives an update rule that works for general RL algorithms, not just policy gradient methods.\n* Theorem 1 with the analyzed chain environment is helpful for gaining insight into the utility of ExpRS. \n* ExpRS is more robust to distractors in the environment than baselines. The results demonstrate that distractors hurt the performance of all baselines, whereas ExpRS is more robust and affected less.\n\nWeaknesses:\n* The simple environments are not enough to verify the algorithm. While these environments are still helpful, results on more complex environments are important to verify the utility of ExpRS. For example, LIPRG [1] a work cited throughout this  paper and published at NeurIPS 2018, shows results on pixel-based Atari games. My concerns are exacerbated with the performance on the LineK task, which is the only task that demonstrates a neural network policy, yet the gap between ExpRS and baselines is small. In LineK, ExpRS is only slightly more sample efficient than the regular sparse reward function. This concerns me that ExpRS cannot scale to neural policies and rewards in complex environments.\n* How does ExpRS balance the SelfRS and count-based bonus reward? Since the main contribution of the paper is combining these two rewards, studying their balance is importance. On line 183, the paper shows the count-based reward is weighted by parameter $\\lambda$. The authors should analyze the impact of different reward weightings to see the impact on performance. \n* The SORS baseline is not explained enough in Section 4.1. SORS is briefly explained on L114, but a more detailed description would be useful in Section 4 for how the SORS reward is obtained and how it compares to ExpRS.\n* The paper might also be lacking some important baselines, which I discuss in the Questions section. \n\n[1] Zheng Zeyu et al. On learning intrinsic rewards for policy gradient methods. In NeurIPS, 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O_5TLT28aK7", "reviewer": "Reviewer_tE4G", "summary": "The paper proposes a new method for reward shaping (called EXPRS) that leverages intrinsic rewards from self-supervised reward shaping techniques, as well as exploration bonus in the shaped reward. The authors first describe the general setting of reward shaping followed by existing reward techniques and their shortcomings, specifically self-supervised reward shaping and exploration bonus based reward shaping. Subsequently the authors introduce their reward shaping formulation, which includes both the intrinsic self-supervised reward and the exploration bonus in a combined formulation. The authors then derive gradient update formulas for their proposed algorithm, in a general and empirical setting, and outline the steps in their EXPRS method.\n\nFollowing the definition of EXPRS, the authors perform a theoretical analysis of how EXPRS should perform in a chain environment compared to other methods and then show experimental results of EXPRS and those methods in a chain environment, as well as a Room environment and a LineK environment. The paper then concludes with a brief discussion of the results and some directions for future work.", "questions": "**General Questions:**\n- Could you define \"b\" in Line 167? This is mainly for completeness since I did not see it defined (I might have missed it)\n- How general is the gradient derivation to different RL algorithms? For example, if someone wanted to apply EXPRS to PPO, would they have to derive empirical updates just for PPO? It would be good to get a better sense of how easy/difficult it is to plug in EXPRS to different RL algorithms\n- Is the theoretical analysis in Section 3.3 mainly limited to the chain environment or does the general utility hold beyond it? I noticed that the appendix includes further details on how you arrived at cost for the different cases, which would be good to mention in the paper.\n- What is the reasoning behind using the baselines that you show in your experiments? Also, is LIPRG included there (it could be that SELF-RS is similar to LIPRG but that is currently unclear)?\n- It looks like for many of the experiments that you performed, EXPRS and some of the baselines fall within the deviation of each other. Could you comment on where you see clear advantages of EXPRS and where it is more difficult to distinguish its performance from other methods.", "limitations": "I think the paper could be improved by a more thorough discussion of the limitations of EXPRS, which are only briefly touched upon in the conclusion. It would be good to get more detail on what the authors anticipate would be needed (in general terms) to perform the extensions that they mention (continuous spaces, converge, other agents). Given space limitations, it would be OK to include this in the appendix even though I would recommend it to be in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:**\n- Strengths: The paper proposes a new method for automated for reward shaping in RL that draws upon two prior methods and combines them into a single formulation. Relevant work is cited and compared against in the beginning of the paper.\n- Weaknesses: The authors could have provided more detail on the limitations of their method, which are only briefly discussed in the conclusion.\n\n**Quality:**\n- Strengths: The paper provides a good amount of detail in the derivation steps related to the method and is generally very thorough in description of relevant details.\n- Weaknesses: The number of experiments that showcase the capabilities of the method is somewhat limited.\n\n**Clarity:**\n- Strengths: The paper is generally well organized and includes relevant figures and diagrams for the environments used in the experiments.\n- Weaknesses: The authors could have helped guide readers by providing summaries of important points/messages throughout the paper, especially after sections with heavy text and equations. For example, after Section 2 it could be helpful to have a small table outlining the prior methods and outline what they can do (e.g. with checkmarks) and how EXPRS has capabilities beyond those method. In Section 3.2 it could be helpful to state in the beginning what the missing terms in the gradient computation are and how the derivation leads to them being available when describing the final algorithm. \n\n**Significance:**\n- Strengths: The paper proposes a new method for exploration based in reward shaping, which is a relevant problem in RL.\n- Weaknesses: The environments and algorithmic varieties for which the method was tested were somewhat limited and provide little insight into how the method may perform in more challenging settings.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept