Redeeming intrinsic rewards via constrained optimization
Authors: Eric R Chen
Zhang-Wei Hong
Joni Pajarinen
Pulkit Agrawal
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
intrinsic reward
curiosity-driven exploration

Abstract: State-of-the-art reinforcement learning (RL) algorithms typically use random sampling (e.g., $\epsilon$-greedy) for exploration, but this method fails on hard exploration tasks like Montezuma's Revenge. To address the challenge of exploration, prior works incentivize exploration by rewarding the agent when it visits novel states. Such intrinsic rewards (also called exploration bonus or curiosity) often lead to excellent performance on hard exploration tasks. However, on easy exploration tasks, the agent gets distracted by intrinsic rewards and performs unnecessary exploration even when sufficient task (also called extrinsic) reward is available. Consequently, such an overly curious agent performs worse than an agent trained with only task reward.  
Such inconsistency in performance across tasks prevents the widespread use of intrinsic rewards with RL algorithms. We propose a principled constrained optimization procedure called Extrinsic-Intrinsic Policy Optimization (EIPO) that automatically tunes the importance of the intrinsic reward: it suppresses the intrinsic reward when exploration is unnecessary and increases it when exploration is required. The results is superior exploration that does not require manual tuning in balancing the intrinsic reward against the task reward. Consistent performance gains across sixty-one ATARI games validate our claim. The code is available at https://github.com/Improbable-AI/eipo.

Redeeming Intrinsic Rewards via Constrained Optimization
Eric Chen, Zhang- Wei Hong *t Joni Pajarinen's Pulkit Agrawal*
Improbable AI Lab, Massachusetts Institute of Technology
MIT-IBM Watson Al Lab" Aalto University NSF AI Institute for Al and Fundamental Interactions (IAIFI)S
Abstract
State-of-the-art reinforcement learning (RL) algorithms typically use random sam- pling (e.g., e-greedy) for exploration, but this method fails on hard exploration tasks like Montezuma's Revenge. To address the challenge of exploration, prior works incentivize exploration by rewarding the agent when it visits novel states. Such intrinsic rewards (also called exploration bonus or curiosity) often lead to excellent performance hard exploration tasks. However, easy exploration tasks, the agent gets distracted by intrinsic rewards and performs unnecessary exploration even when sufficient task (also called extrinsic) reward is available. Consequently, such an overly curious agent performs worse than an agent trained with only task reward. Such inconsistency in performance across tasks prevents the widespread use of intrinsic rewards with RL algorithms. We propose principled constrained optimization procedure called -ntrinsii-Intissicc Policy Optimization (EIPO) that automatically tunes the importance of the intrinsic reward: suppresses the intrin- sic reward when exploration is unnecessary and increases it when exploration required. The results is superior exploration that does not require manual tuning in balancing the intrinsic reward against the task reward. Consistent performance gains across sixty-one ATARI games validate our claim. The code is available at https: //gi thub. com com/Improbable-Aee AIieoo...
Introduction
The goal of reinforcement learning (RL) [1] is to find mapping from states to actions (i.c., policy) that maximizes reward. At every learning iteration, agent is faced with question: has the maximum possible reward been achieved? In many practical problems, the maximum achievable reward is unknown. Even when the maximum achievable reward is known, if the current policy sub-optimal then the agent is faced with another question: would spending time improving its current strategy lead to higher rewards (exploitation), or should it attempt different strategy in the hope of discovering potentially higher reward (exploration)? Pre-mature exploitation akin to getting stuck in local-optima and precludes the agent from exploring. Too much exploration on the other hand can be distracting, and prevent the agent from perfecting a good strategy. Resolving the exploration-ccllotttion dilemma [1] is therefore essential for data/time efficient policy learning.
In simple decision making problems where actions do not affect the state (e.g. bandits or contextual bandits [2]), provably optimal algorithms for balancing exploration against exploitation are known [3, 2]. However, in the general settings where RI. used, such algorithms are unknown. the absence of methods that work well in both theory and practice, state-of-theaae RI algorithms rely on heuristic exploration strategies such as adding noise to actions or random sampling of sub-optimal actions
denotes equal contribution. Correspondence to pulkitagmmit.edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Extrinsk reward
MantezumaRevange
Jamesbond
Intrimic reward
2.5K
Ageal
Estrinsic returt
2.0K
Wall
1.5K
10 OK
Extrinsic
1.0K
Extrinaic-inbiisic
5.0K
Ours
0.5K
0.0k
0.0k
50
20
(millions)
Frames (millions)
(a)
(b)
Figure I: (a) At the start of training all locations are novel for the agent (green triangle), and therefore the pink circles representing intrinsic rewards are evenly distributed across the map. The blue circles represent sources of extrinsic reward or task-reward. Here intrinsic rewards can distract the agent, as the sum of extrinsic and intrinsic rewards can be increased by moving along the bottom corridor. (b) This type of distraction is a possible reason why an intrinsic reward method does not consistently outperform method trained using only extrinsic rewards across ATARI games. Intrinsic rewards help in some games where the task or extrinsic reward is sparse (e.g., Montezuma's 's revenge), but hurt in other games such as James Bond. Our proposed method, EIPO, intelligently uses intrinsic rewards when needed and consistently matches the best-performing algorithm amongst extrinsic and extrinsic+intrinsic methods.
(e.g., e-greedy). However, such strategies fail sparse reward scenarios where infrequent rewards hinder policy improvement. One such task is the notorious ATARI game, Montezuma's 's Revenge [4].
Sparse reward problems can be solved by supplementing the task reward (or extrinsic reward rg) with dense exploration bonus (or intrinsic reward rr) generated by the agent itself [4-8]. Intrinsic rewards encourage the agent to visit novel states, which increases the chance of encountering states with task reward. Many prior works [4, 8, 9] show that jointly optimizing for intrinsic and extrinsic reward (i.e.. rg + Ar,, where À 2 is a hyperparameter) instead of only optimizing for extrinsic reward (i.e., X = 0) improves performance on sparse reward tasks such as Montezuma's 's revenge [4, 9, 10].
However, recent study found that using intrinsic rewards does not consistently outperform simple exploration strategies such as e-greedy across ATARI games [1 1]. This is because the mixed objective (rE + Ary) ) biased for |>| > 0, and optimizing it does not necessarily yield the optimal policy with respect to the extrinsic reward alone [12]. Fig. la illustrates this problem using a toy example. Here the green triangle is the agent and the blue/pink circles denote the location of extrinsic and intrinsic rewards, respectively. At the start of training, all states are novel and provide a source of intrinsic reward (i.e., pink circles). This makes accumulating intrinsic rewards easy, which the agent may exploit to optimize its objective of maximizing the sum of intrinsic and extrinsic rewards. However, such optimization can result in a local maxima: the agent might move rightwards along the bottom corridor, essentially distracting the agent from the blue task rewards at the top. In this example, since is not hard to find the task reward, better performance obtained if only the extrinsic reward (A 0) is maximized. The trouble, however, is that in most environments one doesn't 't know a priori how to optimally trade off intrinsic and extrinsic rewards (i.c., choose A).
A common practice is to conduct an extensive hyperparameter search to find the best X, as different values of À are best suited for different tasks (see Fig. 4). Furthermore, as the agent progresses on a task, the best throughout training. In initial stages of training exploration might be preferred. Once the agent is able to obtain some task reward, it might prefer exploiting these rewards instead of exploring further. The exact dynamics of the exploration-exploin exploitation trade-off desedeepnndent, and per-task tuning is tedious, undesirable, and often computationally infeasible. Consequently, prior works use a fixed À during training, which our experiments reveal is sub-optimal.
trade-off can vary, and a constant À may not be optimal
We present an optimization strategy that alleviates the need to manually tune the relative importance of extrinsic and intrinsic rewards as training progresses. Our method leverages the bias of intrinsic rewards when it is useful for exploration and mitigates this bias when it does not help accumulate higher extrinsic rewards. This is achieved using an extrinsic optimality constraint that forces the extrinsic rewards earned after optimizing the mixed objective to be equal to the extrinsic rewards accumulated by the optimal policy that maximizes extrinsic rewards only. Enforeing the extrinsic optimality constraint in general settings is intractable because the optimal extrinsic reward is unknown. We devise a practical algorithm called Extrinsic-Intrinss Policy Optimization (EIPO), which uses an approximation to solve this constrained optimization problem (Section 3).
2

---

While in principle we can apply EIPO to any intrinsic reward method, we evaluate performance using state-of-the- of-thee-rt art random network distillation (RND) [9]. Fig. 1b presents teaser results on two ATARI games: (i) Montezuma's 's Revenge- where joint optimization with RND (red) substantially outperforms PPO policy [13] optimized with only extrinsic rewards (black); (ii) James Bond where PPO substantially outperforms RND. These results reinforce the notion that bias introduced by intrinsic rewards helps in some games, but hurts others. Our algorithm EIPO (blue) matches the best algorithm in both games, showing that it can leverage intrinsic rewards needed. Results across 61 ATARI games reinforce this finding. Additionally, in some games EIPO outperforms multiple strong baselines with and without intrinsic rewards, indicating that method can not only mitigate the potential performance decreases caused by intrinsic reward bias, but can also improve performance beyond the current state-of-the-ee aftt
2 Preliminaries
We consider a discrete-time Markov Decision Process (MDP) consisting of a state space S, an action space A, and extrinsic reward function Rr S A + R. We distinguish the extrinsic and intrinsic reward components by E and I, respectively. The extrinsic reward function Rg refers to the actual task objective (e.g., game score). The agent starts from an initial state so sampled from the initial state distribution Po S + R. At each timestep f, the agent perceives state 8, from the environment, takes action a, sampled from the policy я, receives extrinsic reward r Rgg( az), and moves to the next state 8t41 according to the transition function ((s+11 (st+1|s; 18, ax). The agent's
goal is to use interactions with the environment find the optimal policy æ such that the extrinsic objective value FF((T) (T) is maximized:
8
max (T),,, where JÆ(T) Eeo. n
x'r
(Extrinsic objective),
Su ~ Po. ar ~ (alss) S411 T(1441/51, Vt >
where y denotes a discount factor. For brevity, we abbreviate Esg.ao
as E unless specified.
Intrinsic reward based exploration strategies[4, 8, 9] attempt to encourage exploration by providing "intrinsic rewards" (or "exploration bonuses") that incentivize the agent to visit unseen states. Using the intrinsic reward function R2 S x A - I, the optimization objective becomes:
max JE-1(TT, where J JE+I (T) = F Σ #Ell
rr + xx())] (Mixed objective),
where denotes the intrinsic reward scaling coefficient. We abbreviate the intrinsic reward at timestep & as r/ = (s,, az). State-of-the-art intrinsic reward based exploration strategies [9, 14] often optimize the objective Eq. using Proximal Policy Optimization (PPO) [13].
3 Mitigating the Bias of Intrinsic Rewards
Simply maximizing the sum of intrinsic and extrinsic rewards does not guarantee policy that also maximizes extrinsic rewards: arg max JE+I(TEE+) t * arg max JE(TE). At convergence the optimal policy म T arg max (E++(TE++) could be suboptimal w.r.t. Jg- which measures the agent's task performance. Because the agent's performance is measured using extrinsic reward only, we propose enforcing extrinsic optimality constraint that ensures the optimal "mixed" policy т T = arg max JE+I(TE+I) leads to as much extrinsic reward as the optimal "extrinsic" policy TE = arg max (T)). The resulting optimization objective is:
max #BLIEIT JE+I(TE+1)
(3)
subject to JÆ(TE41) max (F(TE)
(Extrinsic optimality constraint).
Solving this optimization problem can be viewed as proposing a policy N that maximizes JE+1. and then checking if the proposed πE+s is feasible given the extrinsic optimality constraint.
3

---

The constrained objective is difficult to optimize because evaluating the extrinsic optimality con- straint requires (F(T)) which unknown. To solve this optimization problem, we transform into an unconstrained min-max optimization problem using Lagrangian duality (Section 3.1). We then describe an iterative algorithm for solving the min-max optimization by alternating between minimization and maximization in Section 3.2, and we present implementation details in Section 3.3.
3.1
The Dual Objective: Unconstrained Min-Max Optimization Problem
The Lagrangian dual problem for the primal constrained optimization problem in Eq. is:
min max JE+r(TE+1) + a(JE(TE+1) max JE(TE)) meell
(4)
OER+ TRTJEll
where a E R+ is the Lagrangian multiplier. We rewrite Eq. 4 by merging Jr. ( and JE(T):
J ( = JE+I(FRA)) (JR((EET) =E E
Σ 1=
(r) uz) + (s,, atl
The re-written objective provides intuitive interpretation of a: larger values correspond to increas- ing the impetus on extrinsic rewards (i.e., exploitation). Substituting (++T((++))) into Eq. 4 and rearranging terms yields the following min-max problem:
min GER+
max min ( - a (
(5)
LRR- #REll
3.2 Algorithm for Optimizing the Dual Objective
We now describe an algorithm for solving TE+H, and a in each of the sub-problems in Eq. 5.
Extrinsic policy Th (min-stage). TE is optimized via the minimization sub- bub--roblem,, which can be re-written as a maximization problem:
min J (T++(TE+1) AJE(TE) rell
max aJE((T)) (T++) #gell
(6)
The main challenge is that evaluating the objectives r and JE(TE) requires sampling trajectories from both policies π and Tg. If one were to use an on-policy optimization method such as PPO, this would require sampling trajectories from two separate policies at each iteration during training. which would be data inefficient. Instead, if we assume that the two policies are similar, then we can leverage results from prior work to use the trajectories from one policy (T to approximate the return of other policy (TÆ) [15, 16, 13].
First, using the performance difference lemma from [15], the objective (JE(TTE) J (TT+T) can be re-written (see Appendix 1.1 for detailed derivation):
aJE(TR) (E Eng
Σ
T -UTE- (st: an)
where U (st, a+) = art + EAVE+4(S1 V-EE.I (st):
V (st) == Enets
Σn' + +r))ls0 $t
Next, under the similarity assumption, a lower bound the objective Eq. can be obtained [13]:
E
8
y U as) 2 ErELI
7't min
TT((r)s)) (a)+)(a((5))
U=R+I (st,a;),
(8)
( clip TE+I(a, [xx)
-1
e, c,
U ot)
where e E [0, 1] denotes a threshold. Intuitively, this clipped objective (Eq. 8) penalizes the policy
TE (0)()))
TE that behaves differently from #F+7 because overly large or small More details provided in Appendix A.2.
terms are clipped.
n #E+((a)ss)

---

Mixed policy TE+1 (max-stage). The sub-problem of solving for TB+1 is posed as
max ( aJ/((TE) #64EEll
(9)
We again rely on the approximation from [13] to derive a lower bound surrogate objective for B (T aJE(TT) as follows (see Appendix A.1.2 for details):
0
> min { (B+1(ai(si)
( (JE(TE) 2 ETR
U (st,
TE (1((())))
t=
(10)
clip ( TE(
c. c.1
Umax (8),
where Use and V are defined as follows:
U max (5,, a1) (1 aar + +rt
VV (st)
VE (8t) = F
Σ 1=0
rv. 8t
Lagrangian multiplier a. We solve for a by using gradient descent on the surrogate objective derived above. Let g(a)= == maxxg minngen J / aJE(TE). Therefore, Vg(a)=
JE(TE+1) - -E(TE). We approximate Vy(a) using the lower bound surrogate objective:
min Tg+r(ar(sL) (T(
JE(TE+1) JE(TE) ((EETEE+) = E
TT (st,
t=0
(11)
TE+1 (ag St)
clip
- e, c, ( az)
TE (8t)
where A az) rf VV (s++1) V (st) is the advantage of taking action a, state st.
and then following Th for subsequent steps. We update using a step size 3 (Appendix A.1.4):
a - a (L(TE, (T(TT,TE+T))
(12)
Unlike prior works that use a fix trade off between extrinsic and intrinsic rewards, optimizing a during training allows our method to automatically and dynamically tune the trade off.
3.3 Implementation
Min-max alternation schedule. We use an iterative optimization scheme that alternates between solving for Te by minimizing the objective in Eq. 6, and solving for # by maximizing the objective in Eq. 9 (max (max_stage). We found that switching the optimization every iteration hinders performance which hypothesize is result of insufficient training of each policy. Instead, switch between the two stages when the objective value does not improve anymore. Let J be the objective value J ( (JG(TT) iteration i. The switching rule
1 Ji-1 < max_stage False, if mfmax_ttage = True
-
max_stage True, max_ge False,
where max_stage a binary variable indicating whether the current stage is the max-stage a is only updated when the max-stage done using Eq. 12.
Parameter sharing. The policies π and Fg are parametrized by two separate multi-layer per- ceptrons (MLP) with shared CNN backbone. The value functions E++ and V are represented in the same fashion, and share a CNN backbone with the policy networks. When working with image inputs (e.g., ATARI), sharing the convolutional neural network (CNN) backbone between Tg and π-ー helps save memory, which is important when using GPUs (in our case, an NVIDIA RTX 3090Tí). If both policies share backbone however, maximizing the objective (Eq. 9) with respect to TE++ might interfere with Jr J (TF), and impede the performance of wF. Similarly, minimizing the objective (Eq. 6) with respect to Tg might modify J and degrade the performance of mE+J.
5

---

end for Algorithm Extrinsic-Intrinsic Policy Optimization (EIPO)
1: Initialize policies πE+1 and RE, max stage [0]
False, and [[]] s
2: for 1... do
denotes iteration index > Max-stage: rollout by TE and update म E+I Collect trajectories тミ using ag and compute U Uma (st, a₂) ((st, az) E TE
3:
if max_stage [i 1] then
4:
5:
Update π-+ by Eq. 10 and Tæ by auxiliary objective (Section 3.3)
6:
J[i] + J / (TE+1) aJgg (TE)
7:
max_ max_staaeeli [i] + - 1
8:
else
Min-stage: rollout by NE+ and update Collect trajectories T using + and compute U (8t, a+) (8t, ax) E TE+T Update TE by Eq. and NE+1 by auxiliary objective (Section 3.3)
9:
10:
11:
J[i] + / (TE+1) aJ (TE)
max_stage [i] + [[] Jli 2
13:
end if
14:
max_stagee [i 1]
True and max_stage [i] False then
15:
Update (Eq. 32)
> Update when the max-stage is done
16:
end
17:
Interference can be avoided by introducing auxiliary objectives that prevent decreases in (T(TT) and J when optimizing for TE+,, TE, respectively. For example, when updating TE+T in the max-stage, max J (TE+I) f - aJæ(TE), the auxiliary objective max JE (Tæ) can prevent updates in the shared CNN backbone from decreasing J(TT). We incorporate the auxiliary objective without additional hyperparameters (Appendix A.2.3). The overall objective in Appendix A.2.4.
Extrinsii-Intrimsicc Policy Optimization (EIPO) the policy optimization algorithm we introduce to solve Eq. 5, is outlined in Algorithm 1. Pseudo-cods implementation details including hyperparameters can be found Appendix A.2.
be found in Algorithm 2, and full
4 Experiments
While EIPO is agnostic to the choice of intrinsic rewards, we mainly experiment with RND [9] because is the state-of-the-art intrinsic reward method. EIPO implemented with RND is termed EIPO-RND and compared against several baselines below. All policies are learned using PPO [13].
EO (Extrinsic only): The policy is trained using extrinsic rewards only: x² = arg max JF(T).
RND (Random Network Distillation) [9]: The policy is trained using the sum of extrinsic rewards (r:²) and RND intrinsic rewards (Ar{): ): π = arg max JE+1 (T). For ATARI experiments, we chose single value of À that worked best across games. Other methods below also use this X.
EN (Ext-norm-RND): We found that variant of RND where the extrinsic rewards normalized using running mean and standard deviation (Appendix A.2.2) outperforms the original RND implementation, especially in games where RND performs worse than EO. Our method EIPO and other baselines below are therefore implemented using Ext-norm-RND.
DY (Decay-RND): Instead of having a fixed trade-off between exploration and exploitation throughout training, dynamically adjusting exploration vs. exploitation can lead to better per- formance. Without theoretical results describing how to make such adjustments, commonly used heuristic is to gradually transition from exploration to exploitation. One example e-greedy exploration [17], where is decayed over the course of training. Similarly, we propose a variant of Fxt-norm-RNV where intrinsic rewards are progressively scaled down to eliminate exploration bias over time. Intrinsic rewards rf are scaled by X(i), where denotes the iteration number.
" + A(i)r!)) where X(i) is defined as The objective function turns into JE+I = -
X(i) = clip( (Amax Amin), Amin >max). Here, Amax and Amia denote the predefined maximum and minimum X(1), and 7 is the iteration after which decay is fixed. We follow the linear decay schedule used to decay e-greedy exploration in DQN [17].

---

RND EIPO
Ficibokk rewards
Bitrrante
Extrinsic return
400
0.56
Apeil
300
0.54
200
0.52
0.0
Frames
167
(a)
(b)
(c)
Figure 2: (a) 2D navigation task with similar distribution of extrinsic and intrinsic rewards as Figure la. The gray triangle the agent. The black dot is the goal that provides extrinsic reward. Red dots randomly placed along the bottom corridor at the start of every episode. Due the novelty of their locations, these dots serve as source of intrinsic rewards. (b) RND is distracted by the intrinsic rewards and fails match EO optimized with only extrinsic rewards. Without intrinsic rewards, EO is inferior to our method EIPO in terms of discovering a good path to the black goal. This result indicates that EIPO is not distracted by intrinsic rewards, and uses them as necessary to improve performance. (c) a controls the importance of the extrinsic-intrins. optimality constraint. decreases until the extrinsic return starts to rise between 0.5 and 1.0 million frames. Afterwards, also increases, showing that our method emphasizes intrinsic rewards at the start of training, and capitalizes on extrinsic rewards once they are found.
DC (Decoupled-RND) [18]: The primary assumption in EIPO is that πh+1 and Tg are similar (Section 3.2). A recent work used this assumption in different way to balance intrinsic and extrinsic rewards [18]: they regularize the mixed policy to stay close to the extrinsic policy by minimizing the Kullback-Leible (KL) divergence DEL (T [E+)). [<E|[T++)) the extrinsic-optimalii constraint, which key component in EIPO. Therefore, comparing against this method called Decoupled-RNY RND (DC) [18] will help separate the gains in performance that come from making the similarity assumption versus the importance extrinsic-opimmality constraint. We adapted DC to make use of Ext-norm-RND intrinsic rewards:
However, they do not impose
D0
T arg max ETETI
Σ 1=0
Cre + +r() DKL (FEl #E = arg max E
Σ t=0
7
FB
4.1 Illustrative example
We exemplify the problem of intrinsic reward bias using simple 2D navigation environment (Fig. 2a) implemented using the Pycolab game engine [19]. The gray sprite denotes the agent, which observes x 5 pixel view of its surroundings. For every timestep the agent spends the (black) location at the top of the map, an extrinsic reward of +1 is provided. The red circles are randomly placed in the bottom corridor at the start of each episode. Because these circles randomly change location, they induce RND intrinsic rewards throughout training. Because intrinsic rewards along the bottom corridor are easier to obtain than the extrinsic reward, optimizing the mixed objective J/++ can yield policy that results in the agent exploring the bottom corridor (i.c., exploiting the intrinsic reward) without ever discovering the extrinsic reward the top. Fig. 2b plots the evolution of the average extrinsic return during training for EO, RND, and EIPO-RND across 5 random seeds. We find that EIPO-RNU outperforms both RND and BO. RND gets distracted by the intrinsic rewards (red blocks) and performs worse than the agent that optimizes only the extrinsic reward (EO). EO performs slightly worse than EIPO-RND possibly because in some runs the EO agent fails to reach the goal without the guidance of intrinsic rewards.
To understand why EIPO-RND performs better, we plot the evolution of the parameter a that trades- off exploration against exploitation during training (Fig. 2c). Lower values of a denote that the agent prioritizing intrinsic rewards (exploration) over extrinsic rewards (exploitation; see Section 3.).. This plot shows that for the first ~ 0.5M steps, the value of a decreases (Fig. 2c) indicating that the agent prioritizing exploration. Once the agent finds the extrinsic reward (between 0.5. 1M steps), the value of a stabilizes which indicates that further prioritization of exploration is unnecessary. After ~ 1M steps the value a increases as the agent prioritizes exploitation, and extrinsic return increases (Fig. 2b). These plots show that EIPO is able to dynamically trade-off exploration against exploitation during training. The dynamics of a during training also support the intuition that EIPO transitions from exploration to exploitation over the course of training.
7

---

0.8
All Atari games
0.8
All Atari games
0.6
0.49
0.55
0.58
0.52
0.66
P(Ours 2:
0.61
0 5B
0.60
0,55
0.8
0.6
0.8
0.8
0.65
PlOurs
0.65
0,65
0.6
0.49
0.54
0.57
0.51
0.6
0.59
0.2
DC
Algorithm
Algorithm
Frames (millions)
(a)
(b)
(c)
Figure 3: (a) EIPO-RND (ours) has a higher probability of improvement P(EIPO-RND > EO) over EO than all other baselines. This suggests EIPO-RND is more likely to attain a higher score than EO, compared to other methods. (b) Probability of improvement P(EIPO-RNE REDDD--RND> > B) > 0.5 AB indicates that EIPO-RND performs strictly better than the baselines in the majority of trials. (c) Each colored curve denotes the evolution of a in a specific game using EIPO. The variance in trajectories across games reveals that different games.
dynamics are best suited for different
4.2 EIPO Redeems Intrinsic Rewards
To investigate if the benefits of EIPO carry to realistic settings, conducted experiments on ATARI games [20], the de-facto benchmark for exploration methods [4, 11]. Our goal is not to maximize performance on ATARI, but to use ATARI as proxy anticipate performance of different RL algorithms on a new and potentially real-world task. Because ATARI games vary in terms of task objective and exploration difficulty, if an algorithm A consistently outperforms another algorithm B on ATARI, it provides evidence that A may outperform B given a new task with unknown exploration difficulty and task objective. such an algorithm is found, the burden of cycling through exploration algorithms to find the one best suited for a new task is alleviated.
We used "probability of improvement" metric P(A B) with 95%--confidence interval (see Appendix A.4.2; [21]) to judge if algorithm A consistently outperforms across all ATARI games. P(A> B)= > 0.5, indicates that algorithm A outperforms B majority of games (i.e., consistent improvement.. Higher values of P(A > B) means greater consistency in performance improvement of over B. For the sake of completeness, we also report P(A B) which represents weaker improvement metric that measures whether algorithm A matches or outperforms algorithm B. These statisties are calculated using the median of extrinsic returns over the last hundred episodes for at least 5 random seeds for each method and game. More details are provided in Appendix A.4.
Comparing Exploration with Intrinsic Rewards v/s Only Optimizing Extrinsic Rewards Re- sults in Fig. 3a show that P(RND > EO) 0.49 with confidence interval {0.46, 0.52), where EO denotes PPO optimized using extrinsic rewards only. These results indicate inconsistent performance gains of RND over EO, an observation also made in prior works [11]. Ext-norn-RNy RND (EN) fares slightly better against EO, suggesting that normalizing extrinsic rewards helps the joint optimization of intrinsic and extrinsic rewards. We find that P(EIPO-RND > EO) 0.65 with confidence interval (0.62, 0.67), indicating that EIPO able to successfully leverage intrinsic rewards for exploration and is likely to outperform EO on new tasks. Like EIPO, Decoupled-RNV assumes that Tg and T are similar, but Decoupled- RenippeedddDDiiiie is not any better than Ext-norm RND when compared its own does not improve performance,
against EO. This suggests that the similarity assumption and that the extrinsic-opii extrinsic-optimality-c wwt.oopaaaittttoooorininn plays a key part in improving performance.
EIPO Strictly Outperforms Baseline Methods Results in Fig. 3b show that P(EIPO-RND > B) > 0.5 across ATARI games statistically rigorous manner for all baseline algorithms B. Experiments on the Open AI Gym MuJoCo benchmark follow the same trend: EIPO either matches or outperforms the best algorithm amongst EO and RND (see Appendix 10). These results show that EIPO is also applicable to tasks with continuous state and action spaces. Taken together, the results in ATARI and MuloCo benchmarks provide strong evidence that given a new task, EIPO has the highest chance of achieving the best performance.
4.3 Does EIPO Outperform RND Tuned with Hy pyperpareeeeer Search?
In results so far, RND used the intrinsic reward scaling coefficient that performed best across games. Our claim is that EIPO can automatically tune the weighting between intrinsic and extrinsic
8

---

Enduro
Jamesbond
StanGunner
50.0K
1.0K
0.6K
10.0k
30,0K
0.4K
20.0K
EO
5
RND
10.0k
EIPO-RND
0.0K
0.6K
0.0K
Ax
TimaPlot
YarsRevenge
Ventura
EeMr
25.0K
150.0k
1.24
FrAr
22.0K
3.0k
Ao
15.04
100.0K
0.8K
10.06
5.0K
0.0K
20
20 (millioss
Frames (lons)
Frames ons)
Figure 4: No choice of the intrinsic reward scaling coefficients X consistently yield performance improvements over RND and EO, whereas our method EIPO-RND consistently outperforms RND and EO without any tuning across environments. This indicates that our method less susceptible to X.
rewards. If this is indeed true, then EIPO should either match or outperform the version of RND that uses the best Ả determined
per-game basis through exhaustive hyperparameter sweep. Since an exhaustive sweep in each game is time consuming, we evaluate our hypothesis on a subset of representative games: few games where PPO optimized with only extrinsic rewards (EO) outperforms RND, and a few games where RND outperforms EO (e.g., Venture and Montezeuma" 's Revenge).
For each game we searched over multiple values of À based on the relative difference magnitudes between intrinsic (r²) and extrinsic (r rewards (see Appendix A.4.4 for details). The results summarized in Fig. show that no choice of X consistently improves RND performance. For instance, r s Ard substantially improves RND performance in Jamesbond, yet deteriorates performance in YarsRevenge. We found that with RND, the relationship between the choice is not intuitive. For example, although Jamesbond is regarded as easy exploration game [4], the agent that uses high intrinsic reward (i.e., r < Xr') outperforms the agent with low intrinsic reward (r" > )r. EIPO overcomes these challenges and is able to automatically adjust the relative importance of intrinsic rewards. outperforms RND despite game-specific À tuning.
and final performance
Dynamic adjustment of the mance If the performance gain of EIPO solely resulted from automatically determining constant trade off between intrinsic and extrinsic rewards throughout training on a per- per-game basis, an exhaus- tive hyperparameter search of should suffice match the performance of EIPO-RND. The fact that EIPO-RND outperforms RND with different choices of À leads us to hypothesize that adjusting the importance of intrinsic rewards over the course of training is important for performance. This dynamic adjustment is controlled by the value of a which is optimized during training (Section 3.2). Fig. 3c shows the trajectory of a during training across games. most games a increases over time, indicating that the agent transitions from exploration to exploitation as training progresses. This decay in exploration is critical for performance: the algorithm, Decay-RND (DY) that implements this intuition by uniformly decaying the importance of intrinsic rewards outperforms RND (Fig. 5 and Fig. 3a). However,
Expptiion-xxloitatton trade-off during training improves perfor-
some games a decreases over time indicating that more exploration required at the end of training. An example is the game Carnival where EIPO-RND shows a jump in performance at the end of training, which is accompanied by a decrease in a (see Appendix A.8). These observations suggest that simple strategics that follow fixed schedule to adjust the balance between intrinsic and extrinsic rewards across tasks will perform worse than EIPO. The result that P(EIPO-RND > DY)= 0.59 reported in Fig. 3b confirms our belief and establishes that dynamic, game specific tuning of the
trade-off is necessary for good performance.
4.4 Does EIPO improve over RND only in Easy Exploration Tasks?
The observation that EIPO-RND outperforms RND on most games might result from performance improvements on the majority easy exploration games, at the expense worse performance on the minority of hard exploration games. To mitigate this possibility, we evaluated performance on games where RND outperforms EO as a proxy for hard exploration. The PPO-normalized scores [21] and 95% confidence intervals of various methods are reported in Table 1. EIPO-RND achieves slightly

---

higher mean score than Ext-norm-RND. but lies within the confidence interval. This result suggests that EIPO not only mitigates the bias introduced by intrinsic rewards in easy exploration tasks, but either matches or improves performance in hard exploration tasks.
5
Discussion
EIPO can be a drop-in replacement for any RL algorithm. Due to inconsistent performance across tasks and difficulties tuning intrinsic reward scaling, bonus based exploration has not been part of standard state-of-the are RL pipelines. EIPO-RND mitigates these challenges, and out- performs noisy networks [22] an exploration method that does not use intrinsic rewards, but
Table 1: EIPO-RND is either at-part or outper- forms RND in the games where RND is better than EO (i.e., hard-exploration tasks).
PPO-normalizer score
Algorithm
Mean (CI)
RND
384.57
(85.57, 756.69)
Ext-norm RND
427.08
(86,53, 851.52)
represents the best performing exploration strat- egy across ATARI games when used with Q- learning [17] (see Appendix A.9). The consistent performance gains of EIPO-RND on ATARI and MuJoCo benchmarks make it strong contender become the defacto RL exploration paradigm going forward. Though we performed experiments with PPO, the EIPO objective (Eq. 3.1) is agnostic to the particular choice of RL algorithm. As such, it can be used as drop-in replacement in any RL pipeline.
Decay-RND
383.83
(84.19, 753.17)
Decoupled-RND
1.54
(1.09, 2.12)
HIPO-RND
435.56
(109.45, 874.88)
Limitations. Across all 61 ATARI games, we use the same initial value of a and demonstrate that per-task tuning of intrinsic reward importance can be avoided by optimizing a. However, it is possible the choice of intrinsic reward function.
that the initial value of and learning step-size 3 depend To see if this is the case, we tested EIPO with another intrinsic reward metric (ICM [8] using the same initial and step-size B. The preliminary results in Appendix A.10 show that performance gains of EIPO-ICM over the baselines are less consistent than EIPO-RND. While one possibility that RND is better intrinsic reward than ICM, the other possibility is that close dependencies between EIPO hyperparameters and the choice of intrinsic reward function come into play. We leave this analysis for future work. Finally, it's worth noting that the performance benefits of EIPO are limited by the quality of the underlying intrinsic reward function on individual tasks. Finding better intrinsic rewards remains exciting avenue of research, which we hope be accelerated by removing the need for manual tuning using EIPO.
Potential applications to reward shaping. Intrinsic rewards can be thought of as a specific instance of reward shaping [23] - the practice of incorporating auxiliary reward terms to boost overall task performance, which is key to the success of RL in many applications (see examples in [24, 25]). Balancing auxiliary reward terms against the task reward is tedious and often necessitates extensive hyperparameter sweeps. Because the EIPO formulation makes assumptions specific to intrinsic rewards, its success in balancing the RND auxiliary reward suggests it might also be applicable in other reward shaping scenarios.
Acknowledgments
We thank members of the Improbable AI Lab for helpful discussions and feedback. We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources. This research was supported in part by the MIT-IBM Watson AI Lab, an AWS MLRA research grant, Google cloud credits provided as part of Google-MIT support, DARPA Machine Common Sense Program, ARO MURI under Grant Number W91 INF-21-1111 --1-2-1---- -1-022, ONR MURI under Grant Number 1-001--22-1-2-444 and by the United States Air Foree Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-199 19-2000000-000.... The views and conclusions contained this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Author Contributions
Eric Chen developed heuristic- based methods for balancing intrinsic and extrinsic rewards which led to the intuition for formulating the cstrinsic-intrinsi optimality constraint. He provided critical input during EIPO prototyping. Implemented baselines, ran experiments scale, and helped with paper writing.
10

---

Zhang- Wei Hong conceived of the extrinsic-intrinsic optimality constraint, derived and developed the first prototype of EIPO, small-scale experiments, played primary role in paper writing and advised Eric.
Joni Pajarinen provided feedback on the technical correctness of the EIPO formulation and writing.
Pulkit Agrawal conceived and oversaw the project. He was involved in the technical formulation, rescarch discussions, paper writing, overall advising and positioning of the work.
References
[1] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
[2] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. 2020.
[3] Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pages 3199-3210. PMLR, 2020.
[4] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, 2016.
[5] Jurgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. From animals to animats: Proceedings of the first international conference on simulation of adaptive behavior, 1991.
[6] Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning, Advances in neural information processing systems, 17, 2004.
[7] Fierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 2009.
[8] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. Proceedings of the 34th International Conference on Machine Learning, pages 2778-2787, 2017.
[9] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL
/ttps://openrevial net/forrmidddHiiee
[10] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-586 2021.
[11] Adrien Ali Taiga, William Fedus, Marlos Machado, Aaron Courville, and Mare G Bellemare. Benchmarking bonus-based exploration methods on the arcade learning environment. arXiv preprint arXiv:9908, 02388, 2019.
[12] William F Whitney, Michael Bloesch, Jost Tobias Springenberg, Abbas Abdolmalcki, Kyunghyun Cho, and Martin Riedmiller. Decoupled exploration and exploitation policies for sample-efficient reinforcement learning. arXiv preprint wxXiv:2001.09458 2021.
[13] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1107.066. 2017.
[14] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of curiosity-driven learning. arXiv preprint
2018.
[15] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference Machine Learning. Citeseer, 2002.
[16] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1889-1897, 2015.

---

[17] Volodymyr Mnih. Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
[18] Lukas Schäfer, Filippos Christianos, Josiah P. Hanna, and Stefano V. Albrecht. Decoupled reinforcement learning to stabilise 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22.
exploration. In Proceedings of the
[19] Thomas Stepleton. The pycolab game engine, 2017. URL htpps//ggithbb com/deepmiiddyycolab. 2017.
[20] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[21] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[22] Meire Fortunato. Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint 1rXiv:1706.10022 2017.
[23] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In Ieml, volume 99, pages 278-287, 1999.
[24] Gabriel B Margolis, Ge Yang. Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. Robotics: Science and Systems, 2022.
[25] Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. Conference Robot Learning, 2021.
[26] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. International Confer- ence of Representation Learning, 2015.
[27] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1026-5033. IEEE, 2012.
[28] Cetin Merigli, Tekin Merigli, and H Levent Akin. A reward function generation method using genetic algorithms: a robot soccer case study. AAMAS, pages 1513-1514, 2010.
[29] Jonathan Sorg, Richard L. Lewis, and Satinder Singh. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23, 2010.
[30] Xiaoxiao Guo, Satinder Singh, Richard Lewis, and Honglak Lec. Deep learning for reward design to improve monte carlo tree search in atari games. arXiv preprint 1xXxv:1604.00055 2016.
[31] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018.
[32] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pages 11436-1 1446. PMLR, 2020.
[33] Yujing Hu, Weixun Wang. Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 13:15531-15541 2020.
12

---

[34] Xi Chen, Ali Ghadirzadeh, Märten Björkman, and Patric Jensfelt. Meta-leaming for multi- objective reinforcement learning. 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 977-983. IEFE, 2019.
[35] Christian Shelton. Balancing multiple sources of reward in reinforcement learning. Advances in Neural Information Processing Systems, 13, 2000.
[36] Stefan Ultes, Pawel Budzianowski, Inigo Casanueva, Nikola Mrkšić, Lina Lojas-Barahona Pei-Hao Su, Tsung- Hsien Wen, Milica Gašić, and Steve Young. Reward-balanciee for statis- tical spoken dialogue systems using multi-objective reinforcement learning. arXiv preprint arXiv:1707.06299 2017.
[37] Runzhe Yang. Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. Advances in Neural Information Processing Systems, 32, 2019.
[38] Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International Conference on Machine Learning, pages 11-20. PMLR, 2019.
13


Reviews:
[{"review_id": "bpPQQl6sbLj", "reviewer": "Reviewer_sfNt", "summary": "The authors pose the problem of exploration using intrinsic motivation as one of constrained optimization.\nNaturally, the goal of the agent maximizing the intrinsic+extrinsic reward should be to perform AT LEAST as good as an agent simply maximizing the intrinsic reward.\n\nAs such, they present a constraint where the E+I agent needs to perform at least as good as the E agent when measured only on E (extrinsic) rewards.\n\nEmpirical evidence shows this method indeed works.", "questions": "My main question is around section 3.3 (and 3.2)\n\nWhy collect data using pi_{E+I} when optimizing pi_{E}? Why not optimize pi_{E} when using data collected by pi_{E}?\nWhy not optimize both agents in parallel at each time? Is there a restriction against optimizing pi_{E} and pi_{E+I} in both max and min stages, just making sure to use the appropriate importance sampling ratios?", "limitations": "-", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Motivation is clear. The approach is solid.\n- Empirical results look very good.\n- Paper is overall well written.\n\nWeaknesses:\n\n- Wouldn't expect strong theoretical justification, but overall this is an optimization objective with 3 parts that can be presented as a 2-3 time-scale stochastic optimization scheme and convergence can be analyzed. Convergence analysis might shed some light on section 3.3 which seems a bit hard to comprehend.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q-j0W2KkmeG", "reviewer": "Reviewer_bbkw", "summary": "This paper focuses on the exploration-exploitation trade-off in reinforcement learning. In particular, the authors aim to devise a method that automatically adjusts the importance of the intrinsic reward component (exploitation) and the extrinsic reward component (exploration). To this end, they pose an (extrinsic optimality) constrained policy optimization. Then, they write the Lagrangian dual problem and solve it iteratively. For implementation purposes, they leverage tools from TRPO/PPO literature. The proposed EIPO method is compatible with any intrinsic reward method; for experiments, they use RND. Finally, they conducted an extensive empirical investigation of their proposed method on all 61 Atari games.    ", "questions": "Question: \n\nHow does EIPO relate with (or compare against) the line of works in [Zheng et al. 2018; Hu et al. 2020] that also consider automatic learning/adjustment of intrinsic reward component? \n\n[Zheng et al. 2018] Zheng et al. On learning intrinsic rewards for policy gradient 376 methods. 2018.\n\n[Hu et al. 2020] Hu et al. Learning to utilize shaping rewards: A new approach of reward shaping. 2020.", "limitations": "Their proposed method does not seem to cause any direct potential negative societal impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method is interesting and novel. The existing intrinsic reward-based exploration methods (e.g., RND) need to heavily tune the parameter $\\lambda$ in $r_E + \\lambda r_I$ for each environment to perform better. EIPO alleviates this manual tuning effort and automatically adjusts the importance scale $\\alpha$ between the intrinsic and extrinsic rewards via optimization. Further, EIPO is compatible with any intrinsic reward method. In this work, the authors do not directly learn or optimize the intrinsic reward $r_I$.  \n\nThe paper is overall well written; the motivation and the story are well conveyed. It was easy to follow.\n\nOn the empirical side, the authors have done a large-scale study on all 61 Atari games and reported the results using rigorous metrics as prescribed in [18]. In particular, the empirical results validate the following: (i) the probability of improvement EIPO over PPO is higher than the baselines (variants of RND), (ii) EIPO performs better than all the baselines in the majority of games (as well as in terms of strict probability of improvement), (iii) EIPO performs comparable/better than heavily $\\lambda$-tuned RND (for each environment). These empirical findings make the paper strong. \n\n****\n\nWeakness:\n\nThe paper is missing a detailed discussion of related work. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4WnFu4IOfSd", "reviewer": "Reviewer_JLyn", "summary": "Difficult exploration tasks which rely on sparse rewards can be particularly challenging for traditional RL methods. Injecting intrinsic reward functions can increase reward density, but also introduces bias that can hinder performance in more tractable exploration tasks. Regulating this bias often reduces to a balancing act between intrinsic vs extrinsic reward influence using manual hyperparameter tuning. This paper proposes an objective function that explicitly minimizes the difference between the learned, combined intrinsic + extrinsic policy and an optimal policy under only the extrinsic reward. The objective function is reformulated using lagrangian duality and the policy learning is constrained using TRPO to provide tractability. The method is then evaluated on 61 Atari exploration tasks that vary in difficulty. Results show that the proposed method can match state of the art performance on difficult tasks while still achieving PPO-level performance on simpler tasks, all without manual tuning of the reward balance.", "questions": "I have no major questions or suggestions, other than the minor suggestions provided above.", "limitations": "I would have liked to see more discussion around the limitations of this approach, or possible applications where this may not be appropriate.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Originality\n\nThe core concept is simple but elegant.  Formulation of the minimization constraint between mixed- and extrinsic-trained policies, reformulation using langrange duals, and the use of a TRPO-like objective to improve trainability were all creative. Overall, I like that the idea is approachable but very effective.\n\n## Quality\n\nThe overall quality of the paper is excellent, with a few minor caveats:\n\n- I thought the abstract was a little weak. It's more pronounced because the rest of the paper is very solid, so the abstract feels thin by comparison. A few suggestions:\n\t- Some may not be familiar with \"intrinsic rewards,\" so maybe highlight the link with reward sparsity. \n\t- Maybe include a few more details about your approach beyond just \"constrained policy optimization\"\n- Very minor, but each word in your title should be capitalized except for \"via\".\n\nOn the positive:\n\n- As mentioned above, the core concept is solid and well-presented.\n- Detailing of the methodology was excellent, from preliminaries through implementation details.\n- The number of experiments and the detail of the analysis were great.\n\n## Clarity\n\nThe paper was clearly written and approachable. The description of the problem and the formulation of the solution follow a coherent narrative and are straightforward to understand.\n\n## Significance\n\nI don't feel like anything in this work is particularly ground breaking, but rather an elegant solution to a specific problem. That said, I do appreciate the breadth of evaluation and it will be interesting to see this approach applied to more real-world scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept