QXplore: Q-Learning Exploration by Maximizing Temporal Difference Error
Authors: Riley Simmons-Edler
Ben Eisner
Daniel Yang
Anthony Bisulco
Eric Mitchell
Sebastian Seung
Daniel Lee
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Exploration

Abstract: A major challenge in reinforcement learning is exploration, especially when reward landscapes are sparse. Several recent methods provide an intrinsic motivation to explore by directly encouraging agents to seek novel states. A potential disadvantage of pure state novelty-seeking behavior is that unknown states are treated equally regardless of their potential for future reward. In this paper, we propose an exploration objective using the temporal difference error experienced on extrinsic rewards as a secondary reward signal for exploration in deep reinforcement learning. Our objective yields novelty-seeking in the absence of extrinsic reward, while accelerating exploration of reward-relevant states in sparse (but nonzero) reward landscapes. This objective draws inspiration from dopaminergic pathways in the brain that influence animal behavior. We implement the objective with an adversarial Q-learning method in which Q and Qx are the action-value functions for extrinsic and secondary rewards, respectively. Secondary reward is given by the absolute value of the TD-error of Q. Training is off-policy, based on a replay buffer containing a mix of trajectories sampled using Q and Qx.  We characterize performance on a set of continuous control benchmark tasks, and demonstrate comparable or faster convergence on all tasks when compared with other state-of-the-art exploration methods.

Under review as a conference paper at ICLR 2020
QXPLORE: Q-LEARNING EXPLORATION BY MAXIMIZ- ING TEMPORAL DIFFERENCE ERROR
Anonymous authors Paper under double-blind review
ABSTRACT
A major challenge in reinforcement learning is exploration, especially when reward landscapes are sparse. Several recent methods provide an intrinsic motivation explore by directly encouraging agents to seek novel states. A potential disad- vantage of pure state novelty-seeking behavior that unknown states are treated equally regardless of their potential for future reward. In this paper, propose exploration objective using the temporal difference error experienced on extrinsic rewards secondary reward signal for exploration in deep reinforcement learn- ing. Our objective yields novelty- seeking in the absence of extrinsic reward, while accelerating exploration of sewardrrlevaatt states in sparse (but nonzero) reward landscapes. We implement the objective with two-policy Q-learning method in which Q and Q.. are the action-value functions for extrinsic and secondary rewards, respectively. Secondary reward is given by the absolute value of the TD-error of Q. Training is off-policy. based on a replay buffer containing a mix of trajectories sampled using Q and Qz. We characterize performance on set of continuous control benchmark tasks, and demonstrate comparable or faster convergence on all tasks when compared with other state-of-the-art exploration methods.
INTRODUCTION
Deep reinforcement learning (RL) has recently achieved impressive results across several challenging domains, such as playing games (Mnih et al. 2016 Silver et al. 2017 OpenAl| 2018 Baker et al.) 2019) and controlling robots (OpenAl et al. 2018 Kalashnikov et al., 2018). In many of these tasks, well-shaped reward function is critical to learning performant policies. On the other hand, deep RL still remains challenging for tasks where the reward function is sparse. In these settings, state-of-the-art RL methods often perform poorly and train very slowly, at all, due to the low probability of observing improved rewards by following the current optimal policy with a naive exploration policy such as c-greedy sampling.
The challenge of learning from sparse rewards is typically framed as problem of exploration, inspired by the notion that a successful RL agent must efficiently explore the state space of its environment in order to find improved sources of reward. One common exploration paradigm is to directly determine the novelty of states and to encourage the agent to visít states with the highest novelty. In small MDPs this can be achieved through counting how many times each state has been visited. This approach often performs poorly in high-dimensiona or continuous state spaces, but recent work (Tang al. 2017 Bellemare et al. [0066] Fu et al. 2017) using count-like statistics have shown success on benchmark tasks with complex state spaces. Another paradigm for exploration learns a dynamic model of the environment and computes novelty measure proportional to the error of the model in predicting transitions in the environment. This exploration method relies on the core assumption that well-modeled regions of the state space are similar to previously visited states and thus are less interesting than other regions of state space. Predictions of the transition dynamics can be directly computed (Pathak et al., 2017 Stadic et al.) 2015] (aavinvv et al. 2019 Burda al. (019a)), related to information gain objective the state space, described in VIME (Houthooft et al. 2016) and EMI (Kim et al. 2018).
Several exploration methods have recently been proposed that capitalize the function approxima- tion properties of neural networks. Random network distillation (RND) trains a function to predict the output of andomll-initialize neural network from input state, and uses the approximation

---

Under review as a conference paper at ICLR 2020
error as a reward bonus for separately -rrinne RL agent (Burda al. 2019b). Similarly, DORA (Fox et al. 2018) trains a network to predict zero on observed states and deviations from zero are used to indicate unexplored states.
An important shortcoming of existing exploration methods is that they only incorporate information about states and therefore assume all unobserved states are equally motivating, regardless of their viability for future reward. The viability of this assumption is highly task dependent: While games like Montezuma's ss Revenge or Super Mario Bros, where novelty correlates highly with success, can be attacked effectively by state novelty methods alone (Burda et al. 2019b; Pathak et al.) 2017 Ecoffet et al., 22119] Kim et al. 2018). other tasks such as hide-and-seek or some Atari games where novelty and utility are less correlated tend to frustrate state novelty methods (Burda et al., 2019b]
Baker et al., 2019 Burda et al. 2019a). Baker et al. (2019) explored using both RND and a simple state counting baseline to discover skills such as navigation and block-pushing in hide-and-sedd environment. However, the authors found that careful construction of the state representation used for novelty seeking was necessary to discover any such skills, as novelty in the full state space did not correspond to novelty in the intuitive sense (Baker et al. 2019).
Instead of focusing on the state-space, this work uses the temporal difference error (TD-error) which provides a signal into novelty in the reward landscape. Past works have also utilized information from the reward landscape
learning signal to various extents. Schmidhuber et. al. first describe using reward misprediction and model prediction error for exploration (Schmidhuber) 1991 Thrun & Möller 1991 1992). However, the work was primarily concerned with model model-building and system- identification in small MDPs, and used reward prediction error rather than TD-error. Later, Gehring & Precup (2013) used TD-error as negative signal to constrain exploration focus on states that are well understood by the value function to avoid common failure modes. Related to maximizing TD-error is maximizing the variance or KL-divergence of a posterior distribution over MDPs or O-functions, which can be used as measure of uncertainty (Osband & Van Roy| 2017 O Donoghue al. 2017: Chen et al. 2017 Fox et al. 2018 Osband et al., 2018). Posterior uncertainty over Q-functions can be used for information gain in the reward or Q-function space, as opposed to information gain in the state space as described by VIME among others (Houthooft et al. 2016 Kim et al. 2018), though posterior uncertainty methods have thus-far largely been used for local exploration as an alternative to dithering methods such as e-greedy sampling, though Osband et al. (2018) do apply posterior uncertainty Montezuma's Revenge.
In this paper we propose QXplore, a new exploration formulation that seeks novelty in the predicted reward landscape instead of novelty in the state space. QXplore exploits the inherent reward-space signal from the computation of temporal difference error (TD-error) in value-based RI., and explicitly promotes visiting states where the current understanding of reward dynamics is poor. In the following sections, we describe QXplore and demonstrate its utility for efficient learning on variety of complex benchmark environments with continuous controls and sparse rewards.
PRELIMINARIES
We consider RL in the terminology (uttonn & Barto (1998) in which agent seeks to maximize reward in Markov Decision Process (MDP). An MDP consists of states 8 E S, actions a E A, a state transition function S S A x S + [0. 1] giving the probability of moving to state 8141 after taking action from state st for discrete timesteps t E O, O,.... T. Rewards are sampled from reward function S x A + R. An RL agent has policy T(s;, a.) = p(a,) (s.) that gives the probability of taking action a when in state st. The agent aims to learn policy to maximize the expectation of the time-decayed sum of reward (so) I a;) where a, ((1,,;))
value function Vo(s)) with parameters 0 is a function which computes Vg(*,) R R2 (st) for some policy т. Temporal Difference (TD) error &, measures the bootstrapped error between the value function at the current timestep and the next timestep as
8, Vo(ss) (r(st, a, =T(s)) + Voo
Q-function is a value function of the form Q(5t, at), which computes Q(5c, a)) = r(st, as) + y max. max.: a'), the expected future reward assuming the optimal action is taken each future timestep. An approximation to this optimal Q-function Qo with some parameters may be trained using mean squared TD-error objective L.q. = ||Qo(st, at) - (r(st, az) + --ammmaaaaa Q/ (8e-1, !2
2

---

Under review as a conference paper at ICLR 2020
given some target Q-function Q/r- commonly time-delayed version of Qa (Mnih et al. 2015 Extracting a policy т given Qo amounts to approximating argmax, (a a). Many methods exist for approximating the argmax, operation in both discrete and continuous action spaces (Lillicrap et al. 2015) Haarnoja et al. 2018). Following the convention of Mnih et al. (2016), we train Qo using off polily replay buffer of previously visited (s, as, 8') tuples, which we sample uniformly.
QXPLORE: TD-ERROR AS ADVERSARIAL REWARD SIGNAL
3.1
METHOD OVERVIEW
Env
ar
r
a
Replay Buffer Q
Interaction
Learning
TD-error
Env
at
ft
Q
Replay Buffer Q
Figure I: Method diagram for QXplore. We define two Q-functions which sample trajectories from their environment and store experiences in separate replay buffers. Q is standard state-action value- function, whereas Q2 's reward function is the unsigned temporal difference error of the current Q on data sampled from both replay buffers. A policy defined by Qu samples experiences that maximize the TD-error of Q, while a policy defined by Q samples experiences that maximize discounted reward from the environment.
We first provide an overview of the method- visual representation is depicted in Figure [ At a high level, QXplore is an exploration method that jointly trains two independent agents equipped with their own Q-functions and reward functions:
1. Q: A standard Q-function, that learns a value function on reward provided by the external environment.
2. Qe: A Q-function that learns a value function directly on the TD-error of Q-
The policy TQ_ that samples Qz and the Q-function Q form an adversarial pair, wherein TQ, seeks to sample state-action pairs that produce large TD-errors while Q's training objective Lq, attempts to minimize the TD-error for previously sampled state-action pairs. Thus, TQ², achieves reward when the agent ventures into states whose reward dynamics are foreign to Q (i.e. Q under/overrssimates reward achieved). Separate replay buffers maintained for each agent, but each agent receives samples from both buffers at train time. A similar adversarial sampling scheme was used to train an inverse dynamics model by Hong et al. (2018), and Colas et al. (2018) use separate goal-driven exploration and reward maximization phases for efficient learning, but to our knowledge parallel adversarial sampling policies have not previously been used for exploration.
3.2 TD-ERROR OBJECTIVE
We directly treat TD-error as reward signal and use Q-function trained on this signal to induce an exploration policy, rather than as a supplementary objective or to compute a confidence bound. Cru- cially, when combined with neural network function approximators, this signal provides meaningful exploration information everywhere as discussed in Section 3.4 For a value function with parameters 8, and TD- d, we define exploration reward function as
(2) for some extrinsic reward function TE and target Q-function Q/- Notably, we use the absolute value of the temporal difference (rather than the squared error) used to compute updates for Q9 to keep the magnitudes of oF and r, comparable and reduce the influence of outlier temporal differences on the gradients of Qz, which we describe below.
Ta a (St, s1+1) |6,1 = (Q, a.) (rE(st, at) + yymax Qo (s:-1,
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 QXplorre Algorithm
Input: MDP S, Q-function Qo with target Qg Q₂ function Qz,o with target Q 2z replay buffers ZQ and ZQ batch size B and sampling ratios RQ and RQ.. CEM policies no and TQ., time decay parameter 7,, soft target update rate T, and environments Eq. Eq_ while not converged do Reset Eq. Eq. while Eq and Eq, not done do
Sample environments
2q + (s, (s,a, To Eq 2q. t (s.a,r, 8') TQ, Eq.
Sample minibatches for Qo and Qr
(sq, TQ, () < B * Ro samples from 2q and B * (1 R2) samples from Zo.
(sq_ (Q., TQ= (Q2) t B * RQ, samples from ZQ and B (1 Ro_) samples from Zo
Train
22 QQ((sq.; )Q (rq. + 1 Qu (0(('q.))))
Lę < 1Q0(sQ, AQ) (rq t 1 (5 q: 70 Q))
Lq= + 11Q,,0(9 (Q.) (ra, + 20' (8 Q To, (s' (f)Q.)))1
Update 0 x LQ
Update @ x T.g.
0' f- (1 7)0' +
- (1 - 7) )) TO
end while end while
Intuitively, a policy maximizing the expected sum of T. will sample trajectories where Qo does not have accurate estimate of the future rewards it will experience. This is useful for exploration because Ta will be large not only for state-action pairs producing unexpected reward, but for all state- action pairs leading such states, providing much denser exploration reward function. Further, TD-error-basee baror-based exploration with dedicated exploration policy removes the exploration-vere verrats exploitation tradeoff that state-novelty methods must contend with, where trajectories maximizing state novelty often do not also maximize reward. Separate exploration and exploitation policies allow us to sample trajectories maximizing Tz that provide information about the task for Qo to train on without impacting its ability to maximize reward.
3.3 Qxi LEARNING A Q-FUNCTION MAXIMIZE TD-ERRON
Next, we will describe how we use the TD-error signal defined Section 3.2 to define an exploration policy. The reward function r. is generic, and can be maximized by any RI. algorithm. However, given its derivation from a bootstrapped Q-function, training second Q-function to maximize T= allows the entire algorithm to trained off-policy with two replay buffers that share data between Qo and the Q- Q-function maximizing Tz, which we term Qa- This approach is beneficial for exploration, as avoids needing trade off between exploration and exploitation via weighting hyperparameter, and sharing data between replay buffers improves data efficiency for training both Q-functions.
We define a Q-function, (s, a) with parameters Φ, whose reward objective Tz. using the standard bootstrapped loss function
We train Qa,o
Lo. 11Q (00,)((8, ar) (rz(se, at, 114)) + 7.max, Qn (se+1,
(3)
The two Q-functions, Qo and Q.. are trained off off-policy parallel, sharing replay data so that Qo can train on sources of reward discovered by Qr and so that Q2 can better predict the TD-errors of Qo- Since the two share data, TO, acts as an adversarial teacher for Qs. sampling trajectories that produce high TD-error under Qo and thus provide novel information about the reward landscape. To avoid off-policy stability issues due to the different reward objectives, we sample fixed ratio of experiences collected by each policy for each training batch. We use nonparametric cross-entropy method policy inspired by Kalashnikov et al. (2018), previously described as more robust to hyperparameter variance (Simmons-Fade Fdler et al. 2019 Kalashnikov et al. 2018). We also experimented with a variant using
4

---

Under review as a conference paper at ICLR 2020
DDPG-style parametric policies (Lillierap et al. 2015) for both Qo and Q² but found preventing sampling collapse by Q8 policy difficult. Our full method is shown in Figure and pseudocode in Algorithm [
3.4 STATE NOVELTY FROM NEURAL NETWORK FUNCTION APPROXIMATION ERROR
A key question in using TD-error for exploration is: What happens when the reward landscape is flat? Theoretically, in the case that ((s, a), r(s, a) for some constant E R, an optimal Q-function which generalizes perfectly to unseen states will, in the infinite time horizon case, simply output a) a) En= Cy This results in a TD-error of everywhere and thus no exploration find that perfect generalization
signal. However, using neural network function approximation, pairs does not occur, and in fact observe Figure 2 that the distance of new datapoint from the training data manifold correlates with the magnitude of the network output's deviation from Σt=1 Chy² and thus with TD-error. As a result, in the case where the reward landscape flat TD-error exploration converges to form of state novelty exploration. This property of neural network function approximation has been used by several previous exploration methods to good effect, including RND (Burda et al., 2019b) and DORA (Fox et al., 2018). In particular, the exploration signal used by RND (extrapolation error from fitting the output of a random network) should be analogous to Tx (extrapolation error from fitting a constant value), meaning we should expect to perform comparably to RND in the worst case where no extrinsic reward exists.
to unseen
0.20
0.32
40
Figure 2: A neural network trained to predict constant value does not interpolate extrapolate well outside its training range, which can be exploited for exploration. Predictions of 3-layer MI.Ps of 256 hidden units per layer trained imitate (() 0 on IR R with training data sampled uniformly from the range [-0.75, -0.25] U [0.25, 0.75J. Each line is the final response curve of an independently trained network once its training error has converged (MSE- < le-7).
EXPERIMENTS
We performed several experiments demonstrate the effectiveness of Q. on continuous control benchmark tasks. We compare QXplore with related state of the art state novelty based method, RND (Burda et al. 2019b), DORA (Fox et al. 2018), and with e-greedy sampling as a simple baseline. Each method is implemented in shared code base on top of TD3 Fujimoto et al. (2018b) using cross entropy method policy as proposed by Qt-Opt Kalashnikov et al. (2018) ) for hyperparameter stability. We also compare to results from several previous works Sparsella fCheet Finally, we present several ablations to QXplore, as well as analysis of its robustness in response to several hyperparameters. Implementation details and hyperparameters for QXplore, RND, DORA, and c-greedy can be found Appendix A
4.1 EXPERIMENTAL SETUP
We benchmark on four continuous control tasks using the MuloCo physics simulator that each require exploration due to sparse rewards. First, the SparseHal SeparseaalfChhetaae fCheetah task originally proposed by VIME (Houthooft et al. 2016). Next, we benchmark on three OpenAl gym tasks, Fet chPush, Fetchslide and FetchPickAndP lace, originally developed for goal-directed exploration methods such as HER (Andyychoicc et al.) 2017). We chose these tasks as they are challenging exploration problems that relatively simple control, but still involve large continuous state spaces and in the case of the Fet ch tasks learning to generalize across random object/goal posi- tions. For consistent reward shaping across tasks we used a reward function in the range [-1-0] for
5

---

Under review as a conference paper at ICLR 2020
SparseHal SCheettttecheeeae ah similar to the Fetch tasks, but results the original reward function from Houthooft et al. (2016) can be found in Appendix E where we perform comparably. We ran random seeds for each experiment. More details these environments can be found in Appendix B
4.2
EXPLORATION BENCHMARK PERFORMANCE
300
902
48
Jouns
500000
1003820
1500000
500003
3082830
1500000
greedd
epicites
(a) SparseHalCCheetah..
(b) FetchSlide-vl -1
-12
-10
-02
-0
Dapare loann
FAC
spcartc
RNC
(c) FetchPush-vl
(d) FetchPickAnPPlace-1
Figure 3: Performance of QXplore compared with RND and e-greedy sampling. QXplore outperforms RND and e-greedy continuous control tasks. QXplore performs better due to efficient exploration sampling by Qr and the separation of the exploration and exploitation objectives. Q indicates performance of our exploitation Q- f-ffuction, while Q₂ indicates the performance of our exploration Q-function, whose objective does not directly maximize reward but which may lead to high reward regardless.
Episodes until mean reward of
QXplore
VIME
EX2
EMI
GEP-PG
DORA
SimHash
50
2000
10000*
4740
2580*
NA
x
100
3000
x
6180
4520
4000
X
200
4000
x
X*
8440*
X
300
7900
x
X
x
X
X
X
Table 1: Number of episodes required to reach mean reward milestones on SparseHal FCheetah for several methods. QXplore outperforms previously published methods. Results marked with **" are previously published numbers. VIME from Houthooft et al. (2016), EMI and EX2 from Kim et al. (2018), and SimHash from /Tag al. (017)) Results marked with "x" indicate that the mean reward was not achieved. For GEP-PG (Colas et al. 2018) we used the author's implementation, which did not permit easy evaluation of intermediate performance.
We show the performance of each method on each task in Figure QXplore outperforms RND modestly on the sparseHal! fCheetah task, but performs much better comparatively on the Fetch tasks- only on FetchPush the easiest task, did RND find non-random reward. We theorize that this improved performance on the Fetch tasks is because QXplore's TD-error exploration drives the agent to discover the conditional relationship between the changing goal position and the reward function, whereas RND and other state novelty methods are goal- -ggoosic since the goal is statie for the entire episode. While QXplore is not goal-directed RL method, and does not achieve state-of thettheart performance compared to dedicated coal-directed RL methods, the fact that this relationship is discovered through T'-error exploration is encouraging to its broader applicability.

---

Under review as a conference paper at ICLR 2020
We compare to several other exploration methods is in Table The methods from previous work are built on top of TRPO (Schulman et al., 2013). so a comparison in terms of training iterations as in Figure 3 would not be informative due to TRPO's variable update rule. We instead compare the number of episodes of interaction required to reach a given level of reward, though QXplore was not intended to be performant with respect to this metric. While some decrease in episode efficiency is expected due to differing baseline methods (TRPO (Schulman et al. (0015) versus TD3 Fujimoto al. (2018b)), compared published results for EMI (Kim et al. 2018), EX2 (Fu et al. 2017). VIME (Houthooft et al., 2016). and SimHash (Tang et al., 2017) on the SparseHa ECheetal task, QXplore reaches every reward milestone faster, and achieves peak reward (300) not achieved by any previous method.
We also include here the performance of our implementation of DORA (Fox et al. 2018 on Spa rseHal 1 FCheetah. DORA performed poorly, possibly because it was not intended for use with continuous action spaces, and thus we did not test it on other tasks.
Finally, we compare to GEP-PG (Colas et al., 2018), which used separate exploration and exploitation phases similar to QXplore. We downloaded the author's implementation (built on top of DDPG) and tested it on Sparreefalfe ffhhetaa using the parameters for the Hal welefffhhhahhhte originally tested on. The author's implementation did not facilitate evaluating performance midway through training, and thus we report only their final performance number after 4000 episodes, which was 120.2.
task it was
4.3 ROBUSTNESS
As RL tasks are highly heterogeneous, and good patamettrizatto//efffmmanee can hard to obtain in practice for many methods (Henderson et al. 2018), we performed sweeps over several hyperpa- rameters and introduce several ablations of QXplore on SparsellalfCheetaal to demonstrate the method's robustness and validate aspects of the algorithm.
We swept over the learning rates of Q and Q2, as well as the ratio of self- collected versus other-collected data used train each function. The results suggest that while the performance of Q is somewhat sensitive to learning rate, keeping learning rates for Q and Qr the same works well. The results also show that while our ratio of 75% self/25% non-self performs best, Q is fairly robust to the on/off-policy data ratio, including when Q is trained entirely off-policy on data collected by Q.. Results shown in Figures and in Appendix D
Parameter Sweeps
Weight Initialization Also, since neural network generalization is key to QXplore, we tested several different network weight initialization schemes, including some that were deliberately poor priors. We found that while the performance of Q is sensitive to initialization scheme, Qu robustly finds reward in all cases. See Figure 12 in Appendix G
The 'Noisy TV' Problem One drawback that naive state novelty exploration methods have is that un- predictable observations (such from a TV displaying static) act as maxima in the exploration reward function. Naive methods are unavoidably drawn to such states instead of exploring. TD-error driven exploration is not sensitive to unpredictable observations as they do not affect the underlying reward function. To demonstrate this, we tested QXplore with variant of the Sparse Ha ECheet ah task with noisy observations. We observe that QXplore performs as normal in this case. A description of the task can be found in Appendix F
4.4 ABLATIONS
There are two features of QXplore that distinguish it from prior work in exploration: the use of a pair of policies that share replay data, the use of unsigned TD-Error to drive exploration. We performed several ablations that assess the impact of aspects of each of these features. Detailed results can be found in Appendix
Single-Polic QXplore First, we test single-policy version of QXplore by replacing Qu(s, a) with a value function (s). We use a value function rather than f-fucction in this case to avoid large estimation errors stemming from fully off off-policy training such reported by FFjjimto al. (2018a). We observe in Figure that while the policy is able to find reward quickly and converge faster, the need to satisfy both objectives results in lower converged reward than the original QXplore method.
7

---

Under review as a conference paper at ICLR 2020
1-Step Reward Prediction Second, that simply predicts the current r(s,, at). Using reward error instead value function in Qx can still produce the same state novelty fallback behavior in the absence of reward; however, it provides only limited reward-based exploration utility. We tested this variant and observe in Figure 5 that fails to sample reward. Reward prediction error is not sufficient allow strong exploration behavior.
ablation where replaced Qo(s.a) a) with function
QXplore with State Novelty Exploration To assess the importance of TD-error specifically in our two policy algorithm, replaced the TD-error maximization objective of Qs with the random network prediction error maximization objective of RND, while still performing separate rollouts of each policy. The results are shown in Figure 6 We observe that while the modified Q₂ function does sample reward, it is too infrequent to guide Q to learn the task, and further that the modified Qz function does not display directional preference in exploration once reward is discovered.
QXplore with Signed TD-Error Objective While we used unsigned TD-error to train Qz. we also tested QXplore using signed TD-error. We used the negative signed TD-error -ē, from equation so that better-than-expected rewards result in positive Ta values. The results of this experiment are shown Figure The unsigned TD-error performs better on SparseHal fCheetah.
4.5 QUALITATIVE BEHAVIORAL ANALYSIS
Qualitatively, SparseHal 1Chhetta observe interesting behavior from Q2 late in training. After initially converging to obtain high reward, Q2 appears to get "bored" and will focus on the reward threshold, stopping short jumping back and forth across it, which results in reduced reward but higher TD-error. This behavior is distinctive of TD-error seeking over state novelty seeking, as such states are not novel compared to moving past the threshold but do result in higher TD-error. Such behavior from Q₂ motivates Q to explore the state space around the reward boundary. Example sequences of such behaviors are shown in Figure
"Fake-Out"
and
Re-cross"
Figure 4: Example trajectories showing Q, behavior late in training that distinctive of TD-error maximization. The corresponding Q network reliably achieves reward at this point. In "fake-out", Qn approaches the reward threshold and suddenly stops itself. In "cross and re-cross", Q. crosses the reward threshold going forward and then goes backwards through the threshold.
DISCUSSION AND CONCLUSIONS
Here, we have described new method for using TD-error to explore in reinforcement learning. We instantiate reward function using TD-error, and show that when combined with neural network approximation, sufficient to discover solutions to challenging exploration tasks fewer training iterations than recent state novelty-based exploration methods. We hope that our results can spur further work on diverse exploration signals in RL.
It is also worth noting that there may be additional benefits provided by Qu for Q learning in non- exploration contexts. Maximizing TD-error can be seen as a form of hard example mining, and for complex tasks could result in better generalization behavior and faster transfer to tasks through efficient trajectory sampling by Q.-
One potential future area of investigation is in method's connection biological models of dopamine pathways in the brain where levels of dopamine correlate with TD-error learning trials (Niv et al. 2005). phenomenon previously described in animals (Arias 'Arias-Carrón & Pöppell 2007).
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welin- der, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay, In Guyon, U V Luxburg. S Bengio, H Wallach, R Fergus, S Vish- wanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5048-5058. Curran Associates, Inc., 2017. URL htto--hinniigttteepeeeeeeeeeeeee reeppprraenceeeeplyy
Óscar Arias- Carai-Carriónn and Ernst Pöppel. Dopamine, learning, and reward-seeking behavior. Acta neurobiologiae experimentalis, 2007.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint VxXiv:1999007 2019.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-Scale Study of Curiosity-Driven Learning. In International Conference on Learning Repre- sentations, 2019a. URL.ftttps://oenneeriew hwtpreewwwwwweeeeeeeeeeeeeeeeeee wttp://e///ooooomerrmrmmmo
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2019b. URL https
//openreview..og .et.eeewweeeeeeee...e http:/iiwwwee///orrrrree
Richard Y Chen, Szymon Sidor, Picter Abbeel, and John Schulman. Ucb exploration via q-ensembles.
arXiv preprint arXiv:1706.01500 2017.
Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. Gep-pg:
Decoupling exploration and 2018.
exploitation in deep reinforcement learning algorithms. arXiv preprint
Adrien Ecoffet. Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-Explore:
New Approach for Hard Eanl--Expoortioo Problems, 2019.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. (DORA} The Explorer: Directed Outreaching Reinforcement Action- In International Conference on Learning Representations, 2018.
URL htttpss://ppeeeeeieeee net. forum? id=rylarUcCh
Justin Fu, John Co-Reyes, and Sergey Levine. EX2: Exploration with Exemplar Models for Deep Re- inforcement Learning. In I Guyon, V Luxburg, Bengio, H Wallach, R Fergus, Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2577-2587. Curran Associates, Inc., 2017. htt.ttttss//axxii.ooo
abs/ 01260
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint 1etp:028200000 2018a.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596, Stockholmsmässsn,, Stockholm Sweden, 2018b, PMLR. URLhttp /pprocediings.o mlr www.:.a................e.ee.mee.
press v80 http///00//////t// 1tttttllm...mmmlm......mm
Clement Gehring and Doina Precup. Smart Exploration in Reinforcement Learning using Absolute Temporal Difference Errors. In Autonomous Agents and Multiagent Systems (AAMAS), 2013. ISBN 978- 778-1-4503-1993-55
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.

---

Under review as a conference paper at ICLR 2020
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Max- imum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmässsan, Stockholm ml prw.sss//00/haaaooooaaaa
Sweden, 2018. PMLR. htsp::////aaaaiiiiassrrr//////ar html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenct classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Peter Henderson, Riashat Islam. Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelli- gence, 2018.
Zhang-Wei Wei Hong, Tsu-Jui Fu, Tzu- Yun Shann, Yi-Hsiang Chang, and Chun-Yi Yi Lee. Adversarial exploration strategy for self-supervised imitation learning. CoRR, ubs/1806.10013 2018. URL
http://arxiy> org/ hbggg//bs/// attp/b/s//11/ 10019
Rein Houthooft, Xi Chen, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational Information Maximizing Exploration. In D D Lee, M Sugiyama, V Luxburg. I Guyon, and R Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1109-1117. Curran Associates, Inc., 2016. URL (ttLhtt::// http::://paaerr.ct ptpppppess.ips...ppppp.p.ee 6591 vine variatioonala.... intollitioommmiinooomo -nt.eerarrrtttlllllnoonaaaaaaaaa
maximi mingiiiiiiiiiiiimmm. m exploration, pdf
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Iharz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable Deep Reinforcement Learning for Vision-Basse Robotic Manipulation. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 651-673. PMLR, doi: htl::1806.102220
2018. ISBN 012492543X.
URL /
10293
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. EMI: Exploration with Mutual Information, 2018
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning: Deep Deterministic Policy Gradients (DDPG). ICLR, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 33, feb 2015. ISSN 1476-4687. doi: 10. (0338//ature112366 URL http: pattp://wet //www ncbi mlm. mih. gov
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. IJCAI International Joint Conference Artificial Intelligence, 2016. ISSN 10450823. doi: 10.1038. nature14236.
Yael Niv, Michael O Duff, and Peter Dayan. Dopamine, uncertainty and TD learning. Behavioral and brain functions BBF, 1:6, may 2005. ISSN 1744-9081. doi: 10.1186/1744-9081-11
URL https: http://www...........ooooooooe/a
//wwtt ncbi
nlm. www.:nh....../eeeoeeeee htt.ah/hgggg/mmm/tttaceee///////
Brendan O' D'Doooguue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. arXiv preprint MxXiv:700.0053:0 2017.
OpenAl. OpenAl Five Benchmark: Results, 2018.
URL
openai -fi fi www.hbbbehhhaakkkeeeeeeeee reaultts/
10

---

Under review as a conference paper at ICLR 2020
OpenAl, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning Dexterous In-Hand Manipulation. CoRR, 2018. URL http::/arxiv. httpaaawwwvvorgabb///aaaaa
00177
lan Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2701-2710. JMLR. org, 2017.
lan Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8617-8629, 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-Driven Exploration by Self-Supervised Prediction. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2017. ISBN 9781538607336. doi: 10. 10.1109/VPP
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphael Marinier, Mare Pollefeys, Timothy Lillicrap. and Sylvain Gelly. Episodic Curiosity through Reachability, 2019. URL https?
/ /oppnneeeiiww.occ/ net http://wwwww.ee/o.ot/ee.e.mmeme
Jürgen Schmidhuber. Adaptive confidence and adaptive curiosity. In Institut fur Informatik, Technische Universitat Munchen, Arcisstr: 21, 800 Munchen 2. Citeseer, 1991.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game Go without human knowledge. Nature, 2017. ISSN 1476-4687. doi: 10.1038.aau
Riley Simmons-EEoe Edler, Ben Eisner, Eric Mitchell, Sebastian Seung, and Daniel Lee. Q-learning for continuous actions with cross-entropy guided policies. arXiv preprint urXiv:/900 10605, 2019.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models. CoRR, abs/ 1507.0, 2015.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. {/EEE} Trans. Neural Networks, 105, 1998, doi: 19.1109/TNN.1998.111000 URL Utthttss://doi.ogg
10. 1109/TNN 1998 712192
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. #Exploration: A Study of Count- Based Exploration for Deep Reinforcement Learning. In I Guyon, U V Luxburg, S Bengio, Wallach, R Fergus, Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. Curran Associates, Inc., 2017. htt.tttts.//xx...rrttsse/
04717
Sebastian B Thrun and Knut Möller. On planning and exploration in non-discrete environments. GMD Sankt Augustin, Germany, 1991.
Sebastian B Thrun and Knut Möller. Active exploration in dynamic environments. In Advances in neural information processing systems, pp. 531-538, 1992.
11

---

Under review as a conference paper at ICLR 2020
Table 2: Parameters used for benchmark runs.
DEFAULT PARAMETERS
CEM
ITERATIONS
NUMBER OF SAMPLES
64
TOP K
6
ALL NETWORKS
NEURONS PER LAYER
256
NUMBER OF LAYERS
NON-LINEARITIES
RELU
OPTIMIZER
ADAM
ADAM MOMENTUM TERMS
31
0.9, B2 0.99
TRAINING
Q LEARNING RATE
0.001
BATCH SIZE
128
DECAY
0.99
TARGET Q-FUNCTION UPDATE T
0.005
TARGET UPDATE FREQUENCY
2
TD3 POLICY NOISE
0.2
TD3 NOISE CLIP
0.5
TRAINING STEPS PER TIMESTEP
QXPLORE- ORE-SPECIICC
Q2 LEARNING RATE
0.001
BATCH DATA RATIO
0.75
Qz BATCH DATA RATIO
0.75
3Q (Q INITIAL OUTPUT BIAS)
RND-SPECIFIC
PREDICTOR NETWORK LEARNING RATE
0.001
EXTRINSIC REWARD WEIGHT
2
INTRINSIC WEIGHT
Y
0.99
0.99
DORA-SPECIFIC
0.1
0.05
"/E
0.99
TQQ
0.99
G-GREEDY-SPECIIG
0.1
A IMPLEMENTATION DETAILS AND HYPERPARAMETERS
We describe here the details of implementation and training parameters. We held these factors constant and used a shared codebase for QXplore, RND, and c-greedy to enable a fair comparison. We used off- off-policy Q-learning method based off of TD3 (Fujimoto et (018b) and CGP (Simmons: Edler et al. 2019 with twin Q-functions and a cross-entropy method policy for better hyperparameter robustness. Each network (Qo. Qs,6. RND's random and predictor networks) consisted of 4-layer MLP of 256 neurons per hidden layer, with ReLU non- lineariiiis.es. We used batch size of 128 and learning rate 0.001, and for QXplore sampled training batches for Q and Q2 of 75% self-collected data and 25% data collected by the other Q-function's 's policy as described in Algorithm
For DORA (Fox al., 2018), we used the hyperparameters and training procedure specified by the original paper where possible, though it was necessary to adapt the method somewhat to the continuous action domain. This is because the original formulation proscribed an "LLL" action selection scheme that requires taking discrete ong-robbabilities of the distribution of Q and E values over actions, which is not tractable in continuous action spaces. Instead, we tried selecting actions using either CEM policy that maximizes the sum of the two objectives, using the E values as a reward bonus for training Q and selecting actions that maximize Q only. We thus expect the performance of our implementations to be somewhat worse than hypothetical distributional-DORe
12

---

Under review as a conference paper at ICLR 2020
though the action selection scheme we used does make this version directly comparable to QXplore and RND. Both formulations behaved similarly on Sparsellalfhheet... ah. and did not achieve reward with any frequency.
For e-greedy sampling with continuous actions, we sampled a uniform distribution of the valid action range (-1 to 1 for all tasks) with probability and act greedily otherwise. We note that the stochastic cross-entropy method policies we used for all experiments also introduce some amount of local exploration through noisy action selection.
We present the parameters we used for the benchmark tasks in Table
B ENVIRONMENT DETAILS
We use the SparseHal wCheet ah environment proposed by Houthooft et al. (2016) in which simulated cheetah receives reward of O if it is at least units forward from the initial position and otherwise receives reward -1 We also use the OpenAl gym tasks, Fet and Fet chP i ckAndP lace, which were originally developed for benchmarking HER (Andrychow icz et al. 2017 The objective in these environments is move a block to target position, with reward function returning -1 if the block is not at the target and 0 if it is at the target. For consistency in reward shaping, we structured the reward function of the Spars SparseHa fCheetah task match the Fet ch tasks, such that the baseline reward level is 1 while a successful state provides 0 reward, but report reward values on 0 500 scale for direct comparison with previous work. We trained each method with 5 random seeds for 5,000 episodes on Spars MatarsellaEChheee ah and 50,000 episodes on Fetch tasks. Time to convergence on these tasks for any exploration method is highly variable, and as such we visualize the mean and standard deviation of the runs in our results.
FetchS1 ide,
C ABLATIONS
To study the effects of different components of QXplore, we performed several ablations, discussed in Section 4.4] First, we replaced Qa with simple 1-step reward prediction, and Q₂ 's objective function with maximizing cumulative 1-step reward prediction error plus extrinsic reward, which we label as "QXplore-I-step in Figure 5 This ablation fails to find reward, as the 1-step reward prediction error makes long range exploration more difficult to learn.
OXplore vs. Ablations
-100
-500
-400
Coatore O
Oiplir O+
1030200
Oititr Ou (1-c99)) Oiplbre-Vaiee
Figure 5: Plot showing the performance of two ablations, I-Step Reward Prediction (QXplore-I-step) and Single-Policy QXplore (QXplore-vallle compared to the original QXplore method. In the 1-Step ablation, QR is trained to predict a combination extrinsic reward and reward prediction error, and fails to make progress. In the Single-Policy ablation, the policy converges faster, but a worse policy than vanilla QXplore due to the need to balance TD-error and extrinsic reward maximization.
Next, we tested variant of QXplore using only single sample policy and treating TD-error as a reward bonus, more line with traditional exploration bonus methods. We trained value function Vs(s) trained via bootstrap and computed rx T (8t, Oe, St41) = Vo(st) (re(st, as) VV (s)+ This variant uses only a single sample policy, Qu. which is trained via bootstrapped off off-policy Q-learning using one-step reward targets rı (r=(st, a, S141) arg(st, ae) to maximize a combination of intrinsic and extrinsic rewards, controlled by the hyperparameter a. We used
0.1, which found work well for SparseHa fCheet in tuning experiments. We
13

---

Under review as a conference paper at ICLR 2020
used a value function (s(s) rather than Q-function for this ablation avoid the wildly optimistic max action selection fully off-policy Q-functions have been reported to suffer from (Fujimoto et al.) 2018a). We label this experiment as "XXplore-value" in Figure [ This variant performs comparably to the Qz function of normal QXplore, but performance does not decrease late in training thanks to the extrinsic reward signal. However, overall performance is still well helow that of normal QXplore's exploitation policy, which does not have to satisfy two conflicting training objectives.
OXplore Position Sparse (iChetah-v)
DXpplore
-100
-200
-300
-400
Oxglore 3
OKaiore longinall Ox
-500
500200
LO00000 1500002
Oiglare IVLD
500030 1000000 1500000
OKaiore Qx
OKaiore (onginall OKalore RND -.
up10005
-llgicne RVB: Ok
Figure 6: Plots showing the performance of QXplore where the objective of Q₂ replaced by the RND exploration objective, as well as the mean position of the cheetah during an episode throughout training. While Q₂ does sample reward, does so too infrequently to guide Q to learn the task. While the Q₂ function of QXplore-RND RND does reach states far from the origin, it does not display directional preference, whereas original QXplore's s Q2 function converges to sample states around the reward threshold at 5 units.
Third, we tested a variant of QXplore where the TD-error maximization objective of Q2 was replaced by the RND random network prediction error maximization objective. We call this ablation "QXplore- RND" and results are shown in Figure oo both Q and Qz policies. We observe that neither function converges to achieve reward. While we see that Q. does sample reward, Q samples reward only during two episodes of training, and Q. does not converge to achieve high expected rewards itself. Looking at the mean position of the cheetah during an episode over training, we observe that for EXplore-RND Qu samples states relatively far from the origin compared to Q, based on the wider standard deviation, but does not display a directional preference (besides the inbuilt tendency to move forward more readily than backward that the cheetah has built in), since states found in both directions are equally novel. Comparatively, the Q. function of normal QXplore displays a strong forward preference once reward is found, and converges on sampling states close to the 5-unit reward threshold (this results in a mean position less than due to time spent traveling from the origin), while the corresponding Q function prefers to move well past the reward theshold (a mean position above 5) to reliably achieve reward.
OXplare on SparsahalfCheetah-- with signed TD-error
500
300
200
100
11eegnett
500030
1030300
The performance of QXplore's Q function with Q maximizing signed versus unsigned TD-error on two different reward variants of wwwrrseaalChheeeee the task for all variants, performance is reduced with the signed objective. Performance SparseHal fCheet ah variant with -1 to 0 reward function is shown shifted to match the axes of the 0 to variant for comparison.
Figure 7:
While Q is able to learn
the
Finally, we tested maximization of signed TD-error by Qn rather than unsigned. This objective tracks closer to dopamine-secking in animals, where increases in dopamine (corresponding to
14

---

Under review as a conference paper at ICLR 2020
unexpectedly positive outcome) sought out while decreases in dopamine (from unexpectedly negative outcomes) are avoided. To emulate this, we negate the signed TD error such that negative TD-error (the predicted Q-value was less than the target value) is maximized, while positive TD-error is minimized, Qz is otherwise trained as normal. The results are shown for both variants of the Spa rseHall FCheetal ah reward function (-1 to 0 and to 1) in Figure with and without setting the initial output bias of Q to 10 in the 0 to case. We observe that while QXplore does train with signed TD-error, performance is reduced. While this result bares further investigation, we hypothesize this is because prior to finding reward the sign of the TD-error is not correlated with the novelty of state, thus the state novelty exploration phase is less efficient.
D PARAMETER SWEEPS
We performed two sets of parameter sweeps for QXplore: varying the learning rates of Q and Qz- and varying the ratios of data sampled by each Q-function's 's policy used in training batches for each method. For learning rate, we tested combinations (QLR, QxLR) (0.01, 0.01), (0.01, 0.001), (0.001, 0.01), (0.001, 0,001), (0.001, 0.0001), ) (0.0001, 0.001), (0.0001, 0.0001).
For batch data ratios, we tested combinations (specified as self-fraction for Q, then self-fraction for Q₂) of (0, 1). (0.25, 0.75), (0.5, 0.5), (0.75, 0.25).
Results for these sweeps can be seen in Figures 8 and QXplore sensitive to learning rate, but relatively robust to the training data mix, to the point of Q training strictly off-policy with only modest performance loss.
baseline
baseline
00000000000 quir 0001
dr. 001 quir 0001
01
gxtr 01
ar. 10.0000ggpp w_d.pgrrrooo. 0001
dr. 0001 exir, 0001
10001_000 [01_4xpr_00
or. 10.0001ggg q_tgprrr_00 0000
dr. 0001 qxir, 001
0=01_qxtr_00
exir
-350
reworr
-35
-400
-40
-500
-45
-50
200
400
600
800
1000
2000
4000
6000
8000
10000
episode
(a) SparseHalfCheetah
(b) FetchPush-vl
Figure 8: Learning rate sweeps for Q and Q₂
D. RND PARAMETER SWEEPS
As we have adapted RND operate with vector observations and continuous actions, we performed several hyperparameter sweeps to ensure a fair comparison. We report in Figure 10 the results of varying both predictor network learning rate "Ir" and extrinsic reward weight "rw" independently on the SearsellalFCheee
task. The baseline values for these parameters used elsewhere are 0.001 and 2 respectively. We observe that RND is fairly sensitive to reward weight, but a value of I or two performs well, while learning rate of 0.001 appears to learn faster early in training without loss of final performance.
E REWARD SHIFTING VERSUS 3Q
For most of our experiments with the SparseHalfCheee function that is -1 for non-goal states and 0 for goal states have consistent reward shaping with the ce tasks in the OpenAl Gym. However, as the original SparseHalfooe ah proposed by Houthooft et al. (2016) used reward function that is 0 for non-goal states and 1 for goal states,
environment, we used a reward
15

---

Under review as a conference paper at ICLR 2020
beseline
baseline
grat surat
queret
grat, 0_oxrat_t00 100
qrat_0 grar___qxra__1 100
grat, gerat 25
arat. 75, gxrat
grat. 75
25
-300
-30
reward
-350
-35
-400
-40
-500
-50
200
400
600
800
1000
2000
4000
6000
8000
10000
episode
(a) SparselallCCheeaah
(b) FetchPush-vl
Figure 9: Sample ratio sweeps for Q and Qz
Parameter Sweeps
-100
200
-100
rw]
RND mv01
RND my05
In01
000000
1000000
1500000
20000000
2500000
RND original
upoates
Figure 10: Parameter sweeps for RND. A reward weight of either or 2 works best, with a learning rate of 0.0001 close second.
we present results for QXplore and our implementation of RND on the original reward function as well in Figure 11 Because we initialized the output distribution to be close to 0 initially, QXplore performed worse on this reward function due to the much smaller magnitude of TD-errors during the initial reward-free exploration phase slowing down exploration. However, adjusting the hyperparameter 3g. the initial bias of the output neuron of Q, allows to obtain identical performance to the to 0 reward function. QXplore's state novelty search efficiency is sensitive to this initial TD ie-error magnitude, which varies depending the reward function, but in coarse parameter sweep of initial biases -00 0, 1, 10, and 100 we found a good setting for the parameter which performed comparably to the -1 reward function. This dependency is similar in concept to the reward weighting used by many reward bonus methods to trade off between exploration and exploitation, the setting of which may also depend the reward landscape.
F THE 'NOISY TV' PROBLEM
The 'Noisy TV' problem is a classic issue with some state-novelty exploration methods in which states with unpredictable observations serve as maxima in the novelty reward space. QXplore's TD-error objective is not fundamentally vulnerable to the problem, but to demonstrate that our function approximation early in training also no subject to it, we trained QXplore on variant the SparseHal fCheet ah task where add random normally-distribun vector of the agent. The variance of this noise value increases proportionately to the movement the cheetah in the negative direction (away from the reward threshold). An agent vulnerable to the noisy tv problem will be enticed to explore in the negative direction rather than forward, as this maximizes the noveltylunprediccabiitty of the observations.
value to the observation
16

---

Under review as a conference paper at ICLR 2020
QX SparselaaliCheetah with [0-2]
520
420
220
2.0
Qx y10 Bixs)
Ox (1 biaal
Ox (10 bis
Qr (1.0 blec)
Ox 610 ban 00000
2000200
1500020
2000000
2520203
HADD
usdates
Figure 11: QXplore performance on CearseHHalfCheee Adjusting 3q recovers full performance compared the -1 to 0 reward function (shown here shifted by 500 reward for comparison). We tested several different values for Bo and found that a value of 10 worked best for SparseHal waarseeaalChheeatat fCheetah.
with the to 1 reward function.
We show the results of training QXplore this environment in Figure 13 for both and Q2. well as the mean position of the cheetah along the movement dimension during Q 's training rollouts. As expected, the performance of neither Q nor Q₂ is meaningfully altered relative the baseline, and Q₂ is not biased to explore backwards to greater degree than it typically does early in training.
G WEIGHT INITIALIZATION
As we use neural net function approximation error as a state novelty baseline for early exploration, the behavior of Q2 may be sensitive weight initialization. To test this, addition the Pytorch default initialization method "Kaining-Uniform; (He et al. 2015) which we used for all runs outside and "Xavier-Uniform;
this section, we also tested initializing both Q and Q~ with "Kaiiing-Norma (Glorot & Bengio 2010) two other initialization methods that result in higher variance between initial outputs of the networks, which translates into reduced training performance, We further tested two naive distributions that produced very high variance in outputs, "Normal," sampling weight values from N(0. 1) and "Uniform," sampling values from U(-1, 1). These configurations were not expected to perform as well as "Kaiming- Uniform". but do test the ability of Qu to explore given a poor initialization.
all cases other than "Kaiming-Unin O. The results of QXplore with each initialization scheme SparseHa 1 @Cheeteete are shown in Figure 12
we set the bias of each neuron to
Normal" and "Xavier-Unifon though both Q and Qa were able to converge on reward. "Normal" and "Uniform" however both more-or-less prevented from converging on reward. Their effect on the ability of Q2 to find reward however is much more mild- only "Normal" and to lesser extent "Uniform" caused significant issues with discovering and converging on reward. This suggests that Q, not particularly dependent on careful weight initialization to explore with function approximation error.
both showed moderate decrease in overall performance,
17

---

Under review as a conference paper at ICLR 2020
folming an fm soaming rome!
kainiing uniform kaining nonmal
initializatonn- Q
initialization Ox
uhrppmm
uniforms
nommal
northal
.pster
200
200
-500
500000
1000000 1500000
2030203
(a) Initialization of Q
(b) Initialization of Q.
Figure 12: Several alternate initialization schemes for Q and Qx. While Q is adversely impacted, Qz relatively robust even to very poor initializations such "Normal" and "Uniform."
robustness problem
robustness 'nolsy problem
-300
400
-402
-100
-500
0x cT
500000
1000000
1500000
2000000
2500000
OHaicee
500000
1000000
1500000
2030203
2500020
0re Nolay D6a
updates
@Xotore Neagy
(a) Noisy observation effects on Q
(b) Noisy observation effects on Q.
average position of theetah
3-
Orpiore
(c) Noisy observation effects on absolute position
Figure 13: QXplore trained 'noisy tv' variant of Sparseflalfhhettal where one element of the observation vector is normally distributed random value whose variance increases if the chectah moves in the negative direction. The performance of QXplore is not impacted in any way by this noise, and it trains as normal.
18


Reviews:
[{"review_id": "rkl5dt-ycH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a novel exploration method wherein an additional Q-function/policy is learned that treats abs(TD-error) of the standard Q-function as its reward function. Both policies are executed in parallel and experience is shared between them for off-policy learning. They demonstrate their method's superiority on a few continuous control tasks relative to RND. Overall, I thought the method was interesting and novel, but several concerns prevent me from endorsing its publication.\n\n1) Incorrect or untuned RND baseline.\nAs the tasks under consideration are quite different from those in the RND paper, RND must be adapted significantly. It is therefor troubling that only a single RND-specific hyper-parameter is reported, when there should be many (e.g. ratio of intrinsic to extrinsic rewards, see Table 5 in their paper for many more).\n\nIt could be that you're adapting RND to be more like an ablation of your own approach, such as replacing the Qx TD-error rewards with random network distillation rewards. This is fine (and I'd suggest doing this as an additional baseline if not), but it should be labeled as such.\n\n2) No comparisons to published RND baselines. \nAll of these problems could be avoided if the authors chose to run their method on tasks with published baseline results. Indeed, the lack of a Montezuma's Revenge run is particularly glaring. The author's are right to point out that Atari is perhaps not ideal for exploration since most novel states are rewarding, so I'm not expecting it to outperform RND per se. But the authors also claim that in these situations it won't do worse than RND, and without a Montezuma's Revenge run, this claim isn't well founded empirically.\n\n3)No comparison to value uncertainty methods.\nThis is also made more frustrating when contrasted to the author's claim that \"to our knowledge posterior uncertainty methods have thus-far only been demonstrated in small MDP.\" Osband, Aslanides, and Cassirer (2018) has results on a continuous control task and Montezuma's Revenge. As QXplore is related to both RND and this class of value uncertainty methods, is feels as though the latter should also be included.\n\nSmaller points:\n\n* No mention of prioritized experience replay. This method utilizes TD-error to bias replay rather than act explicitly, but there seems to be an interesting connection (acting to obtain more TD-error vs over-sampling high TD-error transitions) that was disappointingly ignored in this work.\n\n* I'd suggest dropping the \"biologically inspired\" bits from the paper. The evidence you're citing (dopamine represents TD error + dopamine seeking behavior) could be used to justify any approach that optimistic with respect to value function uncertainty (e.g. thompson sampling on value function posterior), and doesn't really add anything.\n\n* While the Fetch results are impressive, they should be contextualized by the results obtained in the HER paper. Obviously, HER isn't as general as this method, but it is worth reminding the reader that there is still a large performance gap between QXplore and the SOTA on these tasks.\n\n* noisy TV problem also isn't a problem for RND. Either compare to prediction-error based approaches and show that you're more robust to this issue, or drop it entirely.\n\nRebuttal EDIT:\n\nI appreciate the authors' responsiveness to my feedback on the text; the framing of biological plausibility and the connection to HER performance are both handled quite well now.\n\nI also appreciate the efforts the authors have demonstrated in adding new baseline results. That said, GEP and DORA results seem preliminary still (which is understandable given the timeframe), and would benefit from tuning and immediate results reporting in the case of GEP.\n\nI still find the treatment of RND problematic. \"Where possible, we used the parameters specified by RND in experiments (i.e. fixing intrinsic/extrinsic reward weight at 2)\" -- this is not the right attitude when testing such different domains and reinforcement learning algorithms. All of these hyper-parameters should be swept over in a manner analogous to your own approach before this comparison is valid.\n\nI acknowledge the Montezuma's Revenge (MR) isn't representative of exploration problems by itself -- this could be a good place to raise the bar and test against all 6 or so exploration-heavy problems in the Atari Suite. But even just running on MR would be better than introducing new tasks without published results from prominent exploration methods like RND. At the very least adding MR to your set of tasks would allow for comparisons against prior exploration results, which is the primary thing preventing me from raising my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SygTs5emKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an exploration method based on the TD-error as an adversarial reward signal. The authors show that this reward signal has interesting exploration properties. They compare it empirically to RND and epsilon-greedy, showing better performance. Besides, they perform additional ablation studies to better investigate the properties of their approach.\n\nThough the paper proposes an interesting concept, it suffers from many weaknesses, some easy to fix and some which will require more work.\n\nHere are some remarks, in random order:\n\n- at the end of the introduction, the authors mention inspiration from computational neuroscience models, but they do not come back to this aspect in their work. To me, these remarks should be removed from the paper and kept for another paper about the biological significance of the model. In the conclusion, the authors come back to the \"biological concepts of curiosity boredom, and exploration\", I would rather say they are psychological concept, and the authors should have a look at developmental psychology and developmental robotics if they really want to contribute in this respect (but not for this paper).\n\n- some related work references are dispersed in the introduction, in Section 2, in the beginning of Section 3.2, in the end of Section 3.3 and in a few other places. The authors should build a proper \"related work\" section. Globally, the paper is poorly organized, e.g. Section 3.2 refers to Section 3.4 etc.\n\n- given that Q_x's reward function is the unsigned TD-error, I would like to see whether QXplore can deal with problems taking both some positive and negative rewards.\n\n- the authors mention RND and DORA as baselines, but only compare to RND. What about DORA? Despite the excitement it generated when published, I suspect RND is a rather weak baseline. There are many other exploration frameworks, the paper would be much stronger if the comparison was with respect to many other methods, such as GEP-PG (Colas et al. , ICML 18), Novelty search approaches, etc. To me, the weak comparison is the biggest weakness of this work.\n\n- the authors compared themselves to approaches based on TRPO while they were using TD3 (this information is hidden in Appendix A and should be moved in the main paper. But then, is the difference in performance due to using TRPO which is known to be less sample efficient? Again, this makes the comparison very weak, the authors should rely on the same algorithm from both sides.\n\n- the caption of Table 1 should be explicit about which algorithm comes from which paper.\n\n- I found Section 4.5 very weak, it looks like mere \"handwaving\" and would deserve a proper quantitative study if the authors want to keep it. In my opinion, the authors should remove it for now and move Appendices E and F to the main paper instead. As is, Appendix E is very poor (by the way, the caption and the figure legend do not match, so we don't know which is which) and I had to look for the number of seeds until I found it hidden in Appendix F.\n\n- Appendix C shows that, though the authors try to minimize the importance of this fact, their algorithm is very sensitive to initialization. It is very honest of them to have kept this study in their paper, but I'm afraid it strongly speaks against the algorithm.\n\n- As described, SparseHaflCheetah is not that sparse (-1 everywhere and 0 when you succeed, as this reward scheme already favors exploration). It would be more informative to use 0 everywhere and 1 when successful.\n\ntypos:\n\nEq (1) and (3) should finish with a dot as it is the end of a sentence.\np3: MDP's => MDPs\nSection 3.3, second sentence: avoid starting a sentence with a symbol.\nSection 4, the second sentence is not a sentence (no main verb)\np14: is also no subject to it => not ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1xbZ_dpYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an exploration algorithm for reinforcement learning agents based on learning a separate Q-value function which treats TD-errors as rewards. Experiments on continuous control tasks with a difficult exploration component are used to highlight the effectiveness of the approach. \n\nThe proposed algorithm is interesting and has an intuitive appeal, the magnitude of the TD-error as an \"auxiliary reward\" seems like a natural choice to guide exploration. The experiments are comprehensive, including some ablation studies and robustness tests, with one caveat. Due to some experimental details (particularly concerning the environments), the empirical results may be invalid and it is difficult to assess them. There are also some certain parts that are unclear in the experimental results. Overall, because of these issues, I cannot recommend acceptance although I would be willing to increase my score if my concerns are addressed.\n\nConcerning the SparseHalfCheetah task, from Appendix F: \"...baseline reward level is -1 while a successful state provides 0 reward, but report reward values on a 0 to 500 scale for direct comparison with previous work\". This reward function is not sparse since it gives -1 at every step except at the goal which would incentivize the agent to explore naturally if it were acting greedily (or approximately greedily) with respect to the value function. For a sparse reward task, I would expect the reward to be 0 everywhere except at the goal state(s), where it would have some positive value. This is the case for the original reward function for SparseHalfCheetah described in [1]. This change to the reward function makes any comparisons to previous work questionable (Table 1) and could also affect the qualitative results in section 4.5. \n\nSimilarly, the other 3 tasks also do not have a sparse reward structure as the agent receives -1 at every timestep. This again is a confounder for assessing the exploration capabilities of the algorithm. It would be preferable to use other environments such as those in [1], which indeed have sparse rewards. \n\nFor these sparse reward problems where most rewards are zero, then before any nonzero reward is observed, it seems like Qx would mostly behave like previous algorithms based on a state novelty term (as discussed in sec 3.4). This could happen since the agent would only be observing rewards of 0 at this point during training. It is unclear to me then which benefits QXplore could have compared to previous algorithms such as DORA.\n\nOther points:\n1) Some discussion of possible pitfalls for the algorithm could be added. For example, while the noisy TV problem may be avoided, there could be a noisy reward problem now. The absolute TD error could be high simply due to stochasticity in rewards. The agent could also be vulnerable to scenarios in which a stochastic transition brings the agent to either a state eventually leading to very high reward or to a state leading to no reward as this would cause the TD error be high (even if the value function converges).\n2) The sentence before sec 3.3, \"Further TD-error-based exploration with a dedicated exploration policy removes the exploitation-versus-exploration\ntradeoff that ... to an optimal Q-function.\" Could the authors clarify how TD-error-based exploration avoids the exploitation-versus-exploration\ntradeoff? I do not see the connection here.\n3) In the experiments, how are actions chosen for the epsilon-greedy baseline since the action space is continuous (and not discrete)? \n4) I am bit confused as to what the difference between the lines Q and Qx are in Fig. 3.\n5) Why was DORA not included as a baseline algorithm? It would seem to be the most closely related to QXplore when rewards are mostly constant (or zero).\n6) Are two separate buffers necessary instead of a single shared buffer (with uniform sampling)? It seems to needlessly introduce complexity and additional hyperparameters (the ratios of samples from each buffer). If this is a key choice, then it should be mentioned in the paper.\nThe text could also be clarified to indicate there are two buffers. In some places the writing suggests there is only one, e.g. sec3.3 \"...policy with a replay buffer shared between Q_\\theta andthe Q-function maximizing rx, which we term Qx.\"\n7) In section 4.4, for the \"Single-Policy QXplore\", why was Q(s,a) replaced by V(s)? If I understand correctly, this experiment tries to test a variant where the TD error is used as a reward bonus. If this is the case, it seems like the best comparison would be to leave the original Q(s,a) which is learned by Q-learning instead of changing it to a state-value function. \n\nSuggestions (did not impact score)\n- I wonder if there is a connection between Qx and the variance of the returns since the latter can be learned by using the squared TD-errror as a reward (see the Direct Variance algorithm in [2]) and the absolute TD-error is a similar quantity. In this way, it could be possible to frame QXplore as following a type of risk-seeking stategy (see [3] for an algorithm that makes use of the variance). \n- I am not sure 'adversarial' is the right term to describe Q and Qx since the two policies are not in direct competition with each other.\n\n[1] \"VIME: Variational Information Maximizing Exploration\" by Houthooft et al.\n[2] \"Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods\" by Sherstan et al. \n[3] \"Deep Reinforcement Learning with Risk-Seeking Exploration\" by Dilokthanakul and Shanahan\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is insufficient support to recommend accepting this paper.  Although the authors provided detailed responses, none of the reviewers changed their recommendation from reject.  One of the main criticisms, even after revision, concerned the quality of the experimental evaluation.  The reviewers criticized the lack of important baselines, and remained unsure about adequate hyperparameter tuning in the revision.  The technical exposition lacked a sober discussion of limitations.  The paper would be greatly strengthened by the addition of a theoretical justification of the proposed approach.  In the end, the submitted reviews should be able to help the authors strengthen this paper.
Decision: Reject