Exploring through Random Curiosity with General Value Functions
Authors: Aditya Ramesh
Louis Kirsch
Sjoerd van Steenkiste
Jürgen Schmidhuber
Venue: NeurIPS 2022 Accept (2022)
Keywords: exploration
curiosity
general value functions
random network distillation

Abstract: Efficient exploration in reinforcement learning is a challenging problem commonly addressed through intrinsic rewards. Recent prominent approaches are based on state novelty or variants of artificial curiosity. However, directly applying them to partially observable environments can be ineffective and lead to premature dissipation of intrinsic rewards. Here we propose random curiosity with general value functions (RC-GVF), a novel intrinsic reward function that draws upon connections between these distinct approaches. Instead of using only the current observation’s novelty or a curiosity bonus for failing to predict precise environment dynamics, RC-GVF derives intrinsic rewards through predicting temporally extended general value functions. We demonstrate that this improves exploration in a hard-exploration diabolical lock problem. Furthermore, RC-GVF significantly outperforms previous methods in the absence of ground-truth episodic counts in the partially observable MiniGrid environments. Panoramic observations on MiniGrid further boost RC-GVF's performance such that it is competitive to baselines exploiting privileged information in form of episodic counts.

Exploring through Random Curiosity with General Value Functions
Aditya Ramesh' Louis Kirsch' Sjoerd van Steenkiste Jürgen Schmidhuber 1,3,4
'The Swiss Al Lab (IDSIA), University of Lugano (USI) & SUPSI
Google Research
Al Initiative, King Abdullah University of Science and Technology (KAUST)
"NNAISENSE faditya, louis, juergen)(idsia.chh avansteenkistte@goglee.ccm
Abstract
Efficient exploration in reinforcement learning a challenging problem commonly addressed through intrinsic rewards. Recent prominent approaches based state novelty or variants of artificial curiosity. However, directly applying them partially observable environments dissipation intrinsic rewards. Here we propose random curiosity with general value functions (RC-GVF), novel intrinsic reward function that draws upon connections between these distinct approaches. Instead of using only the current observation's novelty curiosity bonus for failing predict precise environ- ment dynamics, RC-GVF derives intrinsic rewards through predicting temporally extended general value functions. We demonstrate that this improves exploration in a hard-exploration diabolical lock problem. Furthermore, RC-GVF significantly outperforms previous methods in the absence of ground- truth episodic counts the partially observable MiniGrid environments, Panoramic observations on Mini- Grid further boost RC-GVF's performance such that it is competitive to baselines exploiting privileged information in form of episodic counts.
be ineffective and lead to premature
Introduction
How should an agent efficiently explore environments with high-dimensional state spaces and sparse rewards [70, 2]? An extrinsic coward-maximise agent will make little progress until it stumbles upon a rewarding sequence actions. the literature, this issue is commonly addressed by providing additional intrinsic reward guide the agent's exploratory behaviour [49, 12, 43].
One class of prominent approaches rely on state novelty, where an intrinsic reward the form of a 'novelty bonus' is awarded based on how often state has been visited [65, 6]. More recent works have extended these approaches high dimensional state spaces where tabular counts are inapplicable [7, 41, 10]. Another class of approaches is based artificial curiosity, where agents are rewarded in proportion to the prediction errors or information gains of predictive world model [52, 60). Curiosity-basse techniques have also been scaled to handle larger state spaces [22, 43).
Expecting an agent to have access to the complete environment state is unrealistic. such partially observable settings [3, 26], observations may look alike in different states, and the benefit of approximate state novelty bonuses as intrinsic rewards is unclear. Consider the environment in Figure 1 where an agent is required to traverse a series of blue and white tiles, yet it is only able observe the current tile is standing on. The colours of the tiles alternate in a regular fashion until the end of the corridor, where there is a surprising sequence of consecutive blue tiles. Directly
Part of this work was done while the author was Postdoctoral researcher at IDSIA.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

applying state novelty approaches may not be particularly meaningful in this case, as can be seen when using random network distillation (RND: [10]). In RND, the agent receives novelty bonus based on the error of its predictions about the output of fixed randomly initialised neural network applied to the current observation. shown in the top panel of Figure 1, RND unable to ascribe novelty when the state pattern changes later in the sequence and the intrinsic reward has vanished. Similarly, intrinsic rewards may also vanish prematurely for certain curiosity- approaches when single-step dynamics are simple [42], while longer horizon predictions (as needed in the partially observable case) are susceptible to rapidly compounding errors during sequential rollouts [69].
In this paper, we explore a connection between state novelty and artificial curiosity to address these limitations. We propose random curios- ity with general value functions (RC-GVF) as novel approach to generating intrinsic re- wards. It takes inspiration from exploration through temporally extended experiments that summarise long observation sequences [53]. Our approach uses random general value func- tions (GVFs) [67] to pose questions about the cu- mulative future value of observation-dependee features. Specifically, we train an ensemble of predictors to minimise the temporal differ- ence (TD) errors of the general value functions and derive an intrinsic reward based the TD- errors and disagreements in the long-term pre- dictions. The effective horizon of the predic- tion task can be controlled via the discount fac- tor, where zero discount can be related to state novelty with random network distillation (RND) [10]. We hypothesise that predicting in- formation from an temporally extended horizon improves exploration in POMDPs and guards against premature vamishing of intrinsic rewards due to the increase in difficulty of the auxiliary task. Indeed, in our toy experiment of Figure 1 it can be seen how RC-GVF manages to generate a spike of intrinsic reward at the end of the sequence.
12
9.6
0.0
402
600
800
1000
Frames
Figure I: The alternating tile corridor environment. The agent (red triangle) observes the colour of the tile that it currently occupies (white or blue). The extrinsic reward is always zero. At each time- step (frame), the agent moves one tile forward, until it reaches the last tile. Unlike our approach (RC-GVF), RND does not generate an intrinsic reward when encountering the surprising last few tiles (step 1000).
We evaluate our approach on sparse-reward and partially observable RL problems. Our results on hard exploration diabolical lock problem [36] and the Minigrid suite of environments [13] reveal the benefits of considering extended horizon predictions in comparison to approaches that rely on immediate partial observations generate exploration bonuses. Furthermore, existing baselines in Minigrid are heavily reliant on privileged information about episodic state-visitation counts from the simulator, which is commonly used to scale the exploration bonus. We demonstrate that RC-GVF succeeds in many environments without the use of such episodic counts.
2 Preliminaries
Reinforcement learning in POMDPs We consider the scenario where an agent interacts with a Partially Observable Markov Decision Process (POMDP) with time steps t E N, with environment states S. S, observations O E O, actions A, A, extrinsic rewards R. : S + [0. 1], and policies that map histories of different lengths to distributions over actions
H - A (AA) where H. = O E H [26]. The agent only observes O, at time step I, which may not fully describe the MDP state, and thus it becomes necessary to condition the agent on the history Hç-
The learning objective find the optimal policy π that maximizes the expected discounted return
J (( E.,
Σ
R. (Sx)
where 7 E [0, 1) is the discount factor and the expectation is over stochastic components in the environment and the policy.

---

Random Network Distillation In random network distillation (RND) [10] the agent receives state novelty reward proportional to the error of making predictions 2(o;) where the targets are generated by randomly initialized neural network Zø O Rd. The RND intrinsic reward for an observation is given by
(,(o,) 2 (ot)
(oz) 1|2-
RND-like error signals can be a simple way to obtain effective uncertainty estimates about targets at given input, which could subsequently be used for exploration in RL or detecting out of distribution samples [14]. Beyond its simplicity, an appealing aspect of using RND is that it allows us to implicitly incorporate prior beliefs about useful rewards through the neural network architecture [38].
General Value Functions A general value function (GVF) [67] is defined by policy cumulant or pseudo-reward function : O + IR, and a discount factor 7/2 E [o, 1). It can be expressed as
Un (0) E
k [000++))00,
General value functions extend the concept of predicting expected cumulative values to arbitrary signals beyond the reward. They can be viewed as "answering" questions about cumulative quantities of interest under a particular policy π and discount factor 7s- Predictions from GVFs have previously been used as features for state representation [47] or to specify auxiliary tasks that assist in shaping representations for the main task [24, 73].
3 Random Curiosity with General Value Functions
In the spirit of artificial curiosity [53], we are interested in predicting long-term outcomes in the environment under particular policy as a means for exploration. Our approach, RC-GVF, rewards an agent for taking actions that lead to higher uncertainty about the future. However, instead of predicting the entire sequence of future states, we consider of random numeric questions about the environment and capture uncertainty about their outcome using general value functions (GVFs).
3.1 General value functions predict the future
The central piece of RC-GVF is the prediction of temporally extended outcomes in the environment. This is different from only predicting a feature of the current state (as in state novelty exploration [65]) or dynamics models that predict the entire (sequence of) next observation(s) (as in early versions of artificial curiosity [52]). At time step t, the current observation O, is mapped to a collection of pseudo-rewards 2++1 Rdd which together with the policy and discount factor define question that can asked about the future: "what is the expected discounted cumulative sum of these pseudo- rewards under given policy?". The answer to this question (i.e. the prediction target) is based on outcomes in the environment, as in Equation 3. In this paper, we investigate whether a set of random pseudo rewards are sufficient to drive exploration in this way.
Previously it was found that random features extracted by a neural network are often sufficient to capture useful properties of the input [45]. Similarly, neural network architectures can be used to express prior knowledge about useful features [71, 14]. To this end, we generate pseudo-rewards from fixed and randomly initialised neural network Za O + dgd with parameters & that maps observations to pseudo-rewards. This choice also bypasses the difficult problem of discovering meaningful general value functions [73].
Training the GVF predictors Our exploration mechanism is to reward the agent for taking actions that generate previously unknown outcomes. To facilitate this, we train a separate (recurrent) neural which we call the predictor-to predict these values. Concretely, a predictor θr,= 7/ - Bd maps histories (of observations, actions, and pseudo rewards) to pseudo-values.
The predictor is trained on-policy, implying that the GVFs are evaluated under the current policy. One motivating factor for this choice is that couples the prediction task to the current policy, thus creating an incentive to vary behaviour for additional exploration [18].
3

---

We use the (truncated) X-return as the target for the predictor, which can be recursively expressed as
Gj (A=)= = Ze+1 + 7= ,=) (H+-1) + Aa (Az).
(4)
Here, A E [0, 1] is the usual parameter that allows balancing the bias-variance trade off by interpo- lating between TD(O) and Monte Carlo estimates of the pseudo-return [64]. The predictor is trained to minimise the mean squared TD-error with the A-return target. For convenience in notation, we will denote G (Az) Gi, despite its dependence Xg.
3.2 Disagreement and prediction error as intrinsic reward
To generate an intrinsic reward, a straightforward choice could be to consider the error between the temporal difference target (from Equation 4) and the predictor's 's output at the current observation:
LID(hE) [G; 07, (ha)1²
(5)
However, the presence of aleatoric uncertainty is problem that arises consequence of extending the horizon of the predictive task. From an exploration perspective, the agent should focus on the reducible epistemic uncertainty, and not the irreducible aleatoric uncertainty [15, 23). Directly minimising Equation 5 with the predictor would ignore the inherent variance in the TD-target (due to stochasticity in the policy and environment), and thus using the prediction an intrinsic reward does not distinguish between aleatoric and epistemic uncertainty.
of the GVF target as
Another way to recognize this is by decomposing the expected prediction error as deseribed by input o, [( can be split into
Jain et al. [25]. The expected loss of a predictor f(.) epistemic (E) and aleatoric (A) components: / L(t, (()))
E(f. A(o), where P( is the conditional distribution over targets for input and [(l,-) -) is chosen loss function. The aleatoric uncertainty at input o, (o), is defined as the expected prediction error of Bayes optimal predictor function. E(f, o) denotes the epistemic uncertainty of a predictor at o.
To overcome this issue with aleatoric uncertainty in RC-GVF, propose to train an ensemble of predictors and utilise the variance across their predictions as a multiplicative factor on the prediction error. Concretely, we train K predictors e k E 21, KJ. The prediction target for each member of the ensemble is the same À --seudo-retum (Equation 4) by using bootstrapped values from that member's 's predictions. Using the ensemble of predictors, the intrinsic reward is given by
Ra(o.) Σ (BLL)D(he)) o Vish.,, (ha)));
(6)
1=1
G
M (bh)
Unray (he) E (hi)
K
k=1
where 2 corresponds element-wise multiplication. In this formulation, even when prediction error remains, the exploration bonus will vanish as the predictors converge to the same expectation.
3.3 Effective horizon and relation to random network distillation
The effective horizon over which predictions are considered depends on the choice the discount factor 72. It can be shown that for horizon H E R satisfying
log (
H
H
log
the discounted sum (pseudo) rewards beyond this horizon is bounded by [28, 30].
From Equation 4 we can see that for the special case of 7- = 0 and any choice of Xş, we have G = Zt+1 Zø (ot). Note that this case the TD-error between prediction and target equivalent to the intrinsic reward provided by random network distillation (RND; Equation 2). Since the prediction targets are now deterministic, there no problem with aleatoric uncertainty, and a single predictor suffices (as opposed to ensemble).

---

4 Related work
Exploration In tabular RI., explicit counts for states or state-action pairs can be maintained to enable exploration and achieve efficient learning [61, 4]. Count-based bonuses have been scaled to larger state spaces through the use of density models, which provide pseudo-counts [7, 41J. State novelty bonuses are similarly inspired heuristics that provide a bonus based on estimated novelty of the visited state [10].
Perhaps the simplest way of implementing exploration through 'curiosity' is reward the policy for encountering transitions to states that surprise a learning predictor [52]. This is suitable for deterministic environments but it suffers from the 'noisy TV problem' in stochastic environments [56, 9]. To overcome this limitation, intrinsic rewards have been defined by learning progress (the first derivative of the prediction error) [51], information gain [60, 22], compression progress [55]. Also compare such bonuses to pseudo-counts [7]. Alternatively, it is possible to mitigate sensitivity noise by measuring prediction errors in a latent space [53, 43].
Our method is different from state count and novelty approaches in that it models temporally extended values beyond the current observation. It includes state-novelty measure, RND [10], as a special case when the GVF discount factors are zero. Unlike most artificial curiosity approaches, RC-GVF does not model the entire environment dynamics, in line with the idea of conducting temporally extended experiments whose outcomes are abstract summaries (represented in latent variables) of long observation sequences [54]. We mitigate sensitivity aleatoric uncertainty by measuring the disagreement of an ensemble of predictors, inspired by prior work (for single-step prediction models) [59, 44]. Ensembles for exploration have also been explored in the context posterior sampling [62]. Bootstrapped-DOG
[38] uses an ensemble of Q-value functions for a model-free interpretation of posterior sampling [39]. Unlike in our approach, these value functions are restricted to the original task rewards, and can not capture arbitrary pseudo-rewards. Recent works suggest incorporating mosel-based planning to directly optimise for long-term novelty [63, 58, 31]. Our intrinsic reward-wan which is generated by considering multiple future steps could motivate approaches for optimising long-term novelty without explicit rollout-based planning.
steps-could
General Value Functions Sutton al. [67] proposed general value functions (GVFs) an approach to represent predictive knowledge about the world. Closely related is the vector-valued adaptive critic [50] which predicts and controls cumulative values of special input vectors. Several works have studied the impact of using auxiliary value functions to improve representation learning in RL [24, 8, 35]. Further, was previously found that GVFs of randomly generated pseudo-rewards can also be useful for shaping and learning representations [34, 78] (see also related work on forecasts [47], TD-networks [66], and predictive state representations [32]). This can be viewed as 'answering' questions about cumulative quantities of interest under a particular policy and discount factor %s. In our implementation, the predictor is an entirely separate network and thus GVF prediction can not be used to improve the agent's 's internal representation. Instead, we use the GVF prediction to derive a useful signal to guide exploration.
The discovery/selection of useful GVF 'questions' is an open problem [48]. Existing approaches learn questions related optimizing performance on the main task [73, 27]. However, such an approach might not be suitable for sparse-reward environments. In our approach, we side-step the GVF discovery problem through the use of randomly generated pseudo-rewards and on-policy predictions.
Experiments
In this section we present an empirical evaluation of RC-GVE. First, we consider a hard exploration diabolical lock problem from the literature [36] and demonstrate the benefits of predicting beyond the immediate observation. We then proceed to experiments with the standard Minigrid suite of partially observable and procedurally generated environments, demonstrating how RC-GVF is able to outperform several strong baselines [18, 77].
"Code is available https: httpp://githhb. com Adiiyaiittaaaa Bamettyy---mm-----ee ht//://plorigggtrouuooooooooooo through-rogvf.
5

---

5.1 Diabolical locks
The diabolical lock problem [36] is considered a difficult environment for exploration due to noisy high-dimensional observations, stochastic dynamics and misleading rewards that deter the agent from finding the sparse optimal reward at the end of the lock. In the literature, RND is often shown to fail on these types of environments due to only considering the immediate future [36]. Due to RC-GVF's policy-conditioned predictions of future quantities, this task is instructive to demonstrate the qualitative advantages of our method and how this results in markedly different performance.
Environment details The environment con- sists of states organised in 3 rows {a, b, c] and H columns from {1, 2... H) (see Figure 6 in Appendix A. In each episode, the agent is initialised in one of two possible starting states, either at a1 or b1. At each state the agent has L available actions, only one of which is 'good' and transitions to a 'good' state in the next col- umn with equal probability. Taking the good action gives a negative (anti-shaped) reward of -,, except the end of the lock (states an bir), where the sparse optimal reward of 10 is received. The remaining T, - 1 actions at any state are 'bad', causing a deterministic transition to the 'dead' row at the next column (zero reward). All actions from dead state lead to the dead state in the next row, and hence the optimal policy is to take the respective good actions in each state. The good ac- tion at each state is assigned randomly as part of the MDP specification. The agent never observes the state directly and only has access to high-dimensional noisy observation (details in Appendix A.
Table I: Mean over 10 seeds and (min, max) of the farthest column reached (of H = 100) at different points on the diabolical lock problem.
Frames
5M
10M
20M
RC-GVE 61 (47, 81) (83, 100)
100 (100, 100)
RND 28 (21, 41) 37 (27,47)
46 (33, 58)
In our experiment, consider problem with horizon H 100, L 10, and observation noise G, = 0.1. As discussed by Misra et al. [36], this exploration problem is hard because the probability of reaching the optimal reward through uniformly random policy is L 1/ (10 100 in instance). Further, the stochastic transitions from good states prevent a solution that relies on memorising a state-independent successful sequence of H actions independent of the state.
Implementation We use Proximal Policy Optimization (PPO) [57] in an actor-critic framework as the base agent and use 128 pscudo-rewards for RND and RC-GVF. For RC-GVF we set 7 = 0.6 and use two prediction heads in the ensemble. The agent is trained to maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards. Further details are in Appendix A.I.
Results Table I compares RND to RC-GVF in this environment. can seen how RC-GVF succeeds in completing the lock on all runs, while the best run of RND only reaches 60% of the lock (and does not complete it). Indeed, once the agent takes an incorrect action and 'falls' into the bottom row, the future consequences (and thus, the GVFs) become highly predictable. By staying alive, the agent will encounter new observations and unpredictable transitions which appear interesting. Further experiments with additional baselines are presented in Appendix D. This result is illustrative of RC-GVFs exploration capabilities as we will see next.
5.2 MiniGrid
We evaluate RC-GVF on procedurally generated environments from MiniGrid [13], which a stan- dard benchmark in the deep reinforcement learning literature for exploration [46, 11, 18, 77, 76, 42]. Exploration in these environments is challenging due to partial observability, extremely sparse rewards, and the procedural generation of mazes and objects.
Environment Details Broadly speaking, we will consider three classes of environments organised according to several difficulty levels. First, we study MultiRoom-N7-3 and MultiRoom-Ni2-SW which are navigation tasks in a maze with seven and twelve rooms respectively. Next, we consider two levels of difficulty in KeyCorridor-S4RS and KeyCorridor-S3R33 Here, the agent needs to pick up ball behind locked door. Finally, we examine the ObstructedMaze set of environments where the agent has a similar task of picking up a ball behind a locked door, but keys are hidden in boxes (ObstructedMaze-22 2ith), and doors can be obstructed (ObstructedMaze-2DDD
The observations

---

10
RC-VV
AGAC
64g
(a) KeyCorridor- -SRR3
MultiRoom-NTe
(c) ObstructedMaze- 2D1h
1.9
03
Avenge metmr
22g
Fieeres
(d) KeyCorridor-S5BB
MultiRoom-N1253
ObstructedMaaz--DDiiie
Figure 2: Average return of RC-GVF, RND and other baselines on the selected Minigrid environments (without ground-trutt episodic counts baselines). RC-GVF outperforms baselines the absence of episodic counts. Some environments prove challenging for all approaches in this setting.
corresponds an egocentric view of the cells the front of the agent. Further details of the MiniGrid environments are available in Appendix A.2.
Implementation We compare RC-GVF to RND [10], AGAC [18], and NovelD [77]. We use Proximal Policy Optimization (PPO) [57] to maximize the expected sum a weighted combination of intrinsic and extrinsic rewards. At each time step I, the agent receives reward R₁ (&t) + Ro()),, where 3 E IR+, balances the contribution of the extrinsic R. (st) and intrinsic (ot) rewards. For RC-GVF we set 7/2 0.6, two prediction heads in the ensemble of predictors and 128 pseudo-rewards (same for RND). Other important coefficient, and learning rate the predictor are obtained via an extensive all baselines (see Appendices C.2 and C.3 for details, including on our implementation of baselines.).
base agent for all approaches. The agent is trained
such as the intrinsic reward coefficient (8), entropy search for
We present results averaged over 10 independent runs for each approach in every figure (solid line). Unless mentioned otherwise, the shading indicates 95% bootstrapped confidence intervals.
5.2.1 Egocentric observations
We will first consider the usual setting where the agent receives access to the egocentric observations from the environment. Importantly, we do not provide agents with access to privileged information about the environment in the form of so-called 'episodic counts'. This is different from several recent approaches that have been applied to MiniGrid, which incorporate these values part of their intrinsic reward [46, 18, 77].
The use of episodic counts obtained through the simulator is problematic for solving exploration problems in a partially observable setting, since tells the agent precisely how frequently has been in each state. Indeed, the use of episodic counts alone can be sufficient for exploration in MiniGrid (Figure Appendix 1). Meanwhile, there exists no good method for obtaining episodic counts in the absence of a simulator, observations (eg. as in Figure 1). Details about the exact usage episodic counts by baselines in their intrinsic reward formulation are provided in Appendix C.2.
it is difficult to accurately estimate pseudo-counts from partial
In Figure compare RC-GVF, RND, AGAC, and NovelD in the absence of episodic counts, while using egocentric observations. It can be seen that RC-GVF explores faster and succeeds more often than RND and the other baselines in the KeyCorridor N7-S8 environment. In comparison to RND and NovelD that utilise prediction errors in immediate random embeddings, it appears that RC-GVF benefits from extended horizon predictions random pseudo-rewards. Meanwhile, AGAC only succeeds in ObstructedMare-2DDli. where all baselines perform similarly well. Finally, we observe that all approaches struggle with the most difficult level of considered environments (KeyCoriidor MultiRoom-NI2-SI and ObstructedMaze-2DDiie
environment and MultiRoom-

---

CC-CVF (pano) AND (pano)
AGAC (coures)
NevelD (counts)
69
21
100 124
1h0 1Ta 200
(a) KeyCorridor-S444
(b) MultiRoom-NNZ
(c) ObstructedMaze-2DIii
000
Tramso millee)
0%
3 Trumns Bard!
Pramo
(d) KeyCorrido--55R3
(c) MultiRoom-N12-100
(1) ObstrootedMMuee-DDil
Figure Average return of RC-GVF and RND with panoramic observations. The performance of baselines (AGAC and NovelD) improves significantly with the inclusion of the episodic count component while RC-GVF (using panoramic observations) achieves similar performance without.
5.2.2 Panoramic observations
We now demonstrate how using episodic counts from the simulator is crucial to the success of the base- lines. In Figure it can be seen how AGAC and NovelD improve with episodic counts, and also solve the more difficult problems (KeyCorridor-S3R3, MultiRoom-NI2-SI and ObstructedMaaze2DDibn
Panoramic observations In order to increase performance of RC-GVF further, without introducing episodic counts, we investigate the use of augmented panoramic views as previously explored by Parisi al. [42] In the MiniGrid environments, the observation changes almost in its entirety when the agent changes the direction it faces. In contrast, moving to the next cell leads to fewer sudden changes in the observation.
The lack of gradual changes egocentric observations through rotations is a consequence of having four discrete angles for orientation (0º, 90% 180% 270%), which may be an unrealistic depiction of how rotations affect observations for agents situated in the real world. As a consequence, prediction error dominated by predictions of the outcomes of turning (rather than other actions). To address this, prior work proposed to make the agent's observations invariant to rotation by augmenting the observation with all directional observations [42]. In effect, this assumes that the agent rotates 360 after moving to a cell, or alternatively that it is equipped with additional sensors on its sides and back. Similar Parisi al. [42], will consider panoramic views only for generating intrinsic rewards, and egocentric observations for the base PPO agent.
In Figure 3, we demonstrate how RC-GVF with panoramic views and without privileged information about the underlying state of the MDP can solve harder problems and is competitive with the baselines that episodic counts in five out of the six settings. We see that RC-GVF with panoramic and MultiRoom--
observations comfortably succeeds the KeyCorridor where it was previously unsuccessful. RC-GVF did not succeed in solving the harder ObstructedMaze- 2Dlhb within the given frames despite the inclusion of panoramic views. Figure 3 shows that the use of panoramic views does not make the exploration problem trivial; RND also uses panoramic views (but no episodic counts). It improves with panoramic views on KeyCorridor-4RR2 and MultiRoom-N7S8 but struggles with the harder instances.
environments
5.3 Analysis
Changing the temporal prediction horizon We study the effect of changing the temporal prediction horizon by evaluating RC-GVF with different discount factors. Recall from Equation
'We verified in a preliminary experiment that incorporating episodic counts for RC-GVF similarly improves performance (see Figure 8 in Appendix E.2). However, since we believe that this constitutes privileged information that is not readily accessible outside of simulator, will not explore this direction further.

---

Y. =.99
V:
0.7
Y, = 0.6
Auarage
Average
Y:
0.4
%.2 0.05
92
0.2
V.
1.2
14 16
18
0%
Tratmess [trilliams)
Frames (millions)
(a) KeyCorridor- S5R3
(h) MuliRoom-N12-S10
Figure 4: Analysis reveals that: (a) larger discount factors (except 0.99) are useful in KeyCorridor-
S5R3 (b) low and intermediate discount factors perform well in MultiRoom-N12-S10
how the discount factor can me mapped the prediction horizon, which lets us estimate lower bound on the horizon of about 11 time steps for 7/2 0.6 with 0.01. In Figure 4 it can be seen how a larger prediction horizon desirable for KeyCorridor5R33. Among the higher discount factors, 0.95 is good choice for this environment, but not 0.99. Compared our default value of 7 0.6, a short horizon using 0.3 reduces performance further, while discount factors closest to 0 yield worse performance. In contrast, for MultiRoom- N12--101- perform well. The high GVF discount factors of 0.99 and 0.95 perform worse. Comparing 7 = to RND shows how the of recurrent networks and the variance term derived from the ensemble of predictors contributes most to the performance of RC-GVF in this domain.
find that low and intermediate values of Y
Ablation To better understand which of these components are contributing most to the success of RC-GVF, consider the following additional ablations: (1) RC-GVF (7= 0) without the variance term in Equation 6, and (2) RC-GVF (?= = 0) without the recurrent predictor. Figure 5a
compares these variations to RC-GVF (7= 0), RC-GVF (?=- 0.6), and RND. It can be that the inclusion of the distory-conditione recurrent predictor and the variance term individually contribute the improved performance of RC-GVE.
Using the history conditioned recurrent predictor sharply improves performance of RC-GVF with Ys = O. We hypothesise that this effect arises from the previous pseudo-rewand rewards available to the predictor, which might enable better predictions on input observations it was not explicitly trained on (which is often the case in procedurally generated environments). In support of this hypothesis, observed that using a recurrent predictor with solely observations as inputs does not produce such an improvement (Figure in Appendix E.3).
In Figure 5b, we present ablations for 7 0.6. We consider RC-GVF without the variance term in Equation 6, and without the recurrent predictor. Removing either the disagreement term or the recurrent predictor worsens performance of RC-GVF with 75 = 0.6. Interestingly, unlike as was observed for ? 0, here we find that RC-GVF with only disagreement (no recurrent predictor) performs better than RC-GVF with only recurrent predictor (no disagreement). This further emphasises the importance of handling the aleatoric uncertainty for non-zero GVF discounts.
Increasing the ensemble size We conduct experiments with larger ensemble sizes in two MiniGrid environments. We obtain comparable results for ensemble sizes 2, 4, 6 and predictors in Figure 10 in Appendix E.4, indicating that two member ensembles usually suffice for these problems.
Limitations and Societal Impact
Limitations The incorporation of the current policy into the prediction task of GVF prediction can have benefits with regards to behavioural exploration. However, it may also encourage over-exploration due to the same state appearing 'interesting' under different policies. Possible mitigation strategies include off off-policy variants explicitly policy-conditioned (general) value functions [17,20]. 20].
In some situations it may be difficult to ascertain what is good choice of the GVF discount factor 7z, which effectively determines the prediction horizon length, and can influenced by optimisation effects under function approximation [72]. The best choice may be problem-dependent and could

---

RC-GVF lis -0.00.0 wéo disegreemen:
RC-GVF (ys 0.6) we saggeement
12
RC-OVP ty. 0.00 recurrence
1.2
GVF 6.6)
Iy- 0.6)
(y,
RC-GVF lys 0.0)
Averad
AND
9.8
0.6
02
0.
0,0
10
12
14 26
9.
30
(milions)
Frames (millonst
(a) Ablation with 7 0.0
(b) Ablation with 0.6
Figure 5: Ablation of RC-GVF components in the KeyCorridor S5R3 environment (a) the variance term, recurrent predictor and discount factor contribute most to the performance of RC-GVF (b) the variance term appears be more important the case of 7/2 0.6.
further benefit from dynamic adjustment during learning [5, 37]. Nonetheless, our results in Figures 2 and 3 indicate that default of %/ 0.6 works well in many of the considered settings.
Several other intrinsic reward formulations using RC-GVF are possible (eg. based on informa- tion/prediction gain, learning progress, or only the disagreement term), which we did not investigate. Our specific formulation Equation was motivated by the prediction error in RND, which does not separate the aleatoric uncertainty when %:2 0. Starting with the prediction error term also allowed us to closely study the connection between RND and RC-GVE.
A general limitation of curiosity- based exploration for task-specific RL is that more information is gathered than is necessary to solve the given task [40, 33]. Indeed, the space of potential GVFs enor- mous. In the case of RC-GVF one could aim at focusing it by integrating task-specific information [68] or by considering predictions using a smaller set of policies that adequately cover the state space [1].
Societal impact This paper does not focus real-world applications of exploration in RL and thus does not have a direct societal impact. In applications, advanced exploration methods may lead to unexpected policy behaviour, which could be mitigated by incorporating safety constraints [19].
Conclusion
We introduced random curiosity with general value functions (RC-GVF), an exploration method that intrinsically rewards a reinforcement learning agent based on errors and uncertainty in predicting random transformations of observation sequences generated through the agent's ' actions. Unlike state- novelty approaches, ours takes into account multi-step future behaviours of policy and environment. Unlike commonly used artificial curiosity approaches, RC-GVF does not rely on model rollouts and does not need to predict all details of future observations.
On the diabolical lock problem [36] and the MiniGrid environments [13] we demonstrate that predict- ing abstract quantities of extended time intervals can improve exploration in POMDPs. Compared to recent methods such as AGAC [18] and NovelD [77] that rely on privileged episodic counts in MiniGrid, our RC-GVF achieves competitive results even when using only panoramic observations. Importantly, in the natural setting where episodic counts are not available, RC-GVF significantly outperforms all baseline methods.
Our approach can be generalized by moving beyond random pseudo-rewards, considering general value functions under a set of different policies [1], and introducing time or state-dependent discounting. This offers exciting avenues for future research.
Acknowledgments and Disclosure of Funding
We would like to thank Kenny Young, Francesco Faccio, Anand Gopalakrishnan, and Dylan Ashley for valuable comments. This research was supported by the ERC Advanced Grant (742870), the Swiss National Science Foundation grant (00021_192336), and by the Swiss National Supercomputing Centre (CSCS projects s1090 and sl 127).
10

---

References
[1] A. Agarwal, M. Henaff, Kakade, and W. Sun. Pe-pg: Policy cover directed exploration for provable policy gradient learning. Advances in Neural Information Processing Systems, 33: 13399-13412, 2020.
[2] S. Amin, M. Gomrokchi, H. Satija, H. van Hoof, and D. Precup. A survey of exploration methods in reinforcement learning. arXiv preprint S1t:::20090001557 2021.
[3] K. J. Åström. Optimal control of markov processes with incomplete state information. Journal of mathematical analysis and applications. 10(1):174-205, 1965.
[4] Auer, T. Jaksch, and R. Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008
[5] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[6] A. G. Barto and P. Singh. On the computational economics of reinforcement learning. In Connectionisl Models, pages 35-44, Elsevier, 1991.
[7] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29: 1471-14799 2016.
[8] M. Bellemare, W. Dabney, R. Dadashi, Ali Taiga, Castro, N. Le Roux, D. Schuurmans, T. Lattimore, and C. Lyle. A geometric perspective on optimal representations for reinforcement learning, Advances in neural information processing systems, 32, 2019.
[9] Y. Burda, H. Edwards, D. Pathak, A. J. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. In 7th International Conference Learning Representations, 2019.
[10] Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, 2019.
[11] A. Campero, R. Raileanu, H. Küttler, B. Tenenbaum, T. Rocktäschel, and E. Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint 1rXhv22006.2212,, 2020.
[12] N. Chentanez, A. Barto, and S. Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, 17, 2004.
[13] M. Chevalier-Boisvert,, L. Willems, and S. Pal. Minimalistic gridworld environment for openai gym. httts::/gittie thu. wtt. com./mxximecb/ggmm.. hetp:/gyymmmmiiieeeee grid, 2018.
[14] K. Ciosek, V. Fortuin, R. Tomioka, K. Hofmann, and R. E. Tumer. Conservative uncertainty es- timation by fitting prior networks. In 8th International Conference on Learning Representations, ICLR, 2020.
[15] A. Der Kiureghian and O. Ditlevsen. Aleatory or epistemic? does it matter? Structural safety, 31(2):105-111 112, 2009.
[16] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Sealable distributed deep-rl with importance weighted actor-learner architectures. In International Conference Machine Learning, pages 1407-1416. PMLR, 2018.
[17] F. Faccio, L. Kirsch, and J. Schmidhuber, Parameter-based value functions. In 9th International Conference on Learning Representations, 2021.
[18) Y. Flet-Berliac, J. Ferret, O. Pietquin, P. Preux, and M. Geist. Adversarially guided actor-critic.
In 9th International Conference on Learning Representations, 2021.

---

[19] J. Garcia and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-140, 2015.
[20] Harb, T. Schaul, D. Precup, and P.-L. Bacon. Policy evaluation networks. arXiv preprint arXfv:2002.11833,, 2020.
[21] S. Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
[22] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. Advances Neural Information Processing Systems, 29:
1109-1117, 2016.
[23] E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):47-5006 2021.
[24] M. Jaderberg. V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arX1v:1611.05337, 2016.
[25] M. Jain, S. Lahlou, H. Nekoei, V. Butoi, P. Bertin, J. Rector-Brooks, M. Korablyov, and Y. Bengio. Deup: Direct epistemic uncertainty prediction. arXiv preprint arXiv:2 102008501, 2021.
[26] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
[27] A. Kearney, A. Koop, J. Günther, and P. M. Pilarski. Finding useful predictions by meta-gradient descent to improve decision making.. arXiv preprint arXiv:2111.1 2021.
[28] M. Kearns, Y. Mansour, and A. Y. Ng. A sparse sampling algorithm for near-optimal planning in large markov decision processes. Machine learning, 49(2): 2002.
[29] D. Kingma and J. Ba. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, 2015.
[30] L. Koesis and C. Szepesvári. Bandit based monte-carie planning. In European conference on machine learning. pages 282-293. Springer, 2006.
[31] N. Lambert, M. Wulfmeier, W. Whitney, A. Byravan, M. Bloesch, V. Dasagi, T. Hertweck, and M. Riedmiller. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201. / 1861, 2022.
[32] M. Littman and R. S. Sutton. Predictive representations of state. Advances in neural information processing systems, 14, 2001.
[33] X. Lu, B. Van Roy, V. Dwaracherla, M. Ibrahimi, I. Osband, and Z. Wen. Reinforcement learning, bit by bit. arXiv preprint arXiv:2103.04047, 2021.
[34] C. Lyle, M. Rowland, G. Ostrovski, and W. Dabney. On the effect of auxiliary tasks on representation dynamics. In International Conference on Artificial Intelligence and Statistics, pages 1-9. PMLR, 2021.
[35] M. McLcod, C. Lo, M. Schlegel, A. Jacobsen, R. Kumaraswamy, M. White, and A. White. Continual auxiliary task learning. Advances in Neural Information Processing Systems, 34: 12549-12562, 2021.
[36] D. Misra, M. Henaff, A. Krishnamurthy, and J. Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning, pages 6961-6971. PMLR, 2020.
[37] T. Moskovitz, J. Parker-Holder, A. Pacchiano, M. Arbel, and M. Jordan. Tactical optimism and pessimism for deep reinforcement learning. Advances Neural Information Processing Systems, 34, 2021.
12

---

[38] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. Advances neural information processing systems, 29, 2016.
[39] Osband, B. Van Roy, and Z. Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning, pages 2377-2386. PMLR, 2016.
[40] I. Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. Advances Neural Information Processing Systems, 31, 2018.
[41] G. Ostrovski, M. G. Bellemare, A. Oord, and R. Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.
[42] S. Parisi, V. Dean, D. Pathak, and Gupta. Interesting object, curious agent: Learning task-agnostic exploration. Advances in Neural Information Processing Systems, 34, 2021.
[43] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pages 2778-2781 PMLR, 2017.
[44] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pages 5062-5071. PMLR, 2019.
[45] A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems, 21, 2008.
[46] R. Raileanu and T. Rocktäschel. RIDE: rewarding impact-driven exploration for procedurally- generated environments. 8th International Conference on Learning Representations, 2020.
[47] T. Schaul and M. Ring. Better generalization with forecasts. In Twenty- Third International Joint Conference on Artificial Intelligence, 2013.
[48] T. Schaul, H. van Hasselt, J. Modayil, M. White, A. White, P.-L. Bacon, J. Harb, S. Mourad, M. Bellemare, and D. Precup. The barbados 2018 list of open issues in continual learning. arXiv preprint arXiv: /8/ 1.07004, 2018.
[49] J. Schmidhuber, Making the world differentiable: On using fully recurrent self self-supervised neural networks for dynamic reinforcement learning and planning in -tatiinnn environments. Technical Report FKI-126-90, TUM, 1990.
[50] J. Schmidhuber. Reinforcement learning in Markovian and Mon-Markovian environments. In D. S. Lippman, E. Moody, and D. S. Touretzky, editors, Advances Neural Information Processing Systems 3 (NIPS 3), pages 500-506. Morgan Kaufmann, 1991.
[51] J. Schmidhuber. Curious model-building control systems. In Proceedings of the International Joint Conference on Neural Networks, Singapore, volume 2, pages 1468-1463. IEEE press, 1991.
[52] J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pages 222-227, 1991.
[53] J. Schmidhuber. What's interesting? Technical Report IDSIA-35-97, IDSIA, 1997. www.aaawwwhppippgggeeeeneeeeeeee gz; extended abstract Proc. Snowbird'98, Utah, 1998; see also [54].
[54] J. Schmidhuber. Exploring the predictable. In A. Ghosh and S. Tsuitsui, editors, Advances in Evolutionary Computing, pages 579-612. Springer, 2002.
[55] Schmidhuber. Simple algorithmic principles of discovery, subjective beauty, selective atten- tion, curiosity & creativity, In International conference discovery science, pages 26-38. Springer, 2007.
[56] Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions autonomous mental development, 2(3):230-247, 2010.
13

---

[57] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, arXiv preprint arXiv: 100.::67770347 2017.
[58] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning to explore via supr-vuee world models. In International Conference Machine Learning, pages 8583 8592. PMLR, 2020.
[59] P. Shyam, W. Jaškowski, and F. Gomez. Model- active exploration. In International conference on machine learning, pages 5779-5788. PMLR, 2019.
[60] J. Storek, S. Hochreiter, J. Schmidhuber, et al. Reinforcement driven information acquisition in non-deterministic environments. Proceedings of the international conference on artificial neural networks, Paris, volume 2, pages 159-164. Citeseer, 1995.
[61] A. L. Strehl and M. L. Littman. An analysis model-based interval estimation for markov decision processes. Journal of Computer and System Sciences,
2008.
[62] M. Strens. A bayesian framework for reinforcement learning. ICML, volume 2000, pages 943-950, 2000.
[63] Y. Sun, Gomez, and Schmidhuber. Planning be surprised: Optimal bayesian exploration in dynamic environments. In International conference on artificial general intelligence, pages 41-51. Springer, 2011.
[64] R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
[65] R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
[66] R. Sutton and Tanner. Temporal-differs difference networks. Advances in neural information processing systems, pages 1377-1384, 2005.
[67] R. Sutton, Modayil, M. Delp. T. Degris. P. M. Pilarski, A. White, and D. Precup. Horde: scalable real-time architecture for learning knowledge from unsupervised sensorimotor interac- tion. L. Sonenberg, Stone, Tumer, and P. Yolum, editors, 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2011), I), Taipei, Taiwan, May 2-6, 2011, Volume 1-3, pages 761-768, 2011.
[68] R. S. Sutton, M. C. Machado, G. Z. Holland, D. S. F. Timbers, B. Tanner, and A. White. Reward- respecting subtasks for model-based reinforcement learning, arXiv preprint 202:4:200.003466, 2022.
[69] E. Talvitie. Model regularization for stable sample rollouts. In UAI, pages 780-789, 2014.
[70] S. B. Thrun. Efficient exploration in reinforcement learning. 1992.
[71] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446 9454, 2018.
[72] H. Van Seijen, M. Fatemi, and A. Tavakoli. Using logarithmic mapping to enable lower discount factors in reinforcement learning, Advances in Neural Information Processing Systems, 32, 2019.
[73] V. Veeriah, M. Hessel, 7. Xu, R. Lewis, Rajendran, J. Oh, H. Hasselt, D. Silver, and S. Singh. Discovery of useful questions as auxiliary tasks. arXiv preprint arXivv/999.04444 2019.
[74] L. Willems. RI starter files. http:///ggthbbb.co/cwwweeeeee
2017.
[75] R. J. Williams and D. Zipser. Gradient-based learning algorithms for recurrent networks and their computational complexity. Backpropggation: Theory, architectures, and applications, 433: 17, 1995.
14

---

[76] D. Zha, W. Ma, L. Yuan, X. Hu, and J. Liu. Rank the episodes: A simple approach for exploration in procedurally-generaie environments. arXiv preprint arXiv:2101.08152 2021.
[77] Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. Gonzalez, and Y. Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34, 2021.
[78] Z. Zheng, V. Veeriah, R. Vuorio, R. L. Lewis, and S. Singh. Learning state representations from random deep action-conditional predictions. Advances in Neural Information Processing Systems, 34, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [Yes] Included in the supplementary material. An open-source implementation will made available online.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Implementation details and hyperparameters are provided in Appendices and C
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We include 95% bootstrapped confidence intervals over 5 or 10 seeds for all of our results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Details provided in Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Provided in Appendix A.2 with the
description of the environment.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "eN13k0dAACj", "reviewer": "Reviewer_MZa2", "summary": "This paper proposes a novel exploration strategy for RL that uses intrinsic rewards based on predicting long term values of random vector values rewards. The proposed approach contains the common RND exploration algorithm as a special case and is shown to work well on standard benchmarks, even when not given privileged access to episodic counts.", "questions": "Figure 1: I appreciate the idea of having a simple motivating example, but the details are too sparse. How does the agent get to the end to see the spike? It seems that even predicting future rewards gives no signal until the very end so it would still appear to be a very hard exploration problem without a shaped true reward. What is the statespace here? is it just white and blue? x-pos? What is the task reward?\n\nLine 226: Transitions from \"dead\" row are unclear. From the current description it seems that  an optimal policy be to take a bad action H-1 times (total reward 0) followed by a good action at the last timestep?\n\nLine 232: I don't understand why you can't just always take the good action. It seems like this deterministic policy works regardless of what state you are in\n\nHow sensitive is the algorithm to the weights on true reward and intrinsic reward for RL?\n", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+Simple idea that seems to work well and generalizes prior work\n\n+Good results on exploration benchmarks\n\n+Nice use of ensembles to deal with aleatoric and epistemic uncertainty\n\n-The environment details are sparse for someone who hasn't used these benchmarks before\n\n-It is unclear how sensitive the algorithm is to the weights on true reward and intrinsic reward for RL?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K3kr1GNNR2N", "reviewer": "Reviewer_5dYH", "summary": "The authors propose a novel form of artificial curiosity that utilizes randomly initialized neural networks to represent generalized value functions. The authors utilize the insight that temporally extended predictions of environment quantities (such as state transitions, etc.) are often useful for generating artificial curiosity in partially observable environments. However, instead of using state sequences directly, they utilize randomly initialized NNs to annotate trajectories with pseudo-rewards and utilize the uncertainty in the prediction of these pseudo-rewards by a prediction network ensemble as a signal for artificial curiosity. They evaluate their method in lock-opening environments and in grid world tasks and show a performance improvement on curiosity baselines. ", "questions": "1. Are there any ablations on ensemble size?\n2. As mentioned I would be interested in knowing the results of AGAC and NovelD on the lock task.\n", "limitations": "I believe the limitations of the method have been adequately discussed by the authors. I believe the paper is well written. While not state-of-the-art in terms of results, I believe there is some novelty in the method. I am on the fence about my decision about the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well motivated. Long-term dependencies should be accounted for when generating artificial curiosity signals. \n2. The writing is easy to follow and the approach is intuitive.\n3. The experiments are representative of hard exploration problems.\nWeaknesses:\nMy only real qualm with the paper is the significance of the results. The method strongly outperforms the baseline in the lock task but really only outperforms the baselines in 2 of the grid-world tasks. Additionally, why are there no performance metrics for AGAC and NovelD on the lock task? I would be interested in knowing those numbers as a reader.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EyoBvz8pCby", "reviewer": "Reviewer_9UDz", "summary": "This paper proposes a generalisation of the random network distillation (RND) to deal with partially-observable environments. That is, the paper extends the notion of intrinsic rewards in RND to deal with long-horizon, sparse reward environments that are partially observable by (1) predicting the value of pseudo-rewards (as opposed to per-state rewards in RND), (2) using a recurrent value predictor network to create a belief state due to partial observability, and (3) using the disagreement among an ensemble of value predictors to explore where predictions are epistemically uncertain. The combination of this approach with any model-free on-policy RL agent as a basis leads to an approach to exploration through temporally-extended curiosity. To validate the approach, the authors have combined their approach with PPO and tested it on a long-horizon anti-reward-shaped task, as well as on a set of MiniGrid tasks with egocentric and panoramic observations. They conduct analysis and ablation studies to verify that all added components are necessary to achieve the best performance in such tasks over RND and other techniques.", "questions": "1. In order to avoid state aliasing in the rewards, wouldn't it make more sense for the rewards to also be generated through a fixed, randomly initialised recurrent network that conditions on the sequence of observations, actions, and pseudo-rewards? For example, say there are only aliasing states between a trajectory that the agent has experienced repeatedly and a trajectory that the agent had not experienced before. Now, the history summaries would be the same and so would the value predictions, and the disagreement would be low. But if the trajectories were distinguishable by different pseudo-rewards, then the histories would be distinct and the epistemic uncertainty would be high for the trajectory that had not been explored before.\n\n2. The authors have tried different discount factors for training the GVF predictors and argue that lower discounts are not good because the prediction horizon should be larger. Couldn't the worse performance of lower discount factors be due to optimisation issues under function approximation? (See van Seijen et al. (2019) for details on this issue.) \n\n3. I also feel the range of discount factors used is somewhat arbitrary and not well motivated. The most commonly-used discount factors (e.g. $\\gamma = 0.9$ or $\\gamma = 0.99$) are missing in the tested values in Figure 4. Why? I understand that these are not agent discounts and are only used for exploration, but I cannot process why still a higher value shouldn't in principle be preferred in episodic settings. I think too high wouldn't work well with function approximation due to optimisation issues, but a value of 0.99 should work in practice. Why is it not preferred in principle by the authors?\n\n4. Does the basis PPO agent use the history summary or only use the latest observation? Is this setting the same among all baselines? \n\n5. Where can I find the result that you refer to in footnote 1?  \n\n6. The approach seems also applicable in principle to continuous state-action environments. The authors do not claim this, but could you comment on whether you see any obstacles in such environments?\n\n7. How do you justify using only 10 independent trials (or only 5 in one task)? PPO itself generally has very high variance across trials, and my experience is that using 20 trials is important in deriving any conclusions about performance characteristics. Now, in your context, the variation added due to pseudo-reward predictors could only make this worse. It is my view that, given that the domains are not too computationally expensive, it is critical to at least make sure the results are statistically reliable. Please share your view on this, and why the choice to run only 10/5 seeds.\n\nShould these questions be adequately clarified/addressed, I'd be happy to increase my score. \n\n- H. van Seijen, M. Fatemi, A. Tavakoli. “Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning.” NeurIPS (2019).", "limitations": "The authors discuss some limitations of their work. I have asked for comments on other potential limitations in the Questions section.\nThey appropriately state that the nature of the work does not per se entail a potential negative societal impact. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper moves in the direction of extending a simple yet powerful idea (namely, RND) to the more general problem setting of partially-observable environments with sparse rewards. \n\n- The approach is simple and fundamentally well justified; specifically, (1) GVF formulation is natural for going beyond predicting single-step rewards (as in RND), (2) using random targets as in RND has been previously shown effective (and simplifies the problem of selecting/discovering auxiliary tasks in Horde-style architectures), (3) using recurrence as agent's state-update function to deal with partial-observability is well-accepted and the authors have also justified its particular importance in the context of this paper, (4) using prediction disagreement among an ensemble is an effective proxy for epistemic uncertainty.    \n\n- The approach is well-motivated through illustrative examples and challenging exploration problems (but with relatively simplistic dynamics; namely, Diabolical Locks and 6 MiniGrid environments).\n\n\n### Weaknesses:\n\n- I feel that the pseudo-reward generator should also be based on some representation of history (e.g. via a fixed and randomly initialised recurrent network). I discuss this further in the Questions section (first question).\n\n- The trade-offs involved in choosing the discount factor are not discussed to a good extent. For example, why should we even care for low discounts? Why do they think $\\gamma = 0.6$ works better, and not $\\gamma = 0.9$ or $\\gamma = 0.99$? Discussing intuitions and drawing on previous literature on the role of discounting (especially in combination with function approximation) could help improve this shortcoming.\n  \n- In relation to discounting, I also feel the range of discount factors used is somewhat arbitrary and not well motivated. The most commonly-used discount factors are missing in the tested values in Figure 4. Why?\n\n- I feel that the panoramic-view experiment or at least the justifications around it are forced. The statement in footnote 1 of the paper is not a reasonable argument in my view. If the environment doesn't give you some information, assuming anything beyond it is privileged. Augmenting the agent with more sensors to get a panoramic view is privileged too. As such, I much rather see the result of your approach having access to the episodic count as opposed to a panoramic view.\n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5oZgn1xIM7i", "reviewer": "Reviewer_ZFPx", "summary": "The authors propose Random Curiosity with General Value Functions (RC-GVF) that essentially extends the intrinsic rewards obtained through Random Network Distillation (RND) to temporally extended outcomes of action sequences, in order to tackle long-horizon hard-exploration tasks in partially observable environments. To deal with aleatoric uncertainty, an ensemble of predictors is trained and the ensemble disagreement is combined with the prediction error as a multiplicative factor. The method is evaluated in two sets of environments: the diabolical lock and the MiniGrid environments.", "questions": "- Why is there no recurrency in the predictor network for the diabolical lock experiment?\n- Why are there only two ensemble members? How does the number of ensemble members affect performance in regards to noise levels in the tested environments?\n- I find the ablation study Figure 4c to be contradicting the statements made in section 3.3. Using $\\gamma_z=0$ is the equivalent of intrinsic rewards by RND, in which case as the authors state no ensemble is needed (The deterministic target value of the pseudo-reward takes care of the stochasticity). So why is the ensemble ablation performed for this discount value? In fact why are there no ablations for $\\gamma_z=0.6$?\n- If I understand it correctly, the main difference between RC-GVF with $\\gamma_z=0$ with and without recurrence is that the predictor network unlike in RND is not the same as the pseudo-reward generator and in the case of Minigrid doesn’t use an LSTM core. Did you do an ablation on the extra representational power in the different predictor network architecture of RC-GVF compared to the pseudo-reward generator also for the diabolical lock experiment?", "limitations": "-", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe this is an interesting work touching on an important subject that is utilizing future novelty with long horizon novelty predictions to solve hard-exploration tasks.\n\nHowever, I believe that the authors of the paper didn’t discuss the related work in this direction in enough detail. There are several recent papers that build upon the ensemble disagreement proposed in [1] (which is discussed in the current paper) and extend it to get multi-step novelty estimates into the future, such as [2] and [3]. (Especially in [3], multi-step novelty with RND as intrinsic reward is also proposed.). I believe these works should also be part of the discussion to give a more complete picture of long-horizon intrinsically-motivated exploration.\n\nI still believe that the way the authors combined the RND intrinsic reward with general value functions is interesting and different in that they do not rely on model rollouts.\n\n[1] Pathak, Deepak, Dhiraj Gandhi, and Abhinav Gupta. “Self-supervised exploration via disagreement.” International conference on machine learning. PMLR, 2019.\n\n[2] Sekar, Ramanan, et al. “Planning to explore via self-supervised world models.” International Conference on Machine Learning. PMLR, 2020.\n\n([3] Lambert, Nathan, et al. “The Challenges of Exploration for Offline Reinforcement Learning.” arXiv preprint arXiv:2201.11861 (2022).)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept