Improving Intrinsic Exploration by Creating Stationary Objectives
Authors: Roger Creus Castanyer
Joshua Romoff
Glen Berseth
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Exploration
Intrinsic Rewards
Stationarity
Primary area: reinforcement learning
TL;DR: We propose a framework to transform intrinsic reward methods into stationary learning signals, which enables better policy learning across many challenging environments.

Abstract: Exploration bonuses in reinforcement learning guide long-horizon exploration by defining custom intrinsic objectives. Count-based methods use the frequency of state visits to derive an exploration bonus. In this paper, we identify that any intrinsic reward function derived from count-based methods is non-stationary and hence induces a difficult objective to optimize for the agent. The key contribution of our work lies in transforming the original non-stationary rewards into stationary rewards through an augmented state representation. For this purpose, we introduce the Stationary Objectives For Exploration (SOFE) framework. SOFE requires *identifying* sufficient statistics for different exploration bonuses and finding an *efficient* encoding of these statistics to use as input to a deep network. SOFE is based on proposing state augmentations that expand the state space but hold the promise of simplifying the optimization of the agent's objective. Our experiments show that SOFE improves the agents' performance in challenging exploration problems, including sparse-reward tasks, pixel-based observations, 3D navigation, and procedurally generated environments.

Published as a conference paper at ICLR 2024
IMPROVING INTRINSIC EXPLORATION BY CREATING STATIONARY OBJECTIVES
Roger Creus Castanyer Mila Québec AI Institute
Joshua Romoff
Glen Berseth Mila Quéhec Al Institute Université de Montréal
Ubisoft LaForge
Université de Montréal
joshua romof www.ff@@uuiooo.ccmmmm
{ (roger. www.scccuuusssssaeeeee
glen.bersen @mi quebec
ABSTRACT
Exploration bonuses in reinforcement learning guide long-horizon exploration by defining custom intrinsic objectives. Several exploration objectives like count- based bonuses. pseudo-counts, and state-entropy maximization are non-stationary and hence are difficult to optimize for the agent. While this issue is generally known, usually omitted and solutions remain under-explored. The key con- tribution of our work lies in transforming the original non-stationary rewards into stationary rewards through an augmented state representation. For this pur- pose, introduce the Stationary Objectives For Exploration (SOFE) framework. SOFE requires identifying sufficient statistics for different exploration bonuses and finding efficient encoding of these statistics to as input to a deep net- work. SOFE is based on proposing state augmentations that expand the state space but hold the promise of simplifying the optimization of the agent's objec- tive. We show that SOFE improves the performance of several exploration ob- jectives, including count-based bonuses, pseudo-counts, and state-entropy max- imization. Moreover, SOFE outperforms prior methods that attempt to stabilize the optimization of intrinsic objectives. We demonstrate the efficacy of SOFE bard-exploration problems, including sparse-reward tasks, pixel-based observa- tions, 3D navigation, and procedurally generated environments.
INTRODUCTION
Intrinsic objectives have been widely used to improve exploration in reinforcement learning (RL), especially in sparse-reward and no-reward settings. In the case of Markov Decision Processes (MDPs) with finite and small set of states, count-based exploration methods perform near- optimally when paired with tabular RL algorithms (Strehl & Littman 2008 Kolter & Ng 2009).
Count-based methods keep track of the agent's frequency of state visits to derive an exploration bonus that can be used to encourage structured exploration. While much work has studied how to extend these methods to larger state spaces and continuous environments (Bellemare et al. 2016 Lobel et al. 2023 Tang et al. 2017), count-based methods introduce unstable learning dynamics that have not been thoroughly studied and can make it impossible for the agent to discover optimal policies. Specifically, any reward distribution that depends on the counts (i.e. the state-visiration frequencies) non-stationary because the dynamics for the counts change the agents generate new experiences, and the agent does not have access to the information needed to estimate these dynamics. In an MDP, the convergence of policies and value functions relies on the transition dy- namics and the reward distribution being stationary (Sutton & Barto 2018). The non-stationarity of bount- rewards induces a partially observable MDP (POMDP), as the dynamics of the re- ward distribution are unobserved by the agent. In POMDP, there are no guarantees for an optimal Markovian (i.e. home-homogeneous) policy to exist (Alegre et al. 2021 20 Cheung et al. 2020 Lecar- pentier & Rachelson] 2019). In general, optimal policies in POMDPs will require non- non-Markovian reasoning to adapt to the dynamics of the non-stationary rewards (Seyedsalehi et al. 2023 Despite this issue, count-based methods
usually paired with RL algorithms that designed to converge to Markovian policies and hence might attain suboptimal performance. Previous research has either overlooked or attempted address the -oatsoatinnartty issue in intrinsic rewards (Singh al. 2010

---

Published as a conference paper at ICLR 2024
Some efforts to tackle this problem involve completely separating the exploration and exploitation policies (Schäfer et al. 2021; Whitney et al. 2021). However, these approaches add an additional layer of complexity to the RL loop and can introduce unstable learning dynamics. In this work, we introduce a framework to define stationary objectives for exploration (SOFE). SOFE provides an intuitive algorithmic modification to eliminate the non-stationarity of the intrinsic rewards, making the learning objective stable and stationary. With minimal complexity. SOFE enables both tractable and end-to-end training of a single policy on the combination of intrinsic and extrinsic rewards.
SOFE is described in Section 4 and consists of augmenting the original states of the POMDP by including the visitaiiitttinn frequencies or a representative embedding. SOFE proposes a state augmentation that effectively formulates the intrinsic reward distribution as deterministic function of the state, at the cost of foreing the agent to operate over a larger set of states. We hypothesize that RL. agents with parametrized policies are better at generalizing across bigger sets of states than optimizing non-stationary rewards. We evaluate the empirical performance of SOFE in different exploration modalities and show that SOFE enables learning better exploration policies. We present SOFE as a method to solve the non non-stationarity of count-based rewards. However, we show that SOFE provides orthogonal gains other exploration objectives, including pseudo-counts and state-entropy maximization. Furthermore, our experiments in Section [5] show that SOFE is agnostic to the RL algorithm and robust in many challenging environment specifications, including large 3D navigation maps, procedurally generated environments, sparse reward tasks, pixel-based observations, and continuous action spaces. Videos of the trained agents and summarized findings can be found on our supplementary webpage
2 RELATED WORK
Exploration in RL Exploration is central challenge in RL. Classical exploration strategies explore an alcatoric fashion. e-greedy (Sutton & Barto 2018) samples random actions during training for the sake of exploration. Adding random structured noise in the action space (Lillicrap et al. 2015] Fujimoto et al., 2018) can enable exploration in continuous spaces. Maximum entropy RL provides a framework to find optimal policies that are as diverse as possible, and hence better explore the space of solutions (Haarnoja et al. 2018: Levine et al. 2020 Jain et al. 2024p. For hard-exploration tasks, structured exploration has been studied through the lens of hierarchical RL (Gehring et al.) 2021 Eysenbach et al. 2018). State-entropy maximization has been proposed to explore efficiently, attempt to learn policies that induce uniform distribution over the state- visitatiin distribution (Seo 2021 Lee et al. 2019 Zisselman et al. 2023). In MDPs with sparse reward distributions, exploration bonuses (i.e. intrinsic rewards) provide proxy objectives to the agents that induce state-covering behaviors, hence allowing agents to find the sparse rewards. Count-based methods (Auer 2002) derive an exploration bonus from state- sate-visitation frequencies. Importantly, the inverse counts of a given state measure its novelty and hence provide suitable objective to train exploratory agents. This property makes count-based exploration an appealing technique drive structured exploration. However, count-based methods do not scale well to high- dimenssinnii state spaces (Bellemare et al. (016)). Pseudo-counts provide a framework to generalize count-based methods to high-dimensional and partially observed environments (Tagg et al. 2017 Lobel et al. 2023
Bellemare et al. 2016).
In modern deep RL applications, many popular methods enable exploration by defining exploration bonuses in high-dimensional state spaces (Laskin et al. (021). and among them are curiosity- PPahhak et al. 2017: Burda et al., 2018), data-based based ( Yarats et al. 2021) and skill-based (Eysenbach) et al. 2018; Lee et al. 2019). Recently, elliptical bonuses have achieved great results in contextual MDPs with high-dimensional states (Henaff et al.] 2022). These methods aim to estimate novelty in the absence of the true state-visitation frequencies. Henaff et al.] (2022) showed that elliptical bonuses provide the natural generalization of count-based methods to high-dimensional observa- tions. In this work, we show that SOFE improves the performance of count-based methods in small MDPs and pseudo-counts in environments with nigh-dimensional observations (c.g. images), further improving the performance of the of-te----hh-arr art exploration algorithm E3B in contextual MDPs. Additionally, our results show that SOFE provides orthogonal gains to exploration objectives of different natures like state-entropy maximization.
https://si wow.iteess.ggooeoeeeeeeeooleeeee www..oooooooooooooooooooo wtt.://........iieeeee.eee webp://///eeeeeeeepp//e////meee
2

---

Published as a conference paper at ICLR 2024
Non-stationary objectives A constantly changing (i.e. non-stationary) MDP induces partially observed MDP (POMDP) if the dynamics of the MDP are unobserved by the agent. In Multi- Agent RL, both the transition and reward functions non-stationary because these a function of other learning agents that evolve over time (Zhang et al. 2021a| Papoudakis et al.) 2019). In contextual MDPs, the transition and reward functions can change every episode and hence require significantly better generalization capabilities, which might not emerge naturally during training (Cobbe et al. 2020 [Henaff al., 2022 Wang al. 2022). For MDPs with non-stationary rewards, meta-learning and continual learning study adaptive algorithms that can adapt to moving objectives (Beck al.] 2023). Learning separate value functions for non-stationary rewards has also been proposed (Burda et al.] 2018). |Schafer et al.] (202T) proposed DeRL, which entirely decouples the training process of an exploratory policy from the exploitation policy. While DeRL, mitigates the effect of the non-stationary intrinsic rewards in the exploitation policy, the exploration policy still faces a hard optimization problem. Importantly. there might not exist an optimal Markovian policy for POMDP (Seyedsalchi et al. 2023). Hence, RL algorithms can only achieve suboptimal performance in these settings.
Many exploration bonuses are non-stationary by definition. In particular, count-based methods are non-stationary since the state-visitation frequencies change during training (Singh et al., 22110] Simşek & Barto 2006). We note that this issue is also present in many of the popular deep explo- ration methods that use an auxiliary model to compute the intrinsic rewards like ICM (Pathak et al.,
2017). RND (Burda et al., 2018). E3B (Henaff et al. 2022), density models (Bellemare al. | 2016 2017) and many others (Lobel et al. 2023] Raileanu & Rocktäschel 20200 Flet-Berliag al. 2021 Zhang al. 2021b). In these cases, the non stationarity is caused by the weights of the auxiliary models also changing during training. In this work, we argue that non-stationarity should not be implicit when an exploration bonus is defined. For this reason, we introduce SOFE, which proposes intuitive modification to intrinsic objectives that eliminates their non-stationarity and facilitates the optimization process.
Tang ct al.
3 PRELIMINARIES
Reinforcement Learning (RL) uses MDPs to model the interactions between a learning agent and an environment. An MDP is defined as a tuple M (S, R, T.7) where S the state space, A is the action-space, R. : S x A + IR is the extrinsic reward function, T S x A x S + [0, 1] a transition function and y is the discount factor. The objective of the agent is to learn a policy that maximizes the expected discounted sum of rewards across all possible trajectories induced by the policy. If the MDP is con-stationary. then there exists some unobserved environment state that determines the dynamics of the MDP, hence inducing a partially observed MDP (POMDP), which is also a tuple M' = O, R. T,7) where is the observation space and the true states 8 E are unobserved. In POMDP, the transition and reward functions might not be Markovian with respect to the observations, and therefore, the policy training methods may not converge to an optimal policy. To illustrate this, consider MDP where the reward distribution is different odd and even time steps. If the states of the MDP are not augmented with an odd/even component, the rewards appear to be non-stationary an agent with Markovian policy. In this case, a Markovian policy will not be optimal over all policies. The optimal policy will have to switch at odd/even time steps. In this work, we extend the previous argument intrinsic exploration objectives in RL. In the following sections, we uncover the implicit -tattiiiinn of several exploration objectives and propose a novel method to resolve it.
3.1 EXPLORATION BONUSES AND INTRINSIC REWARDS
In hard-exploration problems, exploration is more successful if directed, controlled, and efficient. Exploration bonuses provide a framework to decouple the original task from the exploration one and define exploration separate RL problem. In this framework, the extrinsic rewards provided by the environment are aggregated with the intrinsic rewards (i.e. exploration bonuses) to build an augmented learning target. By directing the agent's behavior towards custom exploration bonuses, this formulation induces exploratory behaviors that are state-covering and are well-suited for long- horizon problems.
3

---

Published as a conference paper at ICLR 2024
Central to SOFE is the introduction of the parameters d, in the formulation of exploration bonuses B(s,, at (6:), which enables reasoning about the dynamics the intrinsic reward distributions. The parameters of the intrinsic reward distribution o, determine how novelty is estimated and exploration is guided, and if they change over time then B is non-stationary. based methods, pseudo-counts, and state-entropy maximization under the same formulation, which includes 6s. In the next section, we present SOFE as solution to their non-stationarity.
the following, we unify count-
3.1.1 COUNT-BASED METHODS
Count-based methods keep track of the agent's frequencies of state visits derive an exploration bonus. Formally, the counts keep track the visited states until time t, and so Ne(s) is equal to the number of times the state has been visited by the agent until time t. Two popular intrinsic reward distributions derived from counts that exist in prior work are:
B
R(s aes $411 = B(st, aes S1+1 @1) (Neo(44110))
(1)
where / weights the importance of the count-based bonus, and:
R(s at (141 B(ss, ae, 8141/0:) = 1, if (f 0,
(2)
else
Note that the state-visitation frequencies N₁. are the sufficient statistics for oi and hence for the count-based rewards in Equations [ and [ That is, the state-visitation frequencies are the only
dynamically changing component that induces non-stationarity in count-based rewards.
Equation Strehl Littman 2008) produces a dense learning signal since ( Sc. Ge, 8 (0)) *
unless (8 = which is unrealistic in practice. Equation 2 (Henaff et al. 2023) defines
sparse distribution where the agent is only rewarded the first time it sees each state, similar to the objective of the travelling salesman problem. Throughout the paper, we refer to Equations and as v-reward and salesman reward.
3.1.2 PSEUDO-COUNTS
To enable count-based exploration in high-dimenniioal spaces, the notion of visitation counts has been generalized to that of pseudo-counts (Bellemare et al. (016). Prior work has estimated pseudo- counts through density models (Bellemare al. 2016), neural networks (Ostrovski et al. 2017). successor representations (Machado et al. 2020). and samples from the Rademacher distribution (Lobel et al. 2023). Recently, |Henaff et al. (2022) proposed elliptical bonuses (E3B) as natural generalization of count-based methods. An appealing property of E3B is that it models the com- plete set of state-visitation frequencies over the state space, and not only for the most recent state Concretely, the E3B algorithm produces bonus:
B( at, 81+ l(be) = the (F CF (e((41)
T
C G= C- (i((s)))(ss))
where bh is an auxiliary model that produces low-dimensional embeddings from high-dimensional observations. Since the ellipsoid is updated after each transition, the exploration bonus is non- stationary. The matrix C, defines an ellipsoid in the embedding space, which encodes the distri- bution of observed embeddings in a given trajectory. Since C, the only moving component of Equation 3 it is a sufficient statistic characterize the non-stationarity of the reward distribution. Note that in an MDP with finite state space, where y the one-hot encoding of the states, the explo- ration bonus in Equation 3 becomes count-based bonus similar to Equation [ Concretely, C,) becomes a diagonal matrix with the inverse state-vise vistttviitttiin frequencies for each state in the elements of the diagonal (Henaff al. 2022).
'Previous pseudo-coun count methods allowed the agent to query density model with a single state and obtain its pseudo-count. However, F3B maintains model of the state-visitation frequencies over the complete state space. The latter key for SOFE to obtain sufficient statistics of the E3B reward in Equation
4

---

Published as a conference paper at ICLR 2024
3.1.3 STATE-ENTROPY MAXIMIZATION
State-entropy maximization is a widely used exploration objective that consists of training policies to induce a uniform distribution over the state-marginal visitation distribution (Lee et al. 2019). A canonical formulation of this problem is presented in Berseth et al. (2019). Maximizing the state- entropy objective corresponds to training policies to maximize the following reward distribution
(Berseth et al. 2019)
Rst,, at S6+ 1 1(x) BSs,,, dty 8e4 ( log Pé. (50+1)
The parameters 0 define the policy and are the parameters of generative model which estimates the state- state-marginal distribution d (s²). Note that the sufficient statistics of the generative distribu- tion also sufficient statistics for the intrinsic reward in Equation 4 Throughout the paper, refer to this algorithm as surprise maximization (S-Max) and use the Gaussian distribution to model trajectories of states with Pa.- Hence the sufficient statistics of Pé, for the reward reward in Equation are &i (Me U of). We present the details of S-Max in Section A.7
4 STATIONARY OBJECTIVES FOR EXPLORATION
In the following, we present a training framework for Stationary Objectives for Exploration (SOFE). Any exploration bonus (s Ge: st-1 derived from dynamically changing parameters will define a non-stationary reward function. Without any modification, exploration bonuses define a POMDP: M = (S, O, A, T, 7). For simplicity, we have fully replaced the task-reward R. with the ex- ploration bonus B3 and we consider that the only unobserved components in the POMDP are the parameters of the reward distributior Hence, we argue that the unobserved states S satisfy U dh. Note that the transition function of the POMDP is generally only Markovian if defined over the state space and not over the observation space: T S x A S [0,
The sufficient statistics for exploration bonuses are always available during training as they are explicitly computed to produce the intrin- sic rewards. However, current RL methods do not allow the agents to observe them. Hence, any method that aims to solve .M faces op- timizing non-stationary objective, which is difficult to optimize, as it can require non- Markovian properties like memory, continual learning, and adaptation, and may only find suboptimal policies. In this work, we argue that non non-stationarity should not be implicit in the formulation of an exploration objective. For this reason, we propose SOFF, which augments
π
A
Sencia ward
soFe
TRUDE HTIEE
E
B(s ds #sa1
Figure SOFE enables agents to observe suffi- cient statistics of the intrinsic rewards and use them for decision-making.
where S = (O U $). with the state space by defining an augmented MDP M M (S.
O being the observations from M. Note that we get rid of the observation space O in the definition of M because by augmenting the original observations from M with the sufficient statisties for B we effectively define a fully observed MDP. This simple modification allows instantiating the same exploration problem in a stationary and Markovian setting. That is the optimal policies in M are also optimal in M. This is true since the transition and reward functions are identical in M and M. We note that the update rule for the parameters de must be Markovian, meaning that these can be updated after every step without requiring information other than s; and st+1. For example, counts only increment by one for the state that was most recently visited: M = (s) Ys E (S-&8) where sy St+1 and (s;) = (s,) + 1. The latter also applies E3B and S-Max, since the ellipsoid C₁ and parameters of the generative model are updated incrementally with every new transition (see Equation 3 and Section A.7 Given the sufficient statistics, the intrinsic reward dis- tributions in Equations
become fully Markovian, and hence are invariant across time.
"This assumption holds true the agent has access to sufficient statistics of the transition dynamics (e.g. grid environments), and makes SOFE transform POMDP into a fully observed MDP. Even when there are unohserved components of the true states apart from the parameters of the intrinsic reward distributions, empirically show that SOFE mitigates the non-stationary optimization, yielding performance gains.
5

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
SOFE is designed to improve the performance of exploration tasks. evaluate efficacy, we study three questions: (1) How much does SOFE facilitate the optimization of non-stationary exploration bonuses? (2) Does this increased stationarity improve exploration for downstream tasks? (3) How
well does SOFE scale to image-based state inputs where approximations needed to estimate state-visitation frequencies?
To answer each of these research questions, run the experiments as follows. (1) We use three different mazes without goals to inves- tigate how SOFE compares to vanilla count- based methods and S-Max in reward-free ex- ploration. Concretely, we evaluate whether SOFE allows for better optimization of purely exploratory behaviors. We also use large 3D environment with continuous state and action spaces which introduces complex challenges as requires more than purely navigation skills.
Secondly (2), we use 2D maze with goal and sparse extrinsic reward distribution. This hard-cxploration task where the extrinsic reward is only non-zero the agent reaches the goal, which requires sequence of 75 co- ordinated actions. We evaluate whether SOFE enables better optimization of the joint objec- tive of intrinsic and task rewards. Further- more, we use the DeepSea sparse-reward hard- exploration task from the DeepMind suite (Os- band al. 2019) and show that SOFE achieves better performance than DeRL. (Schäfer et al.) 202T) which attempts stabilize intrinsic re- wards by training decoupled exploration and exploitation policies.
Mane
We use 3 mazes a large map to eval- uate both goal-reaching and purely exploratory behav- iors. Maze 1: a fully connected, hard-exploration maze; Maze 2: maze with open spaces and goal; Maze same Maze but with doors which an intel- ligent agent should use for more efficient exploration; 3D map: large map with continuous state and action spaces.
Figure 2:
Thirdly (3), we apply SOFE on the E3B (Henaff et al. 2022) algorithm as argued in Section + to demonstrate the effectiveness of the approach with an imperfect representation of the state-visitation frequencies. We the Maaitaaacc-MliiRoom-- (2023), and the Procgen-Maze task (Cobbe et al. 2020). In both environments, the task is to navigate the goal location in a procedurally generated map and the extrinsic reward is only non-zero if the agent reaches the goal. Both environments return pixel observations. Minihack additionally returns natural language observations. However, the Procgen-Maze task is more challenging because each episode uses unique visual assets, requiring an additional level of generalization, while in Minihack, different episodes only vary in the map layout. Additionally, we include the Habitat environment Szot et al. 202T) to evaluate purely exploratory behaviors and show the results in Section A.1
task, originally used for E3B in Henaff et al.
We provide the details of the network architectures, algorithm hyperparameters, and environment specifications in Section A.3 Furthermore, we provide an in-depth analysis of the behaviors learned by SOFE in Section A.2 which uncovers valuable insights on how SOFE learns drive exploration more efficiently.
5.1 REWARD-FREE EXPLORATION
In this section, we focus on the first research question and consider the reward-free setting to eval- uate purely exploratory behaviors. We use the 3 mazes in Figure 2 and measure map coverage, which correlates with exploration in navigation environments. In Figure we show how SOFE en- ables agents explore the mazes better than vanilla count-based methods. Even though count-based rewards described in Equations I and SOFE generally enables RL agents to better op- timize them, achieving higher state coverage. Section A.9 contains the results across all algorithms
fix the

---

Published as a conference paper at ICLR 2024
and exploration modalities.
We also experiments in large 3D environment from the Godot
Aug.
A2C C.
1M
3M 3M 10M
15M
Figure 3: Episodic state-visitation for A2C agents during training. The first row represents SOFE, which uses both the count- based rewards and state augmentation (+ Aug.), and the second row represents training with the count-based rewards only (+ C.). Although optimizing for the same reward distribution, our method achieves better exploration performance.
RL repository (Beeching et al. 2021a). to evaluate SOFE's ability to scale to continuous state and action spaces. This environment contains challenging dynamies that require exploratory agents to master variety of skills, from avoiding lava and water to using jump pads efficiently (Alonso et al.) 2020 [Beeching et al., (222Ib). Figure 4 shows that SOFE also scales to these more complex set- tings, enabling SAC (Haarnoja et al. 2018) agents to achieve higher map coverage across different exploration modalities.
Exploration Salesman
Exploration
60
Envirommen Staps
Enwinonment Steps
Figure 4: Map coverage achieved by SAC agents in complex 3D map. Blue curves represent agents that use count-based rewards (+ C.); Red curves represent SOFE, which uses both count- based rewards and the state augmentations from SOFE (+ C. Aug.). Even though the same learning objective, SOFE facilitates its optimization and achieves better exploration. Shaded areas represent standard deviation. Results are averaged from 6 seeds.
Additionally, we show that SOFE stabilizes the state-entropy maximization objective. Figure shows the episodic map coverage achieved in Maze 2 by the vanilla S-Max algorithm compared to the augmented S-Max with SOFE. These results provide further evidence that SOPE is a general framework that tackles the -tatiinnionartty of exploration objectives and provides orthogonal gains across objectives of different natures.
5.2 EXPL.ORATION FOR SPARSE REWARDS
In the previous section, we showed that SOFE enables RI. agents to better explore the state space. In this section, we evaluate whether SOFE can achieve better performance on hard-exploration tasks.
We evaluate count-based methods and SOFE in Maze 2 in Figure 2 For each of the RL algorithms, we compare training with the sparse extrinsic reward only and training with the extrinsic and intrinsic rewards with and without SOFE. Figure 6 shows that SOFE significantly improves the performance of RL agents in this hard-exploration task. Our results confirm that extrinsic rewards are not enough to solve such expllorttoot tasks and show that SOFE is significantly more effective than vanilla count-based methods, achieving the highest returns across multiple RL algorithms. PPO (Schulman et al. 2017), PPO+LSTM (Cobbe al. 2020), and A2C (Mnih et al.) 2016) achieve near-optimal

---

Published as a conference paper at ICLR 2024
Entrgpy DOS
ALLO
Figure 5:
Episodic state coverage achieved by S-Max (bluc) and SOFE S-Max (red) in Maze 2. When augmented with SOFE, agents better optimize for the state-entropy maximization objective. Shaded areas represent one standard deviation. Results are averaged from 6 seeds.
goal-reaching performance only when using SOFE. Importantly, policies equipped with LSTMs have enough capacity model the non-stationary intrinsic rewards (Ni al. [202T) and could learn to count implicitly (Suzgun et al. 2018). However, our results show that SOFE further improves the performance of recurrent policies when optimizing for non-stationary intrinsic rewards. Addi-
Episodic Exploration, V-Reward, Full Obs.
Globel Exploration, v R-ewwrd, Full Obs.
KM
PSO
(oUrS) PPOLSIM Aug.
PPOLSTM C.
PRO-LSTM
P5O.L5TM
PVCLSTM
(ours) Aug
Aug.
PPO
PPO C.
(eurs) DON
(ours) DON + C. Aug.
DON
DON
DON
(ours) AZC + C. Aug
(ours) Aug.
AZC
A2C c.
0.0
0.3
0.6
0.9
0.0
0.3
0.5
Mean Episode Return
Mean Eploode Return
Figure 6: Interquantile mean (IQM) of episode extrinsic rewards for episodic (left) and global (right) exploration across multiple algorithms. Green bars represent training with the sparse reward; blue bars represent additionally using count-based rewards C.); and red bars represent additionally ing count-based rewards and SOFE (+ C. + Aug.). We compute confidence intervals using stratified bootstrapping with 6 seeds. SOFE generally achieves significantly higher extrinsic rewards across RL algorithms and exploration modalities. Without exploration bonuses, agents fail to obtain a non- zero extrinsic reward during training.
tionally, we compare SOFE to DeRL (Schäfer et al. (022) in the DeepSea environment and show the results in Table I DeRL entirely decouples the training process of an exploratory policy from the exploitation policy to stabilize the optimization of the exploration objective. SOFE is degrees of magnitude less complex than DeRL as only requires training an additional feature extractor. Still, SOFE achieves better results in the harder variations of the DeepSea environment.
5.3 EXPLORATION HIGI-DIMENSIONAL ENVIRONMENTS
In this section, we evaluate SOFE and E3B Henaff et al. 2022), the state-of-the-art exploration al- gorithm for high-ding digh-dimensional contextual MDPs. E3B tackles the challenging problem of estimating the true state-visite titte-visttaionn frequencies from pixel observations. As argued in Section [.3] the ellipsoid the only moving component the E3B objective. Hence, we evaluate whether including either the diagonal the full ellipsoid in the state enables better exploration. We optimize the E3B objec-
8

---

Published as a conference paper at ICLR 2024
Algorithm
DeepSea 10
DeepSea 14
DeepSea 20
DeepSea 24
DeepSea
DeRL-A2C
0.98 0.10
0.65 + 0.23
0.42 0.16
0.07 0.10
0.09 = 0.08
DeRL-PPO
0.61 0.20
0.92 0.18
-0.01 0.01
0.63 0.27
-0.01 0.01
DeRL-DQN
0.98 0.09
0.95 0.17
0.40
0.53 0.27
0.10 + 0.10
SOFE-A2C
0.94 = 0.19
0.45 + 0.31
0.11
0.08 = 0.14
0.04 = 0.09
SOFE-PPO
0.77 0.29
0.67 + 0.33
0.13 + 0.09
0.07 0.15
0.09 0.23
SOFE-DON
0.97 + 0.29
0.78 + 0.21
0.70 + 0.28
0.65 + 0.26
0.42 + 0.33
Table I: Average performance of DeRL and SOFE with one standard deviation over 1 100,000 evalu- ation episodes in the DeepSea environment. SOFE achieves better performance in the harder explo- ration variations of the DeepSea environment while consisting of less complex method. Additional details on the hyperparameters used for SOFE and DeRL are presented in Section A.3
tive with IMPALA (Espeholt et al. 2018) as proposed in Henaff et al. (002).. Section the details of the policy architecture.
A.3
contains
Figure 7 shows that SOFE also improves the performance of pseudo-count- based methods, pro- viding empirical evidence that reducing the non-stationarity optimization even in high-dimensional environments. In Section AA.] we include experiments with E3B and SOFE in the reward-free setting using the Habitat simulator. These show that SOFE im- proves the sample efficiency of E3B.
reward distribution enables better
Minihock Multroom N6 IMM
Proogan Noze
(ours E38 Augmented Diagonal
(oun)) E35 Augmented Diaganal
(ours] E38 Augmented
Augmented
E30
E38
ICM
KCM
RND
2.5
5.0
7.5
10.0
Mean Episode Retum
Episode Retum
Figure 7: Interquantile mean (IQM) of the episode extrinsic rewards in Minihack (left) and Procgen (right). The vanilla E3B algorithm solve MiniHack as shown in Henaff et al. (2022). However, we find that E3B can only consistently reach the goals in Procgen-Maze when using SOFE. As shown in Henaff et al. (2022), ICM (Pathak al., 2017) and RND (Burda et al. 2018) fail to provide a good exploration signal in procedurally generated environments. Results are averaged from seeds.
CONCLUSION
We identify that exploration bonuses can be non-stationary by definition, which can complicate their optimization, resulting in suboptimal policies. To address this issue, we have introduced novel framework that creates stationary objectives for exploration (SOFE). SOFE based on capturing sufficient statistics of the intrinsic reward distribution and augmenting the MDP's state representa- tion with these statistics. This augmentation transforms the non-stationary rewards into stationary rewards, simplifying the optimization of the agent's objective. We have identified sufficient statis- ties of count-based methods, the state-entropy maximization objective, and E3B. Our experiments provide compelling evidence of the efficacy of SOFE across various environments, tasks, and rein- forcement learning algorithms, even improving the performance of the state-of-the-art exploration algorithm in procedurally generated environments. Using augmented representations, SOFE signif- icantly improves exploration behaviors, particularly in challenging tasks with sparse rewards and across multiple exploration modalities. Additionally, SOFE extends to large continuous state and action spaces, showcasing its versatility.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We want to acknowledge funding support from NSERC and CIFAR and compute support from Digital Research Alliance of Canada, Mila IDT and NVidia. We thank Mehran Shakerinava and Raj Ghugare for helping review the paper before submission.
REFERENCES
Lucas N Alegre, Ana LC Bazzan, and Bruno C da Silva. Quantifying the impact of non-stationarity in reinforcement learning- baaeing-basedd traffic signal control. PeerJ Computer Science, 7:e575, 2021.
Eloi Alonso, Maxim Peter, David Goumard, and Joshua Romoff. Deep reinforcement learning for navigation in aaa video games. arXiv preprint arXiv:2011.04764, 2020.
Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
Christopher Bamford. Griddly: A platform for ai research in games. Software Impacts, 8: 8:100066, 2021.
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shi- mon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023.
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Godot reinforcement learning agents. arXiv preprint arXiv:21 2021a.
Edward Beeching, Maxim Peter, Philippe Marcotte, Jilles Debangoye, Olivier Simonin, Joshua Ro- moff, and Christian Wolf. Graph augmented deep reinforcement learning in the gamerland3d environment. arXiv preprint arXiv22112.17331, 2021b.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski. Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information pro- cessing systems, 29, 2016.
Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement learning in unstable environments. arXiv preprint
2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1rXiv:/606.01555 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /2894, 2018.
Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning, pp. 1843-1854. PMLR, 2020.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning. pp. 2048- 2056. PMLR, 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference on machine learning. pp. 1407-1416. PMLR, 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 2018.
Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversari- ally guided actor-critic. arXiv preprint 2o..iv22122000 04376, 2021.
10

---

Published as a conference paper at ICLR 2024
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. (10 2021), 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631-37646 2022.
Mikael Henaff, Minqi Jiang, and Roberta Raileanu. A study of global and episodic bonuses for exploration in contextual mdps. arXiv preprint arXiv:2306.008 033336 2023.
Arnav Kumar Jain, Lucas Lehnert, Irina Rish, and Glen Berseth. Maximum state entropy exploration using predecessor and successor representations. Advances in Neural Information Processing Systems, 36, 2024.
J Zico Kolter and Andrew Y Ng. Near-bayesian exploration in polynomial time. In Proceedings of the 26th anmual international conference machine learning, pp. 513-520, 2009.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2110.15191, 2021.
Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, worst- case approach using model-based reinforcement learning. Advances in neural information pro- cessing systems, 32, 2019.
Lisa Lec, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint
2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643, 2020.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (.XXv::509.02977 2015.
Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins estimate pseudocounts for exploration in reinforcement learning, arXiv preprint arXiv:2306.03188 2023.
Marlos C Machado, Mare Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pp.
2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be strong baseline for many pomdps. arXiv preprint arXiv:21 2021.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari. Satinder Singh, et al. Behaviour suite for reinforce- ment learning. arXiv preprint 1981v:1908.03568, 2019.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721 2721-2730. PMLR, 2017.
11

---

Published as a conference paper at ICLR 2024
Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint arXiv:1906.04733 2019.
Deepak Pathak. Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines: Reliable reinforcement learning implementations. The Journal of Ma- chine Learning Research, 22(1):12348 12355, 2021.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for seneeduralll-geneeate environments. arXiv preprint arXiv:2002.12292, 2020.
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg. John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury. Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterprt 3d dataset (HM3d); 1000 large- scale 3d environments for embodied AI. In Thirty-fifth Conference Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2021. URL https: arwivv org/ abs/ $109.082330
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:2109 Stt::2209.1302.. 2021.
Manolis Savva, Abhishck Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied Al Research. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019.
Lukas Schäfer, Filippos Christianos, Josiah P Hanna, and Stefano V Albrecht. Decoupled reinforce- reinforce ment learning to stabilise intrinsically-motine motivated exploration. arXiv preprint arXiv:2107.08966, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lec. State entropy maximization with random encoders for efficient exploration. In International Conference Machine Learning, pp. 9443-9454. PMLR, 2021.
Erfan Seyedsalehi, Nima Akbarzadeh, Amit Sinha, and Aditya Mahajan. Approximate information state based convergence analysis of recurrent q-learning. arXiv preprint (.XXi:2306605991 2023.
Özgür Şimşek and Andrew G Barto. An intrinsic reward mechanism for efficient exploration. In Proceedings of the 23rd international conference on Machine learning, pp. 833-840, 2006.
Satinder Singh, Richard Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-88 2010.
Alexander L Strehl and Michael Liutman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 7309):1309-1333 2008.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Mirac Suzgun, Yonatan Belinkov, and Stuart M Shieber. On evaluating the generalization Istm models in formal languages. arXiv preprint 1811v:1111.0000 2018.
12

---

Published as a conference paper at ICLR 2024
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Von- drus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira. Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
Haoran Tang. Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurek, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. Advances neural information processing systems, 30, 2017.
Kaixin Wang, Kuangqi Zhou, Bingyi Kang, Jiashi Feng, and YAN Shuicheng. Revisiting intrinsic reward for exploration in procedurally generated environments. In The Eleventh International Conference on Learning Representations, 2022.
William F Whitney, Michael Bloesch, Jost Tobias Springenberg, Abbas Abdolmaleki, Kyunghyun Cho, and Martin Riedmiller. Decoupled exploration and exploitation policies for sample-efficient reinforcement learning. arXiv preprint arXiv:2101.09455 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321- 384, 2021a.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217-25230, 2021b.
Ey Zisselman, Itai Lavie, Daniel Soudry, and Aviv Tamar. Explore to generalize in zero-shot rl. arXiv preprint arXiv:2306.03000 03072, 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A. REWARD-FREE EXPLORATION WITH SOFE AND E3B
As in Section 5.1 evaluate if SOFE can en- able better optimization of the non-stationary exploration bonus, in this case for E3B. We
Reward-free Exploration Habitat
consider the reward-free setting for purely ex-
ploratory behaviors. For this reason, we use
the Habitat simulator (Savva et al., 2019; Szot
Map Coverage
ctt al., 2021) and the HM3D dataset (Ramakr- ishnan et al. 2021), which contains 1,000 dif- ferent scenes photorealistic apartments for 3D navigation. We train E3B and proposed augmented versions for 10M environment steps and measure their map coverage in a of 100 held-out scenes. We optimize the E3B explo- ration bonus with PPO (Schulman et al.) (2017)
Augmented Matri
Oiagonal
Figure 8: Map coverage on a held-out set 100 3D scenes of the HM3D dataset. The E3B agents trained using SOFE explore the new scenes better.
which requires 31 hours in machine with single GPU. We show the results Figure 8 sponding to the results presented in Section
In Figure A.I
we show the learning curves corre-
Minihack MultiRoom (N6)
Procgen Maze
-10
MD
E36 Augrenntd Digonal
E38
E35 Augieented Diggnnal
Augrentac
Augrenttd
Figure 9: E3B (blue) outperforms the other deep exploration baselines in the hard-exploration and partially-observable tasks of MiniHack and Procgen-Maze. With SOFE (Red and Green), we further increase the performance of E3B.
14

---

Published as a conference paper at ICLR 2024
A.2 ANALYSIS OF THE BEHAVIOURS LEARNED BY SOFE
By using SOFE on count-based based methods, RL agents extract features from the state-visitation fre- quencies and use them for "reision-making To better understand how the agents use the augmented information, we artificially create an object No with No(s;) > AA, E {S 8y and No(s,) = 0. Intuitively, we communicate the agent that all states in the state space but sj have already been visited through the state-visitation frequencies. We evaluate PPO agents pre-trained on reward-free episodic exploration and show the results Figure [0] Results show that augmented agents ficiently direct their exploration towards the unvisited states, self-identifying these as goals. This reveals how the agents leverage the augmented information for more efficient exploration.
Figure 10: Analysis of how RL agent uses SOFE for better exploration. Yellow boxes show the agent's starting position. Red boxes show the goals' positions, which are the only positions for which set Ns(ss) 0, and green traces show the agent's trajectory. Augmented agents pre- trained on episodic exploration efficiently direct their exploration toward the unvisited states.
A.3 TRAINING DETAILS
A.3.1 NETWORK ARCHITECTURE
We use Stable-Baseliness (Raffin al. 2021 run our experiments in the mazes, Godot maps, and DeepSea. For DQN, PPO, A2C, and SAC we use the same CNN to extract features from the observation. The CNN contains convolutional layers with kernel size of (3 3), stride 2, padding of 1, and 64 channels. The convolutional layers are followed by fully connected layer that produces observation embeddings dimension 512. For the augmented agents, we use an additional CNN with the same configuration to extract features from di. The augmented agents concatenate the representations from the observation and the parameters Φ; and feed these to the policy for decision-making, while the vanilla methods (e.g. counts, S-Max) only extract features from the observations. We show the CNN architecture in Figure A.3.1
G4xBAB
32
3x00x32
Convolution p=1)
Convolution
Convolution p=1)
(te3, p=1)
ReLU
For Minihack and Procgen, we use the official E3B codebase, which contains baselines for ICM and RND, and uses IMPALA to optimize the exploration bonuses. We use the same policy architecture as in Henaff et al. 2022), which contains an LSTM. We ran the experiments in Minihack and Procgen for 100M steps. For the augmented agents, design CNN that contains 5 convolutional layers with a kernel size of (3 x 3) and stride and padding 1, batch normalization layers after every convolutional layer, max-pooling layers with kernel size of (2 x 2) and stride of 1, followed by
15

---

Published as a conference paper at ICLR 2024
fully-connecter layer that produces embeddings of dimension 1024. This architecture allows to extract features from the 512x512 ellipsoids, which are later passed together with the observation features to the policy for decision-making We show the CNN architecture in Figure ??.
1012-212
121004
320 200122
Convolutinn
Convolution p=1)
Convolutio p=1)
Convolution p=t)
Dense
p=1)
ReLU
BatchNorm
BatchNorm
BatchNorm
BatchNorm
MaxPool2O (K=2)
(I=2)
(K=2))
(k=2)
(K=2)
ReLU
ReLU
ReLU
ReLU
ReLU
16

---

Published as a conference paper at ICLR 2024
A.4 ALGORITHM HYPERPARAMETERS
A.4.1 DQN
Hyperparameter
Value
Num. Envs
16
Learning Rate
0.0001
Buffer Size
1000000
Learning Starts
50000
Batch Size
32
Tau
1.0
Gamma
0.99
Train Frequency
4
Gradient Steps
4
Target Update Interval
10000
Exploration Fraction
0.1
Exploration Initial Epsilon
1.0
Exploration Final Epsilon
0.05
Max Grad Norm
10
Table Hyperparameters for the DQN Implementation
A.4.2 PPO
Hyperparameter
Value
Num. Envs
Learning Rate
0.0003
Steps
2048
Batch Size
64
Epochs
10
Gamma
0.99
GAE Lambda
0.95
Clip Range
0.2
Normalize Advantage
True
Entropy Coefficient
0.0
Value Function Coefficient
0.5
Max Grad Norm
0.5
Table 3: Hyperparameters for the PPO Implementation
A.5
A2C
Hyperparameter
Value
Num. Envs
16
Learning Rate
0.0007
N Steps
5
Gamma
0.99
GAE Lambda
1.0
Entropy Coefficient
0.0
Value Funetion Coefficient
0.5
Max Grad Norm
0.5
RMS Prop Epsilon
10-5
Use RMS Prop
True
Normalize Advantage
False
Table 4: Hyperparameters for the A2C Implementation
17

---

Published as a conference paper at ICLR 2024
A.6 ENVIRONMENT DETAILS
A.6.1 MAZES
We designed the mazes in Figure with Griddly (Bamford 2021). The mazes are size 32x32.
The agents observe entity maps: matrices of size smap_site, mnep_size) of entity id's (e.g. 0 for the floor, for the wall, and 2 for the agent). The action space is discrete with the four-movement actions (i.e. up, right, down, left). In the following, we now show example of the observation space of 5x5 variation of the mazes we use throughout the paper following the OpenAl Gym interface (Brockman et 2016).
obs_shape env env.
shape
obs shape (3,5,
obs, reward, done, info = env. ...
obs avatar in these lccations 10,0, 0, 0, 0l, [0, 0,1,0, 0,0,, 10,0,0,0,01,
10,0,0,0,01
10,0,0,0,0]
1, I wall in these locations 11,1,1,1,11,
11,0, 0, 11,0,0,0,1] [1,1,1,1,1]
[ # goal these lccations
00,0,0, 0,0], 10,0,0, 0, 10,0,0, 0,0,00 01 10,0, 0, [0,0, 0,0,0,
A.6.2 DEEPSEA
The DeepSea environment taken from (Os- band et al.) 2019) and has been used to evalu- ate the performance of intrinsic exploration ob- jectives (Schäfer et al. 202T). DeepSea repre- sents hard-exploration task in a N x N grid where the agent starts in the top left and has to reach a goal in the bottom right location. At each timestep. the agent moves one row down and can choose whether to descend to the left or right. The agent observes the 2D one-hot en- coding of the grid and receives a small nega- tive reward of -0.01 for moving right and 0 re- wards for moving left. Additionally, the agent receives a reward of +1 for reaching the goal and the episode ends after N timesteps. Hence, is very hard for an agent trained on extrinsic rewards only solve the credit assignment problem and realize that going right is the optimal action.
N
Figure 11: The DeepSea environment.
18

---

Published as a conference paper at ICLR 2024
The episodes last exactly N steps, and the complexity of the task can be incremented by increasing N.
A.6.3 GODOT MAP
We the Godot game engine design the 3D world used in Section 5.1 which we open-source together with the code. We show global view of the map in Figure A.6.3 The map is of size 120x120 and has continuous state and action spaces. To apply count-based methods we discretize the map in bins of size 5, resulting in 24x24 object Mr. The observations fed to the agent are the result of shooting several raycasts from the agent's perspective. The observations also contain global features like the agent's current velocity, rotation, and position. The action space contains continuous dimensions that control the velocity, rotation, and jumping actions.
Figure 12: Global view of the 3D Godot map.
A.6.4 MINIHACK MULTIROOM
We use the Multiroom-N task from the Minihack suite (Samvelyan et al., 2021 to evaluate the performance of E3B and our proposed augmentation, originally used in Henaff et al. (2022). The environment provides pixel and natural language observations and generates procedurally generated maps each episode. The rewards are only non-zero when the agent finds the goal location in a maze that contains 6 rooms. We use the same policy architecture described in Section C.I.1 in Henaff al. 2022).
env i ronment observation St
message
The door resists!
symbolic image
...)......
3-kx...X
stats vector
34, 1,
14,
2, 7, o]
(x, y) position
time step
Figure 13: An observation from MiniHack.
19

---

Published as a conference paper at ICLR 2024
A.6.5 PROCGEN MAZE
We use the Procgen-Maze task from the Procgen benchmark (Cobbe al. 2020) evaluate the performance of E3B and our proposed augmentation. We use the memory distribution of mazes. The mazes are procedurally generated each episode, have different sizes, and use different visual assets. Proegen-Mazz provides pixel observations and the rewards are only non-zero if the agent finds the goal location.
Figure 14: Procedurally generated mazes from the Procgen- environment.
A.7 STATE-ENTROPY MAXIMIZATION
In this section, we provide the pseudo-code for the surprise-maxinizattee algorithm presented in Section 3.1.3 Note that the update rule for the sufficient statistics the generative model pa. is Markovian mentioned in the paper, we use a Gaussian distribution to model Pø., and hence when using SOFE, pass its mean and standard deviation to the RL agents.
updated at each timestep with the new information from the next state. As
Algorithm Surprise Maximization
1: while not converged do
2: B (
b Initialize replay buffer
3: for episode ... M do
4:
So t p(so): + {oo} for t 0, ....... do
5:
6:
no (ar((s)
Get action
7:
8141 T(s++1 86, at)
Step dynamics
8:
log (8t+1)
P. Compute intrinsic reward
9:
T++1 - T ($441
D. Update trajectory
10:
Φ111 t U(T++1)
Update mean and variance
11:
+ U {(st, az, re, 8(41)}
12:
end for end for
13:
14:
0-- (T)
Update policy
15:
end while
20

---

PPO-LSTM C. Aug. Published as a conference paper at ICLR 2024
A.8 EXPLORATION FOR SPARSE REWARDS
In this section, we show the complete set of results for Section 5.2 The results include confidence intervals and learning curves for DQN, A2C, PPO, and PPO-LSTM for the task of reaching the goal in Maze 2 in Figure 2 We also include the partiall--bseervalee setting, where the agent does not observe the full maze but 5x5 agent- observation.
Episodic Exploration, v V-Reward, Full Obs.
Global Exploration, v Rewarrd, Full Obs.
PRO-LSTM C. Aug.
PPO-LSTM
PPO-LSTM
PPO + C. Aug.
PPO C. + Aug.
PPO
PPO
PPO
PPO
DON Aug.
DON
DON
DON
DON
DON
Aug.
Aug.
A2C
A2C
0.0
0.3
0.6
0.9
0.0
0.3
0.6
0.9
Mean Episode Return
Mean Eplsode Return
Episodic Exploration, V-Reward, Partial Obs.
Global Exploration, r-Reward, Partial Obs.
IQM
PRO-LSTM Aug.
PPO C. Aug. PRO-LSTM
PPO-LSTM
PPO-L5TM
PPO-LSTM
PPO Aug.
PPO C. Aug.
PRO
PPO
DON C. Aug.
DON
DON
DON
DON
AZC C. Aug.
Aug.
A2C C.
A2C
A2C
0.0
0.3
0.9
21

---

Published as a conference paper at ICLR 2024
Episodic Exploration, Salesman Reward,
Global Exploration, Reward, Full Obs.
IQM
PPO-LSTM Aug.
PRO-LSTM
PPO-USTM
Aug.
PPO + C. + Aug.
PPO
PR0
PPO
PPO
DON C. Aug.
DON + C. Aug.
DON
DON
DON
DON
AZC
Mean Episode Return
Mean Episode Return
Episodic Exploration, Reward, Partial
Global Exploration, Reward, Partial Obs.
IQM
PPO-LSTM C. Aug.
PPO-LSTM C. Aug.
PPO-LSTM
PPO-LSTM
PPO-LSTM
PPO-LSTM
PPO Aug.
PPO + + Aug.
PPO
PPO
PYU
WO
DON Aug.
DON Aug.
DON
Aug.
Aug.
Mean Episade Return
Mcan Episode Return
Episodic ation/ (-Reoaard Full Ob.
Global Exploration Rewward Futt
PPO
MW00y 1441000 PM
Anauy wwwmmm
Graisrmee SHII
PRO LSTM
DON
22

---

Published as a conference paper at ICLR 2024
Global Exploration F /-Reward Partta IPO
-
SHpp STM
Strgs DON
DOW CCC
Enitor CoormmerSogg
Divioomell Dags
Exvioement Stept
Episodic Exploration: 5 Salesman Reward / Full Obs.
Global Expleration Salesman Reward Full Ots.
Ewiseemed PRO-LSTH
Stepe
LSTM
COAA
Erannoe Strps
Ephodic Exploration
Partial
AWINac Saourd
www
Stope
Draneemen
PRO-LSTH
FRO-LSTM
DOR
133
WOLITe-
23

---

Published as a conference paper at ICLR 2024
A.9 REWARD-FREE EXPLORATION
In this section, we show the complete set of results for Section 5.1 The results include learning for DQN, A2C, PPO and PPO-LSTM measuring the map coverage achieved by these algo- rithms in the mazes in Figure
A.9.1 DQN
DQN Episodic Exploration v- Reward
Maze
Maze
Maze
Episodic map coverage (%)
100
Full obsevabiitty
80
60
Episodic map coverage (9)
Partial abserubiity
60
20
0.0
0.5
10
15
0.0
05
10
15 le7
0.0
05
10
15
Environment steps
le7
Environment steps
Environment steps
le7
DON + Aug.
DON C.
Random
DQN Global Exploration v- Reward
Maze
Maze
Maze
100
Full observailityy
80
Map coverage
60
Partial obvervailiyy
Map coverage (%)
60
20
0.0
10
0.0
Environment steps
Environment steps
Environment steps
DON + Aug.
DON C.
Random
24

---

Published as a conference paper at ICLR 2024
DQN Episodic Exploration Salesman Reward
Maze
Maze
Maze
Episodic map coverage (%6
100
Full abservabiltt
80
60
Episodic map coverage (%)
Partial observablity
20
0.0
Environment steps
Environment stops
Environment steps
DON + C. + Aug.
DON C.
Random
DQN Global Exploration Salesman Reward
Maze
Maze
Maze
100
Full observailityy
80
Map coverage
60
Partial obvervailiyy
Map coverage (%)
60
20
0.0
10
0.0
Environment steps
Environment steps
1c7
Environment steps
DON + Aug.
DON C.
Random
25

---

Published as a conference paper at ICLR 2024
A.9.2 A2C
A2C Episodic Exploration Reward
Maze
Maze
Maze
Episodic map coverage (%6
100
Full observability
80
60
Episodic map coverage (%)
Partial obiervailiyy
20
0.0
0.0
Environment steps
Environment steps
167
Environment steps
A2C + C. + Aug.
A2C + C.
Random
A2C Global Exploration Reward
Maze
Maze
Maze
100
Map coverage (%)
Full abservabilty
80
60
100
Purtial observabiity
Map coverage (%)
20
Environment steps
Environment steps
Environment steps
A2C + + Aug.
A2C +
Random
26

---

Published as a conference paper at ICLR 2024
A2C Episodic Exploration Salesman Reward
Maze
Maze
Maze
Episodic map coverage (%6
100
Full abservabilty
80
60
Episodic map coverage (%)
Partial observablity
20
Environment steps
Environment stops
Environment steps
A2C + C. + Aug.
A2C +
Random
A2C Global Exploration Salesman Reward
Maze
Maze
Maze
100
Full observablityy
80
Map coverage
60
Partial obvervailiyy
Map coverage (%)
60
20
0.0
10
0.0
Environment steps
Environment steps
1c7
Environment steps
AZC + Aug.
A2C +
Random
27

---

Published as a conference paper at ICLR 2024
A.9.3 PPO
Episodic Exploration Reward
Maze
Maze
Maze
Episodic map coverage (%6
100
Full observability
80
60
Episodic map coverage (%)
Partial obiervailiyy
20
0.0
Environment steps
Environment steps
1c7
Environment steps
PPO C. + Aug.
PPO C.
Random
PPO Global Exploration Reward
Maze
Maze
Maze
100
Map coverage (%)
Full abservabilty
80
60
100
Purtial observabiity
Map coverage (%)
20
Environment steps
Environment steps
Environment steps
PPO Aug.
PPO Random
28

---

Published as a conference paper at ICLR 2024
Episodic Exploration Salesman Reward
Maze
Maze
Maze
Episodic map coverage (%6
100
Full abservabilty
80
60
Episodic map coverage (%)
Partial observabliyy
m
20
Environment steps
Environment stops
Environment steps
PPO C. Aug.
PPO +
Random
PPO Global Exploration
Salesman Reward
Maze
Maze
Maze
100
Full observabiityy
80
Map coverage
60
Partial obiervbilty
Map coverage (%)
20
15
0.0
10
Environment steps
Environment steps
Environment steps
PPO Aug.
PPO C.
Random
29

---

Published as a conference paper at ICLR 2024
A.9.4 PPO-LSTM
PPO-LSTM Episodic Exploration Reward
Maze
Maze
Maze
Episodic map coverage (%6
100
Full observability
80
60
Episodic map coverage (%)
Partial obiervabliiy
20
0.0
Environment steps
Environment steps
lc7
Environment steps
PPO-LSTM + C. + Aug
PRO-LSTM C
Random
PPO-LSTM Global Exploration Reward
Maze
Maze
Maze
100
Map coverage (%)
Full abservabilty
80
60
100
Purtial observabiity
Map coverage (%)
20
Environment steps
Environment steps
Environment steps
PPO-LSTM + + Aug.
PRO-LSTM C
Random
30

---

Published as a conference paper at ICLR 2024
PPO-LSTM Episodic Exploration Salesman Reward
Maze
Maze
Maze
Episodic map coverage (%6
100
Full observabilty
80
60
Episodic map coverage (%)
Partial observablity
20
Environment steps
Environment stops
Environment steps
PPO-LSTM + C. + Aug.
PRO-LSTM +
Random
PPO-LSTM Global Exploration
Salesman Reward
Maze
Maze
Maze
100
Full observablityy
80
Map coverage
50
Partial obvervailiy
Map coverage (%)
20
15
0.0
10
Environment steps
Environment steps
1c7
Environment steps
PPO-LSTM + Aug.
PRO-LSTM
Random
31


Reviews:
[{"review_id": "1bgBZvVZLq", "reviewer": "Reviewer_7vAW", "summary": "This paper focuses on a problem faced by approaches that use intrinsic rewards for guiding exploration; they introduce non-stationarity in the reinforcement learning objective. This non-stationarity in rewards can destabilize learning in many RL approaches. The non-stationarity arises because the agent cannot predict the intrinsic reward as the features required to predict it (such as visitation counts) are not observable.\n\nFocussing on generalizations of count-based approaches, the paper proposes Stationary Objectives For Exploration (SOFE) that aim to augment the state space with sufficient statistics for intrinsic reward prediction to eliminate the partial observability and associated non-stationarity.\n\nExperiments in sparse and no-reward navigation tasks show that including SOFE improves the performance of a previously proposed count-based intrinsic reward approach (E3B).", "questions": "Q1. There still remains a source of non-stationarity for RL as the considered tasks have a maximum number of steps, but agents are not provided the time step in the state. In the episodic exploration setting $C_t$ can probably be used to infer the time step. However, it might make sense to include the time step in the global exploration setting. I am curious to know the authors’ thoughts regarding this and whether they have tried something along these lines.\n\nQ2. Is there a reason why Figure 3 shows visitations with A2C as the base algorithm, but the evaluation in Figure 4 uses PPO? Would these qualitative results change with the base algorithm?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.** Using count-based bonuses to explore sparse-reward environments is a widespread technique in RL. Improving the performance of count-based approaches would interest the research community.\n\n**S2.** The paper focuses on a relatively under-explored issue of addressing the non-stationarity introduced due to count-based bonuses. The proposed solution of augmenting states with sufficient statistics for intrinsic rewards seems novel, simple, and well-motivated.\n\n**S3.** The paper presents concepts clearly and is easy to follow.", "weaknesses": "**W1.** The paper should adjust the claim that it identifies that intrinsic reward functions induce a non-stationary RL objective. While solutions may be under-explored, the issue is generally known and noted in previous papers (e.g., [1]). Other works have chosen not to tackle the non-stationarity in intrinsic reward as it slowly varies and could be tracked [2]. Also, see the references in W2.\n\n**W2.** A crucial weakness of the paper is that it misses comparisons with previous works that have proposed decoupling exploration and exploitation policies [3, 4] to tackle the non-stationarity introduced due to intrinsic bonuses. The paper would significantly benefit from comparing their approach with baselines based on decoupling exploration and exploitation.\n\n**W3.** An aspect that merits further discussion is that RL algorithms with recurrent architectures (like PPO + LSTM used in the paper) could learn representations that resolve this partial observability [5]. One way to promote this effect could be to have an auxiliary task of predicting intrinsic rewards (which could make for an interesting baseline). \n\nWhile this might perform worse compared to directly providing sufficient statistics in the state (in terms of sample efficiency), it is potentially a more general solution to the problem for other intrinsic rewards.\nFor instance, it has previously been shown that LSTMs can learn to count in discrete settings [6], which could be helpful in episodic exploration settings. Previous work has also shown that recurrent architectures can learn contexts that resolve non-stationarity due to partial observability in bandit problems [7].\n\n\nOverall, I appreciate the direction the authors took to address the non-stationarity introduced by intrinsic rewards. Should the weaknesses and questions be adequately addressed/clarified, I would gladly increase my score.\n\n\n—------------------—------------------—------------------—------------------—------------------\n\n### References\n\n[1]Singh, S., Lewis, R. L., Barto, A. G., & Sorg, J. (2010). Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2), 70-82.\n\n[2] Şimşek, Ö., & Barto, A. G. (2006, June). An intrinsic reward mechanism for efficient exploration. In Proceedings of the 23rd international conference on Machine learning (pp. 833-840).\n\n[3] Whitney, W. F., Bloesch, M., Springenberg, J. T., Abdolmaleki, A., Cho, K., & Riedmiller, M. (2021). Decoupled exploration and exploitation policies for sample-efficient reinforcement learning. arXiv preprint arXiv:2101.09458.\n\n[4] Schäfer, L., Christianos, F., Hanna, J. P., & Albrecht, S. V. (2021). Decoupled reinforcement learning to stabilise intrinsically-motivated exploration. arXiv preprint arXiv:2107.08966.\n\n[5] Ni, T., Eysenbach, B., & Salakhutdinov, R. (2021). Recurrent model-free rl can be a strong baseline for many pomdps. arXiv preprint arXiv:2110.05038.\n\n[6] Suzgun, M., Belinkov, Y., & Shieber, S. M. (2019). On Evaluating the Generalization of LSTM Models in Formal Languages. In Proceedings of the Society for Computation in Linguistics (SCiL)\n\n[7] Ramesh, A., Rauber, P., Conserva, M., & Schmidhuber, J. (2022). Recurrent Neural-Linear Posterior Sampling for Nonstationary Contextual Bandits. Neural Computation, 34", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XiVTo1TEw8", "reviewer": "Reviewer_4EvL", "summary": "The paper claims that exploration bonus induces a non-stationary reward function, which causes instability in policy learning. The paper proposes to solve this instability by incorporating a sufficient statistic over the exploration bonus. The paper conducts experiments on 2D and 3D maze-like environments and demonstrates the proposed method can cover the state space more efficiently when compared to existing approaches.", "questions": "- I find the paper confusing to read at times:\n\t- There is some mention of $\\phi_t$ but in experimentation we only mention $N_t$ (i.e. count) and $C_t$ (i.e. elliptical bonus). Are they the $\\phi_t$?\n\t- Under section 5.1, what is $N_0$? Is it a binary mask over the map in the maze with only the $j$'th cell is 0? Is the observation the \"state\"?\n\t- What is salesman reward and $\\sqrt{}$-reward on figure 5? I believe it is Eq. 3 and 2 respectively\n\n**Possible typos**\n- Abstract, fourth last line: \"holds\" instead of \"hold\"\n- Generally, \"state visitation frequency\" should be \"state-visitation frequency\"\n- Page 3, first paragraph, fourth last line: \"should not\" instead of \"shouldn't\"\n- Five lines after Eq. 1: $\\forall s_t, a_t$ instead of $\\forall_{a, s, t}$\n- Page 4, second paragraph, line 1: \"Eq. 2 and 3\" instead of \"2 and 3\".", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea is simple and intuitive---by augmenting the state space the method converts the non-stationary exploration bonus to stationary.\n- The results on navigation tasks seem promising---in particular the policy appears to explore the maze more efficiently and recovers behaviour akin to a goal-conditioned policy when the goal state is set to unvisited", "weaknesses": "**Comments**\nI am happy to increase my score after these points are addressed:\n- The formulation in E3B doesn't really cover the action which is also important for count-based exploration for particular tasks (e.g. deterministic dynamics will be okay but not stochastic). Consequently, I don't think this is particularly a sufficient statistic but only for environments tested.\n\t- This also raises another question---why does the paper only focus on navigation tasks? There are many other difficult exploration tasks (e.g. Montezuma's Revenge on Atari, Minecraft, etc.)\n\t- The paper claims that it is a sufficient statistic and empirically demonstrated that it does perform well, but it will be great if there is a proof showing that this is true under some assumptions.\n- In section 4, the paper indicates that \"we consider that the only unobserved components in the POMDP are the parameters of the reward distribution.\" I am curious as to why this is a good assumption? In particular if $\\phi_t$ is only a sufficient statistic for exploration bonus, it may not be a sufficient statistic for deriving the state $s_t$ (with observation $o_t$).\n\t- By adding $\\phi_t$ to the state space, we are essentially exploding the state space (potentially to $\\infty$.) from a finite state space (e.g. the maze example in experimentation.) What is the intuition that the algorithm is still able to tractably find a optimal policy?\n\t- In continuous state-action space, the counting mechanism depends on the quantization---how do we still ensure stationary reward in this case? I believe it will be non-stationary reward since we cannot differentiate two states within the same bin. Otherwise, we can just use global timestep as the count.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOcMgLiIhK", "reviewer": "Reviewer_Yao3", "summary": "This paper studies intrinsic motivation for RL. First, the authors notice that common intrinsic bonuses (count-based is considered as the illustrative instance) induce non-stationary objectives, which can make the learning process unstable. Then, they propose a solution to this problem by designing a method, called SOFE, to make the objective stationary through state augmentation. Finally, they empirically evaluate the proposed algorithm in a variety of domains, including 3D navigation and ProcGen tasks.", "questions": "1) Count-based bonuses have been largely employed for hard-exploration tasks, either in terms of regret minimization in theoretical papers or shaping of sparse rewards in empirical literature. From my perspective, count-based bonuses are not a great fit for pure exploration instead: They are not only non-stationary, as the authors noted, but also vanishing, and suffer from some other well-known issues (see Ecoffet et al., Go-Explore: A new approach for hard-exploration problems, 2021). However, only in the pure exploration setting the POMDP argument and optimality of history-based policies seem to make sense. If an external reward is there, it is a actually good to converge to a Markovian deterministic policy. Can the authors comment on why they think it is worth studying count-based methods for pure exploration?\n\n2) To follow-up the previous question, count-based bonuses have been used in recent reward-free RL literature as well (e.g., Jin et al., Reward-free exploration for reinforcement learning, 2020). Also in the latter setting, the bonuses are freezed at the start of the episode, so that a Markovian policy optimizing a stationary objective is always deployed to collect data.\n\n3) To address pure exploration settings, other stationary objectives have been designed, e.g., state entropy maximization. Can the authors relates their contributions with this stream of works:\n- Hazan et al., Provably efficient maximum entropy exploration, 2019;\n- Mutti et al., Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate, 2021;\n- Liu & Abbeel, Behavior from the void: Unsupervised active pre-training, 2021;\n- Seo et al., State entropy maximization with random encoders for efficient exploration, 2021;\n- Yarats et al., Reinforcement learning with prototypical representations, 2021;\n- and many others.\nMoreover, is state entropy maximization a meaningful baseline for the reported experimental evaluation?\n\n4) While I am not particularly familiar with the literature of count-based methods, my feeling is that this kind of bonuses have been deeply studied. From a brief research, I am not sure the authors made a thorough due diligence of prior works, especially those addressing the limitations of count-based methods (e.g., Shafer et al., 2020 and Ecoffet et al., 2021 mentioned before). Interestingly, previous works also showed that $1/n$ bonuses bring faster learning in pure exploration settings (Menard et al., Fast active learning for pure exploration in reinforcement learning, 2021).\n\n5) How are the pseudocounts implemented in the experiments? This seems to make a huge different. For instance, Bellemare et al. (2016), Ostrovski et al., Count-based exploration with neural density models (2017), Tang et al., (2017), Machado et al., Count-based exploration with the successor representation (2020) all make different design choices with varying results. Do the authors think their results are general for every implementation of count-based bonuses or for just one?\n\n6) How many seeds are considered in the experiments and what is the meaning of the shaded areas? This crucial information seems to be missing for some of the results.\n\n7) In the ProcGen Maze experiment, the paper comapres SOFE with E3B, ICM, RND. These does not seem to be the state of the art for procedurally generated tasks. Do the authors considered also tailored methods, such as (Ghosh et al., Why generalization in RL is difficult, 2021; Zisselman et al., Explore to generalize in zero-shot RL, 2023)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- (Originality) Although limitations of count-based bonuses for intrinsic exploration have been considered before, the algorithmic idea of augmenting the state representation to make the objective stationary is new to the best of my knowledge;\n- (Significance) The experimental results look promising at least and experiments are carried out in some challenging and interesting domains, such as ProcGen Maze;\n- (Clarity) The main ideas of the paper are presented with clarity, but most of the relevant implementation details are deferred to a very brief section in the supplementary (Appendix A.5).", "weaknesses": "- (Motivation) The paper does not clarify that most of the reported considerations are valid when count-based bonuses are the actual learning objective rather than shaping of (sparse) external rewards. The former is arguably not the setting count-based methods have been designed for;\n- (Novelty and scope) Identifying the non-stationarity of the bonuses as a limitation of count-based methods does not look to be novel (e.g., Schafer et al., Decoupled reinforcement learning to stabilise intrinsically-motivated exploration, 2022) and the paper does not credit previous works on that;\n- (Robustness of the empirical evaluation) The experiments section does not specify how count-based bonuses are implemented. Several strategies exist, and they are known to make a significant difference in the resulting performance, which leaves one wondering how general the reported comparison really is. Moreover, the paper does not compare SOFE against alternative pure exploration methods aside from E3B, ICM, RND in Fig.7.\n\n**EVALUATION**\n\nWhile the paper is overall interesting, as it tackles a relevant problem on the inherent non-stationarity of count-based bonuses and provides some promising experimental results, I think it is falling short of the quality required for an ICLR paper in terms of clarity of the motivation, discussion of prior works, and robustness of the empirical evaluation. Hence, I am currently providing a slightly negative evaluation, while I report some detailed comments and questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W6vVuvdyLW", "reviewer": "Reviewer_GnJy", "summary": "This paper focuses on count-based exploration (or its variants) in reinforcement learning problems. It investigates the issue that count-based bonuses make the reward function non-stationary. The authors propose to augment the state space with visitation count for states (or representative embeddings of states). Therefore, the reward function in count-based exploration becomes stationary and satisfies Markovian property (i.e. the reward is fully determined by the state in this and next step given the augmented state space).\n\nThe authors conduct extensive experiments on 3D navigation maps, and procedurally generated environments with high-dimensional observation space. The experiments combine the proposed approach with multiple count-based exploration algorithms, on both reward-free and sparse-reward RL problems. The empirical results validate that it is beneficial for exploration to include state visitation count as a component in state space.", "questions": "It has been mentioned several times that \"we identify the matrix C_{t-1} as the sufficient statistics\", but it is still vague to me. Where did you identify this? Could you provide any rigorous mathematical proof showing that C_{t-1} is a sufficient statistics for the count-based rewards?\n\nIn Section 3.2., the paper introduces the notation \\phi_t without definition. For the context, I think it refers to sufficient statistics for count-based bonuses, but this point is very unclear in Section 3.2. Also, sufficient statistics in the problem setting are not defined. \n\nIn Equation 7, does s_t include sufficient statistics \\phi_t or not? I think the state s_t means a fully observable state in the augmentation MDP \\hat{M}, so s_t should already contain the information in \\phi_t. Then it is weird to have the reward function and policy depending on both s_t and \\phi_t in Equation 7.\n\nThe empirical results in Figure 4 look interesting. Could it successfully go to any goal position in the training map? If the exploration policy is trained on many different navigation maps. Could the policy be general enough to navigate to any goal position in unseen map?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed method is straightforward and relatively easy to implement.\nThe proposed method is flexible enough to be combined with many existing count-based exploration approaches.\nThe method is simple and effective in improving the existing count-based exploration approaches, given extensive experimental results.", "weaknesses": "The technical contribution is not significant enough. It is straightforward to add state visitation count as state input. When it comes to high-dimensional observation space, the proposed method is fully built on E3B and relies on its encoding of the distribution of observed embeddings. So the technical contribution is incremental upon E3B. Is E3B the best approach to solve any hard-exploration problems with high-dimensional observation space? If not, could the proposed method be general enough to improve other exploration approaches for hard-exploration problems with high-dimensional observations?\n\nThe proposed method is specifically constrained to the count-based exploration approach. As for other exploration methods with intrinsic motivations, such as RND (random network distillation), the proposed method is not compatible with RND and cannot be used to improve the exploration result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)