RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning
Authors: Mingqi Yuan
Roger Creus Castanyer
Bo Li
Xin Jin
Wenjun Zeng
Glen Berseth
Venue: Accepted by TMLR ()

Abstract: Extrinsic rewards can effectively guide reinforcement learning (RL) agents in specific tasks. However, extrinsic rewards frequently fall short in complex environments due to the significant human effort needed for their design and annotation. This limitation underscores the necessity for intrinsic rewards, which offer auxiliary and dense signals and can enable agents to learn in an unsupervised manner. Although various intrinsic reward formulations have been proposed, their implementation and optimization details are insufficiently explored and lack standardization, thereby hindering research progress. To address this gap, we introduce RLeXplore, a unified, highly modularized, and plug-and-play framework offering reliable implementations of eight state-of-the-art intrinsic reward methods. Furthermore, we conduct an in-depth study that identifies critical implementation details and establishes well-justified standard practices in intrinsically-motivated RL. Our documentation, examples, and source code are available at [https://github.com/RLE-Foundation/RLeXplore](https://github.com/RLE-Foundation/RLeXplore).

Published in Transactions on Machine Learning Research (04/2025)
RLeXplore: Accelerating Research in Intrinsicall-MMoiivatee Reinforcement Learning
Minggi Yuan"
minggi. ponnoot-yyaneccomee.pp.......u
Deportment of Computing The Hong Kong Polgtechnic Universily
Roger Creus Castanyer Mila Québec Al Institute & Université de Montréal
mager.corru-ccastttnggrmmmiiii quecbb
Bo Li
comp-bo- bo. li@ppoyy....eeee edu.k.
Department of Computing The Hong Kong Polytechnic University
Xin Jin
juwinneitechh.cmm edu.cr
Ningho Institute of Digital Twin Eastern Institute of Technology, Ningbo
Wenjun Zeng
wzeng-upBeitech.co edu.cr
Ningbo Institute of Digital Twin Eastern Institate of Technology, Ningbo Fellow, IEEE and CAE
Glen Berseth
glen, www.ebrrsserrrooo @mriilalttmmaaammmmmmm quebee
Mila Québec AI Institute & Université de Montréal
Reviewed on OpenReview: https: // openreview. net/ forum? id= B9BHyTN426
Abstract
Extrinsic rewards can effectively guide reinforcement learning (RL) agents in specific tasks. However, extrinsic rewards frequently fall short in complex environments due to the signifi- cant. hurman elfort needed for their design and annotation. This limitation underscores the necessity for intrinsic rewards, which offer auxiliary and dense signals and can enable agents to learn in unsupervisod manner. Although various intrinsic reward formulations have been proposed, their implementation and optimization details are insufficiently explored and lack standardization, thereby hindering research progress. To address this gap, we introduce RLeXplore, a unified, highly modularized, and plug-and-play framework offering reliable im- plementations eight state-of- htate-of-the-ar intrinsic reward methods. Furthermore, we conduct an in-depth study that identifies critical implementation details and establishes well-justified standard practices intrinsically-motivvaed RL. Our documentation, examples, and source code are available at https://githhei thuaa/h///tt........ Ctwm//LEE-Pommaaiio///eelloooooo
1 Introduction
Reinforcement learning (RL) provides framework for training agents to solve tasks by learning from inter- actions with environment. At the core of RL is the optinization of reward function, where agents aim to maximize cumulative rewards over time (Sutton & Barto 2018). However, in complex environments, defin- ing extrinsic rewards that effectively guide an agent's 's learning process can be impractical, often requiring
Equal contribution. "oorrepponding author.

---

Published in Transactions on Machine Learning Research (04/2025)
domain-specific expertise. In practice, poorly defined extrinsic rewards can lead to sparse-reward settings, where RT. agents struggle due to the lack of meaningful learning signal (Burda et al. 2019ab.
As the RL community tackles increasingly complex problems, such as training generally capable RL agents, there is a need for more autonomous agents capable of learning valuable behaviors without relying on dense supervision (Jiang et al.
2023)- To address this challenge, the concept of intrinsic rewards has emerged as a
promising approach in the RL community (Burda et al. 2019b Pathak et al. 2017 Raileanu & Rocktäschel, 2020 Badia et al.) 2020 Henaff et al. 2022 Pathak et al.) 2019). Intrinsie rewards provide agents with additional learning signals, enabling them to explore and acquire skills across diverse environments beyond what extrinsic rewards alone can offer. However, computing intrinsic rewards often requires learning auxiliary models, heavy engineering, and performing expensive computations, making reproducibility challenging.
While several formulations of intrinsic rewards have been proposed (Pathak et al., 2017 Badia et al. 2020; Laskin et al. 2021), each with its potential benefits for improving agent learning, the field lacks a compre- hensive understanding of the comparative advantages and challenges posed by these methods. Importantly, existing literature reports varying performance when using the same intrinsic rewards, reinforcing the need for standardized framework and deeper understanding of the optimization and implementation details.
In this paper, we introduce RLeXplore, an open-source library containing high-quality implementations of state-of-the-art (SOTA) intrinsic rewards. RLeXplore offers a plug-and-play framework for researchers working on intrinsically-mmoiivted RL, enabling them to seamlessly integrate intrinsic rewards into their projects. Specifically, RLeXplore (1) facilitates fair comparisons across multiple intrinsic reward methods, (2) can be easily integrated with various RI. frameworks, and (3) streamlines the development, of new intrinsic reward methods. In Table compare the performance of the implementations in RLeXplore with the original results reported in previous works. In Appendix D we provide the full details on reproducibility with RLeXplore.
Table 1: Summary of comparative results from our RLeXplore implementations and prior work. Right two columns report success rates (% of episodes solving the task), except for Procgen where we report mean episode rewards after training. See Appendix D for reproducibliity and evaluation details,
Environment
Intrinsic Reward
Original
RLeXplore
SuperMarioBros (10M Steps)
RIDE
23%
50%
SuperMarioBros (10M Steps)
ICM
30%
30%
MiniGrid-DDorrkey-16611 16 (10M Steps)
ICM
0%
60%
MiniGrid-DDorKKc-16x111 (10M Steps)
RND
0%
60%
MiniCrid-DoorKey-166666 (10M Steps)
RIDE
25%
12%
MiniGrid--DoorKey-BkS (1M Steps)
RE3
50%
95%
MiniGrid-DDooree-SS88 (1M Steps)
RND
0%
82%
MiniGrid-DoorKey--0 (1M Steps)
ICM
20%
83%
Procgen 200 Mazes (25M Steps)
E3B
3.00
4.10
Procgen 200 Mazes (25M Steps)
ICM
2.50
5.90
Procgen 200 Mazes (25M Steps)
RND
1.70
5.00
To support these capabilities, we have provided extensive documentation that includes detailed guides on using RLEXPLOORE, along with comprehensive code tutorials. These resources are designed to make it straightforward for users to get started with RLEXPLORE, regardless of their prior experience with intrin- sic rewards in RL. In Appendix D.6] we provide an overview of the main differences and advantages of RLEXPLORE compared to existing RL libraries.
We aim for the community to adopt RLEXPLORE as a standard tool for evaluating intrinsic reward methods, reducing implementation efforts, and mitigating inconsistencies in results and conclusions.
Our work presents a systematic study aimod at addressing gaps in understanding the critical implementa- tion and optimization details of intrinsic rewards. We investigate the design of different intrinsic reward methods and (1) highlight challenges in the reproducibility of prior work, and (2) share highly performant
2

---

Published in Transactions on Machine Learning Research (04/2025)
reimplementations of many popular methods. To guide investigation, we formulate numerous questions, aiming to uncover the intricacies of intrinsic rewards and their impact on RL agent performance. Our results highlight the importance of thoughtful implementation design for intrinsic rewards, showing that naive implementations can lead to suboptimal performance. Through carefully studied design decisions, we demonstrate significant performance gains.
Our results show that with RLeXplore, RL agents can learn emergent behaviours autonomously, solving multiple levels of SuperMarioBros without task rewards. Additionally, show that intrinsic rewards enable RL agents to obtain great performance complex sparse-reward tasks like Procgen-Maze, MiniGrid, the ALE-5 hard-cxploration tasks and Ant-UMaze.
2 Related Work
While some works have benchmarked intrinsic rewards specific environments Taiga et al. 2021 Wang
et al. 2022 Laskin al. 202T), their lack of detailed discussions on implementation and optimization leads to reproducibility problems (Voelcker et al.) 2024). In this work, we introduce RLEXPLORE, a comprehensive framework that incorporates the most widely-used intrinsic rewards, which provides standardized approach to enhance reproducibility, accelerate research, and facilitate the comparison of baselines in intrinsically- motivated RL. In the following, we overview existing formulations for intrinsic rewards of different natures and introduce the methods included in RLEXPLORE.
2.1 Count-Basse Exploration
Count-based exploration methods provide intrinsic rewards by mcasuring the novelty of states, defined to
be inversely proportional to the state visilation counts (Strehl & Littman 2008 Tang el al. 2017 Machado
et al.
2020 Jo et al.) 2022p. In finite state spaces, count- count-based methods perform near optimally
(Strehl &
Littman 2008). For this reason, these methods have been established as appealing techniques for driving
structured exploration in RL. However, they do not scale well to high-dimensional state spaces [Bellemare
et al. 2016 Lobel et al. 2023). Pseudo-counts provide a framework generalize count-based methods
higl-dimensional and partially observed environments (Bellemare et al., 2016 Ostrovski et al.) 2017 Martin
et al.]
2017). Burda et al. (2019b) proposed random network distillation (RND), which uses the prediction
error against fixed network as learning signal that is correlated to counts. Recently, Henaff et al. (2022) proposed E3B and showed that the intrinsic objective provides a generalization of counts to high-dimensional spaces. In RLEXPLORE, we include Pseudo-counts, RND, and E3B as representatives of the state-of-the-ar count-based based methods.
2.2 -uriosity-Driven Exploration
Curiosity-based objectives train agents to interact with the environment seeking to experience outcomes that are not aligned with the agents' predictions (Aubret et al. 2023). Hence, curiosity-driven exploration usually involves training an agent to increase its knowledge about the enviromment (e.g., environment dynamics)
(Stadie et al.) 2015 Pathak et al. 2017 Yu et al. 2020 The intrinsic curiosity module (ICM) (Pathak et al.] 2017 Burda et al. 2019a) learns a joint embedding space with inverse and forward dynamics losses and was the first buried method successfully applied to deep RL settings. Disagreement (Pathak et al. 2019) further extended ICM by using the variance over ensemble of forward-dynamice models to compute curiosity. However, curio methods consistently found to be unsuccessful when the environment has irreducible noise (Savinov et al., 2019). To address the problem, Railcanu & Rocktäschel (2020) proposed RIDE, which uses the difference between two consecutive state embeddings as the intrinsic reward and encourages the agent to choose actions that result in significant state changes. In general, curiosity-based objectives remain amongst the most popular intrinsic rewards in deep RL applications this day. In RLEXPLORE, we include ICM, Disagreement, and RIDE as representatives of the state-of of cdriosity-driven methods.
3

---

Published in Transactions on Machine Learning Research (04/2025)
2.3 Global and Episodic Exploration
Towards more general and adaptive agents, recent works have studied decision decision-making problems in contextual Markov decision processes (MDPs) (o.g., procedurally-generaaee
environments) (Railcanu & Rocktäschel Henaff et al. 2022: Matthews et al. 2024). Contextual MDPs require episodic-level exploration, where novelty estimates are reset at the beginning of each episode. Henaff et al. (2023) showed that both global and episodic exploration modalities have unique benefits and proposed combined objectives that achieve remarkable performance across many MDPs of different structuros. NGU (Badia et al. 2020) and RIDE RRileanu & Rocktäschel 2020) also instantiate both global and episodic bonuses. Inspired by this recent line of work, in this paper, we study novel combinations of objectives for exploration that achieve impressive rosults in contextual MDPs.
2020
3 Background
We frame the RL problem considering a MDP (Bellman 1957 Kaelbling et al. 1998) defined by a tuple M (S, R, P. da, where the state space, A is the action space, and reward function, P: S A A(S) is the transition function that defines probability distribution over S, do E ASS) is the distribution of the initial observation 8o, and y [0, is discount factor. The goal of RL is to learn policy Te(a(s) to maximize the expected discounted return:
x - R is the
(0) = Ex
Σ x Re 1=
(1)
Following prior work in intrinsiiallly-moiivveeed RL (Burda et al. function that combines both extrinsic and intrinsic components. defined as:
2019b). we consider an augmented reward The overall reward signal at time step is
Riotal Rs + Br It,
(2)
where / S A R is the intrinsic reward, Bg Su k) is the exploration cocfficient controlling the relative weight of intrinsic motivation over time, and is decay rate. This formulation enables the agent to balance task-specific objectives with exploratory behavior, particularly in sparse or deceptive reward settings. Finally, the augmented optimization objective is:
(+) E, Σ r (R, B 1.)
In Appendix A
we present detailed overview of the SOTA intrinsic reward methods that we implement in
RLEXPLORE.
4 RLeXplore
In this section, present RLeXplore, unified, highlly-oduularizedd and plug-and-play framework that currently provides high-quality and reliable implementations of eight SOTA intrinsic reward method:"] Com- paring multiple intrinsic reward methods under fair conditions challenging due to various confounding factors, such as using distinct RL frameworks (e.g., PPO (Schulman et al. 2017). DQN (Mnih et al. 2013 IMPALA (Espeholt el. al.] 2018)), optimization (e.g., reward and observation normalization, network archi- tecture) and evaluation details (e.g., environment configuration, algorithm hyperparameters). RLeXplore is designed to provide unified framework with standardized procedures for implementing. computing, and optimizing intrinsic rewards.
RR.eXpe complies with the MIT License.

---

Published in Transactions on Machine Learning Research (04/2025)
RLeXplore
Start Tremung
PSC
ICM
RIDE
RAD
DSG
NGU
I
Date hommut
updater]
Sampling ing Rollouts
S.
S
Rs
I E
π
Samples
Obseration
No
π
computed
Deworrt Scaler
Updanel
Intrinsic
(10
Conceptual framework.
(b) Implementation framework.
Mixed
Fabric(
Intrinsic
Reward
(c) Create mixed intrinsic rewards using the Fabric class.
Figure 1: The workflow of RLEXPLORE. (a) RLEXPLORE provides decoupled module for intrinsic rewards that integrates seamlessly with the RL training loop. RLEXPLORE implements SOTA intrinsic rewards and adapts to the ummodified RL training loop. (b) RLeXplore monitors the agent-environment interactions and gathers data samples using the watch() function. After collecting experience rollouts, RLeXplore computes the corresponding intrinsic rewards using the compute () function and updates the auxiliary models via the uudate function. (c) RLeXplore provides a Fabric class that allows developers to combine multiple intrinsic rewards in an elegant manner. In Appendix C.6 provide more details on how to add new intrinsic rewards RLeXplore.
4.1 Architecture
The core design decision of RLeXplore involves decoupling the intrinsic reward modules from the RL opti- mization algorithms, which enables our intrinsic reward implementations be integrated with any desired RL algorithm (or existing library, see Appendix and the official integration examples). Figure illustrates the basic workflow of RLXXplore, which consists of two parts: data collection (i.e., policy rollout) and reward computation.
Commonly, at each time step, the agent receives observations from the enviromment and predicts actions. The environment then executes the actions and returns feedback to the agent, which consists of a next observation, a reward, and terminal signal. During the data collection process, the wwachh function is used monitor the agent-environment interactions. For instance, E3B (Henaff al., (022) updates an estimate of an ellipsoid in an embedding space after observing every state. At the end of the data collection rollouts, .cmmptee computes the corresponding intrinsic rewards. Note that compute is only called once per rollout using batched operations, which makes RLeXplore a highly efficient framcwork. Additionally, RLeXplore provides several utilities for reward and observation normalization. Finally, the .) function is called immediately after compute() to train the reward module necessary (e.g.. train the forward dynamics models Disagreement (Pathak et al. 2019) or the predictor network in RND [Burda et al. 2019b)). Appendix illustrates the usage of the aforementioned functions. All operations are subject to the standard workflow of the Gymnasium API (Towers el. al. 2023
In particular, recent research (Henaff et al. 2023) has highlighted that mixed intrinsic rewards signif- icantly promote the agent's exploration capability by providing comprehensive exploration incentives. In RLeXplore, we provide a Fabric class that allows developers to combine multiple intrinsic rewards in an clogant manner, as illustrated in Appendix
RLeXplore offers several benefits the rescarch community:
For researchers seeking reliable tools for benchmarking and general applications: RLeXplore pro- vides high-quality implementations popular intrinsic reward methods, useful in both research and practical applications. It can be scamlessly integrated with existing RL libraries. We provide specific examples of integrating RLeXplore with
(Rafin et al. 2021), CleanRL
(Huang et al. 2022b), and RLLTE (Yuan al. 2023 in Appendix
and G
5

---

Published in Transactions on Machine Learning Research (04/2025)
For developers experimenting with new intrinsic rewards: RLeXplore offers modular components, such as various embedding networks and a standardized workflow. This setup facilitates the creation, modification, and testing of new ideas. Detailed examples are available in the code repository and documentation.
For promoting collaboration and accelerating progress: We have published space using Weights & Biases (W&B) to store reusable experiment results on recognized benchmarks. This initiative aims to enhance collaboration within the research community and speed up progress by providing easy access to established benchmark results.
4.2 Algorithmic Baselines
In RLeXplore, we implement cight widely-rocognized intrinsic reward methods spanning the different cate-
gories described in Section namely ICM (Pathak et al. 2017). RND (Burda et al.) 2019b), Disagreement
Pathak et al. 2019), NGU (Badia et al. 2020). PseudoCounts (Badia et al.) 2020), RIDE (Raileanu &
Rocktäschel 2020), RE3 (Seo et al.) 2021).
and E3B (Henaff et al. 2022) respectively, We selected them
based on the following tenet:
The algorithm represents a unique design philosophy;
The algorithm achieved superior performance on well-recognized benchmarks;
The algorithm can adapt to arbitrary tasks and can be combined with arbitrary RL algorithms.
For detailed descriptions of each method, refer the reader to Appendix A
5 Experiments
Our experiments aim to achieve two main objectives: (i) highlight how intrinsic reward methods sensitive to implementation details, and (ii) identify the best algorithmic and design choices to ensure high performance across various sparse reward environments to demonstrate the generality and robustness of our framework.
AIS
G fighot
D'S CIS
(a)
(b)
(c)
(d) MerioBrrs (b) MiniGrid. (c) ALE-5. (d) Proegen-
(e)
Figure 2: Screenshots of the selected exploration games. (a)
Maze. (e) Ant-UMaze.
First, we use SuperMariiorrs (SMB) without access to the enviromment's rewards to study the low-level implementation details of intrinsic reward methods that drive robust exploration. We selected SMB because effective exploration within this environment strongly correlates with task performance, making an excel- lent. benchmark for measuring the efficacy of exploration techniques. This enviromment has been widely used in previous studies on exploration in RL (Pathak et al., 2019: Raileanu & Rocktäschel] 2020 Burda et al. 2019a). To further generalize which is challenging due to the sparse rewards, making it difficult to solve with classical RL algorithms
findings, also the MiniGrid-DoorKey 1iree-11xx11 (MGD) environment,
The effectiveness of intrinsic rewards in MiniGrid environments has also been highlighted in prior works [Raileanu & Rocktäschel, 2020 Henaff et al. 2022 2023). With these two environments we aim to study the implementation details in both reward-free and sparse-reward tasks.
hhttps://ninigrt farama. http://///..attteatta//at/s///rr

---

Published in Transactions on Machine Learning Research (04/2025)
Disagreement
PseudoCounts
RIDE
Extrinsic
SuperMarioBros 1-1-v3 (Reward-
MiniGrid MeniGrid-DDorkey. Dewo-oooooooeeeeee
1.0
1.0
Normalized Episode Return
Normalizes lized Episode Return
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.2
0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.8
0.8
1.0
Environment Steps (x10')
Environment Steps (x107)
Figure 3: Episode returns achieved by the intrinsic rewards in RLeXplore. (left) Mupirnos without access to the task rewards. (right) MiniCrrid-DoorKey-16xx 16 with sparse rewards.
Secondly, to showcase the generalizability of RLeXplore, evaluate our implementations in additional sparse-reward environments, including Procgen, MiniGrid, Mnt-UMaze, and the set of five hard-oxploration games in the areade learning environment (ALE) suite. These experiments are designed to test how well our methods balance the use dense intrinsic rewards with sparse extrinsic rewards across a variety tasks. The complete set learning curves for all the experiments is shown in Appendix
Lastly, we explore recent advancements in using combined intrinsic rewards (Henaff al. 2023) enhance exploration in contextual MDPs. Specifically, single and combined intrinsic rewards can explore various game versions and generalize their exploration across different levels.
the full set of levels in SMB to evaluate how well both
In the following sections, we present results from SMB and MiniGrid for objective (i) and from Procgen-Mazz for objective (ii). Additionally, in Appendix D we show that using RLeXplore, we are able to reproduce and improve the performance reported in previous works for many intrinsic rewards and across multiple environments.
The design of these experiments is driven by our primary goal: to provide a general and reliable set of intrinsic reward implementations within a user-friendly framework. Instead of attempting to benchmark all methods across every possible domain, we focus on verifying the generality of each method within a carefully selected subset of popular exploration tasks. Finally, we provide the experimental dotails of test benchmarks and method configurations in Appendix
5.1 Low-level Implementation Details of Intrinsic Rewards
The performance of intrinsic rewards is affected by various factors that tend to vary with the complex- ity of the task. For instance, the RL algorithm used for optimization, the architecture of the net- works, algorithm--peeifie hyperparameters, and the joint optimization of intrinsic and extrinsic rewards. As a result, implementing and reproducing intrinsic reward meth- ods is challenging. To tackle this problem, we first formulate five questions investigate how various low-level implementation de- tails impact the training of intrinsically motivated agents. We first define an initial baseline configuration for optimizing the intrinsic rewards, shown in Table 2 These bascline settings are selected based on the most common configurations reported in the litera- ture. Next, we address each question by modifying only one hy- perparameter in the bascline configuration at time. Finally, we evaluate the performance of these intrinsic rewards with the best parameters hered in each question. All the experiments
Table 2: Details of the baseline settings.
Hyperparameter
Value
Observation norm.
RMS
Reward norm.
RMS
Weight init.
Orthogonal
Update proportion with LSTM
100%
False

---

Published in Transactions on Machine Learning Research (04/2025)
oK-riee. BN-BE5 ur-1n0e, Mi-drthep... LSTM-l
00-3P5, it--ttn- -fae, 1P-1088, Ni-Orthog.. 15MMmi
0a-aMs, Admaes, 1P-2%, Mi-drtheg... LSTAM
(W-Vond1la, la RNARSS, UP-188N, Ki-orthog... LST
09-9/5, RN-AY5, UP-d0%, kI-Orthog... LSTM-to
onwe45 Riwes. UP-1005. MI-defoult, LSTM-Ro
Un-eine.
ite-aana Taa, un 200n, Wi-orsteg., LS1
0lwtPs, RRwMS, un-20N, Ni-0rthog... Ls inftoo
crevets Niwets atw2p0h, Windrshop... 1511heea
ICM
RND
Dimgreement
NGU
Nemeiteed Seate
PseudeCounts
RIDE
Normeleed Score
N/A
033
91
92
93
(a) MarirMasio
ICM
RND
Disagreement
NGU
Noma feed Score
Normalzed Cormalizddoo Scoe
N/A
(b) MfiniGrid-Door 6ex
Figure 4: Results for QI, Q2, Q3, Q4, and Q5 in SMB (top) and MGD (boltom)), which are normalized by the maximum possibly achieved in the task. Here, ON is the observation normalization, RN is the reward normal- ization, UP is the update proportion, and WT is the weight initialization. The bar with a hatch mark is Combined, which refens the results of using the best hyperparameters gathered each question. Since RE3 only employs fixed, randomly initialized neural network for encoding observations, there are no values in Q3. All the results are aggregated over 10 seeds, and each run uses 10M environment interactions.
ducted using SMB and MGD to investigate the effects in sparse-rewards and reward-free (i.e., without access to extrinsic rewards) scenarios, respectively.
Importantly, as shown in Figure we keep the PPO hyperparameters fixed and the overall RL training loop unmodified throughout all the experiments in the paper in order to isolate the effect of the questions on the intrinsic reward components. Previous work has shown that PPO has many implementation details that are key to achieving great performance (Huang et al.] 2022a Engstrom et al., 20200). In the following, we study implementation details for the intrinsic reward components, The fixed PPO hyperparameters are shown in Table g
Q1: The impact of observation normalization.
Observation normalization is crucial in deep learning to avoid numerical instabilities during optimization. Image observations, where each pixel value typically ranges from 0 to 255 per colour channel, commonly normalized to range of ( to 1 using Min-Max normalization by dividing each pixel value by 255. How- ever, previous studies suggest that Min-Max normalization may not be ideal for all representation learning algorithms [Burda et al. 2019b).
In Q1, we compare Min-Max normalization with using an exponential moving average (EMA) of the mean and standard deviation for observation normalization (RMS) for the inputs to the intrinsic reward modules. RMS normalizes observations by subtracting the running mean and dividing it by the running standard

---

indicate Published in Transactions on Machine Learning Research (04/2025)
Aggregate
Aggregate Median
Aggregate IQM
Aggregate
Combined
LSTM (05)
Wi=Default (04)
UPe 1% (93)
UP-10%
RNe Vanilla
BN -N--Min-axx (92)
Vanilla (91)
Baseline
0.0
0.25
0.5
0.75
1.00.0
0.25
0.5
0.75
1.00.0
0.25
0.5
0.75
1.00.0
0.25
0.5
0.75
1.0
(a) SuperMario
Aggregate
Aggregate Modian
Aggregate IQM
Aggregate
Cambined
LSTM (05)
WI-Default (Q4)
UPw
UP-10%
UP-50% (Q3)
RN Vanilaa (02)
RN- Max (02)
Vanialil
0.0
0.25
0.5
0.75
1.00.0
0:25
0.5
0.75
1.00.0
0.25
0.5
0.75
1.00.0
0.25
0.5
0.75
1.0
(b) MiniCrid- Key----660
Figure Aggregated performance of the eight intrinsic rewards with different low-level hyperparameters over 10 random seeds. The vertical dashed line represents the performance of the extrinsic agent, which only has access to the task rewards. Here, ON the observation normalization, RN the reward normalization, UP the update proportion, WI is the weight initialization, IQM is the interquartile mean, OG is the optimality gap (lower is better), and Combined refers the results of using best hyperparameters gathered each question. All the compulation is performed using the Rliable (Agarwal et al. 2021 library.
deviation of all observations collected by the agent thus far. Our results shown in Figures that using RMS for observation normalization generally reduces the variance and achieves better asymptotic performance across all the environments of study. Importantly, some intrinsic rewards, such as RND, NGU, PseudoCounts, and RIDE, benefit significantly from RMS normalization. Critically, RND achieves zero rewards in SMB if observations are not normalized with RMS. These results indicate that RMS normalization is important for intrinsic reward methods that random networks, since the lack of normalization result in the embeddings produced by the random networks carrying very little information about the inputs Burda et al. 2019b
$ and 5
Q2: The impact of reward normalization.
Similarly to Q1, reward normalization can have a large impact when using deep neural networks to compute the intrinsic rewards, since the scale these rewards can be arbitrary and vary significantly over time. To mitigate the con-stationarity of intrinsic rewards, in Q2, we compare three normalization approaches for the reward oulputs of the intrinsic reward modules: (1) Min-Max normalization, (2) using RMS of the standard deviation, and (3) reward normalization.
Reward normalization smooths the optimization process, which can be beneficial for stability but can lead to slower convergence (Burda et al. 2019b). Our findings show that almost all intrinsic rewards critically require some form of reward normalization, as agents fail to explore without normalized rewards. Importantly, the latter applies to all the euvironments that we experiment with. Additionally, while RMS is generally the default strategy for reward normalization, our results in Figure show that Min-Max normalization is a more robust option in SMB, improving the performance and reducing the variance of the majority of the methods.
Q3: The co-learning dynamics policies and auxiliary tasks for intrinsic rewards.
9

---

Published in Transactions on Machine Learning Research (04/2025)
Optimizing intrinsic rewards in deep RL often involves training additional networks for auxiliary tasks (e.g.. predictor network in RND, inverse dynamics encoder in ICM, forward dynamics encoders in Disagreement). However, managing the co-learning dynamics of the auxiliary networks and policies is challenging. In Q3, we explore three update strategics for the auxiliary networks in the intrinsic reward modules: (1) updating them the same frequency as the policy, (2) updating them 50% of the time, (3) updating them 10% of the time, and (4) updating them 1% of the time. This comparison sheds light on the trade-off between the number of gradient updates in the auxiliary networks and the performance of the policy. Additionally, lower update frequencies have the benefit of reducing computational overhead and training time by limiting the number of gradient updates required.
Our findings indicate that the auxiliary networks generally perform robustly across the range of studied update frequencies. Additionally, there is no clear configuration that seems generally better for all intrinsic rewards across environments, rendering this implementation detail worth tuning for specific environments.
Q4: The impact of weight initialization.
Weight initialization plays a crucial role in optimizing deep neural networks, enabling faster convergence. In Q4, we compare two approaches for weight initialization in the auxiliary networks of the intrinsic reward modules: (1) orthogonal weight initialization and (2) uniform weight initialization (PyTorch's default). Note that again, the policy and value networks remain unchanged.
Our results highlight the importance of weight initialization in intrinsically-motiiate RL. Specifically, we found that orthogonal weight initialization is beneficial for most intrinsic rewards, regardless of their specific optimization tasks (e.g., inverse dynamics, forward dynamics), and even in random networks (e.g.. RND and RE3). This benefit is evidenced by reduced variance episode returns and generally higher mean returns. This observation aligns with previous research indicating that orthogonal weight initialization can improve performance stability in deep RL agents (Huang et al., 2022a Engstrom et al. 2020). Importantly, RND is the intrinsic reward method that shows the highest variability for this implementation detail, where orthogonal weight initialization works better in SMB but worse than uniform initialization in MGD.
NGU
RE3
E38
Extrinsic
KeyCorridorS8Rs
KayCorridorSoRs
KnyCorridor110R
MultiRoom-N--s
0.-
0.-
0.4
14-
0.
6.0
31
MultRRoom-N10-P0
MultiRoom
1611
Homoliee Score
13
54
54-
60
Figure 6: Performance of four selected intrinsic rewards in RLeXplore on the top eight most challenging tasks of the MGD suite. The solid line and shaded regions represent the mean standard deviation computed with five random seeds, respectively.
Q5: Is memory required to optimize intrinsic rewards?
In Q5, we investigate whether the intrinsic rewards included in RLeXplore benefit from memory-enabled architectures. We compare the optimization of intrinsic rewards using a vanilla policy network and one equipped with long-short-temi memory (LSTM) (Hochreiter Schmidhuber 1997) module while keeping PPO the RL backbone algorithm.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Some intrinsic reward methods exhibit significantly lower performance when using LSTM policies. This observation aligns with the fact that LSTMs provide episodic context to policies, whereas most intrinsic reward methods define exploration as global problem.
Finally, we use the best-performing implementation details observed from Q1-5 to experiment in the set of most challenging exploration tasks from MiniGrid Our results Figure show that with our implementa- tions of intrinsic rewards in RLeXplore, researchers can make progress training RL agents in challenging tasks where vanilla RL agents unable to learn due to the sparsity of the task rewards. In summary, by systematically addressing the implementation details, work significantly enhances the reproducibility of intrinsic reward methods. These thoughtful design choices not only improve performance but also ensure that our implementations can be reliably reproduced and generalized across various environments.
5.2 Combination of Intrinsic and Extrinsic Rewards
Q6: Joint optimization of intrinsic and extrinsic rewards.
In sparse-reward environments, the objective is for agents to explore the state space by oplimizing intrinsic rewards until they discover the task rewards, at which point they should focus solely on optimizing the task rewards. However, many intrinsically motivated RL applications naively optimize the sum of intrinsic and extrinsic rewards, potentially lcading to learning fuzzy value functions and suboptimal policies (Castanyer et al. 2023). In this section, we compare this common approach with learning two separate value functions, for each reward function (Burda et al. 2019b). The advantages of the latter include the ability to disentangle the effects of intrinsic and extrinsic rewards the agent's 's behaviour, leading to cleaner learning dynamics and potentially more efficient exploration. In these settings, both value functions are used during the advantage estimation phase of PPO. Specifically, we compute separate GAE values one using the intrinsic value function and one using the extrinsic value function. The resulting advantages are then summed to compute the policy loss term for PPO. This separation facilitates more accurate advantage estimates for each reward type, leading improved learning dynamies.
For this analysis, used the Proegen-Maze task (Cobbe et al. 2020 a sparse-reward benchmark. RL agents often struggle to learn meaningful behaviours from the extrinsic reward alone in this task. We evaluate different variants of the task (e.g,, 1 maze vs. 200 mazes) to examine singleton versus contextual MDPs. We note that in our framework, we do not provide different context information to the agents for singleton versus uontextual MDPs (e.g,, the context ID). We refer to these frameworks to formalize the agent-environment interaction when the environment remains static throughout training (i.e., singleton maze) versus when it varies each episode (i.e., contextual a different maze at each episode).
Figure 7 demonstrates that learning two separate value functions (Huang et al. 2022b which we refer to the TwoHead architecture, outperforms the naive approach of simply adding the two rewards in the complex sparse-reward environment of Procgen-Maze both in singleton and contextual settings. Importantly, all methods outperform the extrinsic agent, especially the Maze environment.
5.3 Unlocking the Potential of Intrinsic Rewards
Q1-6 extensively discuss the tuning of intrinsic rewards under both normal and reward-free scenarios, reveal- ing significant insights into the optimization processes. However, aim to delve deeper into the capabilities of intrinsic rewards to address the evolving challenges in the RL community. Specifically, in Q7, we in- vestigate recent developments in the exploration literature in RL, such as combined intrinsic rewards and exploration in contextual MDPs. For our experiments, we use the SMB-Random.Slayer enviromment vari- ant, where agents play a different level in the game at each episode. Our results indicate that the recent developments in combined intrinsic rewards merit further rescarch, as we demonstrate that such methods can enable agents learn exploratory behaviours of exceptional quality in both singleton and contextual MDPs.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Two-Head PPO
Extrinsic
Procgen 1 Maze Hard
Procgen 200 Mazes Hard
Normalized Score
Normalzee Soomalizeda
100 AND Disagreament NSU MecudCrunt RIDE
RE3
E38
1CM
AND Desggieemen NOu PaoudiCounts NIDE
REB
ES8
Figure (Left) During training, the extrinsic agent struggles to find the goal the selected Maze, resulting in a reward O. While some intrinsic reward methods yield occasional non-zero rewards, the methods perform significantly better when intrinsic and extrinsic value estimation are decoupled using two distinct value heads in agent's network. (Right) In the Procgen variant, where each maze represents unique level, the baseline extrinsic agent achieves the goal 50% of the time, and intrinsic rewards don't outperform the baseline significantly. We note that the presence of easier levels, where the goal may occasionally be near the agent's starting point results in generally less sparse rewards and an easier task to learn.
Q7: The performance of mixed intrinsic rewards.
We run experiments using all the levels in the game of SMB, and we sample them uniformly during training. As in Q1-5, we do not use the extrinsic reward for training the agents but use it as an evaluation metric to show how much agents actively explore the environment.
Our results show that combined objectives enable emergent behaviours of much better quality than single objectives. Interestingly, E3B and RIDE are the best performing single objectives, and E3B+RIDE also achieves the highest performance among all the combinations. Similarly, RND and ICM, combined with other intrinsic rewards, outperform their original performance. This indicates that different intrinsic rewards can provide orthogonal gains that can be leveraged logether.
Rewards
Single Rewards
IQM (G.G.)
Eas Und
sala
Mean (G.G.)
89 uca
tho
IQM (G.E.)
poca
sCl
fot wad
Mean (G.E.)
0.2 0.S 0.6 0.35 1.0
0.
0.2
U.A. 0.
38
20
Normalized Return
Normalized Return
Figure& 8: (Left) The performance ranking single and mixed intrinsic rewards the RaneeMM As expected, episodic bonuses (such as E3B and RIDE) demonstrate superior performance, attributed to the environ- ment's non-singleton MDP nature. (Right) Overall performance comparisons between the single and mixed intrinsic rewards. Here, G.E. denotes the six "glosod--episdic"" combinations, and G.G. denotes the three 'global+global" combinations, as illustrated in Table 5
6 Conclusion
Our work introduces RLeXplore, a comprehensive open-source repository that not only implements state-of- the-art intrinsic rewards but also provides a systematic evaluation framework for understanding their impact on agent performance. Our results show that with RLeXplore, RL agents can learn emergent behaviours autonomously, solving multiple levels of Super Bros without task rewards. Additionally, show that
12

---

Published in Transactions on Machine Learning Research (04/2025)
intrinsic rewards enable RL agents to obtain great performance complex sparsc-reward tasks like Proegen- Maze, MiniGrid, the ATF-5 hard-erploration lasks and Ant- UMaee. Finally, RLeXplore facilitates further research in mixed intrinsic rewards (Henaff et al. 2023). uncovering the potential of such methods.
Through our study, we emphasize the importance of thoughtful implementation design, demonstrating that well-considered approaches lead to significant performance gains over naive implementations. Our contribu- tions extend to establishing standardized practices for implementing and optimizing intrinsic rewards, laying the groundwork for future advancements in intrinsically motivated RI.
RLeXplore is designed to benchmark end-to-end intrinsic reward methods. These end-to-end methods are more commonly used and under-evaluated by the community. Skill-based algorithms,, which typically involve separate phases for skill discovery and skill learning, are more complex and left for future work. For an alter- native perspective that includes skill-based approaches, we refer readers to the unsupervised RL benchmark by Laskin et al. (2021). Additionally, RLeXplore was designed with accessibility in mind, ensuring that the implemented methods can be run standard computational resources by any researcher. To maintain this accessibility, we have not included more complex and potentially powerful methods like BYOL-Explore (Guo et a.. 2022) RECODE (Kapturowski et al.). These methods not open-source and have been optimized exclusively with non-open-souree RL algorithms, which further limits their integration into RLeXplore.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Acknowledgments
This work is funded, in part, by HKSAR RGC under Grant No. PolyU 15224823, the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2024A1515011524 the NSFC under Grant No. 62302246, the ZJNSFC under Grant, No. L.Q23F010088, and the Ningbo under Grants No. 20237237 & 2024Z284 20242289 2023CX050011 & 20252038. We thank the perforreffrm computing center at Eastern Institute of Technology and Ningbo Institute Digital Twin for providing the computing resources. We also want to acknowledge funding support from NSERC and CIFAR, and compute support from Digital Research Alliance of Canada, Mila IDT and Nvidia.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29322 2021.
Arthur Aubret, Laetitia Matignon, and Salima Hassas. An nnformation-theeredt perspective on intrinsic motivation in reinforcement learning: A survey, Entropy, 25(2):327, 2023.
Peter Aucr. Using confidence bounds for exploitationnccxlloration tradc-offs. Journal of Machine Learning Research, 3:N77) 422, 2002.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up Learning directed exploration strategies. In International Conference on Learning Representations, 2020.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Uni-
fying count-based exploration and intrinsic motivation. Proceedings of Advances in Newral Information Processing Systems, 29:1471-1433 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-225 279, 2013.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp. 679 684, 1957.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven lcarning. Proceedings of the International Conference on Learning Representa- lions, pp. 17, 2019a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. Proceedings of the 7th International Conference on Learning Representations, pp. 1-17, 2019b.
Roger Creus Castanyer, Joshua Romoff, and Glen Berseth. Improving intrinsic exploration by creating stationary objectives. arXiv preprint arXiv:2310.1811 2023.
Maxime Chevalier-Boisver Bolun Dai, Mark Towers, Rodrigo Vicente Lueas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforeement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning. In International conference on machine learning, pp.
PMLR, 2020.
Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. In COLT, volume 2, pp. 3, 2008.
Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2024. URL http:///ggttuoccm///oaaaaaaaaaaa
Roboticc.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo arXiv preprint Xrx:::2005. 12729, 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Muih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407- 1416. PMLR, 2018.
Zhaohan Guo, Shantamu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandricllo, Jean-Basticn Grill, Yunhao Tang, et al. Byol-cxplore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35:31855-31870, 2022.
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861 1870. PMLR, 2018
Mikael Henaff, Roberta Raileamu, Minqi Jiang, and Tim Rocktäschel. Exploration via clliptical episodic bonuses. Advances in Newral Information Processing Systems, 35:37631 37646, 2022.
Mikael Henaff, Minqi Jiang, and Roberta Raileanu. A study of global and episodic bonuses for exploration in contextual mdps. Xix preprint or Xix:22300. 032366 2023.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 988):1735-1780, 1997.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022a.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1 18, 2022b. URL http:://jmrr.org//peee v23/21- html
Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethinking explo- ration. Royal Society Open Science, 10(6):230539, 2023.
Daejin Jo, Sungwoong Kim, Daniel Nam, Taehwan Kwon, Seungeun Rho, Jongmin Kim, and Donghoon Lec. Loco: Learnable episodic count for task-spocific intrinsic reward. Advances in Newral Information Processing Systems, 35:30432 30445, 2022.
Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):90-134 1998.
Steven Kapturowski, Alaa Saade, Daniele Calandriello, Charles Blundell, Pablo Sprechmann, Leopoldo Sarra, Oliver Groth, Michal Valko, and Bilal Piot. Unlocking the power of representations in long-term novelty- based exploration. In Second Agent Learning in Open-Endednees Workshop.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Christian Kauten. Super Mario Bros for OpenAl Gym. GitHlu, 2018. URL https://github.com
Misha Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. In J. Vanschoren and S. Yeung (ods.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL http://tt//////aaaaaaaaaaaaaaaaa www.ggg....ri..... http:ccpe..pepeee/epeeeeeeeeeee/ http:///244..aaacc..0e2aaaaaaaap
15

---

Published in Transactions on Machine Learning Research (04/2025)
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661-62 (71- 2010.
Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins estimate pseudocounts for exploration in reinforcement learning. arXiv preprint orXiv:2306.031 2023.
Marlos C Machado, Mare Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5125- 5133, 2020.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based exploration in feature space for reinforcement learning. In LJCAI, 2017.
Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Cow- ard, and Jakob Foerster. Craftax: A lightning-fast benchmark for open-ended reinforcement learning. arXiv preprint (xXi::2402.1601,, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the International Conference on Machine Learning, pp. 2721-2730, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity driven exploration by self- supervised prediction. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognation Workshops, pp. 16 17, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagroement. In International conference on machine learning, pp. 5062-5071. PMLR, 2019.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Btable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1-8, 2021. URL http://jmlr... wttg/papers//2222011333 html
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. In International Conference on Learning Representations, 2020. URL https: //openreview wttp://www......ooooooooodeeeee.
Nikolay Savinov, Anton Raichuk, Damicn Vincent, Raphael Marinicr, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In Proceedings of the International Conference on Learning Representations, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint ar 1011177700000 06347, 2017.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximiza- tion with random encoders for efficient exploration. In Proceedings of the 38th International Conference on Machine Learning, pp. 9443-9454, 2021.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 3rXim:1507.00814 2015.
Alexander Strehl and Michael L Littman. An analysis of basel interval estimation for markov
decision processes. Journal of Computer and System Sciences, 74(8):1309-13 13333 2008
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Mare Bellemare. On bonus- based exploration methods in the arcade learning environment. arXiv preprint ar X 1052, 2021.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAIXi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vreez-Vieente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL
Claas A Voelcker, Marcel Hussing, and Eric Eaton. Can we hop in general? discussion of benchmark selec- tion and design using the hopper environment. In Finding the Frame: An RLC Workshop for Examining Conceptual Promeworks, 2024. URL httpp://ppaaeeeeeeeea///turorrpp htt.://reeeet/ouuumreoogggg331
Kaixin Wang, Kuangqi Zhou, Bingyi Kang, Jiashi Feng, and YAN Shuicheng. Revisiting intrinsic reward for exploration in procedurally generated environments. In The Eleventh International Conference on Learning Representations, 2022.
Xingrui Yu, Yueming Lyu, and Ivor Tsang, Intrinsic reward driven imitation learning via generative model. In Proceedings of the International Conference on Machine Learning, pp. 10925-10935, 2020.
Mingqi Yuan, Zequn Zhang, Yang Xu, Shihao Luo, Bo Li, Xin Jin, and Wenjun Zeng. Rllte: Long-tern evolution project of reinforcement learning. arXie preprint arXi::2309. 16382, 2023.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Bebold: Exploration beyond the boundary of explored regions. arXiv preprint arXiv:2012.086: 2020.
17

---

Published in Transactions on Machine Learning Research (04/2025)
A Algorithmic Baselines
ICM Patthak et al. 2017). ICM leverages an inverse-forward model to learn the dynamics of the environment and uses the prediction error as the curiosity reward. Specifically, the inverse model inferences the current action ag based on the encoded states e and e++1, where = (>()) and (+)) is an embedding network. Meanwhile, the forward model f predicts the encoded next-state e, based on (er, az). Finally, the intrinsic reward defined as
== [ffe; a,) ec+1ll2.
(4)
RND Burdda et al. 2019b RND produces intrinsic rewards via self-supervised manner, in which predictor network is trained to approximate fixed and randomly-initialized target network f. As a rosult, the agent is motivated to explore unseen parts of the state space. The intrinsic reward is defined
1. (1(()))1) ((s+1))l2.
Disagreement (Pathak al. 2019). Disagreement is variant of 1CM that leverages an ensemble of forward models and calculates the intrinsic reward as the variance among these models. Accordingly, the intrinsic reward is defined
1 = Va=Var[J((e, a:)), 0, O....,
(6)
NGU (Badia et al. 2020]. NGU is a mixed intrinsic reward approach that combines global and episodic exploration and the first method to achieve non-zero rewards in the game of Pitifll! without using demon- strations or hand-crafted features. The intrinsic reward is defined as
min mainmmon (max (ax), CJ/ Nop(se),
(7)
where is a life-long curiosity factor computed following the RND method, C' is a chosen maximum reward scaling, and Nep is the episodic state visitation frequency computed by pseudo-counts.
PsoudoCounts (Badia et al. 2020). Pseudo-counts has been widely used in count-based exploration ap- proaches (Bellemare et al.] [216)] Ostrovski et al.] (017)) with diverse implementations like neural density models. In this paper, we follow NGU (Badia et al. (0200) that computes pseudo-counts via k-nearest neighbor estimation, which is highly efficient and can be applied to arbitrary tasks. Given the encoded obscrvations (eo...., er-1] visited in the an episode, we have
Nop(8s) =, /
E(()n)e)) tc,
(8)
where é; is the first & nearest neighbors of K is Dirac delta function, and guarantees minimum amount of pseudo-counts. Finally, the intrinsic reward is defined as
I 1/v (sp(
(9)
RIDE [Raileanu & Rocktäsche] 2020). RIDE is designed based on ICM that learns the dynamics of the environment and rewards significant state changes. Accordingly, the intrinsic reward is defined as
J = |let+1 - ex Nop(8141);
(10)
where (8+1)) is used to discount the intrinsic reward and prevent the agent from lingering in a sequence of states with a large difference in their embeddings.
RE3 (Soo et al. 2021). RE3 is an information bhsory-based and computation-efficient exploration approach that aims to maximize the Shannon entropy of the state visiting distribution. In particular, RE3 leverages random and fixed neural network encode the state space and employs k-nearest. neighbor estimator
18

---

Published in Transactions on Machine Learning Research (04/2025)
to estimate the entropy efficiently. Then, the estimated entropy is transformed into particle-based intrinsic rewards. Specifically, the intrinsic reward is defined as
7
log(Ile: - Egll2 + 1).
(11)
k
E3B (Henaff al. 2022. E3B provides generalization of count-based rewards to continuous spaces. E3B learns a representation mapping from observations to latent space (c.g.. using inverse dynamics). At cach
episode, the sequence of latent. observations parameterizes an ellipsoid (Li el al. 2010 Auer 2002 Dani
et al. 2008), which is used to measure the novelty of the subsequent observations. In tabular settings, the
E3B cllipsoid reduces to the table of inverse state-visitation frequencics (Henaff et al. 2022). Given feature encoding f, each time step of the episode the elliptical bomus is defined follows:
- (.=)
(12)
C--1 = Σ f(s.)f(s.) + AI,
(13)
=1
where f the learned representation mapping, Ct-1 is the episodic ellipsoid (Henaff et al, scalar coofficient, and I is the idcntity matrix.
2022 is a
19

---

Published in Transactions on Machine Learning Research (04/2025)
B Experimental Settings
B.1 Benchmark Selection
We evaluate the RLeXplore framework multiple recognized benchmarks, which specifically designed to evaluate the exploration capability of RL agents. We select SuperMarioBros (Kauten 2018), MiniGrid Uchevalier-Boisvel et al. 2023), Procgen (Cobbe et al. 2020], Arcade learning environment (ALE) (Belle- mare et al., 2013), and Gymnasium-Roiis Robbotics (de Lazcano et al. 2024) for our experiments, which sufficiently spans the existing RL benchmarks. Table provides the details of these selected environments, including their observation, action, and reward spaces.
Table 3: Details of the euvironments used in our experiments.
Benchmark
Environment
Observation Space
Action Space
Reward Space
SurerMarioBros
World1-Stagel-v3
Box(0, 255, (240, 256, 3))
Discrete(7)
N/A
SuperMarioBros
HandomStages-v3
Box(0, 265, (240, 256, 3))
Discrete(7)
N/A
MiniGrid
DoorKey-16x16-v0
Box(0, 255, (3, 7, 7))
Discrete(7)
Sparsc
MiniGrid
DoorKey-8x8
Box(0, 255, (3, 7, 7))
Discrete(7)
Sparse
MiniGrid
KeyCoriddor8R35-v0
Box(O, 255, (3, 7, 7))
Discrete(7)
Sparse
MiniCrid
KcyCoridooSSRRG-v00
Box(0, 255, (3, 7, 7))
Discrete(7)
Sparsc
MiniGrid
KeyCoriddon11RR7-x0
Box(0, 255, (3, 7, 7))
Discrete(7)
Sparse
MiniGrid
MultiHooom-N7-SS8-v0
Box(0, 255, (3, 7, 7))
Discrete(7)
Sparse
MiniGrid
Mutt:Room-N1--100-00
Box(0, 255, (3, 7, 7))
Discrete(7)
Sparse
MiniGrid
MuldiRoom:-N12-110-00
Box(0, 255, (3, 7, 7))
Discrete(7)
Sparse
MiniGrid
LockedRoom-v0
Box(0, 255, (3, 7, 7])
Discrete(7)
Sparse
MiniGrid
Dynamie-Obstacles
Obstacles 16x16-v0
Box(O, 255, (3, 7, 7))
Discrete(3)
Sparse
Procgen
Maze, num levels=200
Box(0, 255, (3, 64, 6d))
Discrete (15)
Sparse
Proegen
Maze, num levels=
Box(0, 255, (3, 64, 64))
Discrete( (15)
Sparse
ALE
Gravitar
Box(0, 255, (4, 84, 84))
Discrete(18)
Dense
ALE
RovezzumaRevenge
Box(0, 255, (4, 84, 84))
Discrete(18)
Dense
ALE
PrivateEye
Box(0, 255, (4, 84, 84))
Discrete(18)
Dense
ALE
Seaquest
Box(O, 255, (4, 84, 84))
Discrete (18)
Dense
ALE
Venture
Box(0, 255, (4, 84, 84))
Discrete(18)
Dense
Gymnasium- Gymoasium-Robotics
AntMaze
UMaze-v5
Box(-inf, inf, (27.))
Box(-1, 1, (8.))
Sparse
B.2 Baselines
We designed the following settings for the baseline experiments, and all the subsequent questions were adjusted based on the baselines. Moreover, all the experiments are performed using the proximal policy optimization (PPO) (Schulman et al. 2017 implementation from RLLTE (Yuan et al. 2023
Table 4: Details of baseline settings.
Hyperparameter
Value
Observation normalization
RMS
Reward normalization
RMS
Weight initialization
Orthogonal
Update proportion
1.0
with LSTM
False
B.3 Details of Questions
Table illustrates the details of the candidates for all questions.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Table Details of candidates for all questions, where a batch of intrinsic rewards.
Candidate
Detail
Q1
Vanilla
obs. obs. /25.00, only for anage-based observations, else obs. obs..
RMS
obs. = Clip o rumning mean -5.0, 5.0)
running std.
Q2
Vanilla
I=1 running
RMS
Min-Max
[-min[]]
Q3
0.01
Use 1% the samples to update the intrinsic reward module.
0.1
Use 10% of the samples to update the intrinsic reward module.
0.5
Use 50% of the samples to update the intrinsic reward module.
1.0
Use 100% of the samples update the intrinsic reward module.
Q4
Vanilla Orthogonal
Fill the input tensor with values drawn from the uniform distribution.
Fill the input tensor with a (semi) orthogonal matrix.
Q5
Vanilla LSTM
Policy network with only convolutional and linear layers. Policy network that includes LSTM layer.
Q6
Vanilla
R = E +
Two-head
Value network uses two separate branches for E and
E3B+RND, E3B+ICM, E3B+RIDE, RE3 RND, RE3+ICM, RE3+RIDE RND+ICM, RND+RIDE, ICM+RIDE
Q7
Glisol+Episodic
Global+Global
B.4 Best Configurations
Table 6: The best configurations for each intrinsic reward on SuperMarioBrus.
Reward
Obs. Norm.
Reward Norm.
Update Prop.
Weight Init.
Memory Required
ICM
Min-Max
Vanilla
1.0
Default
RND
RMS
Vanilla
1.0
Orthogonal
Disagreement
RMS
Vanilla
0.5
Default
X
NGU
RMS
Min-Max
0.1
Default
X
PseudoCounts
RMS
Min-Max
0.01
Default
RIDE
RMS
Min-Max
0.01
Default
X
RE3
Min-Max
Min-Max
N/A
Default
X
E3B
Min-Max
Min-Max
0.1
Orthogonal
Table 7: The best configurations for each intrinsic reward on SiniiGrid-DDoKKeyy-te
Reward
Obs. Norm.
Reward Norm.
Update Prop.
Weight Init.
Memory Required
ICM
RMS
RMS
1.0
Orthogonal
X
RND
RMS
Vanilla
0.5
Orthogonal
Disagreement
Vanilla
Min-Max
0.5
Default
X
NGU
RMS
RMS
0.01
Orthogonal
x
PsoudoCounts
RMS
Min-Max
1.0
Orthogonal
x
RIDE
RMS
Min-Max
1.0
Orthogonal
RE3
RMS
Min-Max
N/A
Orthogonal
E3B
RMS
RMS
1.0
Orthogonal
X
21

---

Published in Transactions on Machine Learning Research (04/2025)
Table PPO hyperparameters for SuperMariolBrs, MiniCrid, and Proegen games.
These remain fixed for all
experiments.
Hyperparameter
SuperMarioBros
MiniGrid Procgen
Observation downsampling
(84, 84)
(7, 7)
(64, 64)
Observation normalization
255.
No
255.
Reward normalization
No
No
No
Weight initialization
Orthogonal
Orthogonal
Orthogonal
LSTM
No
No
No
Stacked frames
No
No
No
Environment steps
10000000
10000000
25000000
Episode steps Number of workers
128
32
256
1
1
Environments per worker
8
256
64
Optimizer
Adam
Adam
Adam
Learning rate
2.5e-4
2.5e-4
5e-4
GAE cocfficient
0.95
0.95
0.95
Action entropy coefficient
0.01
0.01
0.01
Value loss coefficient
0.5
0.5
0.5
Value clip range
0.1
0.1
0.2
Max gradient norm
0.5
0.5
0.5
Epochs per rollout
4
3
Batch size
256
1024
2048
Discount factor
0.99
0.99
0.999
22

---

Published in Transactions on Machine Learning Research (04/2025)
C Usage Examples
C.1 API Compatiblity
The following table provides detailed algorithm and environment compatibility of the implemented intrinsic rewards. Since NGU, PsoudoCounts, RIDE, and E3B require an episode memory for the reward computation, when combined with off oif-policy RL algorithms, they can only work non-vectorized environment currently, Therefore, those reward modules are marked with symbol.
Table 9: Algorithm and environment compatibility of the RLeXplore framework.
Image-based State-based Continuous Discrete Action
On-policy
Off-policy
Algorithm
Observation
Observation
Action
RL
RL
ICM
RND
Disagreement
NGU
PseudoCounts
RIDE
RE3
E3B
C.2 Workflow of RLeXplore
The following code provides an example when using RLeXplore with on-policy algorithms. At each time step, the agent first observes the vectorized environments before taking actions. Then the environments execute the actions and return the step information, which is processed by the watch () function extract necessary data for the current intrinsic reward. Finally, the intrinsic rewards will be computed, and the module will updated concurrently at the end of the episode.
load the library
from rllte. <llte.xplore.rr xplore. reward import RE3 # create the revard module irs (......)
# reset the environment obs, infos rese # rollout storage RolloutStorage (...)
# training loop for episode in range (..))
for step range (...).
sample actions agent(obs)) step the next_obs, ruds, terms, truncs, infos envs step (actionss # get data from the transitions irs (oba, actions, rvds, next_obs, terms, trunce, infos)
prepare the samples
samples dict (observationn==ro.o obs, actions=rs actions rewards-ra. zewards, terminateds-rs aonniaatddsss..ttnnnateeeet terminateds,
truncateds-rs. truncateds, next next_ oest_obeerattinneee... next_obe
)
compute the intrinsic revards
## ayne (bool): Whether update the reward sodule after the ** "compute" function, default is "True". intrinsic_rewarde irs. compute (samples, sync-True)
In contrast, the workflow a bit different when using RLeXplore with off-policy algorithms. As shown the following example, the intrinsic reward will computed at each time step rather than at the end of each episode. Moreover, the intrinsic reward module will be updated using the same samples for policy updates.
23

---

Published in Transactions on Machine Learning Research (04/2025)
library
from rllte.xplore splore.rewazz import RE3 create the zeward module irs (E.....)
reaet the environment
obs, infos enva. # training loop while True:
# sample actions actions agent (obs)
step the environment
next_obs, ruds, terms, truncs, infos = enva step (actions) get data from the transitions
irs. watch (obs, actions, ruds, next_obs, terms, compute intrinsic rewards each step
infos)
sync (bool): Whether update the reward module after the "compute" function, default intrinsic_revarda compute) camples
actions=actions,
revards=reds, terminateds
Truccatessteem. next nex.
Fync=False)
update the reward module
batch replay_ sample() irs. update saamplee-dico
aecions=batch..ctioon
revards-batch.rovarrdd
truncatedsbbttch.truaeee
natt_cbaervtiion-hatte next_obs)
) ...
C.3
Mixed Intrinsic Reward
The following code example shows how to create mixed intrinsic reward using two independent intrinsic rewards:
from rllte. env import make make_atari____ env
from rilte. rilte.xplose reward
import reward
Fabric, RE3, ICM
# define the mixed intrinsic revard class TwoMixed (Fabric):
def __init__ (self, mi m2): super __ (m1, m2)
def compute (self, samples, sync):
radi, rud2 comppue..oo (samples, synс)
return rwdl- rwd2
name__ "___ain__"
env setup
device "
enva aate_atari_en (device=device)
create two intrinsic rasard functions
irsl (envs, device)
irs2 (envs, device)
create the mixed intrinsic reward function irs TuoMixed (irsi, (ire1,
irs2)
24

---

Published in Transactions on Machine Learning Research (04/2025)
C.4 RLeXplore with Stable- 3aselees
Stable-Baselines3 (SB3) (Rallin al. 2021 is one of the most successful and popular RL frameworks that provides set of reliable implementations of RL algorithms in Python. SB3 provides a convenient callback function that can be called at given stages of the training procedure, the following code example demonstrates how to use RLeXplore SB3 for on-policy RL algorithms:
ALexploraaithhDoodiccyttt (BaseCallback)
**" Combining ALeXplore and on on-policy algorithme from 983.
**.
def
_..nit__ (eelf, irs, verbose-0): super www.spppporotttthnaaooo..... self) init..iiiii...... (verbose) self.irs irs
self.buffe. None
def init_callback (self, (self, model: BaseAlgorithm) >> None:
super O iititinalllaaeeeeeeeaaeaeaaeeaae (model)
self buffer self model rollout_buffer
def (se1)) bool:
This method will called by the model after each call (env.step()
:return: (bool) If callback returns False, training aborted early. eum
observations self device observations device
actions tee (self locals ["actions"], device-device)
revards = th. as. lalalllllllllllaaaeoooooolllllll locals ("revards") device=device)
dones th ae_aaesstrrreeooooeee (self. locals poccall('doaa ("dones") device=device ab tensor (self.locals ("mex_obs" device=device)
next_obaervation
# get data from the transitions
self irs watch (observatiio,,, actions, rewards, dones, dones, next_observations)
return True
def _o_rrolouut_em aontllout_enddseee (self) None:
# prepare the data samples
get we..a_ttensorselll buffer buffer observations)
observations
new_abs abs.clone()
new_oba [:-1] [ba
new_obs [-1] th. (self ("nev_obe"))
actions th. (ee. actions
revards = th. (obs. www.teessor(sllffuufffreee
reward)
dones
actions. rewards shape, dones, shape, obs shape)
# compute the intrinsic rewards
intrinsic_revarde (observations=obs. aanplaassdittobbbrrrttttoooss. actions=actions
irs.comp
samples-dice truncateds-dae dones,, nett.obervatiossmsr.....)))
rewards-rewardd terminateds=dones.
More dotailed code examples can be found in the attachod supplementary matcrials.
C.5 RLeXplore with CleanRL
CleanRL (Huang et al. 2022b is open-source project focused on implementing RL algorithms with clean, understandable, and reproducible code. It aims to make RL more accessible by providing implementations that are simpler and more transparent than those typically found in research papers or larger librarios. The following code example demonstrates how to use RLeXplore in CleanRL for on-policy RL algorithms:
library
25

---

Published in Transactions on Machine Learning Research (04/2025)
from rllte. xplore. reward import RE3 create zeward module enn,, device=device)
get data from the transitions irs. vatch (obeerrooorrtton [atep], actione [step],
rewarde [step], serminateds-doneee
truncateds=dones [step], nat__oberation--eettt
next_obe obs. c)sne()
next_obe obs [:-1] obs [1:]
next_obs [-1] = next_obs
# compute the intrinsic rewards
intrinsic_ compute)
samples-dict doser,,, actions-actions,
rewards-rewards, terminateds
cruncateds-dones, mxx.oolaerraaiiose-mtt.....o..
sync-True)
add intrinsic rewards intrineic,revarde
revards
More detailed code examples can be found in the attached supplementary materials.
C.6 Implementing New Intrinsic Reward Modules
In RLeXplore, all intrinsic reward methods inherit from a base reward class that requires two functions to be implemented: compute() () and u) The compute function processes batch of on-policy trajectories to calculate intrinsic rewards and is automatically called prior to the PPO update, while the update () function uses the same trajectories to update the associated modules. To integrate a new intrinsic reward method, users only need to create a new script that inherits from the base reward class and implements these functions. Moreover, many pre-defined network modules (e.g., Atari CNN, ResNet CNN) are readily available for import, allowing users to use the currently implemented intrinsic rewards templates for their implementations.
26

---

Published in Transactions on Machine Learning Research (04/2025)
D Comparative Analysis of Intrinsic Reward Implementations
This section provides a detailed comparative analysis of our intrinsic reward implementations in the RL- eXplore framework against other publicly available implementations. The results are compiled in tables for different environments to demonstrate the performance each algorithm. We cite the works from which we obtained the original results in each of the tables, and we provide our results by averaging the performance of the last 100 training episodes over sceds.
D.1 SuperMarioBros (Only Intrinsic Rewards)
Table 10: Comparison of % of level completed in SuperrarriBrrus without task rewards.
Algorithm
% of Level Completed (10M Steps)
% of Level Completed (1M Steps)
(Original) RIDE
23%
(Original) ICM
30%
(RLeXplore) RIDE
100%
50%
(RLeXplore) ICM
30%
2%
The percentage of the level completed is computed by dividing the episode return by 3,000, which corresponds to the maximum reward that be obtained in Super-ariiBBos--I1 (if the agent solves the level without wasting time). Note that in Figure we divide this quantity by 100 and show maximum reward of 30.
Note that our implementation of TCM reproduces the results reported in the original paper in Mario (Pathak et al. 2017 and our implementation of RIDE further outperforms the original implementation.
D.2 MiniGrid- Soteliid-DDorKey166x6 (Extrinsic + Intrinsic Rewards)
Table 11: Episode returns in MiniGridd-DoorKegy1666666 with extrinsic and intrinsic rewards.
Algorithm
Episode Return (10M Steps)
(Original) RIDE ZZaang et al.
2020
0.25
(Original) ICM
Zhang
2020
0.0
(Original) RND Zhang
2020
0.0
(Original) IMPALA (Zhang et al.
2020
0.0
(RLeXplore) PPO
0.37
(RLeXplore) ICM
0.6
(RLeXplore)
RND
0.6
(RLeXplore)
RIDE
0.12
Using the implementations in RTeXplore we obtain significantly better performance the same tasks and with the same algorithms.
27

---

Published in Transactions on Machine Learning Research (04/2025)
D.3 SonGGrid-DDorKey--oo (Extrinsic + Intrinsic Rewards)
We also evaluate our implementations in -oorrrryd-- with budget of 1M environment sleps to be able to compare to the original results reported in (Seo et al. 2021).
Table 12: Episode returns in MiniGrid-DoorKey--S with 1M environment steps.
Algorithm
Episode Return (1M Steps)
(Original) RE3
(Seo et al.
2021
0.5
(Oiginal)
RND
Seo et al.
202
0.0
Original)
ICM
Seo et al.
2021
0.2
Original)
A2C
(So0 et al.
2021
0.0
(RLeXplore)
RE3
0.95
(RLeXplore)
RND
0.82
(RLeXplore)
ICM
0.83
(RLeXplore) PPO
0.22
Importantly, we reproduce the results reported in /eoo al. 2021 very accurately, showing that RE3 can provide more sample-efficien exploration in this domain, compared RND and ICM. Still, our implemen- tations of RE3 and ICM achieve even better performance than the original ones.
60
D4
Epiiode howward
Edlodd EdlodeRovard Roward
Ealodd
u Ay
40
20
112
1.5
30
Eiviromment Stops (X20%
Environment Stops (x109)
Enstronment Steps (x10")
Environmant Stops Dx10f)
Figure 9: Using RLeXplore in MiniGrid- MiniGrid-Doorkeey- DoorKey-8x8, we are able to not only reproduce the conclusions obtained in
previous work (Seo al. 20221) regarding the capabilities of RE3 compared to ICM and RND, but we also generally achieve better performance, hence providing stronger baselines to the RL community.
D.4 Procgen 200 Mazes (Extrinsic + Intrinsic Rewards)
Table 13: Performance comparison in Procgen 200 Mazes with 25M training steps.
Algorithm
Procgen 200 Mazes (25M Steps)
(Original) E3B (Castanyer et al.
2023
3.0
(Original) ICM Castanyer et al.
2023
2.5
(Original) RND Castanyer et al.
2023
1.7
(RLeXplore)
E3B
4.1
[RLeXplore)
ICM
5.9
(RLeXplore)
RND
5.0
D.5 ALE-5 (Extrinsic + Intrinsic Rewards)
In this section, present the evaluation results of the intrinsic reward methods on a set of ALE games known for their challenging exploration requirements. These "hard-exploration" games, including Gravitar, Montezuma's Revenge, Private Eye, Seaquest, and Venture, serve as a benchmark for testing the effectiveness of intrinsic rewards in aiding exploration and improving agent performance.
We observe that while intrinsic rewards lead a decline performance Gravitar, they generally provide substantial benefits, particularly in environments where exploration is difficult. For example, in Seaquest.
28

---

Published in Transactions on Machine Learning Research (04/2025)
Table 14: Mean performance across different environments for each algorithm, averaged over seeds after 25M environment steps. Results averaged over the last 100 episodes of training. In Gravitar, intrinsic rewards appear to hinder the performance of the extrinsic agent, whereas, in other environments, they significantly enhance performance. Notably, in Seaquest, the extrinsic agent ranks among the lowest, highlighting the benefit of intrinsic rewards. All experiments were conducted using sticky actions with repeat probability of 0.25.
Algorithm
Gravitar
MontezumaaRevenge
PrivateEye
Seaquest
Venture
Extrinsic
1060.19
42.83
88.37
942.37
391.73
Disagreement
689.12
0.00
33.23
6577.03
468.43
E3B
503.43
0.50
66.23
8690.65
0.80
ICM
194.71
31.14
-27.50
2626.13
0.54
PseudoCounts
295.49
0.00
1076.74
668.96
1.03
RE3
130.00
2.68
312.72
864.60
0.06
RIDE
452.53
0.00
-1.40
1024.39
404.81
RND
835.57
160.22
45.85
5989.06
544.73
the use intrinsic rewards enables algorithms to significantly outperform the extrinsic agent, which ranks among the lowest.
Note that we do not compare these results other works because evaluation settings differ significantly between papers. For instance, in our case, we used sticky actions with probability of 0.25%, which makes the exploration problem more difficult, and it is not always used. Also, trained our agents for 25M steps instead of the standard 200M due to computational constraints. Still, our results provide evidence that intrinsic rewards are generally helpful in achieving better episode returns in hard-exploration environments.
Table 15: Aggregated performance comparison of mean episode return between RLeXplore and original implemen- tations on the ALE-5 benchmark. The results show average returns across five Atari games. Since the original implementations of R2D2 and NGU are not publicly available, report their published results obtained with 35B environment steps, while our experiments were conducted with 100 million frames.
Algorithm
RL Algorithm
Frames
Mean Episode Return
Extrinsic PPO
PPO
100M
0.5k
Extrinsic R2D2 Kapturowski et al. 2018
R2D2
35B
210k
(RLeXplore) Disagreement
PPO
100M
1.5k
(RLeXplore) NGU
PPO
100M
0.8k
(RLeXplore) PscudoCounts
PPO
100M
0.4k
(Original) Disagreement (Pathak et al.
2019
PPO
100M
0.1k
(Original) NGU (Badia et al. 2020
R2D2
35B
225k
Table 15 further provides direct performance comparison between our RLeXplore implementations and the original ones reported in the literature. Since public codebase dataset is available for NGU, we extracted its baseline performance numbers directly from the paper. Despite operating under more limited training budget, our NGU implementation still achicves competitive performance compared to the published results. For Disagreement, we used its official repository and trained the model for 1M frames. The scores we obtained match the results reported in (Pathak et al. 2019), confirming that our implementation faithfully reproduces the expected performance. Overall, the improved performance of RLeXplore's 's implementations is largely attributable to the extra effort put into fine-tuning low-level details, which helps to optimize the interplay of the various components.
D.6 Comparison with Other Projects
Table [6 illustrates the details of official implementations of the included intrinsic rewards in RLeXplore. It natural to find that they are implemented (1) in different codebases with (2) different librarics (c.g.. PyTorch vs Tensorflow), (3) using different RL algorithms (PPO, IMPALA, A3C, A2C), and (4) supporting different environments (ALE, Mario, MiniGrid, DMC). These details further motivate the development of
29

---

Published in Transactions on Machine Learning Research (04/2025)
Table 16: Details on official implementations of the included intrinsic rewards. Decoupled: Did the code decouple the intrinsic reward modules from the RL optimization algorithms, which can be directly reused in other projects?
Official Repository
ML
Backhone
Supported Tasks
Reward
Decoupled
framework
RL algorithm
SuperMarioBros,
ICM
Repository
Tensorflow
A3C
X
VizDoom
RND
Repository
Tensorflow
PPO
ALE
X
SuperMarioBros,
Disagreement
Repository
Tensorflow
PPO
X
ALE, Maze
NGU
N/A
N/A
N/A
N/A
N/A
PseudoCounts
from NGU
N/A
N/A
N/A
N/A
RIDE
Repository
PyTorch
IMPALA
MiniGrid
X
DMControl,
RE3
Repository
PyTorch
A2C, Dreamer, RAD
X
MiniGrid
MiniHack,
E3B
Repository
PyTorch
IMPALA
x
VizDoom
unified framework for training RL agents with intrinsic rewards under standardized conditions and reinforce our motivation to develop RLeXplore.
Furthermore, we provide a comparison of the advantages of other popular codebases for training RT agents with intrinsic rewards in terms the number of intrinsic reward methods implemented, their modularity and ability to reuse components between RL librarics casily, their documentation, and the number of experiments provided. As compared other existing projects, RLeXplore offers distinctive advantage by providing a more unified and standardized approach training RL agents with intrinsic rewards. It allows users to casily swap intrinsic reward modules regardless of RL libraries, which promotes reproducibility and consistency across different research works. Finally, RLeXplore is evaluated on wide range of benchmarks with over 1,000 experiments, ensuring its reliability and robustness across various scenarios.
Table 17: Comparison between RLeXplore and other reported libraries intrinsic rewards. Note that we focus on the intrinsic reward methods that implemented. For instance, CleanRL has many implementations of different RL algorithms, but RND is the only supported intrinsic reward.
ML
Number of Algorithms
Benchmark Results
Framework
Plug & Play Documentation
Framework
task,
CleanRL
PyTorch
x
experiments
tasks,
DI-Engine
PyTorch
X
19 experiments
rllib
TensorFlow
N/A
17 tasks,
RLeXplore
Py PyTocc
over 2000 experiments
30

---

Published in Transactions on Machine Learning Research (04/2025)
E
Learning Curves
E.1
Q1
Ooservation Romalizatonni RMS
Ohsenvatio Nermal 2ition Max
Exrrinii
22
ICH
RAD
sagreement
NGUI
Epwodde Devond
15 12
20
12
Envirommant Stapa (K107)
Erstro mmnt 50aps (x20))
Eneifonmnnt Stops (x107)
Envrrnmment Stopa (K200)
PscudoCounts
HDE
E30
Tp-oot beown
12
12
15
Enviroremntt Stapa (H10))
Sistro-merte Staps (x20))
Enwironmnn Staps (x10))
Envranmment (x20))
Figure 10: Learning of the baselines and Q1 Murioraas. The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
Observatton Norma tzationm RMS
Observal Normaiiaationn-eeet.t
120
ICM
120
RND
Diagreementt
NGU
100
100
100
100
Ediacde Rewaid
Edlavii Rowadd
&
33
02
34
10
32
6.5 Enwfronment Stops 1x19
080
3.3
33
8.0
ErwFoommnn Steli 1010
Ensfromment Stels [x107)
Enviromment Steas (x1691
PreustCCouuss
MIDE
RE3
EBB
120
125
101
Eptsook Ronurd
Eptodd
33
1.2
32
6.5
080
32
1.0
Enviromment Stess 1011
Enviromment Stops 1x10
Ensirnmen: Steas 1x1073
Envirommnt Steas (x1991
Figure 11: Learning curves of the baselines and QI on MiniGrid DoorKey-166 16x 16. The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
31

---

Published in Transactions on Machine Learning Research (04/2025)
E.2 Q2
Reaend Normeization RMS
Reaard Nemellcation----e Minslax
Swaend hormelsation Meenthe
Eatrinal
20
Eptisee Revwrrd
Epltode Rowen
15
15
Von
(6207)
PsoudoCounts
Eplevee Sevard
20
Estoodo Reward
Wwm
Edlado 8
20
Esloada
to
30
DE EndrennenntStpp (x10"1
5. Environment Stops (x10")
0.2
6.1
30
StproamensSSpss (x20
Environment (x10))
Figure 12: Learning curves of the Q2 on SuperMario Bros. The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
Reward Norm IIM
Heaward Ne Kermalianion-tii-We
Hameard wewwarddeeeee hermaloasinnneeiit
Iost mac
ICM
RND
320
Dhangremment
NGU
100
300
20
Epicode Reward
Epiiodde Noward
Edtrodt Estrodo Roward
60
09
40
20
-20
-28
02
54
22
02
122
22
Enstronment Steps Steps|x101) (x10%)
Envionment Steps (x300)
Enstronment Steps (x10")
Envtronment Steps Ex101)
PocusoCounts
RIDE
128
REB
EBB
100
Eplcode Reaard
Sptodd Roward
Eptode Roward
83
wwwm
15
Environmenttaao Stops 1x30")
EmiisnmeentStpss (x20')
Environment Staps (x20")
Environment Steos (x10")
Figure 13: Learning curves of the Q2 on MiniGrid- MiniCrid-Doorkey-.ee DoorKoy- 16x The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
32

---

Published in Transactions on Machine Learning Research (04/2025)
E.3 Q3
Undale Propartion1..
Utdete Proprtion-AS
Update Proportam--.11
Update Prosortaamm...
Estrinic
20
30
Epttode Rowend
15
(6207)
PsoudoCounts
Ealevee Sevard
Estoodo Reward to
23
Esloada
50
DE
0.5
0.1
0.2
30
Entrennennttopp (x20"1
StproamensSSpss (x20
Environment Stops (x10")
(x10))
Figure 14: Learning curves of the Q3 on SuperMario Bros. The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
Update Pipprtoore110
Update Proportionnm...
Updula Proportion
Update Praportion-..11
Extrinsic
ICM
RND
Dhaggre-ment
NGU
320
100
300
20
Epicvde Reward
Epiiodie Noward
Edtrodo Estrodi Roward
60
62
00
40
20
-20
-28
-20
02
54
13
22
02
122
22
Enstronment Steps (x10%)
Envionment Steps (x300))
Enstronment Steps (x107)
Enstronment Steps Ex1005
PocusoCounts
RIDE
125
RE3
EBB
100
Eplcode Reandd
Sptcode Noward
Eptode Roward
-28
52
0.
54
T4
52
18
Envirosment Steps 1x20")
Emvisnmenn:tag (x10')
Envirommnn tx20"
Environment Steos (x10')
Figure 15: Learning curves of the Q3 on MiniGrid Min-Crid-Doorkyy-.eee 16x The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
33

---

Published in Transactions on Machine Learning Research (04/2025)
E.4 Q4
Weight trateliattoo-Orttgeee
Dieeeaan-DDeeeuu Extrinsic
20
30
Ealoods Rowend
20
15
(6207)
PsoudoCounts
Eatevee Sevard
Estoodo Reword
16
5 8
20
Esioada
5.0
DE
54 Enviromennt Stops (x10")
022
30
EntrenneentStpp (x10"1
EtproamensSSpps (x20
Environment (x10))
Figure 16: Learning curves of the Q4 on SuperMario Bros. The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
Weight triisilatiionn-rrggggg
Weight intiallaationnnan.. Dellait
Extrinse
IEM
RND
Dhaggeemeett
NGU
320
1.0
300
20
Epicode Reward
Roward
Epiiodi Noward
Edtrodt EstrodioRowar Roward
60
00
60
20
-20
-20
-20
02
54
22
127
02
E6
12
22
Enstronment Steps Steps|x100) (x30%)
Envionmmen Steps (x300))
Enstronment Steps (x10")
Environment Steps Ex100)
PocusoCounts
RIDE
REB
EBB
125
100
Eplcdde Reanrd
Sptodd Rowardd
Eptode Roward
-20
52
18
Environmenttaae Stops 1x30")
Emvismmenn:tee (x10')
Environment Staps 1x20")
Environment Stees (x10')
Figure 17: Learning curves of the Q4 on MiniGrid Min-Crid-DoorKey--000 DoyrKoy--KKxx- 16. The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
34

---

Published in Transactions on Machine Learning Research (04/2025)
E.5
Q5
Ventle
LSTM
Extrasic
20
20
20
Ealtode Rowend
20
22
15
1.
(X207)
(6207)
PsoudoCounts
RIDE
Eatevee Sevard
20
Estoodo Reward
-0
Roword
23
byy
Esioada
to
5.0
DE
OR
D2 EtprommensSopss (x20
54 Enviromennt Stops (x10")
022
6.1
30
EntrennenntStpp (x10'1
Environment (x10')
Figure 18: Learning curves of the Q5 on SuperMario Bros. The solid line and shaded regions represent the mean and standard deviation computed with 10 random seeds, respectively.
Vunilaa
LITh
Entrinak
IEM
RND
320
Dhagreement
NGU
1.0
300
20
82
Epicode Reward
Rowar
Epiiodde Noward
Edtrodi Estrodio Roward
60
80
20
-20
-28
02
54
13
32
02
12
22
Enstronment Steps Steps|x100)
Envionmmen Steps (x300))
Enstronment Steps (x107)
Ensironment Steps (x10f)
PocusoCounts
RIDE
REB
EBB
125
100
Eplcode Reanrd
wwwmmm
Eplodd Roward
Eppode Roward
83
M
15
Envirosmenttane Stops 1x20")
Emvisnmeentteegs (x30')
Environmnn Stags tx20")
Environment Steos (x10")
Figure 19: Learning curves of the Q5 on MiniGrid- MiniCrid-DoorKeyy. 16x The solid line and shaded regions represent the
mean and standard deviation computed with 10 random seeds, respectively.
35

---

Published in Transactions on Machine Learning Research (04/2025)
E.6 Q6
PPO
Two Hoad PPO
Extrnatc
X30
Daagremmmn ment
Epponde Reundd
Episade Reword
Brviromment (x207)
Enxfronment (x109)
Enstronment (x107)
Environment
PsoudoCrunts
10
10
10
Estivete Rewnnd
Estode Reword
Esleode
60
1.D
15
25
20
05 Enstronment Steps
10
03
33
20
25
Envirommnn Staps (X20')
Emvironmnt Steps (x10f)
Environment Staps (K10")
Figure 20: Learning curves of Q6 on Progge- HazdHand. The solid line and shaded regions represent the mean and standard deviation computed with five random seeds, respectively.
Tea-tfead
Extrinic
KCM
Diaggee mentt
10
Epande Beaers
Eptoode Roward
Ephode Breperd
1.50
230
23
0.2
0.5
3.2
3.5
2.0
255
200
05
1.0
2.3
2.0
6.0
03
35.
2.0
255
Bnvitommen Stepss (x3071
Envtronment Steps Oct D1)
Enstronment Steps OKt.07
Environment Steps Dc1
#scudoCounts
RIDE
REB
NGU
Edtodd Edtoode Rorerd
Enwiroment Steps (x20')
Endironmett Steps (x19F)
Envrroment Steps (tops (X101)
Envirommant Stops 0<19")
Figure 21: Learning curves of Q6 Allggen-AlMazeJHoae The solid line and shaded regions represent the and standard deviation computed with five random seeds, respectively.
36

---

Published in Transactions on Machine Learning Research (04/2025)
E.7
Q7
E3B+RND E3B+ICM
E3B+RIDE
RE3+RND
RE3+ICM
RID+
SuperMarioBros-I-1-2
SuperMaridrnsRandoontgggss
25
Episode Reward
Episode Reward
30
20
15
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10?)
Environment Steps (x10?)
tepisdl exploration) on SuperMuurioBro--Ittuo and The solid line and shaded regions represent the mean and standard devia- tion computed with five random seeds, respectively.
Figure www.Marrrroooosaaoommmmoooo
Learning curves
Q7
ICM+RND
RIDE+RND
RIM+RIDE
SuperMarioBrrs-11112
SupprrarioorooRanddnSiaggg
30
12
25
10
Episode Reward
Episode Reward
15
10
0.2
0.4
1.0
Environment Steps (x107)
Environment Steps (x107)
Superruuiiorrus I-1-13 and SuperMario Bros Broosonndo.Stage-sn The solid line and shaded regions represent mean and standard devia- tion computed with five random seeds, respectively.
Figure 23:
Learning curves of Q7
exploration)
37

---

Published in Transactions on Machine Learning Research (04/2025)
E.8
Additional Experiments for MiniGrid
Exhrisic
KeyCoridar509n
KayCorddo
Muliitemm-U N7-58
50
Epiode Downrs
Ma Ma
42
Epiodda Nowars
Epaont Rownrd
20
22
-10
22
RuthRoem-440--11
MatrRooomM 2-510
LoccedRoom
Dy Dysme--Bistehweirrtt
50
50
50
10
Ron3rd
Eptoode Rewsrd
23
Steps
(x207)
Figure 24: Learning curves four selected intrinsic rewards on eight extremely hard tasks. The solid line and shaded regions represent the and standard deviation computed with five random seeds, respectively.
Intrinsic Reward
Extrinsic
ICN
AND
Disagreament
NGU
100
120
60
80
30
Epaide (paaseeteram Tevaan
Epsoode fareade/taeenn fareaadd
Epaode Sarenn
Epauzde Taraadeearraan Sareard
60
kn
0.0
00
20
20
20
20
3.0
02
03
54
1.0
0.0
022
5.6
54
3.
20
52
011
3."
Instronmentt Steps Ox10")
Envfronment Steps (x107)
Enimmmment Steps (x107) RE3
Envimnmmnnt Steas (X10")
PoeusoCrruttss
RDZ
E30
80
Edtocde Edleode Rewaard
Edtocde Hovoord Hewwrr
Esteode 50
720 02
100
02
5.
'20
32
14
Snstronmmetttee Steps (x10")
Environment Stops tx2071
Emironment Staps
Enstronmmnttte Stees (x10")
Figure 25: Learning curves on MiniGrid-Mu MowteeMoooiitoieeeeen The solid line and shaded regions represent the mean and standard deviation computed with five random seeds, respectively.
38

---

Published in Transactions on Machine Learning Research (04/2025)
Infiidiic Roward
Extinsic
HAQ
Disegresment
Tplcnde Requard
Sploods Heetardd
sploode Rasuand
Eploode
022
100
Environment Steps (x10")
Enwironment Steps (Mx10))
Enwironment Stops Stops (xI07)
Environment Stettttteee Steos (x101)
Enikode Rewnrd
tsraezie staward
Epoonte
Environment (x10))
Enviromment (x30
Enwrrnment Steps (x10F)
(419))
Figure 26: Learning curves on MiniGrid- Mutt-rii--MiiRRoo----.... Na-ooo--N4-S5--00 N4-85-v0. The solid line and shaded regions represent the mean and standard deviation computed with five random seeds, respectively.
Intrinaic Reward
Extinssc
RND
Dlougneement
Eplivde Secirr
Eplende Reeverd
Eplcode Sewand
toiend Benard
02
646
100
0.0
63
8.6
066
23
6a
10
Instronment Steps (x100)
Enviromment Steps (x:107)
Enwironent Steps (x1.7))
Envtronment Steps (x107)
R.DEE
REB
Eppccde Seavir
tploode Rarard
Eplodd
HM
Eppoode
W
Instronmentt
Endirnment (x:00)
Enwimemennt (x107)
Envimnment Steas (x10")
Figure 27: Learning curves on MiniGrid-Keyu Cerorii-eeeCrrrddSS3333-3-- The solid line and shaded regions represent the mean and standard deviation computed with five random seeds, respectively.
39

---

Published in Transactions on Machine Learning Research (04/2025)
Inofiniic Roward
Extinsic
ICN
WND
Disegremment
Tploode Tploode- Requard
Sploods Heesard
Spleode Tastard
Eplcod
022
000
-33
120 02
20
33
Envfronmmntt Steps (410")
Enwironment Steps (Mx101)
Enwironment Stops (XID7) RES
Environment Steos (x101)
Enykote Rewnaed
Espaezde Strwarrd
Eppadde
Epponte
Environment Staps (x10))
Environment Staps (x:207)
Enwiromment
Staps (x10F)
(H207)
Figure 28: Learning curves on MiniGrid R3 The solid line and shaded regions represent the mean and standard deviation computed with five random seeds, respectively.
Intrinaic Raward
Extrinic
ICM
RND
Diougrement
NIU
Eplivde Secvard
Epieode Deevarrd
Eplcode Seswart
tolonde Benard
3.0
02
044
0.
6.3
100
0.0
63
8.6
0.2
62
2/4
0.6
0'6
12
33
63
2.0
Invtronment Steps 6x10')
Envirommnn Steps (xx107)
Enwironment Steps (x1177)
Envtronment Steas (XI07)
PocudoCourss
R.D.E
REB
Ephccde Seavid
tphocde Sarward
Ephode
Ephoode
Instrommett
Endrrnment (4:001
Enwimement (x10
Invirnment Stexstett"
Figure 29: Learning curves on FemnCrnd-KeCCrrrdnnnnii The solid line and shaded regions represent the mean and standard deviation computed with five random seeds, respectively.
40

---

Published in Transactions on Machine Learning Research (04/2025)
F On-Policy RL Algorithms and Discrete Control Tasks
In this section, we demonstrate the combination of RLeXplore and on-policy RL algorithms and their of- fectiveness on discrete control lasks. Specifically, we couple the PPO algorithm and intrinsic rewards and evaluate their performance on Montezuma Revenge, a hard exploration task from the ALE benchmark [Bellemare et al. 2013). We use the PPO implementation of CleanRL (Huang et al, 2022b) to show the adaptability of RLeXplore. Table 18 illustrates the training hyperparameters used for the experiments.
Table Training hyperparanelerrs for Monlezuma Revenge.
Part
Hyperparameter
Value
Observation downsampling
(84, 84)
Stacked frames
Enviromment steps
le+-8
Episode steps Number of workers
128
1
Environments per worker
8
Optimizer
Adam
PPO
Learning rate
le-4
GAE coefficient
0.95
Action entropy coefficient
0.01
Value loss coefficient.
0.5
Value clip range
0.1
Max gradient norm
0.5
Epochs per rollout
4
Batch size
256
Discount factor
0.99
Observation normalization
RMS
Reward normalization
RMS
Intrinsic reward
Weight initialization
Orthogonal
Update proportion
0.25
with LSTM
False
Montezuma's Revenge
CleanAL's PPO RLeXplore's RND
CleanRL's PPO
1250
Episode Reward
1000
750
500
250
250
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10*)
Figure 30: Since only RND can achieve significant results in this task among the eight intrinsic rewards, only show results RND. The solid line and shaded regions represent the mean and standard deviation computed with five random seeds, respectively.
41

---

Published in Transactions on Machine Learning Research (04/2025)
G Off- Off-Py RL Algorithms and Continuous Control Tasks
To showcase the generality of RLeXplore, we run additional experiments in scttings different from the ones in the main paper. Concretely, we couple intrinsic rewards with soft actor-critic (SAC) (Haarnoja et al. 2018), an off-policy RT. algorithm, and test their performance in Ant- UMae,, a. continuous control task with sparse rewards. Table 19 illustrates the training hyperparameters used for the experiments. We show the performance of Disagreement, RND, ICM, and vanilla SAC in Figure 31 The results indicate that intrinsically-motivalee agents able to navigate the maze than the vanilla agents that only learn from the sparse task rewards.
efficiently, finding the goals more often
We only use 3 intrinsic rewards with SAC because of the episodic nature of the other intrinsic reward methods, For example, the episodic memory in RIDE, PseudoCounts, NGU; and the episodic ellipsoid in E3B require the replay buffer to sample entire opisodes instead of random rollouts. We aim to implement this logic in our RLeXplore codebase in the future.
Table 19: Training hyperparameters for Ant- Umaze.
Part
Parameter
Value
Total timesteps
1 106
Buffer size
106
Discount (v)
0.99
Target smoothing coefficient (r)
0.005
Batch size
256
Learning starts
5000
SAC
Policy learning rate
3 10-4
Q function learning rate
1 10-3
Policy frequency
2
Target network frequency
1
Noise clip
0.5
Entropy coefficient (a)
0.2
Auto-(une entropy coefficient
True
Observation normalization
RMS
Reward normalization
RMS
Intrinsic reward
Weight initialization
Orthogonal
Update proportion
0.25
with LSTM
False
SAC AntUMaze-v4
Disagreement
Extrimsic
RND
ICM
Extrinsic
Disagreement
ICM
RND
Fpisade Return
Figure 31: Performance comparison between the three selected intrinsic rewards and the extrinsic reward.
42


Reviews:
[{"review_id": "5NpBoyWKQs", "reviewer": "Reviewer_Lm7H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written and easy to follow\n* The investigation of hyper parameters is interesting, well presented, and most importantly important for researchers and practitioners.\n* The described API of the framework looks clean and easy to use \n\nThe only shortcoming that I see within the scope of the paper (a framework for evaluating intrinsic reward methods) is that it lacks a clear comparison between the performance of the re-implementations and the original results for Disagreement, NGU, and PseudoCounts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a software framework for evaluating intrinsic motivation methods in reinforcement learning. They accompany their framework with extensive evaluations of the implemented methods across different tasks, particularly focusing on the often overlooked importance of hyper parameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* I would like to see a performance comparison of Disagreement, NGU, and PseudoCounts to the results reported in the corresponding papers in Appendix D.\n* After reading Section 5.1, I was not sure whether the \"Combined\" experiments were using the best hyper parameters across methods or per method. It would be hence worth highlighting whether the authors could identify one universally well working set of hyper parameters for all their implemented methods or not.\n* Figure 5 could benefit from highlighting which investigations belong to which research questions (like in Figure 4).\n* I did not understand why the authors outline URL algorithms in Section 2.4. As far as I understand, the authors did not implement or evaluate any URL methods. Mentioning the URL benchmark could be done without talking about URL methods. Therefore, I would recommend to not make this discussion a sub-section in order to not confuse readers about the contents of the framework.\n* I would recommend the authors to add references to Appendix B, F, and G from the main text.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9ddNx5HbTt", "reviewer": "Reviewer_K5wK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- [S1] RLeXplore provides a unified and modular framework that integrates various state-of-the-art intrinsic reward algorithms. Also, RLeXplore is an open-source library that provides high-quality implementations. This addresses the existing gap in standardized implementations, enhancing reproducibility and facilitating fair comparisons across different methods.\n- [S2] RLeXplore can be seamlessly integrated with existing RL libraries such as Stable Baselines3, CleanRL, and RLLTE. This flexibility allows researchers to easily incorporate RLeXplore into their existing workflows.\n- [S3] The paper presents a systematic study of implementation details that significantly impact the performance of intrinsic reward methods. By investigating factors such as observation normalization, reward normalization, update frequency, and weight initialization, the authors offer valuable insights into the practical aspects of intrinsically-motivated RL.\n- [S4] The framework demonstrates the ability to reproduce and improve upon previously reported results across multiple environments, including SuperMarioBros, MiniGrid, and Procgen. This highlights the effectiveness of the RLeXplore implementations and the importance of the identified implementation details.\n\n### Weaknesses\n- Some actionable weaknesses are written in **Requested Changes**.\n- [W1] As author mentioned, more complex algorithms like BYOL-Explore or RECODE were not included due to their computational demands and limited open-source availability. However, those algorithms can achieve SoTA performance ouperforming the algorithms impremented in this library. This poses a question if RLeXplore may only have a limited contribution to the progress of research on intrisic reward because the SoTA is not included.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces RLeXplore, a unified, modular, and open-source framework designed to facilitate research in intrinsically-motivated reinforcement learning (RL). RLeXplore offers clean implementations of eight state-of-the-art intrinsic reward algorithms, addressing the lack of standardization and insufficient exploration of implementation details in the field. The authors conduct an in-depth study based on 7 research questions to identify critical implementation details and aim to establish standard practices in intrinsically-motivated RL. The framework also encourage the community to enable fair comparisons, easy integration with various RL frameworks, and streamlined development of new intrinsic reward algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Mandantory\n- [MR1] In Table 1, please explain the reason or intuition why RND-MiniGrid-DoorKey-16x16-RLeXprole is better than RND-MiniGrid-DoorKey-8x8-RLeXprole (0.6 vx. 0.0). This is not consistent with other algorithms (ICM, RE3, RIDE) and not intuitive because MiniGrid-DoorKey-8x8 should be easier than MiniGrid-DoorKey-16x16.\n- [MR2] Related to Figure 3, please include the comprehensive Table of environment to explain observation space (image or sensor vectors), action space (discrete, continuous), reward (dense/sparse, available or not), termination, and which intrisic reward can be used (e.g. In Figure 31, only 3 intrisic rewards are used in AntUMaze). If the impremented intrisic rewards are agnostic to the environment, please ignore the last.\n- [MR3] Please include the comprehensive Table of intrisic rewards to explain which type of algorithms can be combined such as (1)discrete/continuous action space, (2) observation space, (3) on-policy v.s. off-policy, (4) model-free v.s. model-based, etc. Please adjust the axes appropriately. This should improve the convinience for the users.\n- [MR4] Please explain why the combination of reward is smaller than $\\binom{8}{2} = 28$ pairs in Figure 8. If there are some criteria or strategy for the combination, please include it. If there are some infeasible combination due to some technical difficulties, please consider adding summary table to describe the available combinations. \n\n### Recommendation\n- [RR1] It would be great to add one or two sentences to explain why RLeXplore outperforms Original in Table 1. I guess this is due to the implementation details explored in Section 5 & Figure 4, but please highlight such exprolations realizes better implementations.\n- [RR2] Please consider including the guide to descibire how the user can add novel intrinsic reward modules into RLeXplore.", "broader_impact_concerns": "This paper does not have Broader Impact Statement section, but considering the research topic the paper studies, I think it is not necessarily required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VlSFHlNfEz", "reviewer": "Reviewer_dYVt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper is well-written and is easy to follow.\n- The need for a standardized framework in intrinsic reward reinforcement learning is clearly presented and is well-motivated.\n- The experiments are detailed and comprehensive.\n- RLeXplore is designed to encourage the experimentation of new intrinsic reward methods without too much overhead.\n\n### Weaknesses:\n\n- Although the authors have already included enough baselines, the design of RLeXplore does not allow the implementation of unsupervised reinforcement learning methods that first explore and discover skills and then learn them. The authors recognize this limitation in conclusion. However, it can be said that this limits the range of methods that can be implemented with RLeXplore. Therefore, it is worthwhile to discuss and provide some alternative design choices.\n\nAdditional points for discussion and clarification:\n- What is the purpose of Figure 1 in the introduction? Could you please elaborate on that?\n- In Table 1, is the performance increase attributed only to the selection of the different hyperparameters (implementation details). Are the environments exactly the same as the original?\n- I find the presentation of the results in Figure 4 confusing and not easy to follow. More specifically, I expected the legend of Figure 4 to be similar to the candidates in Table 5 of the Appendix. For example, in Q2, the approaches are Vanilla, RMS, and Min-Max, whereas in the plot, there is only Min-Max and Vanilla. I understand that some configurations are already covered in Baseline and Combined, but this way of presenting the results is hard to follow.\n- Could you clarify why Min-Max reward normalization is better than Combined for SMB in Figure 5? As I understood, Combined is the selection of the best hyperparameters across all questions.\n- “Interestingly, in MGD, Min-Max normalization seems to decrease the performance of Disagreement and NGU”. Could you specify the plot to which this statement is referring?\n- I would recommend the authors to add some implementation details on the two separate value functions, since this part is vague. For example, are both value functions used in the advantage estimation of PPO?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce RLeXplore, a modular framework that can be used by researchers and practitioners who work on intrinsically motivated reinforcement learning.\n- They develop an open-source library that includes eight state-of-the-art intrinsically motivated reinforcement learning techniques. \n- They design RLeXplore in a way that can be easily integrated with other RL frameworks.\n- Their framework provides a Fabric class that can be used to combine intrinsic rewards from different techniques.\n- They provide a systematic experimental analysis of how various implementation details affect the performance of the different intrinsic reward methods. \n- In their results, they demonstrate that by carefully choosing the design configurations, they can outperform previous implementations of the same methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to ask the authors to address the aforementioned weakness and to clarify and discuss the open questions/points.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in consensus that the revision strongly improved the paper. K5wK: "They sufficiently addressed my concerns and comments with paper revision. The unified benchmark and codebase would promote future research on intrinsic-reward-based exploration in reinforcement learning" and dYVt "The authors have effectively addressed all concerns and open questions raised during the rebuttal phase. The revised manuscript is an improved version that incorporates the necessary clarifications and refinements in response to reviewers’ feedback."

In the final version of the paper the author should address two minor clarifications following:

* In response to reviewer K5wK, the authors performed an evaluation of the RND algorithm in the MiniGrid-DoorKey-8x8 environment with an increased number of steps, confirming the expected effect that under the same number of steps (10M) RND would perform better in MiniGrid-DoorKey-8x8 than in MiniGrid-DoorKey-16x16. However, in the updated paper, the authors list the newly achieved performance as if it has been achieved with 1M steps.

* The authors should add an explanation of the new Table 15 in Appendix D.5.
Decision: Accept (poster)