Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus
Authors: Yiming Wang
Kaiyan Zhao
Furui Liu
Leong Hou U
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Reinforcement Learning
exploration bonus
intrinsic reward
metric-based behavioral similarity
Primary area: reinforcement_learning
TL;DR: We introduce the Effective Metric-based Exploration-bonus which addresses the inherent limitations and approximation inaccuracies of current metric-based state discrepancy methods for exploration

Abstract: Enhancing exploration in reinforcement learning (RL) through the incorporation of intrinsic rewards, specifically by leveraging *state discrepancy* measures within various metric spaces as exploration bonuses, has emerged as a prevalent strategy to encourage agents to visit novel states. The critical factor lies in how to quantify the difference between adjacent states as *novelty* for promoting effective exploration.
Nonetheless, existing methods that evaluate state discrepancy in the latent space under $L_1$ or $L_2$ norm often depend on count-based episodic terms as scaling factors for exploration bonuses, significantly limiting their scalability. Additionally, methods that utilize the bisimulation metric for evaluating state discrepancies face a theory-practice gap due to improper approximations in metric learning, particularly struggling with *hard exploration* tasks. To overcome these challenges, we introduce the **E**ffective **M**etric-based **E**xploration-bonus (EME). EME critically examines and addresses the inherent limitations and approximation inaccuracies of current metric-based state discrepancy methods for exploration, proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to be dynamically adjusted by the variance of prediction from an ensemble of reward models, thereby enhancing exploration effectiveness in particularly challenging scenarios. 
Extensive experiments are conducted on hard exploration tasks within Atari games, Minigrid, Robosuite, and Habitat, which illustrate our method's scalability to various scenarios. The project website can be found at https://sites.google.com/view/effective-metric-exploration.

Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus
Yiming Wang', Kaiyan Zhao! Furui Liu?, Leong Hou U
State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao SAR, China School of Computer Science, Wuhan University, Wuhan, China *Zeejiang Lab, Hangzhou, China
wang - www.iigiooonnntt.tt.mmmmmmm um. edu. mo, zhao kaiyan whu edu en liufurui (eeeaiiieeeeaaeaa com, ryanihus hulluuuu um, edu. mo
Abstract
Enhancing exploration in reinforcement learning (RL) through the incorporation intrinsic rewards, specifically by leveraging state discrepancy measures within various metric spaces as exploration bonuses, has emerged as a prevalent strategy to encourage agents to visit novel states. The critical factor lies in how quantify the difference between adjacent states novelty for promoting effective exploration. Nonetheless, existing methods that evaluate state discrepancy the latent space under L, or L2 norm often depend on count-based episodic terms as scaling factors for exploration bonuses, significantly limiting their scalability. Additionally, methods that utilize the bisimulation metric for evaluating state discrepancies face a theory-practice gap due to improper approximations metric learning, particularly struggling with hard exploration tasks. To overcome these challenges, introduce the Effective Metric-based Exploration-born and addresses the inherent limitations and approximation inaccuracies of current metric- state discrepancy methods for exploration, proposing robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhancee scaling factor integrated into the exploration bonus be dynamically adjusted by the variance of prediction from an ensemble of reward models, thereby enhancing exploration effectiveness in particularly challenging scenarios. Extensive experiments conducted hard exploration tasks within Atari games, Minigrid, Robosuite, and Habitat, which illustrate our method's 's scalability various scenarios. The project website can be found at http://sites.ooogleee cwwp///wwwwflcccii-eeeieeepppppp
(EME). EME critically examines
1 Introduction
Reinforcement learning (RL) has made significant strides, yielding breakthroughs across various domains, including video gaming [3]. autonomous driving [65], and robotic control 337.. However, for many real-world tasks, defining a dense reward function is non-trivial, yet sparse reward function based success failure is directly available, which makes learning effective policies difficult, as they demand efficient exploration of the state space, highlighting exploration in sparse-reward environments as a core challenge in RL [59]
Methods that quantify the state discrepancy between adjacent steps using specific measures within different metric spaces have shown remarkable success in tasks characterized by sparse rewards. These methods, by leveraging a measure of state discrepancy as an exploration bonus, facilitate agents
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

in discovering novel states. For instance, RIDE [1]] employs the difference between two consecutive state embeddings, measured under L2 norm, as an exploration bonus. Similarly, NovelD [69 introduces a bonus based on the state discrepancy, as represented by the RND [ bonus, but under the In norm. However, the effectiveness of these T,p norms-based methods contingent on the scaling factor expressed as the episodic count term (sp which is the number of times state s has been visited during the current episode. The episodic term becomes ineffective when each state is unique and cannot be counted, posting a significant limitation when it comes to more complex, dynamic, and noisy environments [32]. LIBERTY
63 aims to surmount this hurdle by
evaluating state discrepancy through the bisimulation metric [25 which links state differences to value differences within the bisimulation metric space, thus incentivizing exploration of novel states with greater value differences and substantially enhancing learning efficiency. Nevertheless, due to the high computational cost or infeasibility of accurately computing bisimulation metrics, the approximation, and relaxations over the metric [13 68, 63) have been used to optimize efficiency. Our analysis reveals that approximation gap in LIBERTY might break the theoretical guarantees of the bisimulation metric, potentially undermining exploration performance. Furthermore, the efficacy of bonuses based on state discrepancies declines in hard exploration tasks or scenarios where state differences are minimal, such as in the "Noisy-TV" (II] problem and vision-based real indoor environments [60], posing a significant constraint on the scalability of these methods.
(c)
Figure I: Trajectories of policies trained with different exploration algorithms in the real- life indoor environment. (a) episodic count-based method under Lp norm (b) bisimulation metric-based method (c) our method.
To address this aforementioned limitations, we introduce the Effective Metric- Exploration- bonus (EME) for exploration. Our method features a more resilient metric for evaluating state differences, simultancously eliminating approximation gap with theoretical guarantee. EME significantly boosts learning efficiency by forging a closer connection to the value differences between states, while maintaining fidelity to its theoretical framework. Furthermore, we have developed diversity-enhaneee scaling factor to augment exploration efficacy hard exploration tasks, where state differences are notably subtle. Our scaling factor dynamically adjusts the exploration bonus based on the variance of predictions from an ensemble of reward models, which is higher when agents encounter novel state space, thereby encouraging more effective exploration. As illustrated in Figure [ we present the trajectories policies trained using count-based state discrepancy bonus, bisimulation based exploration bonus, and our method within real-life indoor environment [15) (Full results in Figure of Appendix C.3). Our method successfully explores a significantly larger portion of the space compared to the others, demonstrating the superior effecttivneess of EME.
The main contributions of this paper are as follows. Firstly, we conduct a comprehensive analysis of the limitations inherent in current metric- based state discrepancy methods for exploration. Based on the analysis, we introduce an effective metric for evaluating the behavioral similarity between scaling factor
states supported by theoretical assurances. Secondly, propose a diversity-enhancn for exploration bonuses based on the variance of predictions from an ensemble of reward models, which is scalable and effective in hard exploration tasks. Lastly, extensive experiments conducted in Robosuite, Atari games, MiniGrid, and Habitat. The results demonstrate that our algorithm can effectively enhance exploration and accelerate training across various environments.
2 Background
We focus on exploration bonuses to incentivize exploration in reinforcement learning (RL). To foster the reader's understanding, we first introduce standard notation and common practices.
Markov Decision Processes. We assume the underlying environment is a Markov decision process (MDP), defined by the tuple M = (S, A, P, r), where S is the state space, A the action space, P s, a) is state transition function from state S to state s' E S, R is the reward function

---

and y E (0. is the discount factor. Generally, the policy of an agent in an MDP is a mapping S x A- [0, 1]. An agent chooses actions E according to a policy function a ~ (()), which updates the system state x' ~ P(s, a) yielding a reward = R(s,, a). In this paper, we denote a policy by zo, where is the parameter of the policy function. The goal of the agent to optimize the parameter 0 for maximizing the expected accumulative rewards, (rTe = Egg 00 7 R (sc. at)]-
State Discrepancy as Exploration Bonuses. Addressing the sparse reward challenge prevalent in many environments necessitates augmenting the external reward function, r², with intrinsic reward bonus, b, resulting in a combined reward: r = re + b. A number of intrinsic bonuse that encourage exploration have been proposed. More recent methodologies employ exploration bonus that varies based on the novelty of the current state relative to previously visited states, evaluated within different metric spaces, which can be generally defined as:
= SD * SF = d()(s), ((s))) * A
(I)
where SD denotes the state discrepancy, SF denotes the scaling factor, d: S S - IE represents the evaluation metric that quantifies the degree of difference or distance between two projected states, and 4 : S + Z is the projection function which maps the state representation space Z.
Algorithm
Objective = State Discrepancy- Scaling Factor Episodic Approximation Gap Scalability
RIDE [ST
[ Φ(s)]|2 1|2
(s'(s')
NovelD 69
I RND (s')
AND (s)
=
LIBERTY
63
(ydine (s', so) dino (s, so)]
X
EME (ours)
dg(s', 8) min (orn(maa (mar (C(r.), 13. M)
X
Table I: Comparison of exploration methods using different measures (marked in blue) of state discrepancy as
exploration bonus. o= fs)) features encoder; Ns. : episodic (pseudo) count of visits to state s; RND (s):
the RND [1] bonus; a: normalized coefficient; dine: inverse dynamic bisimulation metric used in [63]; so:
initial state; A: the scaling hyper-paramett dg: our proposed effective metric for state discrepancy evaluation; <(rs): variance of predictions from ensemble of reward models: M: maximum reward scaling.
The comparison of recent metric-based exploration methods is summarized in Table These bonuses reward high dissimilarity between adjacent states, encouraging exploration by leveraging measures such as the L2 distance for state embeddings (as used by RIDE (51) and the L1 distance for the disparity in RND bonuses between adjacent states (as utilized by NovelD [69]). However, these approaches, grounded in Lp norms and dependent episodic state visit counts, face significant scalability challenges when exploring various environments. On the other hand, LIBERTY [63 proposes the inverse dynamic bisimulation metric to evaluate the state difference, which links state and value differences to enhance learning efficiency. Yet, the approximation inaccuracies inherent in bisimulation metric learning compromise theoretical assurances, breaking theoretical integrity of bisimulation metric. In the following sections, we detail the shortcomings of these methods.
3 Limitations of Existing Metric-based Exploration Bonus
3.1 Latent Vector Space under Lp Norms
Ip norms such as L1 and L2 norms are widely used in measuring the "distance" " between states or embeddings. For example, considering a continuous map between the states of MDP and latent vector space: Φ : S + Z C IR.. We can equip this vector space with a Euclidean norm to obtain a Euclidean metric space (2, || ||2). and the state discrepancy is measured as SD = [ - +
Reliance on Count-based Scaling Factor. The state- difference based exploration bonus employed by RIDE (51) and Noveld [69] can be regarded as the dif- ference between adjacent states in the Lp norm space. The efficacy of the exploration bonus heavily lies in the episodic scaling factor (SF) measured by the count of visited states Nep(s) presented in T However, the count-based episodic bonus suffers from fundamental limitation, which is similar to that faced by count-based approaches [5) in general: if each state is unique, then
Table 2: Mean success rate comparison (values
helow 0.5 are marked in red)
Environment
Robosuite
MiniGnd
Habitat
RIDE
0.32
0.93
0.19
RIDE wio EP
0.05 1
0.00
0.00
NovelD
0.55
0.93
0.17
NovelD wo EP
0.11 1
0.00
0.00
LIBERTY
0.73
0.13
"The detailed discussion of related work can be found in Appendix
3

---

(sp( will always be and the episodic bonus is no longer meaningful, which is the case for many real-world applications. For example, household robot's state as recorded by its camera might include moving trees outside the window, clocks showing the time, or images on television screen which are not relevant to its tasks, but nevertheless make each state unique. As shown in Table 2 without the episodic count term (denoted as "w/o EP"), the performance of RIDE and Noveld exhibits a significant decline in all different types of environments.
Limited Expressiveness of Novelty. As indicated in Table the performance of RIDE and NovelD significantly degrades in environments with high-dimensional visual observations or continuous control tasks, where the success rate falls below 0.5. The significant decline highlights the inadequa- cies of utilizing state differences within vector spaces, computed using the Lp norm, to effectively express novelty. Therefore, more robust and effective metric is essential to accurately assess and characterize the novelty among visited states.
3.2 State Discrepancy within the Bisimulation Metric Space
The bisimulation metric 25 26 defines a pseudometric d S S -> IR to measure the similarity between two states. The recently proposed variant, x-bisimulation metric [I3] focuses on behaviors relative to particular policy which consists of a reward difference term and a Wasserstein distance in dynamics models between states.
Definition 1 (n-bisimulation metric). Given fixed policy æ, the following r-bisimulation metric exists and is unique:
oj dss, 85) Barra Baynan 12/ + (d) 8t), PT (T
(2)
where Ea & Wasserstein distance.
Eag ((s PT), s;) = Ea P(: 8;, a) and W1 is the 1-
proposes exploration bonuses based on state discrepancy within the inverse dynamic bisimulation metric space, which adds the difference between action output by the inverse dynamic model based on the bisimulation metric. The difference between states evaluated under the bisimulation metric directly associated with the value difference (see Theorem 3,3 in (26) and Theorem 2 in (63) for detailed proof). The loss function of LIBERTY metric learning is:
Approximation Gap. LIBERTY
63
85: - Ir: - r2 WPP P. (n)) - - +1 (0) 12, (0)) 2 ( = E (dmmo
where ă denotes state with stop gradients, P (1)) indicates probabilistic dynamics model parameter- ized with 17 and 1 (0) is the inverse dynamic model parameterized with 0. As the I-Wasserstein distance usually difficult estimate, LIBERTY and prior methods [68] propose to use 2-Wasserstein distance W2 to replace W1, as W2 has convenient closed-form of a Gaussian distribution with respect to the L2 distance.
Proposition (Relaxation Divergence). Relaxing the W1 metric to W2 breaks the theoretical integrity of the inverse dynamic bisimulation metric [63] scenarios where the transition dynamic model P(s, a) or the policy π is stochastic.
Proof in Appendix B Based on Proposition I we can see that the relaxation divergence of 1- Wasserstein distance between dynamic models can only be ignored when both the transition dynamics model and policy æ deterministic. However, this assumption may be too strong hold in practice.
Besides, the metric learning process encounters expectations. Specifically, computing the expected reward differences (first term of Equation (2). IEn,=ar:: - EBy=<rr;11 proves to be computationally daunting and challenging to accurately esti- mate, even with sampling techniques. Following other bisimulation metric-based methods [13 68) LIBERTY [ shifts the expectation operator outside the absolute value of reward differences (3) and differences between action outputs predicted by inverse dynamic models, as in Ea nvx [Ta - | - -
gap due to the relaxation of reward
(second term of Equation
+++1 []. which facilitates more efficient sampling by avoiding the direct estimation of reward and action expectations.
Proposition 2 (Shifted LIBERTY Distance). The relaxation of expectation during learning process shifts the original LIBERTY distance and introduces looser value difference bound.

---

We extend the concept of the shifted MICo distance [4] introduced in B5 16] appendix B Based on Proposition [ as the shifted LIBERTY distance has looser value difference bound, it may be less relevant to the value function. As result, the learned metric may not be able to capture the state similarity within bisimulation metric space accurately. as the shifted LIBERTY distance has looser value difference bound, it may be less relevant to the value function. As result, the learned metric may not be able to capture the state similarity within bisimulation metric space accurately.
The proof is in
Scalability Constraints. As shown in Table 2 LIBERTY's performance significantly drops in hard exploration tasks within MiniGrid and in the realistic scenarios of the Habitat environment (60). Specifically, in Habitat, agents must navigate through photorealistic simulations of actual indoor spaces. In these settings, the subtle variations between states diminish the impact of the exploration bonus, significantly lowering the efficiency of exploration. This limitation becomes even more apparent in the procedurally-genenaan further impedes LIBERTY's performance. Those findings highlight the scalability challenges that bisimulation metric-based methods encounter in complex environments.
MiniGrid environment, where the inherent stochasticity
4 Effective Metric-based Exploration Bonus
In this section, we introduce the Effective Metric-based Exploration- (EME), conceived to overcome the limitations identified in existing methods that use metric-based state discrepancy as exploration bonuses. Our development is driven by two primary objectives. Firstly, we aim to refine the metric learning process to achieve a more effective exploration without incurring the approximation gap. thereby upholding the theoretical integrity of value function bound. Secondly, seek to obviate the reliance on scaling factor of episodic counts, ensuring effective exploration across different environments. To this end, we propose a novel metric that more precisely evaluates the behavioral similarity between states, establishing a more effective exploration strategy compared with previous approaches. Furthermore, between outputs of an ensemble of reward models in metric learning as dynamically-adjuste scaling factor, which significantly enhances exploration efficiency, especially hard exploration tasks, thereby improving scalability, The next two sections describe our method in detail.
introduce innovative approach that utilizes the variance
4.1
The EME Metric
In order avoid the approximation gap introduced by relaxation of Wasserstein distance and reward expectations mentioned in Proposition I and [ along with the reliance on the episodic count, we climinate the calculation of the Wasserstein distance, and propose the EME metric:
Definition 2 (EME Distance Function). Let met be the space of bounded pseudo-metrics
state space S. the EME metric dg :S x S + IR, the EME distance function F(dg, т) met - met is defined as:
F(dE,T)((L 8j)
Bageo Th Ba, 29 7Eo, + ( (DLL((((s))))( (4)
where (-L(-Il-) represents the Beulback-Leibles (KL) divergence.
Behavioral Similarity Between States. We strictly calculate the expectation of reward difference without relaxation. The EME metric measures the distribution distance between dynamics models by computing the distance between sampled subsequent states following representation learning method [14] to avoid the computation of the Wasserstein distance. Additionally, we integrate the Kull- back-Leibler (KL) divergence between policy distributions to more robustly model the "behavioral similarity" between states, which more effective and scalable across diverse environments with different observations, especially in addressing the critical "Noisy-TV ration. The "Noisy-TV" problem started as a thought experiment in [I] and is commonly discussed in exploration literature [51 52, 42). Imagine that an RL agent is rewarded with seeking the novel ex- perience, TV with unpredictable random noise outputs would able to attract the agent's is attention forever. The agent obtains new rewards from state discrepancy caused by noisy TV consistently, but fails to make any meaningful action. To counter this, we utilize differences policy distributions across states, promoting the exploration of diverse actions rather than repetitive behavior.
problem [II] during explo-
5

---

Theorem 1. The EME distance function F(dg, met - met has a unique fixed-poine dg.
Proof in Appendix B Theorem provides a convergence guarantee for the EME distance that by iterating F(dg, T distance will converge to the fixed-point dg.
Theorem 2. (Guaranteed Value difference bound) Given the EME metric dg, states s; and sy and policy π, we have
|V (st) - V (s,) S dg (8,,
(5)
Improved Exploration Efficiency. The proof can be found in Appendix B Theorem 2 demonstrates that the EME distance between states provides an upper bound on the difference in their state values. Let's recap the definition of one-step TD error: & r++1 (8++1) Vss.). Intuitively, if there a large value difference between states 8, and 8t+1. the corresponding exploration bonus b which is calculated using (st, S(+1) will also be large. Consequently, the total reward for the next step, T++1 r + (where r represents the external environment reward), increases accordingly. The increase of reward leads larger TD error when there is significant value difference between adjacent states, incentivizing the agent to prioritize transitions with large TD errors, which not only enhances the agent's exploration capabilities but also significantly boosts training efficiency.
4.2 Tractable Optimization of EME
Based on Equation (4). between any pair of states s; and sj. we can define the loss function of the EME metric learning as:
( (15(()1))) |Rasar
[()) =
E Eeysa
1 ray - y Bear (6 8/) - 7DKI (=((1)))]]
where de the EME metric encoder parameterized by Φ. The last three terms in the loss function (6) can be regarded as the regression of target metric, which approximates the difference between rewards, distances of next states and distances of policy distributions, respectively. However, the reward expectations Ex,~=r%; and Eaywrr? are computationally intractable and also difficult to estimate even based on sampling 13
And the relaxation of reward expectations Ea,~</r9; rsj
ayn0t
will lead to the learned metric having looser value difference bound than the original bisimulation metric as proved Proposition 2 Inspired by metric-based representation learning methods [35 [6], we propose to use an ensemble of reward models to approximate the reward difference more accurately.
used by previous methods 63 68 13
Proposition 3. Let r. be a random variable over the action distribution defined by p(r, = r = (a]s). var(r,,) denote the variance of variable T'a can have:
Eaywr pi Ba Bajaar 10g
Eajo ajax
|ra:
r 2 []
var(r.:) var(rs))
Proof. (Sketch) The proof is based on 14
16 by expanding Baper
[|:2 [2] [2]
Ba
aymas
Baysn r See detailed proof in Appendix B
Ensemble of Reward Models. We train an ensemble of reward models (((m))....,(n))))) to predict the reward from the sate-action pairs sampling from the buffer D,: g(s, S x A + R by minimizing the prediction error [|g[ ay: 77) re-1l12. Consequently, the variance of reward across the output of different models in the ensemble is defined as follows:
var(rs.) ~ ((r..)) Eg {E.,, []lg(s, a., n) Eg ar n (a,,as)>DD,
(8)

---

Based on Equation 6 Equation 7 and Equation 8]. the tractable EME loss without approximation gap can be defined as:
[()) ED, (s; Ir rs,1 ((r2)) ((r2)) -- -Baa (s (5)
as
(x((le)) T- (-
log x (-1s;)11}
Eiversity-Enhance Scaling Factor. The scaling factor of previous methods is either not scalable [5]
69 [2] or hand-crafter typer-parameter [63]. in light of this, we propose a scalable diversity-enhaneed
scaling factor for the intrinsic reward further improve the efficacy of exploration. Let's recap the ensemble of reward models (g(m)..... g(m)) used in metric training (9). To maintain the diversity across the individual models g(n),, we initialize each model's parameters differently and train each of them on subset of data randomly sampled with replacement. Each model in our ensemble is trained
to predict the ground truth reward. Hence, the parts of reward obtained within the state space that have been well explored by the agent will have gathered enough data to train all models, resulting low variance of reward predictions from the models, when generalizing to unseen similar parts of the reward from unvisited state-space, the areas which are novel and unexplored would still have high prediction error for all models as none of them are yet trained on such examples, resulting in higher variance of the reward prediction. Therefore, as the scaling factor intrinsic reward to encourage exploration on unvisited state space. With the maximum reward scaling M, the exploration bonus is defined
the variance of reward predictions
bo+1 = dg(8t; 8t+1) min (maz (mmo((((re))))) 13,
(10)
As a result, the agent receives greater rewards when encountering novel states during training. which facilitates more effective exploration, the ablation study scaling factor and visualization of exploration bonus Appendix C.5
See Algorithm of Appendix [D2 for detailed description.
5 Experiments
The overall objective of our experiment is to evaluate the performance of EME in comparison to other baselines, we conduct comprehensive experiments various settings of continuous control tasks, discrete-action hard exploration games, and real indoor environment to assess the effectiveness and scalability of our algorithm. The implementation details be found in Appendix D
Baselines. We compare it against following competitive baseline methods. ICM 47 a famous curiosity-driven method. RND [T]: intrinsic rewards are the prediction errors of the distillation net- work. E3B [32] proposing count-based episodic bonuses under continuous state spaces. RIDE [5]]: using state difference under latent space as exploration bonuses. NovelD 69 using the state dif- ference measured by RND bonus as intrinsic rewards. LIBERTY 63] using the state discrepancy evaluated under the bisimulation metric space shaping reward.
Door Opening
Tahle Wiping
Pick- and and--lacee
LIBERTTY
Tpechs
Figure 2: Results for various exploration tasks from Robosuite. The x- x-axis represents the number epochs training. The y-axis represents the mean success rate(standard deviations in shade).
5.1 Continuous Control
In our continuous control experiments conducted on the Robosuite platform 70 we assess the exploration capabilities of agents across various challenging tasks, which include Door Opening,
Details of the environments are provided in Appendix D.I and additional experiments are described in
Appendix C
7

---

Table Average testing results of different Minigrid environments for EME and other baselines.
MRN7S8
MRN7S8-NT
KCS4R3
KCS4R3-NT
OM2DIh
OM2D1h-NT
ICM
0.00 1 0.0
0.00 1 0.0
0.00 1 0.0
0.00 0.0
0.00 1 0.0
0.00 + 0.0
RND
0.00 + 0.0
0.00
0.0
0.00
4 0.0
0.00 +
0.95
0.001
0.00
RIDE
0.67
0.001
0.65
0.001
0.93
0.002
0.81 1 0.001
0.95
0.013
0.89 1 0.015
NovelD
0.67 0.0001
0.65
0.001
0.93
0.003
0.00 0.0
0.96
1 0.011
0.91 I 0.008
E3B
0.61
0.003
0.50 1 0.003
0.91
0.007
0.00 0.0
0.91
0.012
0.57 1 0.011
LIBERTY
0.12
0.003
0.00 + 0.0
0.00
0.00
0.00
0.0
0.93
0.007
0.51 + 0.005
EME
0.67 0.011
0.65 + 0.003
0.91 0.011
0.91 + 0.003
0.95
0.028
0.95 + 0.011
Table Wiping, and Pick-and- The detailed deseription of environments can be found in A.pendix(.I.1 Each task represents a demanding context for robotic control, characterized by sparse rewards and significant exploratory challenges. The overall results are depicted in Figure 2 where our method consistently outperforms others, demonstrating its superiority in handling continuous control tasks. LIBERTY achieves the second-best performance in the Door Opening and Table Wiping tasks, underscoring the advantages of state discrepancy-bast based novelty within the bisimulation metric space. Conversely, episodic count-based methods such as RIDE, NovelD, and E3B lag behind, as episodic counts become less effective in environments with high-dimensional states. Furthermore, curiosity-driven approaches like ICM and RND struggle due to insufficient exploration. We also provide additional ablation studies the ensemble size and max reward scaling in Appendix C
EME Combined with Feature Encoder. The EME metric can be integrated with any state repre- sentation derived from various encoders ( expressed as dg (#))), , & (s++1)). The encoder may include the inverse dynamic model 47 51 63 which isolates environmental factors that do not
affect the agent's 's behavior, the bisimulation- based encoder 68 35 14 [3] and random embeddings. The results can be found in Figure 8 in Appendix C.2 the performance of EME under the inverse dynamic encoder and bisimulation-bant under these encoders get close to zero quickly since their representations are very compact and every state looks more similar with the convergence of the encoder, which harms the efficacy of exploration. With respect to random embedding, the variance is larger, resulting in more unstable performance.
encoder exhibits a decline, because the exploration bonus
LME
KM
BND
REDE
XEEX vita US61
NewdDD
2:11
LIBRITY
LIBERIT with 2017
Aliei
Beeshon
BeanRale
MuPacanan
Scor
PybaaeSye
Sasoeliv deers
UpNDoow
Francis
Figure 3: Comparison results for different hard exploration Atari games.
5.2 Discrete-action Games
Atari Games. To assess EME in scenarios involving pixel-based observations and discrete actions, we evaluate our method on the hard exploration games identified by 66 69 63 within the common Atari benchmark [ including Asterix. Alien, Breakout, BeamRider, Mspacman, PrivateEye, Pong, SpaceInvaders, Qbert, and UpNDown. The results are presented in Figure 3 Our method consistently achieves similar or higher average returns compared to other baselines across all tested tasks. Notably, EME demonstrates the fastest convergence, particularly in Asterix and UpNDown, highlighting our method's 's superior learning efficiency. While LIBERTY and NovelD exhibit competitive performance in 6 games, their effectiveness significantly diminishes in the challenging games, PrivateFye and UpNDown. As for other methods like RIDE, E3B, ICM, and RND struggle across all tasks, primarily
8

---

due to inadequate exploration capabilities. To isolate the impact of our proposed EME metric, we also include the performance of RIDE and LIBERTY with iversity-enhaneee Scaling Factor, denoted as RIDE and LIBERTY with DSF EME still achieves the best performance which demostrate the superiority of EME metric.
MiniGrid Environments. MiniGrid 17 presents series of ceneeduurall-ggnerattee challenging environments. We focus on three settings from MiniGrid: Multi-Roon (MR), Key Corridor (KC), and Obstructed Maze (OM). For example, MRN7S6-NT stands for MoA-Rooo-N7----ooo TV. Details on the specific environmental settings are available in Appendix D.1.3 Table presents the testing performance of EME and all baseline methods across five different seeds. EME successfully solves all exploration tasks within MiniGrid, achieving the best performance in 4 out of 6 settings. While Multi-Room Room environments are relatively easy, all baselines except for ICM and RND, demonstrate competency. In the challenging Obstructed Maze environments, where obstructions block doors, our agent also excels by learning to remove these obstructions, further demonstrating the effectiveness and scalability of our approach.
Noisy TV Problem. In addition to standard MiniGrid tasks, we also tested the model's ability to deal with stochasticity in the environment by adding manually- -mde Noisy TV setting introduced in [51, where some blocks change color at every time step. As illustrated in Table 3 EME maintains strong performance even under these conditions, underscoring the robustness of our method.
CloseCab
OpenPridge
Rearragg
100
Mesn Saccisa Ranc())
80
60
Figure 4: Results of exploration tasks on Habitat, Error bars represent std, deviations over 5 seeds.
5.3 Real-life Habitat Environment
To test our model's applicability to real-world environments, we investigate hard exploration tasks in Habitat [60]. Habitat is platform for embodied AI research which provides an interface for agents to navigate and act in photorealistic (Figure 5 simulations of real indoor environments. Full details
the environmental setting be found in Appendix D.1. We evaluate our methods on three embodied Al tasks of Habitat benchmark. As shown in Figure EME consistently outperforms all baselines, demonstrating its superior scalability to high-dimensional visual- based observations and confirming its broad applicability. The performances of count-based methods like E3B, NovelD and RIDE are comparable, whereas curiosity-driven methods such as ICM and RND lag behind. The underperfor- mance is primarily due to their inadequate exploration capabilities, which are particularly challenged by the complex and rich visual observations inherent in real-world settings.
Figure 5: Habitat
Reward-free Exploration. we investigate reward-free exploration by evaluating the performance of all baselines using exploration bonus-only. Jiisplays the trajectories for all baselines on one of the test maps, clearly demonstrating that EME explores significantly larger portion of the space compared to other methods. The full experiment results can be found in Appendix C.3
Conclusion
In this work, we identify the limitations of existing state discrepancy- based exploration methods: the reliance on episodic count-based scaling factor and theory-practice approximation gap, which leads to limited scalability especially for hard exploration tasks within realistic environments. To address the issues, we propose the Effective Metric-based Exploration- bonus (EMF), which addresses the inherent limitations by proposing robust metric for state discrepancy evaluation backed by

---

comprehensive theoretical analysis. Furthermore, we propose the enhanced scaling factor integrated into the exploration bonus to enhance exploration effectiveness in particularly challenging scenarios. Extensive experiments on hard exploration tasks from continuous control, discrete-action games and realistic environments have demonstrated the effectiveness and scalability of our method.
7 Acknowledgements
This work was supported by National Science and Technology Major Project (20232D0121401), the Science and Technology Development Fund Macau SAR (0003/2023/RIC, 0052/2023/RIAI, 0031/2022/A, 001/2024/SKL for SKL-IOTSC), the Research Grant of University Macau (MYRG2022-00252-F5) Shenzhen-Hong Kong-Macau Science and Technology Program Category CSSGDX2023082 1095 59012), and Wuyi University Hong Kong and Macau joint Research Fund (202IWGALH14) This work was performed in part SICC which is supported by SKL-IOTSC, University of Macau.
References
[1] Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Joshua Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39:20 2018.
[2] Adrià Puigdomênech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. International Conference on Learning Representations, 2019.
[3] Babak Badnava, Mona Esmaeili, Nasser Mozayani, and Payman Zarkesh- Ha. new potential- based reward shaping for reinforcement learning agent. In 2023 IEEE 13th Anmual Computing and Communication Workshop and Conference (CCWC), pages 01-06. IEEE, 2023.
[4] Chenjia Bai, Peng Liu, Kaiyu Liu, Lingxiao Wang, Yingnan Zhao, Lei Han, and Zhaoran Wang. Variational dynamic for self- supervueer exploration in deep reinforcement learning. IEEE Transuctions neural networks and learning systems, 3478))4777-4799, 2021.
[5] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[6] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[7] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[8] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[9] Bernadette Bucher, Anton Arapin, Ramanan Sekar, Mare Badger, Feifei Duan, Oleh Rybkin, and Kostas Daniilidis. Perception-driven curiosity with bayesian surprise. 2019.
[10] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of curiosity-driven learning. arXiv preprint S.Ki::8888.03355 2018.
[11] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
10

---

[12] Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprini arXiv:2006. /2/22, 2020.
[13] Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10069-10076 2020.
[14] Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-base state similarity for markov decision processes. Advances in Neural Information Processing Systems, 34:301 13-30126, 2021.
[15] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[16] Jianda Chen and Sinno Pan. Learning representations via a robust behavioral metric for deep reinforcement learning. Advances in Neural Information Processing Systems, 35:36654-36666, 2022.
[17] Maxime Chevalier-ooisvert,, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306. 13831, 2023.
[18] C Colas, Karch, O Sigaud, and PY Oudeyer. Intrinsically motivated goal-conditioned reinforcement learning: A short survey. arxiv 2020. arXiv preprint arXiv:20/2.098.00
[19] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049-13061 2020.
[20] Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Proceedings of the I Ith international conference on autonomous agents and multiagent systems, pages 433-440. IFAAMAS, 2012.
[21] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning. pages 1329-1322 1338. PMLR, 2016.
[22] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv: (eXXiv::91..ii 1099, 2019.
[23] Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. Advances in neural information processing systems, 32, 2019.
[24] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes, In UAI, volume 4, pages 162-169, 2004.
[25] Norman Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. arXiv preprint T:Xivv/207.4111 2012.
[26] Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In UAI, pages 210-219, 2014.
[27] Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In UAI, pages 210-219, 2014.
[28] Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adver- sarially guided actor-critic. arXiv preprint arXiv:2 04376, 2021.
[29] Yang Gao and Francesca Toni. Potential based reward shaping for hierarchical reinforcement learning. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
11

---

[30] Marek Grzes and Daniel Kudenko. Learning potential for reward shaping in reinforcement learning with tile coding. In Proceedings AAMAS 2008 Workshop on Adaptive and Learning Agents and Multi-Agent Systems (ALAMAS-ALAg 2008), pages 17-23, 2008
[31] Anna Harutyunyan, Sam Devlin, Peter Vrancx, and Ann Nowé. Expressing arbitrary reward functions as potential-based advice. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[32] Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631-37646, 2022.
[33] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[34] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay, In Interna- tional Conference on Machine Learning, pages 4940-4950. PMLR, 2021.
[35] Mete Kemertas and Tristan Aommenttado-Amssroong Towards robust bisimulation metric learning. Advances in Neural Information Processing Systems, 34:4764 2021.
[36] Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. arXiv preprint arXiv:1810.01176. 2018.
[37] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32:1233 1274, 2013.
[38] Weichen Li, Rati Devidze, and Sophie Fellenz. Potential reward shaping for learning to play text-based adventure games. arXiv preprint arXiv:2302.107220 2023.
[39] Zhixuan Lin, Pierluca 1)' D'Oro, Evgenii Nikishin, and Aaron Courville. The curse of diversity in ensemble-based exploration. arXiv preprint arXiv:2405.04342, 2024.
[40] Marlos C Machado, Mare G Bellemare, and Michael Bowling, Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1133, 2020.
[41] Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based exploration in feature space for reinforcement learning. arXiv preprint orXiv:/706..000 2017.
[42] Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-drive exploration via latent bayesian surprise. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 7752-7760, 2022.
[43] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Ku- maran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529 533, 2015.
[44] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. Ieml, volume 99, pages 278-287. Morgan Kaufmann, 1999.
[45] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
[46] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count- exploration with neural density models. International conference on machine learning, pages 2721-2730 PMLR, 2017.
[47] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity -rivvnn exploration by sulf-suuee prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017.
12

---

[48] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International conference on machine learning. pages 5062-5071. PMLR, 2019.
[49] Rémy Portelas, Cédric Colas, Katja Hofmann, and Fierre-Yve Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. Conference on Robot Learning, pages 335-853. PMLR, 2020.
[50] Sebastien Racaniere, Andrew K Lampinen. Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint arXiv: /909. 2289, 2019.
[51] Roberta Raileanu and Tim Rocktäschel. RIDE: rewarding impact-driven exploration for
procedurally generrated environments. In 8th International Conference on Learning Representa- lions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[52] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys. Tim- othy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint 1.10.00110.2211 2018.
[53] Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. 1991.
[54] Younggyo Sco, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee, State en- tropy maximization with random encoders for efficient exploration. International Conference on Machine Learning, pages 9443-9444 PMLR, 2021.
[55] Christopher Stanton and Jeff Clune. Curiosity search: producing generalists by encouraging individuals to continually explore and acquire skills throughout their lifetime. PloS one, (0)):001 2016.
[56] Christopher Stanton and Jeff Clune. Deep curiosity search: Intra-life exploration im- proves performance on challenging deep reinforcement learning problems. arXiv preprint arXiv:1806.00553, 2018.
[57] Alexander Strehl and Michael I. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, (M89:1309-1331 2008.
[58) Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric eelf-play arXiv preprint arXiv:1703.05407 2017.
[59] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[60] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turnrr, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[61] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[62] Ruo Yu Tao, Vincent François-Lavet, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances in Neural Information Processing Systems, 33:81 8126,, 2020.
[63] Yiming Wang, Ming Yang, Renzhi Dong, Binbin Sun, Furui Liu, et al. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[64] Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforcement learning agents. Proceedings of the 20th international conference on machine learning (ICMI-03), pages 792-799, 2003.
[65] Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin. Piyush Khandelwal. Varun Kompella, HaoChih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi. Leon Barrett, Rory Douglas, Dion Whitehead, Peter Düm, Peter Stone, Michael Spranger, and Hiroaki Kitano. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602:223 228, 2022.
[66] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
[67] Daochen Zha, Wenye Ma, Lei Yuan, Xia Hu, and Liu. Rank the episodes: simple approach for exploration in procedurally-generaaed environments. arXiv preprint arXiv:2101.08/1 2021.
[68] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint 107442 2020.
[69] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217-25230, 2021.
[70] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín Martin-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009, /2299, 2020.
14

---

A Broader Impact Statement
This work introduces a reinforcement learning (RL) exploration method by introducing an effective metric-based exploration bonus, versatile framework applicable to a wide array of decision decision-making problems, which can be applicable to diverse scenarios, including autonomous driving, household robotics, video gaming, online recommendation or advertisement optimization and etc. Similar to other RL exploration algorithms, our method designed to facilitate the learning that maximizes reward function defined by the designer. The implementation of such policy, depending on the objectives set by the reward function designer, could lead to either positive or negative outcomes. Additionally, researchers and algorithm engineers, and concurrently offers insights into the quality of the designed rewards through the shaping weights derived from our methods.
policy
approach significantly reduces the burden of reward tuning for
B Proofs
Proposition (Relaxation Divergence). Relaxing the W1 metric to W2 breaks the theoretical integrity of the inverse dynamic bisimulation metric [63] scenarios where the transition dynamic model a(s,a) the policy π is stochastic.
Proof. First, let's recall the proof of the existence of unique fixed-poing in the inverse dynamic bisimulation metric:
Lemma ((63) ). Given a fixed policy 7. Define H* mmt -> met by (8i, Ir re
( &, P..) + 11 Si, 8141) (- |8, 85+1) then 7 has least fixed point d=.
Proof. This proof mimics the proof of Theorem 4.5 from (24]. We make use of the same pointwise ordering on met: d S d' iff d(ss 1) S d'(s, 1) for all 8, E S, which gives us an w-cpo with bottom metric. Since Lemma 4.4 from (24] (Wasserstein metric W is
I. which is the everywhere-zew continuous) also applics in our definition, only remains to show that H(d, f) is continuous:
H*
L [Un} (si, = -Irs. -r,, -r,,1+710 W nEN
[ (amb) (L
(P (PA,, P.t))
nEN
+ 111 18, 11188111 - 1a( 88,88422ll1 ss, -r,
Ira,
+ sys W (In) (P P.,))
neW
+ |I (11 8141) (. 88, 85ا8111
by continuity of W
(11)
sup((r
neN rag + W (In) (P Paj))
+ 11 8i, 91+1) (- (1,11+1)
= sup {H (xr) 8;}} wEN
LJ {7L (an)) (s), NEE
The rest of the proof follows in the same way as in 24
In equation [1] the existence of a unique fixed-point in the bisimulation metric requires the continuity and monotonicity of Wi with respect to The properties of continuity and monotonicity do not hold with Wz. Therefore there is no more guarantee about the fixed-point existence in LIBERTY and other approximation-be
methods 688 except that both the dynamics model and the policy π are deterministic, in which case (zdd) degenerates to d (24) and Banach's 's fixed-point exists K5 As for scenarios when the transition model and policy are stochastic, the theoretical guarantee LIBERTY is broken.
15

---

Proposition 2 (Shifted LIBERTY Distance). The relaxation of expectation during learning process shifts the original LIBERTY distance and introduces looser value difference bound.
Proof. The proof is based on top of
14 16
Lemma 2.
Ea.- (1r r23 + 11
12 [] 2 Ba BBa
2/ + +Baann 18111 Bajax
8j
(12)
Proof. Since rā; and raj are rewards which are scalars, we have Ir r331 2 r2
rag and
|r9 r2. 2 Tsg r.. By taking the expectation over ag and aj. we can have
2 Bas (r;; -13]]
(13)
and
Bajona 12
Bap-na (r rg]]
(14)
By combining Equation
(3)
and Equation (14. we can have:
E Be (1
r0y] | Boynat
(15)
Similarly, we have:
Ea Ea 11 SL41
15)
2 Ear 11 1
Ba
144
Ea nox 12
(16)
So we can get
Eg Be nox [1a r2)1 + 11
11 |] |Bewrrs Ea
[laas 18tt Ba, nex 15) 11+1
(17)
Definition (Shifted LIBERTY Distance).
The shifted LIBERTY distance function Ft" is defined as r2; > Wi (P,,P,,
H. (d)(ss,, E Eagoo Ir:
- -11 -
(18)
a
Define the MDP for RL by a tuple AS R, We consider lifted MDP constructed by tuple
(S, A, R. P,7), 7), where state space S = S x S, action space Á A A, transition distribution playas) Pos P and reward function R((s;, 8;)) = EarN Eujorryy). The Bellman
(s),8))
operator 7 under policy I(a; asls), = ((al(s))((alss)) is:
T"d (nd (dT))((()8))))
Σ I(an ((a,,,l1)))))) (ar,ay)
Σ
(311,) (s/1 [R((,,,, 8,)) + + 151
(0/119))
Baiwn - Eaywar
WP PT.)) + 711 1211
HT H'ddd) (dd) 8))).
(19)
So construct the lifted MDP of shift LIBERTY by a tuple (S, R, p) which is the same as the lifted MDP in Definition except for the reward function R((s, 85), (as, a,)) = Ir r3 and shifted metric d. The Bellman operator 7 under policy म is:
Σ #(a,, 8.) (o:.a))
Σ plan.y)) (5))))(())) (s), ((((s, 83)) (o,, + t 1331
(s)10))
H (d ((s,, 8)))
(20)
As proven in LIBERTY
63 for the inverse dynamic bisimulation metric d, we can get
[**(s)) V (s;)) d' (st, 8,)
(21)
16

---

As proved in 63 27 d' is the value function of lifted MDP (S.A.R, P.7). d" can be expanded as the sum of discounted future rewards,
(4)
d= (5: 83) = E Σ 7p (Ba.) 8/0
NT
S1. 8
8j
(22)
dh can be expanded as,
Σn' (E
1000
(si, 83) Ex
E
8,
(23)
with s; 8/. 8) ... The difference between d and d can be regarded as
(0) (0)
(0)
(9)
(1)
ds (5), (s
E
Σ 7t
E
E
19)
(2)
Σ > (a((s!)) .5) (a. 13
E
(24)
As proved Lemma we can get:
dm (s1, d= (81; 20
(25)
Combined with Equation 21 have:
(s:) - VF(s)) S d= (st, 8y) S d= (st,
(26)
So the shifted LIBERTY distance introduces a looser value bound.
Proposition 3. Let rs be a random variable over the action distribution defined by p(rs = r2)
(als), var(r.;) denote the variance of variable Uso can have:
E. Earwar Eaywa
ER
E ay-
|r::
1. var(r..) var(raz)
(27)
Proof. The proof is based on oy and [E (Bagsrr
16 by expanding the difference between
Baywn (ra: rog
g. Ba Baj-rai
(1o: r2j) ra;)
(BaNat E Eay )²
E Ea
Eajnn (1=202] + Buynn
(
27
22 2Ea
(Eagnnrag )²
21 21 a: Bay
(28)
Eajor (1=0)2]] 21 (BaNTayy
E (Baina )a)) 2 + Eagnut
27 (1:))
Ea (arna
2
EaIn Be Bay
var[rec] var(re,)
2cov(r,,,
rs, and rsy independent variables, get cov(rs.; T'y)) 0, can have
E Enpzr p0: - En (
Ener ay
rr - r23 2
var(r..) var(r,))
(29)
17

---

Theorem 1. The EME distance function F(dg, T): met - met has a unique fixed-point dg.
Proof. Let dg d'g E M. We have
[F(dE) (s): - F (d's) (st, 8;]1=
Σ (a: si) v (a, (d) - d') (c))
(30)
ama,
S lldg d'vl100
Therefore, F is contraction mapping w.r.t. the Lxx, norm and there exists unique fixed-point for F due to Banach's ' fixed-point theorem. This completes the proof.
Theorem (Guaranteed Value difference bound) Given the EME metric dg, states s; and s and a policy have
|V (s:) - V (s,)) S dr (s;,
(31)
Proof. The proof mimics [1]. We follow the assumption that E. PP(s') VT [8]]. We will first show that if Vs, 8; E S, (s.) VT(s;)] I d(s,,,5) then (V, VT
F(dg, (8). s,). Suppose |V (s:) - (=(ss) & d(s;. 83) holds, we can have:
(*(()) (8))
= Eu Ba 12 + Σ T(as(ss))
Σ
pai (s))
Σn(ayls) Σ
Pay (s))
of
< Eo Baywart aa
+ Bana
Σ
pai (+(()((((((()))) - -aaya
Pay (8/))*
< E, Nx Ey Euynar nag
Bexr (VT (Eogegy V (E (Eagupiy (8$1))
E Earwat pi -B Baynart 10g
Earsd d B Begwres (84), E Beg (65))
Ea - Eaynot 10)
yBa (5/, 8/) (.) flog T( (-18;) log (-1ss))
S F(d, ()),,
(32)
Similarly, (T(s)) (sis)) S F(d, T
we can have
(v'(s) V= S F(d,
(33)
Assuming that we have an initial distance do which (T((s; (=())) S do(s;,))) and base on Theorem [ F(d, T) is contraction mapping d. By repeatedly applying F(d, TT) on d, d will
eventually converge to the fixed -eintt dg, the fixed point de satisfies:
[V*(s; - V"((s)] S dg(s,,8))
(34)
C Additional Experiments
C.1 Additional Ablation Experiments
To isolate the impact of different hyper-parameters, we carry additional experiments on the ensemble size of reward models and the max reward scaling M. First, M sets an upper limit on the bonus. We set M 10 as the default setting. As we can see from Figure if M the scaling factor is fixed to 1, resulting in significant performance decline. Higher M encourages more extensive exploration. The performance with M slightly lags behind the default setting. The performance with M 20 and M = 40 is comparable and almost the same, indicating that the performance stabilizes as M increases. Practically, the value of M can be adjusted depending on the specific task and environment to determine the intensity of exploration.
18

---

Door Opening
Table Wiping
Pick-and-Piase
EME 08--08-30
EME-06-40
EME Swa- (MC-1
Epocts
Figure 6: Ablation study on the max reward scaling M (M = 1,5, 10, 20, 40).
Doer Opening
Table Wiping
Pick and Place
Tpeck
Figure 7: Ablation study the ensemble size (ES = 6, 9, 12) and the other baseline with best performance.
Door Opening
Table Wipins
Pick-and. Place
ENE Rundess
Spodts
Figure 8: Results for EME and its variants combined with different feature encoders. The x-axis represents the number of epochs in training. The y-axis represents the mean success rate(standard deviations in shade).
As shown in Figure 7 we observe that as the ensemble size (ES) increases, the performance of ES 6 and ES 9 surpasses that of ES 3. The performance of ES = 6 and ES = 9 is comparable, with no significant difference. However, when the size is further increased to ES- = 12, there is decline in performance, particularly in the Table Wiping and Pick and Place tasks. The performance change can be analyzed from perspectives. First, the number of reward models is related to the accuracy of the reward variance prediction used to calculate the loss function. As the ensemble size increases, the overall prediction error decreases because the models can average out individual errors more effectively. This leads to a more accurate approximation of the variance ((*)). Second, with a larger ensemble size, the variance of reward predictions decreases, resulting in a lower scale of the exploration bonus. Thus, there is a balance or trade-off between estimation accuracy and exploration, explaining why the performance of ES 12 lags behind ES 9. The optimal number of ensemble models may vary depending on the specific tasks and environments. Regarding computational cost, increases with the ensemble size. Therefore, we ES 6 as our default setting, where the performance is nearly the same as ES = 9 but with lower computational overhead. It is also noteworthy that even with an ensemble size of 3, EME still outperforms the best baseline methods, further demonstrating the robustness of EME.
C.2 EME Combined with Feature Encoder
Our method EME can be integrated with any state representation derived from different state encoders + (.), expressed as de (st). (($++1)). The encoder may include the inverse dynamic model 47] 31
63 which isolates environmental factors that do not affect the agent's 's behavior, the bisimulation- based encoder [68, 35| 14 13 which learns a compact state representation grouped by the bisimulation
metric, and random embeddings captured by random embedded states. We denote the EME variants as follows: EME with an inverse dynamic model-based encoder is referred to as EME-IDM; with bisimulation metric-based encoder as EME-BM; and with a random embedding as EME-Random. The results can be found in Figure g EME's performance declines due to the rapid reduction of the
19

---

exploration bonus. The performance of EME under the inverse dynamic encoder and bisimulation- based encoder exhibits decline, because the exploration bonus under these encoders gets close to zero quickly since their representations are very compact and every state looks more similar with the convergence of the encoder, which harms the efficacy of exploration. With respect to random embedding, the variance is larger, resulting in more unstable performance.
T
(a) EME
(b) EME-Static
(c) EME-EP
(d) ICM
(c) RND
(f) RIDE
g
NovelD
(h) E3B
ii LIBERTY
Figure 9: Trajectories of policies trained with different exploration algorithms on the Habitat environment. Our method EME reveals the largest portion of the map than other methods.
CloseCab
OpenPridge
Rearrange
Mcan Stuccas Raud%)
RND
RIDE:
NexdD
LINEXTY
Figure Results reward-ffre free exploration tasks on Habitat, Error hars represent std, deviations over 5 seeds.
C.3 Reward-free Exploration
We examine reward-free exploration by assessing the performance of all baselines within Habitat using only exploration bonuses. During training, agents are initialized in varied environments for each episode and are subsequently tested on a set of environments not encountered during training. Figure 9 displays the trajectories of all baselines on one of the maps, highlighting that EME explores substantial portion of the space, unlike other methods. Specifically, ICM and RND agents remain confined to single room, while RIDE, LIBERTY, and NovelD agents explore two rooms before becoming stuck. distant from the goal. E3B exhibits the second-bes, exploration coverage of the map. Quantitative results from other reward- wwward-free tasks are shown in Figure where EME consistently demonstrates robust performance and achieves the best outcomes. In contrast, other methods experience significant declines, underscoring the efficacy of our approach in a reward-free setting.
20

---

Table 4: Table of quantitative results comparison between EME and other baseline methods in different environments Mujoco with delayed reward setting. The best and the runner-up results are (bold) and (uderline)
Methods
Delay 10
HaltCheetah
Hopper
Walker2d
Ant
Humanoid
Swimmer
ICM
1374 +: 368
1258 + 325
1127 + 225
-105 + 43
162 - 54
27 + 11
RND
1691 + 195
1976 + 458
1405 + 262
143 :
532 + 29
32 15
RIDE
2467 + 456
1876
1651 325
92 =
570 45
65 16
NovelD
1785
649 145
360
-34
258
13 11
E3B
1025 1: 347
1474 129
1997 115
66 - 27
518 = 23
43 17
LIBERTY
2973 + 437
2479 + 315
2766 + 487
292 + 68
681 + 73
73 + 21
EME
2779 + 412
2369 + 314
2785 + 399
539 + 62
599 + 65
77 + 18
Methods
Delay
HalfCheetah
Hopper
Walker2d
Ant
Humanoid
Swimmer
ICM
1185 + 287
1097 + 275
995 + 201
175 + 23
434 + 48
23 + 10
RND
1595 415
1925 401
1379 + 193
127 12
519 +
29 + 12
RIDE
2285 + 402
1621 382
1724 + 307
105 105 27
509
59 13
NovelD
945 1 355
513 1 86
794 320
107 1 29
315 1 55
17
11
E3B
887 242
1015 1 185
743 1.
381 45
321 1 36
20
1 11
LIBERTY
2619 1 354
2112 208
2345 1 414
263
1: 55
617 1 53
67
1
18
EME
2607 276
2467 298
1718
352 35
499
59
Methods
Delay
30
HaltChectah
Hopper
Walker2d
Ant
Humanoid
Swimmer
ICM
1017 1. 276
965 213
199
198 1 25
417 1 45
19
RND
1483 393
1773
1038 191
99 + 13
501 - 27
24 11
RIDE
1973 + 369
1405 +
1345 - 305
87 =1
487 - 25
41 15
NovelD
217
505 148
664 + 129
43
279 - 65
18 10
E3B
1246 101
1007 + 142
221
+ 25
366 +
19 12
LIBERTY
273 +
1873 228
2077 + 398
215 48
587 +
+ 15
EME
2448 263
1999 + 288
1685 + 158
315 + 29
495 + 36
58 + 13
MulliRoom-N7-5
KeyCorridor- -4-R3
ObsiructedMuze--D...
Exploration Bomus
Exploration Bonus
Figure 11: The scale of exploration bonus across all MiniGrid environments
C.4 Delayed Reward Setting of Mujoco
We present results from the delayed reward setting in the MuloCo environment (21), following the methodology outlined (63], where accumulated rewards are delayed every 10. 20, and 30 steps. The experiments assess six tasks: HalfCheetah, Hopper, Walker2d, Ant, Swimmer, and Humanoid. As shown in Table our method achieves the best second-best performance in 15 out of 18 delayed reward tasks. This indicates that EME can facilitate effective exploration and maintain high performance even under sparse reward conditions. Notably, as rewards become sparser, EME's performance improves and becomes more robust. In contrast, other metric- based exploration bonus methods such as RIDE and NovelD struggle due to their less expressive metries. Curiosity-driven methods like ICM and RND also become less effective as rewards grow sparser. These results provide further evidence EME's ability to promote efficient exploration effectively, even in environments characterized by delayed rewards.
21

---

Dowr Opening
Table Wiging
Pick-snddPlaos
Figure 12: Results of EME, EME with static scaling factor, EME with episodic count from Robosuite. The x-axis represents the number epochs in training. The y-axis represents the mean success rate(standard deviations in shade).
C.5 Ablation Study on Scaling Factor
To demonstrate the effectiveness of our proposed scaling factor, conducted ablation study focusing on this component. We compared EME with two variants: 'EME-EP" which incorporates episodic counts, and 'EME-Static," which tasks in the Robosuite and MiniGrid environments are illustrated in Figure [2] Deeserttvssy Without the diversity-enhanced scaling factor. EME shows a noticeable decline in performance. 'EME-EP' performs better in MiniGrid environments compared to continuous tasks in Robosuite, highlighting the efficacy episodic state visitation counts in grid-based games but demonstrating limited effectiveness in environments with -igh-dimensionll states. Additionally, we visualize the exploration bonus of all MiniGrid environments in Figure I1
static scaling factor. The results from continuous
D Implementation Details
D.1 Environments Settings
(a) Table Wiping
(b) Door Opening
(c) Pick and Place
D.1.1 Robosuite Manipulation Tasks
We choose the three manipulation tasks in the Robosuite platform 77
Table Wiping. The environment consists of table with whiteboard surface and some markings placed in front of single robot arm, which has whiteboard eraser mounted its hand. The shape of the dirty region is correlated to the position of the cube. When the dirty region is diagonal, the cube on the right-hand side of the robot arm. The goal of the agent learn wipe the whiteboard surface and clean all of the markings. The whiteboard markings are randomized at the beginning each episode.
Door Opening, The environment consists of door with a handle mounted in free space in front of single robot arm. The height of the handle and the position of the door are correlated. When the door is closed to the robot arm, the handle is in low position. When the door is far from the robot arm,
22

---

the handle is in high position. The goal of the agent is to learn to turn the handle and open the door. The initial state distribution of the door location is randomized at the beginning of each episode.
Pick and Place. The environment consists of four objects placed in a bin in front of single robot arm. There are four containers next to the bin. The goal of the agent is to place each object into its corresponding container. This task also has casier single-object variants. The initial state distribution of object locations is randomized at the beginning of each episode.
D.1.2 Habitat
The Habitat platform relies several key abstractions that model the domain of embodied agents and the tasks they can perform in three-dimensional indoor simulation environments. This platform consists of virtually embodied agent, such as a robot, equipped with a suite of sensors that can observe the environment and take actions to alter the agent's 's state or the environ- ment's state. Each sensor is associated with spe- cific agent and can return observation data from the environment at a specified frequency. The 3D en- vironment includes a scene mesh, objects, agents, and sensors, organized into regions and objects through hierarchical representation called the Scene, which can be programmatically manipu- lated. All components of the Scene are present on the SceneGraph. Additionally, a simulator backend instance can update the state of agents and SceneGraphs based on given actions for a set of configured agents and SceneGraphs, and it provides observations for all active sensors possessed by the agents. We use the Matterport3D (MP3D) dataset [15], which consists of high-quality renderings of indoor scenes. To measure exploration coverage, we compute the area revealed by the agent's line of sight using the function provided by the Habitat codebase (2). As depicted in Figure 13] exploration is measured as the proportion of the environment revealed by the agent's 's line of sight over the course of the episode.
unexplored arca
explored
Figure 13: Explored area Habitat
D.1.3 MiniGrid Environments
grid-worlds. MiniGrid, the world is a par- tially observable grid of size N x N. Each tile in the grid contains exactly zero or one ob- ject. The possible object types are wall, door, key, ball, box, and goal. Each object in Mini- Grid has associated discrete color, which can be one of red, green, blue, purple, yellow, or grey. By default, walls are always grey and goal squares are always green. Rewards are sparse for all MiniGrid environments. There are seven actions in MiniGrid: turn left, turn right, move forward, pick up an object, drop an object, toggle and done. The agent can use the turn left and turn right action to rotate and face one of 4 possible directions (north, south, east, west). The move forward action makes the agent move from its current tile onto the tile in the direction is currently facing, provided there is nothing on that tile, or that the tile contains an open door. The agent open doors if they right in front of by using the toggle action. Observations in MiniGrid are partial and egocentric. By default, the agent sees square of 7 tiles in the direction it is facing. These include the tile the agent is standing on. The agent cannot see through walls closed doors. The observations are provided
MiniGrid [7 is a set of procedurally--geeraaee
tensor of shape 7 x 7 x 3. However, note that these are not RGB images. Each tile is encoded using 3 integer values: one describing the type of object contained in the cell, one describing its color, and a flag indicating whether doors are open or closed. This compact encoding was chosen for space efficiency and to enable faster training. For all tasks, the agent gets an egocentric view of its surroundings, consisting of 3 pixels. A neural network parameterized as a CNN is used to process the visual observation.
Figure 14: Rendering the Mul- tiRoomN12S10 in MiniGrid.
MultiRoom. The MultiRoomNXSY environment consists of X rooms, with size at most Y, connected in random orientations. The agent is placed in the first room and must navigate to green
23

---

goal square in the most distant room from the agent. The agent receives egocentric view of its surroundings, consisting of 3 pixels. The task increases in difficulty with X and Y. Episodes finish with positive reward when the agent reaches the green goal square. Otherwise, episodes are terminated with zero reward after a maximum of x steps.
Key Corridor. the agent has to pick up object which is behind locked door. The key is hidden in another room, and the agent has to explore the environment to find it. Episodes fimish with a positive reward when the agent picks up the ball behind the locked door or after maximum of 270 steps.
ObstructedMaze. The agent has to pick up box which is placed in corner a maze. The doors locked, the keys hidden in boxes and the doors are obstructed by balls. Episodes finish with positive reward when the agent picks up the ball behind the door or after a maximum of 576 steps.
D.2 Algorithm Details
Algorithm 1 Effective Metric-based Exploration- (EME)
1: Initialize parameters of policy T0),, metric dz and reward models (.( {mh))
2: while not converged do
3: for to MAX STEP PER EPISODE do
4: Sample action ag To (a: st)
5:
Step environment Sc+-1 ~ p (8t+1 st. 0z)
6:
Record transition in the buffer: D- +- DD U fst, 8141, rt+1)
7:
Sample mini batch with size B D from the buffer Compute the variance (rr.,) scale factor from the ensemble of reward models Compute bonus dr+1 dr (8t, 8141) min (maa(((rr 13,
8:
9:
10:
Reshape reward with the bonus r'+1 T++1 +
11:
Train policy (0) via policy gradient
12:
Update the metric de: Ep [o)] >Equation
13:
Update reward models by minimizing the prediction error: Ep, [lg(s;; axi - rr-1112
14:
end for
15:
end while
D.3 Computation Details
All of our experiments are conducted on 4 GPUs with 16 CPU threads, which include AMD Ryzen
CPU@ 1.20GHz (16 core) CPU, NVIDIA GeForce GTX 3080Ti GPUs, and 64GB memory.
D.4 RL Hyperparameters
24

---

Table 5: Plyper-parameten Robosuite experiments
Parameters
Environment Door Opening
Pick and Place
Table Wiping
Horizon steps
300
500
500
Control frequency (Hz)
20
20
20
State dimension
92
22
30
Action dimension
7
7
Controller type
OSC position 10
Joint velocity 10
Joint velocity
Maximum reward scaling
20
Actor learning rate
3 10-4
Critic learning rate
10-3
Batch size
128
Discount factor
0.99
Soft update weight
0.005
Alpha learning rate
3 x 10
Hidden layers
[256. 256, 256] 4
Returns estimation step
Buffer size
106
Steps per update
10
Table 6: Common Hyperparameters for MiniGrid
Parameter
Value
Learning Rate
0.0001
RMSProp momentum
0
RMSProp
10-3
Unroll Length
100
Number of buffers
80
Number of learner threads
Number of actor threads
256
Max gradient norm
40
Entropy Cost
0.0005
Baseline Cost
0.5
Discounting Factor
0.99
Maximum reward scaling
5
Table 7: Common Hyperparameters for Habitat
Parameter
Value
Clipping PPO epochs
0.2
2
Number of minibatches
16
Value loss coefficient
0.5
Entropy coefficient
5e-5
Learning rate
7e-5
10-5
E
Max gradient norm
0.8
Rollout steps
128
Use GAE
True
?
0.99
0.95
Use linear clip decay
True
Maximum reward scaling
30
Hidden size
512
25

---

Algorithm
Hyperparameters
Maximum reward scaling: 10
threshold probability ratio clipping (c) : 0.5
update timesteps: 2e10
number of epoches per update: 50
number of minibatches: 16
Base Learner (PPO)
batch size: 2048
GAE parameter (A) : 0.95
optimizer: Adam
learning rate: 5 10-3
policy gradient clip norm: 0.9
discount rate (t) 0.99
ridge regularizer A: 0.1
tanh activation entropy cost: 0.005
E3B
intrinsic reward coefficient: 1.0
gradient clip norm: 1.0
forward model loss coefficient: 0.2
inverse model loss coefficient: 0.08
ICM
entropy cost: 10-4
intrinsic reward coefficient 3: 0.005
proportion of experience used for training predictor: 0.25
predictor Model updates per PPO epoch: 6
RND
entropy cost: 0.005
intrinsic reward coefficient /3: 0.1
forward model loss coefficient: 0.5
inverse model loss coefficient: 0.8
RIDE
entropy cost: 0.0005
intrinsic reward coefficient 8: 0.1
episodic memory capacity: 5000
action prediction network filter sizes: (3,3)
NovelD
entropy cost: 0,005
RND clipping factor L: 5
intrinsic reward coefficient (3: 1.0
intrinsic reward coefficient 3: 0.05
LIBERTY
entropy cost: 0.005
inverse model loss coefficient: 0.5
forward model loss coefficient: 0.5
Table The hyperparameters of the tested benchmark algorithms the Atari experiment
26

---

D.5
Codebases Used
Our codebase was built atop the following codebases:
The official NovelD codebase: https://ithhb..o http://hhhbu..co//aauuuuuuZuuuuu com/tianjnnn//Noell
(Creative Commons
Atrrbtiion-NoCCommrrraal 4.0 license) for NovelLD, RND, RIDE and count-based base-
lines (this codebase is built atop the official RIDE codebase below)
The official RIDE codebase:
(Creative Commons Attrbution-NonCCCmmrriic 4.0 license)
The official LIBERTY codebase: http:////gggggiii..ooooooooolll/ (MIT License)
The official E3B codebase: http::/gghhbboooo//cccccooereass Commons
Attriuuionn Pimbbution-NonCCmmerrillll 4.0 license)
E
Limitation
While our method has shown success in several challenging exploration tasks, it does not consistently achieve top performance in environments with less sparse rewards, as observed the MuJoCo continuous control benchmarks. Additionally, learning domains characterized by large action spaces. Addressing these limitations and developing a more generalized solution for diverse RL. environments remains goal for future work.
method has not been tested in reinforcement
F
Related Work
Exploration in RL. Exploration remains long--tandde standing problem in RL. Common approaches include
e-greedy [59], count-based exploration [7|46]
61
[0] [7], ensemble- based exploration [45] 39 48)
and curiosity-based exploration
[53
55
56
10
Several exploration strategies use dynamics
model to provide intrinsic rewards
2] Latent variable dynamics have also Maximum entropy in the state representation has also been Other intrinsic motivation methods have recently been developed
33
36
been studied for exploration 4
9
62
used for exploration (54] [66].
69 32). which automatically generate curricula over variations of the MDP to encourage efficient learning, effectively performing a form of curiosity- driven exploration in the context space, including goal-conditioned (23 50
for exploration in context MDPs [67
28
51
22 and goal-free variants [58, 49 34 [9]. Reward shaping refers to modifying the original reward function with shaping reward function which incorporates domain knowledge. Considering the most general form, namely the additive form, of reward shaping. The first approach to guarantee policy invariance potential-based reward shaping (PBRS) 44, which defines the shaping reward funetion as the difference between values assessed through the potential function based on prior knowledge. There are lots of variants of PBRS 20 areas within the field of reinforcement learning.
30, 29 which have shifted their focus different
Resed Exploration Bonus. Metric space is widely used in encoding state representations [68 [3] T6, 35). Metric-based exploration bonus is based on the evaluation metric that quantifies the degree of difference or distance between two projected states for the measure novelty. RIDE [51 evaluates the novelty between two successive state representations under the L2 norm. NovelD (69 uses the disparity in RND bonuses between adjacent states under the L, distance as the exploration bonus. LIBERTY (63 uses the difference between potential functions of adjacent states under the bisimulation metric space. Note that we propose an effective metric that measures the distribution distance between dynamics models by computing the distance between sampled subsequent states. Additionally, we integrate the Kullback-eibler (KL) divergence between policy distributions to more robustly model the "behavioral similarity" between states, which more effective and scalable across diverse environments with different observations, especially in addressing the critical Noisy-TV problem during exploration.
27

---

NeurIPS Paper Checklist
1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?[Yes) In the introduction section (page I
2. Limitations Question: Does the paper discuss the limitations of the work performed by the
Yethors?[Yes]] In Appendix [
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper pro- vide the full set of assumptions and a complete (and correct) proof? Yes] In the Appendix [
4. Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?[Yes] In Appendix D
5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main exper- imental results, as described in supplemental material? In the anonymous link http://wwww...oaoooooiiio////aaa
6. Experimental Setting/Details Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? [es]] In Appendix[D.4
7. Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Yes] In experiment section (page 5 and Appendix C
8. Experiments Compute Resources Question:
For each experiment, does the paper provide
sufficient information on the computer resources (type of computing workers, memory, time of execution) needed to reproduce the experiments?!Yes: In Appendix D.3
9. Code Of Ethies Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://wurri..cccppppp/pe
SchicaCuidellies YYe]]
10. Broader Impacts Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?[[es) In Appendix
11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g.. pre-trained language models, image generators, or scraped datasets)?
12. Licenses for existing assets Question: Are the creators or original owners assets (e.g., code, data, models), used in the paper, properly credited, and the license and terms of use explicitly mentioned and properly respected? [Yes] In Appendix[D..
13. New Assets Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?[NA]
14. Crowdsourcing and Research with Human Subjects Question: For crowdsourcing exper- iments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?[NA]
28


Reviews:
[{"review_id": "fmxvM5FEmg", "reviewer": "Reviewer_7tZv", "summary": "The authors introduce the Effective Metric-based Exploration-bonus (EME) to address the limitations of existing state discrepancy methods by proposing a robust metric for evaluating state differences and a diversity-enhanced scaling factor for exploration bonuses. Extensive experiments demonstrate that EME outperforms other methods in various environments, including Atari games, Minigrid, Robosuite, and Habitat, highlighting its scalability and applicability to diverse scenarios.", "questions": "1. Can you provide more details on the computational complexity and runtime performance of the EME method compared to existing approaches? Specifically, how does EME scale with the size of the state space and the complexity of the environment?\n\n2. Could you provide detailed ablation studies to isolate the impact of the diversity-enhanced scaling factor and other key components of EME?\n\n3. Some parts of the theoretical analysis are dense. Could you provide more intuitive explanations or examples to help bridge the gap between the theory and its practical implications?", "limitations": "Yes", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality:\n\nEME has not been proposed before. It appears to be effective.\n\nQuality:\n\nThe paper is supported by thorough theoretical underpinnings and extensive experimental validation across many environments.\n\nClarity:\n\nThe paper is well-written and clearly structured.\n\nSignificance:\n\nExploration is a core challenge in RL. This paper makes a significant contribution to it.", "weaknesses": "The ultimate method seems incredibly complex, making it difficult for future practitioners to reproduce. Emphasizing Algorithm 1 (potentially moving it from the Appendix into the main paper) could help with this.\n\nThe paper does not provide a thorough discussion of the computational efficiency and potential overheads introduced by the EME method. It would be useful to provide a section discussing this.\n\nThe paper introduces several novel components, such as the diversity-enhanced scaling factor, but lacks detailed ablation studies that isolate the contributions of these individual components.\n\nThe authors repeatedly refer to \"a real-life indoor environment\". While somewhat true (as it is later revealed to be the Habitat environment), it is rather misleading, as the experiments are still entirely done in the computer. (ie there is no real-world robot involved).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for valuable suggestions, the requested exp have been included in the uploaded pdf, for all the references mentioned in the response, please find the reference list in the global comment, and we provide our responseas follows.\n\n*W1:The ultimate method seems incredibly complex..moving Algorithm 1 to main paper could help..*\n\n*Q1:..more details on the computational complexity and runtime performance..how does EME scale with size of the state space..* \n\n**Response**: Thank you for your valuable suggestions. We would like to move Algorithm 1 to the main text in the revision. Regarding the runtime performance, we direct you to Fig 2 and 3 in the paper, which demonstrate that EME achieves higher returns in the same timesteps and faster convergence compared to other baselines, indicating reduced running time during training. In terms of computational complexity comparison, RIDE utilize calculating state difference in representation space (see Table 1), which is learned using both a forward and an inverse dynamics model. While for the loss function of LIBERTY, please refer to Eq (3), we can see that the calculation of LIBERTY requires the approximation of Wasserstein distance of transition distributions and the difference of inverse dynamic output, which requires additional training of transition model and inverse dynamic model. In contrast, as shown in Equation (9), EME simplifies the process by only training additional reward models to calculate the bonus. So our method is more computationally friendly and we require less training procedures comparing to RIDE and LIBERTY.\n\nThe performance of EME is scalable because EME provides a more robust and effective metric to evaluate state difference within different environments, and we provide analysis in line 200-212 and 218-226. The scalabity of EME is also evidenced in the experiments where EME outperforms other methods in different environments with different state size. Specifically, when EME scales with environments with different state size or complexity, we only need to adjust the size of network accordingly to fit in the state size for calculation of bonus. Because the objective of metric learning only considers the sum of reward differences, the distance between sampled subsequent states, and the KL divergence between policy distributions, which means that the size of states has minimal impact on the metric learning. \n\n*Q2:Could you provide ablation studies to isolate the components of EME?*\n\n**Response**: Yes, we had already provided the ablation study on scaling factor in Appendix C.4 (line 667-676), where we compared EME with two variants: EME-EP, which incorporates episodic counts, and EME-Static, which uses a static scaling factor. The experimental results are shown in Figure 8, 10 and 11. For metric comparison, we also include the performance of other metric-based methods with diversity-enhanced scaling factor to isolate the impact of our scaling factor, the result is shown in Figure 1 in uploaded pdf, where EME beats other metric-based methods equipped with diversity-enhanced scaling factor, indicating the superiority of our metric comparing to bisimulation metric and $L_p$ norms. Also we provide additional ablation study on the ensemble size, max reward scaling M in Figure 2, 3 and 4 of uploaded pdf. Due to limited space and time, we will include the ablation results with other environments in the revised version.\n\n*Q3:..Could you provide more intuitive explanations..on the theory and its practical implications?*\n\n**Response**: Thank you for your insightful question. Proposition 1 and 2 illustrate the impact of the approximation gap on the bisimulation metric-based method. Fundamentally, the distance derived from the inverse dynamic bisimulation metric could break the fixed-point guarantee, resulting in a less stringent bound on the value difference. This suggests that the metric may not be able to converge and not be closely relate to the value function. Consequently, the learned metric could diverge and fails to precisely reflect state similarity within the bisimulation metric space, leading to a decrease in performance.\n\nTheorem 1 provides a convergence guarantee, stating that our learned metric will converge to a fixed point during training, which ensures that the metric becomes stable and reliable as training progresses. Theorem 2 explains that the EME distance between states serves as an upper bound on the value difference between those states. Practically, this means the agent is encouraged to visit states with both higher state differences and higher value differences. Visiting states with higher value differences enhances the value diversity between collected transition samples, leading to a more diverse policy during training. Furthermore, an increase in bonus reward leads to larger value differences and thus larger TD errors (see lines 218-226), incentivizing the agent to prioritize transitions with large TD errors, so the training efficiency will be improved.\n\n*W2:..a real-life indoor environment..is rather misleading..*\n\n**Response**:Thank you for pointing out the potential for misunderstanding. The detailed description of Habitat is included in Appendix D.1.3. To clarify, we use the Habitat platform [A], which is designed for embodied AI research and provides an interface for agents to navigate and act within photorealistic simulations of real indoor environments. Specifically, we utilize the MP3D dataset[B], which includes high-quality renditions of 1,000 different indoor spaces. This platform offers a practical application for RL in embodied AI, providing a highly realistic and diverse set of environments for testing and development. So we call it a real-life indoor environment. To avoid confusion, we will change the term to \"a photorealistic simulator of indoor environments\" in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YeIKCS55uk", "reviewer": "Reviewer_TUTk", "summary": "This paper introduces the Effective Metric-based Exploration-bonus (EME), a novel approach for enhancing exploration in reinforcement learning (RL) tasks. The paper identifies key limitations in existing metric-based exploration methods, such as their reliance on count-based scaling factors and approximation gaps in bisimulation metric learning. To address these issues, the paper proposes a more robust metric for evaluating state discrepancy that provides theoretical guarantees on the value difference bound. Additionally, they introduce a diversity-enhanced scaling factor based on the variance of predictions from an ensemble of reward models to improve exploration effectiveness, especially in hard exploration scenarios. The authors conduct extensive experiments across various challenging environments, including continuous control tasks, discrete-action games, and realistic indoor scenarios, demonstrating the superior performance and scalability of their method compared to prior approaches.", "questions": "My questions are stated above.", "limitations": "Yes, the paper has adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper includes a nicely-written and comprehensive analysis of the limitations in existing metric-based exploration methods; this really helps setting up the motivation for EME.\n\n2. Extensive experimental evaluation across a diverse set of environments, including continuous control, discrete-action games, and realistic indoor scenarios, demonstrating the effectiveness and scalability of EME.\n\n3. Introduction of a theoretically grounded metric for evaluating state discrepancy, backed by rigorous proofs and guarantees on the value difference bound.\n\n4. The reward-free exploration experiment is very nice because in many real-world scenarios, reward function may be hard to write down. \n\nOverall, this is a solid paper with good methodological contributions and convincing experiment results.", "weaknesses": "1. The performance of EME with feature encoder is mixed; this may hamper the scalability of EME to realistic scenarios, where pre-trained feature encoder is commonly used.\n\n2. Ablations are limited. It would be good to know how sensitive EME is to the choice of RL algorithm, the value of M, and other hyperparameters. Only scaling factor is studied right now. \n\n3. On the Robosuite environments, EME do not seem much better than the baselines. How do we characterize when we expect EME to do much better than existing methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for recognizing our contributions! The additional ablation experiments have been included in the uploaded pdf, we provide our response below:  \n\n*W1:The performance of EME with feature encoder is mixed; this may hamper the scalability of EME to realistic scenarios, where pre-trained feature encoder is commonly used*\n\n**Response**: Thanks for the valuable question. Yes, we agree that the performance of EME can be influenced by the choice of pre-trained feature encoders, which is also a common issue for existing state difference-based exploration approaches like RIDE where the bonus heavily relies on the design of encoders. However, as demonstrated in Figures 6 and 7, the impact on EME's performance is not significant and can sometimes be positive. Even when different pre-trained encoders are used, EME consistently outperforms other baseline methods, which suggests that despite the inherent variability introduced by different feature encoders, the performance of EME remains at a relatively high level, providing stronger results compared to other methods.\n\n*W2:Ablations are limited. It would be good to know how sensitive EME is to the choice of RL algorithm, the value of M, and other hyperparameters. Only scaling factor is studied right now*\n\n**Response**: Thank you for the valuable suggestions. Additional experiments can be found in the uploaded pdf. We conducted an ablation study on the value of M, shown in Figure 2, for continuous control tasks in Robosuite. M sets an upper limit on the bonus. We set M=10 as the default setting. As we can see, if M=1, the scaling factor is fixed to 1, resulting in a significant performance decline. Higher M encourages more extensive exploration. The performance with M=5 slightly lags behind the default setting. The performance with M=20 and M=40 is comparable and almost the same, indicating that the performance stabilizes as M increases. We also provide the performance of EME with different RL algorithms. We chose widely used benchmark algorithms for continuous control tasks: PPO, A2C, SAC, and DDPG. As shown in Figure 4, EME acts as a plug-in algorithm and demonstrates considerable improvement across all baseline methods. The performance of EME with different RL algorithms is almost at the same level, indicating EME's wide adaptability. Additionally, we conducted an ablation study on ensemble size, presented in Figure 3, and included the performance of other metric-based methods integrated with our proposed scaling factor in Figure 1. These studies show that EME is robust to different parameter choices and consistently demonstrates superior performance compared to other baselines. Due to limited time and space, we will include the detailed ablation experiment of other environments in the revision.\n\n*W3:On the Robosuite environments, EME do not seem much better than the baselines. How do we characterize when we expect EME to do much better than existing methods?*\n\n**Response**: Thanks for the insightful question. We have discussed the point of EME's performance lead in the limitations section in Appendix E. Specifically, we expect EME to outperform existing methods more significantly in environments with sparse rewards and challenging exploration tasks requiring extensive exploration and diverse actions. Examples include navigation tasks in Habitat, hard exploration tasks in Atari and the gym-maze environment. In these settings, where external rewards from the environment are very sparse or zero during the learning process, the advantage of EME's bonus becomes more pronounced compared to other bonuses due to the superiority of our proposed metric and the diversity-enhanced scaling factor.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ccjNcr9s1U", "reviewer": "Reviewer_tp7G", "summary": "This paper proposes a new exploration algorithm in reinforcement learning (RL), particularly in environments with sparse rewards. The authors critique existing exploration bonus methods using state discrepancy, highlighting their limitations in scalability and theoretical guarantees. They propose a novel method called Effective Metric-based Exploration-bonus (EME), which features a robust metric for evaluating state discrepancy, grounded in theoretical assurances. EME avoids approximation gaps found in previous bisimulation metric-based approaches and eliminates the reliance on episodic count scaling factors.  Furthermore, they introduce a diversity-enhanced scaling factor, dynamically adjusted by the variance of predictions from an ensemble of reward models, to improve exploration in tasks with subtle state differences. Experiments on Robosuite, Atari games, MiniGrid, and Habitat environments demonstrate the effectiveness and scalability of EME.", "questions": "- The explanation in lines 99-102 regarding the breaking of theoretical assurances is unclear. Can the authors elaborate on what specifically breaks down and how EME overcomes this issue?\n\n- On L130, the authors mention that existing methods might already be performing a similar function.  Can the authors clarify this statement and clearly differentiate EME from such methods?\n\n- Why is the Habitat environment considered a more realistic scenario given the variety of task one could attempt to solve with reinforcement learning?\n\n- The paper lacks citations for using an ensemble of models to drive exploration (L261), see for example [1, 2]\n\n- Is the feature encoder jointly learned with the EME metric? I couldn't find the information, though results in the appendix show that even a random feature encoder is providing good results\n\n- How does the number of reward models in the ensemble affect EME's performance and computational cost? Is there an optimal number, or are there diminishing returns as the ensemble size increases?\n\n\n[1] Deep Exploration via Bootstrapped DQN, Osband et al. \n\n[2] The Curse of Diversity in Ensemble-Based Exploration, Lin et al. ICLR 2024", "limitations": "The authors acknowledged limitations regarding the performance in environments with less sparse rewards and untested performance in RL domains with large action spaces.\nAn ablation study of the different component of the algorithm as well as a more thorough evaluation on diverse environments would be helpful to properly evaluation the contribution presented in this paper.\nThe potential negative societal impact is also addressed in the broader impact statement", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "*Originality*: The EME metric presents a novel approach for measuring state discrepancies. The paper adequately cites related work and differentiates its contributions from existing methods.\n\n*Quality*: The paper provides theoretical analysis to support the claims regarding the limitations of current methods and the guarantees offered by the EME metric. The experimental evaluation is conducted across different domains, supporting the effectiveness of the proposed method.\n\n*Clarity*: The paper is generally well-written and organized. The motivation and background information are clearly presented, and the proposed method is explained in detail.\n\n*Significance*: The results shows good performance improvement over existing algorithms and show that EME could be become the default exploration algorithm for most problems.", "weaknesses": "While the presented results look impressive a more thorough empirical evaluation would be more helpful to assess the performance of EME:\n\n- On Atari the paper only evaluates on hard exploration games though it has been shown that evaluating on more Atari games is usually necessary to properly assess the performance of an algorithm. [1]\n- The paper does not include a ablation, because of that it is difficult to know if the improvement comes from the better exploration bonus and using multiple reward predictions\n\n[1] On Bonus-Based Exploration Methods in the Arcade Learning Environment, Ali Taiga et al. ICLR 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thanks to the reviewer for their insightful feedback. Additional exps have been included in the uploaded pdf, for the references mentioned in the response, please find the reference list in the global comment and we provide our response below:\n\n*W1:(1)evaluating on more Atari games..(2)does not include a ablation..*\n\n**Response**: Thank you for the valuable suggestions. The results of more Atari games is in Fig 1 of PDF. We selected 10 widely used Atari games based on [C,D]. EME still achieves top performance and achieves the best performance in 8 out of 10 games. For the ablations, we have already provided results of scaling factor in Appendix C.4, as shown in Fig 8, 10, and 11. For metric comparison, we also include the performance of other metric-based methods with our scaling factor to isolate the impact of our scaling factor, denoted as LIBERTY with DSF and RIDE with DSF, and the results are shown in Fig 1 of the PDF. EME still achieves the best performance, indicating the superiority of our metric. Additionally, we provide further ablation studies on the ensemble size, maximum reward scaling M, and different RL algorithms in Fig 2, 3, and 4 of the PDF. We will include more detailed ablation results of other envs in the revision\n\n*Q1:..line99-102 is unclear..how EME overcomes the issue?*\n\n**Response**: We are afraid that there is some mistake in the line number 99-102. We assume that you may be unclear about the statement \"breaking theoretical integrity of bisimulation metric\", and our response are provided as follows. First we have clarified the statement in Propostion 1. The relaxation of Wasserstein distance $W_1$ to $W_2$ will break the theoretical guarantee of bisimulation metric when the transition dynamic model or the policy is stochastic, e.g. the fixed-point guarantee is broken under the circumstance, the metric may be not able to converge in the training process. EME overcomes the issue by eliminating the need of calculating $W_1$ distance and preserves the guaranteed fixed-point and value difference bound theoretical property, as stated in Theorem 1 and 2. The detailed proof can be found in Appendix B.\n\n*Q2:..L130..differentiate EME from such methods?*\n\n**Response**: Again we are afraid that there is some mistake in the line number 130. We assume that you are referring to line 91-92 that state discrepancy-based exploration bonus methods can be defined as Equation (1). These methods' bonuses can generally be defined as the product of a measure of state discrepancy and a scaling factor. What differentiates EME from such methods is twofold. First, EME uses a novel metric that more effectively and robustly captures state similarity backed by thm 1 and 2. Second, we propose a dynamic diversity-enhanced scaling factor, whereas the scaling factors of previous methods are either not scalable or are hand-crafted parameters. This combination allows EME to adaptively balance exploration and exploitation, leading to improved performance. For a detailed comparison, please refer to Table 1.\n\n*Q3:Why is Habitat a more realistic scenario?*\n\n*Q4:..lacks citations..[1,2]*\n\n**Response**: Thanks for pointing out the citations! we will include them in the revision. We had included a detail introduction of Habitat in Appendix D.1.3. Habitat is a platform designed for embodied AI research, where agents navigate and act within photorealistic simulations of real indoor environments. It is recognized as a more realistic indoor scenario in the RL research community, as noted in [E,F]. \n\n*Q5: Is the feature encoder jointly learned.. random encoder shows good results*\n\n**Response**: We are sorry for the confusion. The feature encoder is pre-trained to study how EME performs under different representations. When using bisimulation and inverse dynamic encoders, the exploration bonus is restricted because their representations are very compact, making state differences subtle and thus harming exploration efficacy. In contrast, as also oberseverd in experiments of [I], random embedded states introduce randomness into the calculation of the exploration bonus, sometimes resulting in higher bonuses in sparse reward environments. This can occasionally provide good results but also introduces higher variance, as shown in Fig 6 and 7.\n\n*Q6:How does number of reward models affect performance..*\n\n**Response**: Thank you for the valuable question. We carry the ablation study on ensemble size in Figure 3 in pdf. From the study, we observe that as the ensemble size (ES) increases, the performance of ES = 6 and ES = 9 surpasses that of ES = 3. The performance of ES = 6 and ES = 9 is comparable, with no significant difference. However, when the size is further increased to ES = 12, there is a decline in performance, particularly in the Table Wiping and Pick and Place tasks. The performance change can be analyzed from two perspectives. First, the number of reward models is related to the accuracy of the reward variance prediction used to calculate the loss function. As the ensemble size increases, the overall prediction error decreases because the models can average out individual errors more effectively. This leads to a more accurate approximation of the variance $\\zeta(s_t)$. Second, with a larger ensemble size, the variance of reward predictions decreases, resulting in a lower scale of the exploration bonus. Thus, there is a balance or trade-off between estimation accuracy and exploration, explaining why the performance of ES = 12 lags behind ES = 9. The optimal number of ensemble models may vary depending on the specific tasks and environments. Regarding computational cost, it increases with the ensemble size. Therefore, we set ES = 6 as our default setting, where the performance is nearly the same as ES = 9 but with lower computational overhead. It is also noteworthy that even with an ensemble size of 3, EME still outperforms the best baseline methods, further demonstrating the robustness of EME.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dlmh2BX4ul", "reviewer": "Reviewer_DhVP", "summary": "The authors propose a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Previous exploration algorithms measure state discrepancy strictly on $L_p$ norms which is limiting since for environments where state differences are minimal in the observations the intrinsic reward is not useful. The authors propose an exploration bonus which is centred aroudn the variance of the predicted reward for a ensemble of reward models. A scaling factor for the exploration bonus is also introduced using the ensemble of reward models, where the variance in the prediction of the ground-truth reward is taken as the scaling factor. This scaling factor is higher in novel spaces since the prediction variance increases for states that have not been explored sufficiently.. The authors conduct extensive experiments on 2D and 3D environments shows superior performance to other known exploration methods.", "questions": "- In practice, how do you train the ensemble of reward models in the reward free setting? Is there any relation to successor representations? What do you do in settings where the reward is not a good indicator for the environment structure? I'm a bit surprised that the method works well given there is no reward to learn the ensemble models. Does the KL-divergence of the policies take the overhand here?", "limitations": "The authors have discussed limitations of previous methods but have not to my knowledge discussed the limitations of their own method.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and structured, although a rough overview of the components of the exploration bonus at the beginning would have further helped my understanding.\n- The approach to consider the metric in which state discrepancy is measured is an interesting idea and seems fitting and quite novel. \n- The intrinsic reward is based on the discrepancy of an ensemble of reward models instead of the observations which is usually done, which I find an interesting idea.\n- The code is available and of high quality which should allow for reproducibility of the results\n- The authors motivate their method very well, highlighting why previous methods perform worse.", "weaknesses": "- To my knowledge other methods that try to aggregate states in high dimensions are not considered. Wouldn’t this mitigate or solve the issue with unique counts in high-dimensional spaces? \n- The maximum reward scaling $M$ seems to be an important parameter, however I do not see an ablation for it. The ablations in the appendix ablate the nature of the scaling factor, but not $M$\n- To my knowledge there is no talk about limitations of the method. Could you provide insight on where possible limitations to your method lie?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for recognizing our contributions, writing and novelty. The additional ablation experiments have been included in the PDF file and all the references mentioned in the response can be found in the reference list in the global comment. We provide our response below:\n\n*W1: To my knowledge other methods that try to aggregate states in high dimensions are not considered. Wouldn’t this mitigate or solve the issue with unique counts in high-dimensional spaces?*\n\n**Response**: Thank you for the insightful question. Indeed, one alternative to address the issue of unique counts in high-dimensional spaces is to design a function that extracts relevant features from each state and feeds them into the count-based episodic bonus. Specifically, instead of defining a bonus using $N_e(s_t)$, we could define the bonus using $N_e(\\phi(s_t))$, where $\\phi$ is a hand-designed feature extractor. In our experiments, following the RIDE implementation in [G], in the Habitat environment, the RIDE and NovelD implementations use $\\phi(s_t) = (x_t, y_t)$, where $(x_t, y_t)$ is the spatial location of the agent at time t, so the unique state count issue is solved. However, despite this, the performance of RIDE and NovelD still lags behind in Habitat and the state counts is still very sparse. Additionally, the aggregation of states using a feature extractor relies heavily on task-specific knowledge, which significantly restricts scalability. Another approach involves using pseudo-counts [H], which estimate counts in high-dimensional state spaces using a density model. This method is adopted by RIDE and NovelD in our MiniGrid experiments and the performance still falls behind ours. Consequently, even with pseudo-counts, the training efficacy and accuracy of the estimated counts remain challenging issues. Overall, even if we solve the unique count issue, the performance of count-based bonuses still struggles in high-dimensional environments due to the sparsity and limited accuracy of counts.\n\n*W2: The maximum reward scaling seems to be an important parameter, however I do not see an ablation for it. The ablations in the appendix ablate the nature of the scaling factor, but not M*\n\n**Response**: Thank you for the valuable suggestions. Additional experiments can be found in the uploaded pdf. We provide an ablation study on the value of M in Figure 2 for continuous control tasks in Robosuite. M sets an upper limit on the bonus. We set M=10 as the default setting. As we can see, if M=1, the scaling factor is fixed to 1, resulting in a significant performance decline. Higher M encourages more extensive exploration. The performance with M=5 slightly lags behind the default setting. The performance with M=20 and M=40 is comparable and almost the same, indicating that the performance stabilizes as M increases. Practically, the value of M can be adjusted depending on the specific task and environment to determine the intensity of exploration. We are glad to include the detailed ablation experiment in the revision.\n\n*W3:To my knowledge there is no talk about limitations of the method. Could you provide insight on where possible limitations to your method lie?*\n\n**Response**: Thank you for the observation. We want clarify that we have provided the cross-reference to limitations in the paper checklist (see Appendix E). This discussion was acknowledged by reviewers 7tZv, TUTk, and tp7G. To summarize briefly, our method does not consistently achieve top performance in environments with less sparse rewards with untested performance in RL domains with very large action spaces.\n\n*Q1: In practice, how do you train the ensemble of reward models in the reward free setting? Is there any relation to successor representations? What do you do in settings where the reward is not a good indicator for the environment structure? I'm a bit surprised that the method works well given there is no reward to learn the ensemble models. Does the KL-divergence of the policies take the overhand here?*\n\n**Response**:Thanks for the insightful question! First, we want to emphasize that in the reward-free setting of Habitat, since there is no reward, we compare the exploration capability of these methods by evaluating the portion of explored space. Therefore, the **exploration bonus** takes the lead here. The performance can be analyzed from two perspectives:\n\n- First, regarding the scaling factor, in the reward-free setting for training the ensemble models, each model’s parameters are initialized differently, and we train each model on a subset of data randomly sampled with replacement. Without the external reward, most labels of the training samples are zero during the training process, leading to larger variance between predictions during the training phase. The increased variance requires more time or a life-long learning approach for the ensemble models to converge, allowing for a larger scaling factor in such scenarios to promote more extensive exploration.\n\n- Second, from the point of metric, referring to EME metric definition in Equation (4), when the reward is zero, the KL-divergence of the policies indeed plays a significant role. It encourages agents to take more diverse actions, compensating for the lack of a reward signal.\n\nOverall, based on our bonus Equation (10), the EME bonus can promote more extensive exploration compared to other methods when the reward signal is sparse or not a good indicator of the environment structure. The effectiveness is also evidenced in Figure 8, which demonstrates the performance of EME in reward-free experiments. Thus, while the KL-divergence does take a more prominent role in these settings, it effectively drives the exploration process combined with diversity-enhanced scaling factor, ensuring the method remains robust and efficient. We are glad to include the discussion in the main paper to further improve the clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful and valuable feedback. Overall, reviwers appreciate the novelty, clarity and contribution of our work, we summarize the major concerns and provide new experimental results with discussions below:

- ### Additional ablation studies and other experiments

According to reviewer 7tZv, detailed ablation studies are required to isolate the impact of the diversity-enhanced scaling factor and other key components of EME. We clarify that we had already included ablation study on diversity-enhanced scling factor in Appendix C.4. Reviewer TUTk and DhVP suggest ablation studies on the maximum reward scaling parameter M. Reviewer tp7G suggests that our method should be tested on more Atari games and propose questions on the impact of ensemble size. To address the concern, we have included additional ablation studies on parameter M, ensemble size, and metric impact comparison to isolate the influence of key components of EME. Additionally, following [C,D], we have chosen 10 other representative Atari games to test our method's performance compared to other baselines. The results, included in the uploaded PDF, show that EME's performance still leads in the majority of games. The analysis of our method's performance and the ablation experiments are provided in the individual responses.

- ### Clarification on theorems and Habitat environment

Reviewer 7tZv suggests to provide more intuitive explanations or examples on the theoretical results, and reviewer tp7G is unclear about how other method break theoretical assurance and how EME overcome the issue. And reviewer 7tZv and tp7G have some questions about why Habitat is called a more realistic environment. We have clarified the theoretical results and the setting of Habitat in the individual response.

 We thanks again for all the reviewers putting time and care into reviewing the paper, and we had answered all the reviewer's questions and minor comments in the individual response. To facilitate cross-referencing, we present all the references utilized in the response here

> Reference list
> 
> [A] Savva, Manolis, et al. Habitat: A platform for embodied ai research. ICCV 2019
>
> [B] Chang, Angel, et al. Matterport3d: Learning from rgb-d data in indoor environments. arXiv 2017
>
> [C] Taiga, Adrien Ali et al. On Bonus-Based Exploration Methods in the Arcade Learning Environment. ICLR 2020
>
> [D] Wang Y et al. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. NIPS 2024
>
> [E] Henaff, Mikael, et al. Exploration via elliptical episodic bonuses. NIPS 2022
>
> [F] Henaff, Mikael, et al. A study of global and episodic bonuses for exploration in contextual mdps. ICML 2023
>
> [G] Mikayel Samvelyan et al. MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research. NIPS 2021
>
> [H] Ostrovski, Georg, et al. Count-based exploration with neural density models. ICML 2017
>
>  [I]    Zhang, T et al. Noveld: A simple yet effective exploration criterion. NIPS 2021

Meta-review: All four reviewers agree that this is a solid paper, making significant contributions on an important topic, presented with clarity and extensive analysis.
Decision: Accept (spotlight)