Title: Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

exploration bonus

intrinsic reward

metric-based behavioral similarity
Abstract: Enhancing exploration in reinforcement learning (RL) through the incorporation of intrinsic rewards, specifically by leveraging *state discrepancy* measures within various metric spaces as exploration bonuses, has emerged as a prevalent strategy to encourage agents to visit novel states. The critical factor lies in how to quantify the difference between adjacent states as *novelty* for promoting effective exploration.
Nonetheless, existing methods that evaluate state discrepancy in the latent space under $L_1$ or $L_2$ norm often depend on count-based episodic terms as scaling factors for exploration bonuses, significantly limiting their scalability. Additionally, methods that utilize the bisimulation metric for evaluating state discrepancies face a theory-practice gap due to improper approximations in metric learning, particularly struggling with *hard exploration* tasks. To overcome these challenges, we introduce the **E**ffective **M**etric-based **E**xploration-bonus (EME). EME critically examines and addresses the inherent limitations and approximation inaccuracies of current metric-based state discrepancy methods for exploration, proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to be dynamically adjusted by the variance of prediction from an ensemble of reward models, thereby enhancing exploration effectiveness in particularly challenging scenarios. 
Extensive experiments are conducted on hard exploration tasks within Atari games, Minigrid, Robosuite, and Habitat, which illustrate our method's scalability to various scenarios. The project website can be found at https://sites.google.com/view/effective-metric-exploration.

Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus

Abstract

Enhancing exploration in reinforcement learning (RL) through the incorporation of intrinsic rewards, specifically by leveraging state discrepancy measures within various metric spaces as exploration bonuses, has emerged as a prevalent strategy to encourage agents to visit novel states. The critical factor lies in how to quantify the difference between adjacent states as novelty for promoting effective exploration. Nonetheless, existing methods that evaluate state discrepancy in the latent space under L 1 or L 2 norm often depend on count-based episodic terms as scaling factors for exploration bonuses, significantly limiting their scalability. Additionally, methods that utilize the bisimulation metric for evaluating state discrepancies face a theory-practice gap due to improper approximations in metric learning, particularly struggling with hard exploration tasks. To overcome these challenges, we introduce the E ffective M etric-based E xploration-bonus (EME). EME critically examines and addresses the inherent limitations and approximation inaccuracies of current metric-based state discrepancy methods for exploration, proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to be dynamically adjusted by the variance of prediction from an ensemble of reward models, thereby enhancing exploration effectiveness in particularly challenging scenarios. Extensive experiments are conducted on hard exploration tasks within Atari games, Minigrid, Robosuite, and Habitat, which illustrate our method's scalability to various scenarios. The project website can be found at https://sites.google.com/view/effective-metric-exploration.

1 Introduction

Reinforcement learning (RL) has made significant strides, yielding breakthroughs across various domains, including video gaming [43], autonomous driving [65], and robotic control [1, 37]. However, for many real-world tasks, defining a dense reward function is non-trivial, yet a sparse reward function based on success or failure is directly available, which makes learning effective policies difficult, as they demand efficient exploration of the state space, highlighting exploration in sparse-reward environments as a core challenge in RL [59].

Methods that quantify the state discrepancy between adjacent steps using specific measures within different metric spaces have shown remarkable success in tasks characterized by sparse rewards. These methods, by leveraging a measure of state discrepancy as an exploration bonus, facilitate agents

in discovering novel states. For instance, RIDE [51] employs the difference between two consecutive state embeddings, measured under L 2 norm, as an exploration bonus. Similarly, NovelD [69] introduces a bonus based on the state discrepancy, as represented by the RND [11] bonus, but under the L 1 norm. However, the effectiveness of these L p norms-based methods is contingent on the scaling factor expressed as the episodic count term N ep ( s ) , which is the number of times state s has been visited during the current episode. The episodic term becomes ineffective when each state is unique and cannot be counted, posting a significant limitation when it comes to more complex, dynamic, and noisy environments [32]. LIBERTY [63] aims to surmount this hurdle by evaluating state discrepancy through the bisimulation metric [25], which links state differences to value differences within the bisimulation metric space, thus incentivizing exploration of novel states with greater value differences and substantially enhancing learning efficiency. Nevertheless, due to the high computational cost or infeasibility of accurately computing bisimulation metrics, the approximation, and relaxations over the metric [13, 68, 63] have been used to optimize efficiency. Our analysis reveals that approximation gap in LIBERTY might break the theoretical guarantees of the bisimulation metric, potentially undermining exploration performance. Furthermore, the efficacy of bonuses based on state discrepancies declines in hard exploration tasks or scenarios where state differences are minimal, such as in the 'Noisy-TV' [11] problem and vision-based real indoor environments [60], posing a significant constraint on the scalability of these methods.

(a)

(b)

(c)

厂

To address this aforementioned limitations, we introduce the Effective Metric-based Explorationbonus (EME) for exploration. Our method features a more resilient metric for evaluating state differences, simultaneously eliminating approximation gap with a theoretical guarantee. EME significantly boosts learning efficiency by forging a closer connection to the value differences between states, while maintaining fidelity to its theoretical framework. Furthermore, we have developed a diversity-enhanced scaling factor to augment exploration efficacy in hard exploration tasks, where state differences are notably subtle. Our scaling factor dynamically adjusts the exploration bonus based on the variance of predictions from an ensemble of reward models, which is higher when agents encounter novel state space, thereby encouraging more effective exploration. As illustrated in Figure 1, we present the trajectories of policies trained using count-based state discrepancy bonus, bisimulation-based exploration bonus, and our method within a real-life indoor environment [15] (Full results in Figure 9 of Appendix C.3). Our method successfully explores a significantly larger portion of the space compared to the others, demonstrating the superior effectiveness of EME.

The main contributions of this paper are as follows. Firstly, we conduct a comprehensive analysis of the limitations inherent in current metric-based state discrepancy methods for exploration. Based on the analysis, we introduce an effective metric for evaluating the behavioral similarity between states supported by theoretical assurances. Secondly, we propose a diversity-enhanced scaling factor for exploration bonuses based on the variance of predictions from an ensemble of reward models, which is scalable and effective in hard exploration tasks. Lastly, extensive experiments are conducted in Robosuite, Atari games, MiniGrid, and Habitat. The results demonstrate that our algorithm can effectively enhance exploration and accelerate training across various environments.

2 Background

We focus on exploration bonuses to incentivize exploration in reinforcement learning (RL). To foster the reader's understanding, we first introduce standard notation and common practices.

Markov Decision Processes. We assume the underlying environment is a Markov decision process (MDP), defined by the tuple M = ( S , A , P, R, γ ) , where S is the state space, A is the action space, P ( s ′ | s, a ) is state transition function from state s ∈ S to state s ′ ∈ S , R is the reward function

and γ ∈ [0 , 1) is the discount factor. Generally, the policy of an agent in an MDP is a mapping π : S × A → [0 , 1] . An agent chooses actions a ∈ A according to a policy function a ∼ π ( s ) , which updates the system state s ′ ∼ P ( s, a ) yielding a reward r = R ( s, a ) . In this paper, we denote a policy by π θ , where θ is the parameter of the policy function. The goal of the agent is to optimize the parameter θ for maximizing the expected accumulative rewards, J ( π θ ) = E π θ [ ∑ ∞ t =0 γ t R ( s t , a t )] .

State Discrepancy as Exploration Bonuses. Addressing the sparse reward challenge prevalent in many environments necessitates augmenting the external reward function, r e , with an intrinsic reward bonus, b , resulting in a combined reward: r = r e + b . A number of intrinsic bonuses 2 that encourage exploration have been proposed. More recent methodologies employ an exploration bonus that varies based on the novelty of the current state relative to previously visited states, evaluated within different metric spaces, which can be generally defined as:

where SD denotes the state discrepancy, SF denotes the scaling factor, d : S × S → R represents the evaluation metric that quantifies the degree of difference or distance between two projected states, and Φ : S → Z is the projection function which maps the state to a representation space Z .

Table 1: Comparison of exploration methods using different measures (marked in blue) of state discrepancy as exploration bonus. ϕ = f ( s ) : features encoder; N ep ( s ) : episodic (pseudo) count of visits to state s ; r RND ( s ) : the RND [11] bonus; α : normalized coefficient; d inv : inverse dynamic bisimulation metric used in [63]; s 0 : initial state; λ : the scaling hyper-parameter; d E : our proposed effective metric for state discrepancy evaluation; ζ ( r s ) : variance of predictions from an ensemble of reward models: M : maximum reward scaling.

The comparison of recent metric-based exploration methods is summarized in Table 1 These bonuses reward high dissimilarity between adjacent states, encouraging exploration by leveraging measures such as the L 2 distance for state embeddings (as used by RIDE [51]) and the L 1 distance for the disparity in RND bonuses between adjacent states (as utilized by NovelD [69]). However, these approaches, grounded in L p norms and dependent on episodic state visit counts, face significant scalability challenges when exploring various environments. On the other hand, LIBERTY [63] proposes the inverse dynamic bisimulation metric to evaluate the state difference, which links state and value differences to enhance learning efficiency. Yet, the approximation inaccuracies inherent in bisimulation metric learning compromise theoretical assurances, breaking theoretical integrity of bisimulation metric. In the following sections, we detail the shortcomings of these methods.

3 Limitations of Existing Metric-based Exploration Bonus

3.1 Latent Vector Space under L p Norms

L p norms such as L 1 and L 2 norms are widely used in measuring the "distance" between states or embeddings. For example, considering a continuous map between the states of an MDP and a latent vector space: Φ : S → Z ⊆ R n . We can equip this vector space with a Euclidean norm to obtain a Euclidean metric space ( Z , ∥ · ∥ 2 ) , and the state discrepancy is measured as SD = ∥ Φ( s ) -Φ( s ′ ) ∥ 2 .

Reliance on Count-based Scaling Factor. The state-difference based exploration bonus employed by RIDE [51] and Noveld [69] can be regarded as the dif- ference between adjacent states in the L p norm space. The efficacy of the exploration bonus heavily lies in the episodic scaling factor ( SF ) measured by the count of visited states N ep ( s ) presented in Table 1. However, the count-based episodic bonus suffers from a fundamental limitation, which is similar to that faced by count-based approaches [5] in general: if each state is unique, then

N ep ( s t ) will always be 1 and the episodic bonus is no longer meaningful, which is the case for many real-world applications. For example, a household robot's state as recorded by its camera might include moving trees outside the window, clocks showing the time, or images on a television screen which are not relevant to its tasks, but nevertheless make each state unique. As shown in Table 2, without the episodic count term (denoted as "w/o EP"), the performance of RIDE and Noveld exhibits a significant decline in all different types of environments.

Limited Expressiveness of Novelty. As indicated in Table 2, the performance of RIDE and NovelD significantly degrades in environments with high-dimensional visual observations or continuous control tasks, where the success rate falls below 0 . 5 . The significant decline highlights the inadequacies of utilizing state differences within vector spaces, computed using the L p norm, to effectively express novelty. Therefore, a more robust and effective metric is essential to accurately assess and characterize the novelty among visited states.

3.2 State Discrepancy within the Bisimulation Metric Space

The bisimulation metric [25, 26] defines a pseudometric d : S × S → R to measure the similarity between two states. The recently proposed variant, π -bisimulation metric [13], focuses on behaviors relative to a particular policy π , which consists of a reward difference term and a Wasserstein distance in dynamics models between states.

Definition 1 ( π -bisimulation metric) . Given a fixed policy π , the following π -bisimulation metric exists and is unique:

where E a i ∼ π r a i s i = E a i ∼ π ( ·| s i ) r ( s i , a i ) , P π ( · | s i ) = E a ∼ π ( ·| s i ) P ( · | s i , a ) and W 1 is the 1Wasserstein distance.

Approximation Gap. LIBERTY [63] proposes exploration bonuses based on state discrepancy within the inverse dynamic bisimulation metric space, which adds the difference between action output by the inverse dynamic model based on the bisimulation metric. The difference between states evaluated under the bisimulation metric is directly associated with the value difference (see Theorem 3.3 in [26] and Theorem 2 in [63] for detailed proof). The loss function of LIBERTY metric learning is:

where ¯ s denotes state with stop gradients, P π ¯ s i ( η ) indicates probabilistic dynamics model parameterized with η and I s i +1 ¯ s i ( θ ) is the inverse dynamic model parameterized with θ . As the 1-Wasserstein distance is usually difficult to estimate, LIBERTY and prior methods [68] propose to use 2-Wasserstein distance W 2 to replace W 1 , as W 2 has a convenient closed-form of a Gaussian distribution with respect to the L 2 distance.

Proposition 1 (Relaxation Divergence) . Relaxing the W 1 metric to W 2 breaks the theoretical integrity of the inverse dynamic bisimulation metric [63] in scenarios where the transition dynamic model P ( s, a ) or the policy π is stochastic.

Proof in Appendix B. Based on Proposition 1, we can see that the relaxation divergence of 1Wasserstein distance between dynamic models can only be ignored when both the transition dynamics model and policy π are deterministic. However, this assumption may be too strong to hold in practice.

Besides, the metric learning process encounters a theory-practice gap due to the relaxation of reward expectations. Specifically, computing the expected reward differences (first term of Equation (2)), | E a i ∼ π r a i s i -E a j ∼ π r a j s j | , proves to be computationally daunting and challenging to accurately estimate, even with sampling techniques. Following other bisimulation metric-based methods [13, 68], LIBERTY [63] shifts the expectation operator outside the absolute value of reward differences (second term of Equation (3)) and differences between action outputs predicted by inverse dynamic models, as in E a i ∼ π,a j ∼ π [ | r a i s i -r a j s j | + | I s i +1 ¯ s i -I s j +1 ¯ s j | ] , which facilitates more efficient sampling by avoiding the direct estimation of reward and action expectations.

Proposition 2 (Shifted LIBERTY Distance) . The relaxation of expectation during learning process shifts the original LIBERTY distance and introduces a looser value difference bound.

We extend the concept of the shifted MICo distance [14] introduced in [35, 16]. The proof is in appendix B. Based on Proposition 2, as the shifted LIBERTY distance has a looser value difference bound, it may be less relevant to the value function. As a result, the learned metric may not be able to capture the state similarity within bisimulation metric space accurately. , as the shifted LIBERTY distance has a looser value difference bound, it may be less relevant to the value function. As a result, the learned metric may not be able to capture the state similarity within bisimulation metric space accurately.

Scalability Constraints. As shown in Table 2, LIBERTY's performance significantly drops in hard exploration tasks within MiniGrid and in the realistic scenarios of the Habitat environment [60]. Specifically, in Habitat, agents must navigate through photorealistic simulations of actual indoor spaces. In these settings, the subtle variations between states diminish the impact of the exploration bonus, significantly lowering the efficiency of exploration. This limitation becomes even more apparent in the procedurally-generated MiniGrid environment, where the inherent stochasticity further impedes LIBERTY's performance. Those findings highlight the scalability challenges that bisimulation metric-based methods encounter in complex environments.

4 Effective Metric-based Exploration Bonus

In this section, we introduce the Effective Metric-based Exploration-bonus (EME), conceived to overcome the limitations identified in existing methods that use metric-based state discrepancy as exploration bonuses. Our development is driven by two primary objectives. Firstly, we aim to refine the metric learning process to achieve a more effective exploration without incurring the approximation gap, thereby upholding the theoretical integrity of value function bound. Secondly, we seek to obviate the reliance on scaling factor of episodic counts, ensuring effective exploration across different environments. To this end, we propose a novel metric that more precisely evaluates the behavioral similarity between states, establishing a more effective exploration strategy compared with previous approaches. Furthermore, we introduce an innovative approach that utilizes the variance between outputs of an ensemble of reward models in metric learning as a dynamically-adjusted scaling factor, which significantly enhances exploration efficiency, especially in hard exploration tasks, thereby improving scalability. The next two sections describe our method in detail.

4.1 The EME Metric

In order to avoid the approximation gap introduced by relaxation of Wasserstein distance and reward expectations mentioned in Proposition 1 and 2, along with the reliance on the episodic count, we eliminate the calculation of the Wasserstein distance, and propose the EME metric:

Definition 2 (EME Distance Function) . Let met be the space of bounded pseudo-metrics on state space S , the EME metric d E : S × S → R , the EME distance function F ( d E , π ) : met → met is defined as:

where D KL ( ·∥· ) represents the Kullback-Leibler (KL) divergence.

Behavioral Similarity Between States. We strictly calculate the expectation of reward difference without relaxation. The EME metric measures the distribution distance between dynamics models by computing the distance between sampled subsequent states following representation learning method [14] to avoid the computation of the Wasserstein distance. Additionally, we integrate the Kullback-Leibler (KL) divergence between policy distributions to more robustly model the ' behavioral similarity ' between states, which is more effective and scalable across diverse environments with different observations, especially in addressing the critical 'Noisy-TV' problem [11] during exploration. The 'Noisy-TV' problem started as a thought experiment in [11] and is commonly discussed in exploration literature [51, 52, 42]. Imagine that an RL agent is rewarded with seeking the novel experience, a TV with unpredictable random noise outputs would be able to attract the agent's attention forever. The agent obtains new rewards from state discrepancy caused by noisy TV consistently, but it fails to make any meaningful action. To counter this, we utilize differences in policy distributions across states, promoting the exploration of diverse actions rather than repetitive behavior.

Theorem 1. The EME distance function F ( d E , π ) : met → met has a unique fixed-point ˆ d E .

Proof in Appendix B. Theorem 1 provides a convergence guarantee for the EME distance that by iterating F ( d E , π ) , distance will converge to the fixed-point ˆ d E .

Theorem 2. (Guaranteed Value difference bound) Given the EME metric d E , states s i and s j , and a policy π , we have

Improved Exploration Efficiency. The proof can be found in Appendix B. Theorem 2 demonstrates that the EME distance between states provides an upper bound on the difference in their state values. Let's recap the definition of one-step TD error: δ t = r t +1 + γV ( s t +1 ) -V ( s t ) . Intuitively, if there is a large value difference between states s t and s t +1 , the corresponding exploration bonus b which is calculated using d E ( s t , s t +1 ) will also be large. Consequently, the total reward for the next step, r t +1 = r e + b (where r e represents the external environment reward), increases accordingly. The increase of reward leads to a larger TD error when there is a significant value difference between adjacent states, incentivizing the agent to prioritize transitions with large TD errors, which not only enhances the agent's exploration capabilities but also significantly boosts training efficiency.

4.2 Tractable Optimization of EME

Based on Equation (4), between any pair of states s i and s j , we can define the loss function of the EME metric learning as:

where d ϕ E is the EME metric encoder parameterized by ϕ . The last three terms in the loss function (6) can be regarded as the regression of target metric, which approximates the difference between rewards, distances of next states and distances of policy distributions, respectively. However, the reward expectations E a i ∼ π r a i s i and E a j ∼ π r a j s j are computationally intractable and also difficult to estimate even based on sampling [13]. And the relaxation of reward expectations E a i ∼ π a j ∼ π | r a i s i -r a j s j |

used by previous methods [63, 68, 13] will lead to the learned metric having a looser value difference bound than the original bisimulation metric as proved in Proposition 2. Inspired by metric-based representation learning methods [35, 16], we propose to use an ensemble of reward models to approximate the reward difference more accurately.

Proposition 3. Let r s be a random variable over the action distribution defined by p ( r s = r a s ) = π ( a | s ) , var ( r s i ) denote the variance of variable r s i , we can have:

Proof. (Sketch) The proof is based on [14, 16] by expanding E a i ∼ π a j ∼ π [ | r a i s i -r a j s j | 2 ] -| E a i ∼ π r a i s i -a

Ensemble of Reward Models. We train an ensemble of reward models { g ( η 1 ) , . . . , g ( η k ) } to predict the reward from the state-action pairs sampling from the buffer D τ : g ( s, a ; η ) : S × A → R by minimizing the prediction error ∥ g ( s t , a t ; η ) -r t +1 ∥ 2 . Consequently, the variance of reward across the output of different models in the ensemble is defined as follows:

Based on Equation (6), Equation (7) and Equation (8), the tractable EME loss without approximation gap can be defined as:

Diversity-Enhanced Scaling Factor. The scaling factor of previous methods is either not scalable [51, 69, 2] or hand-crafted hyper-parameter [63], in light of this, we propose a scalable diversity-enhanced scaling factor for the intrinsic reward to further improve the efficacy of exploration. Let's recap the ensemble of reward models { g ( η 1 ) , . . . , g ( η k ) } used in metric training (9). To maintain the diversity across the individual models g ( η ) , we initialize each model's parameters differently and train each of them on a subset of data randomly sampled with replacement. Each model in our ensemble is trained to predict the ground truth reward. Hence, the parts of reward obtained within the state space that have been well explored by the agent will have gathered enough data to train all models, resulting in a low variance of reward predictions from the models, when generalizing to unseen but similar parts of the reward from unvisited state-space, the areas which are novel and unexplored would still have high prediction error for all models as none of them are yet trained on such examples, resulting in higher variance of the reward prediction. Therefore, we use the variance of reward predictions as the scaling factor of intrinsic reward to encourage exploration on unvisited state space. With the maximum reward scaling M , the exploration bonus is defined as:

As a result, the agent receives greater rewards when encountering novel states during training, which facilitates more effective exploration, the ablation study on scaling factor and visualization of exploration bonus is in Appendix C.5. See Algorithm 1 of Appendix D.2 for detailed description.

5 Experiments

The overall objective of our experiment is to evaluate the performance of EME in comparison to other baselines, we conduct comprehensive experiments on various settings of continuous control tasks, discrete-action hard exploration games, and real indoor environments 3 to assess the effectiveness and scalability of our algorithm. The implementation details can be found in Appendix D.

Baselines. We compare it against following competitive baseline methods. ICM [47]: a famous curiosity-driven method. RND [11]: intrinsic rewards are the prediction errors of the distillation network. E3B [32]: proposing count-based episodic bonuses under continuous state spaces. RIDE [51]: using state difference under latent space as exploration bonuses. NovelD [69]: using the state difference measured by RND bonus as intrinsic rewards. LIBERTY [63]: using the state discrepancy evaluated under the bisimulation metric space as shaping reward.

Door Opening

Table Wiping

Pick-and-Place

1.0

1.0

1.0

EME

ICM

Rate

0.8

0.8

0.8

RND

Success

0.6

0.6

0.6

RIDE

NovelD

E3B

Mean

0.4

0.4

0.4

LIBERTY

0.2

2'0

0.2

0.0

0

200

400

600

800

0.0

200

400

Epochs

009

800

0.0

200

400

600

800

5.1 Continuous Control

In our continuous control experiments conducted on the Robosuite platform [70], we assess the exploration capabilities of agents across various challenging tasks, which include Door Opening,

Table Wiping, and Pick-and-Place. The detailed description of environments can be found in Appendix D.1.1. Each task represents a demanding context for robotic control, characterized by sparse rewards and significant exploratory challenges. The overall results are depicted in Figure 2, where our method consistently outperforms others, demonstrating its superiority in handling continuous control tasks. LIBERTY achieves the second-best performance in the Door Opening and Table Wiping tasks, underscoring the advantages of state discrepancy-based novelty within the bisimulation metric space. Conversely, episodic count-based methods such as RIDE, NovelD, and E3B lag behind, as episodic counts become less effective in environments with high-dimensional states. Furthermore, curiosity-driven approaches like ICM and RND struggle due to insufficient exploration. We also provide additional ablation studies on the ensemble size and max reward scaling in Appendix C.

EMECombined with Feature Encoder. The EME metric can be integrated with any state representation derived from various encoders Φ( · ) , expressed as d E (Φ( s t ) , Φ( s t +1 )) . The encoder may include the inverse dynamic model [47, 51, 63] which isolates environmental factors that do not affect the agent's behavior, the bisimulation-based encoder [68, 35, 14, 13], and random embeddings. The results can be found in Figure 8 in Appendix C.2, the performance of EME under the inverse dynamic encoder and bisimulation-based encoder exhibits a decline, because the exploration bonus under these encoders get close to zero quickly since their representations are very compact and every state looks more similar with the convergence of the encoder, which harms the efficacy of exploration. With respect to random embedding, the variance is larger, resulting in more unstable performance.

EME

ICM

RND

RIDE

RIDE with DSF

NovelD

E3B

LIBERTY

LIBERTY with DSF

Asterix

Alien

Breakout

BeamRider

MsPacman

Average Game ScorePer Episode

PrivateEye

Pong

Spacelnvaders

UpNDown

Frames

5.2 Discrete-action Games

Atari Games. To assess EME in scenarios involving pixel-based observations and discrete actions, we evaluate our method on the hard exploration games identified by [6, 69, 63] within the common Atari benchmark [8], including Asterix, Alien, Breakout, BeamRider, Mspacman, PrivateEye, Pong, SpaceInvaders, Qbert, and UpNDown. The results are presented in Figure 3. Our method consistently achieves similar or higher average returns compared to other baselines across all tested tasks. Notably, EME demonstrates the fastest convergence, particularly in Asterix and UpNDown, highlighting our method's superior learning efficiency. While LIBERTY and NovelD exhibit competitive performance in 6 games, their effectiveness significantly diminishes in the challenging games, PrivateEye and UpNDown. As for other methods like RIDE, E3B, ICM, and RND struggle across all tasks, primarily

due to inadequate exploration capabilities. To isolate the impact of our proposed EME metric, we also include the performance of RIDE and LIBERTY with Diversity-enhanced Scaling Factor, denoted as RIDE and LIBERTY with DSF. EME still achieves the best performance which demostrate the superiority of EME metric.

MiniGrid Environments. MiniGrid [17] presents a series of procedurally-generated, challenging environments. We focus on three settings from MiniGrid: Multi-Room (MR), Key Corridor (KC), and Obstructed Maze (OM). For example, MRN7S6-NT stands for MultiRoom-N7-S6-NoisyTV. Details on the specific environmental settings are available in Appendix D.1.3. Table 3 presents the testing performance of EME and all baseline methods across five different seeds. EME successfully solves all exploration tasks within MiniGrid, achieving the best performance in 4 out of 6 settings. While Multi-Room environments are relatively easy, all baselines except for ICM and RND, demonstrate competency. In the challenging Obstructed Maze environments, where obstructions block doors, our agent also excels by learning to remove these obstructions, further demonstrating the effectiveness and scalability of our approach.

Noisy TV Problem. In addition to standard MiniGrid tasks, we also tested the model's ability to deal with stochasticity in the environment by adding a manually-made Noisy TV setting introduced in [51], where some blocks change color at every time step. As illustrated in Table 3, EME maintains strong performance even under these conditions, underscoring the robustness of our method.

CloseCab

OpenFridge

Rearrange

100

EME

ICM

80

RND

Rate(

RIDE

60

NovelD

E3B

LIBERTY

40

20

5.3 Real-life Habitat Environment

To test our model's applicability to real-world environments, we investigate hard exploration tasks in Habitat [60]. Habitat is a platform for embodied AI research which provides an interface for agents to navigate and act in photorealistic (Figure 5) simulations of real indoor environments. Full details on the environmental setting can be found in Appendix D.1.2. We evaluate our methods on three embodied AI tasks of Habitat benchmark. As shown in Figure 4, EME consistently outperforms all baselines, demonstrating its superior scalability to high-dimensional visualbased observations and confirming its broad applicability. The performances of count-based methods like E3B, NovelD and RIDE are comparable, whereas curiosity-driven methods such as ICM and RND lag behind. The underperformance is primarily due to their inadequate exploration capabilities, which are

Figure 5: Habitat particularly challenged by the complex and rich visual observations inherent in real-world settings.

Reward-free Exploration. we investigate reward-free exploration by evaluating the performance of all baselines using exploration bonus-only. Figure 9 displays the trajectories for all baselines on one of the test maps, clearly demonstrating that EME explores a significantly larger portion of the space compared to other methods. The full experiment results can be found in Appendix C.3.

6 Conclusion

In this work, we identify the limitations of existing state discrepancy-based exploration methods: the reliance on episodic count-based scaling factor and theory-practice approximation gap, which leads to limited scalability especially for hard exploration tasks within realistic environments. To address the issues, we propose the Effective Metric-based Exploration-bonus (EME), which addresses the inherent limitations by proposing a robust metric for state discrepancy evaluation backed by

comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to enhance exploration effectiveness in particularly challenging scenarios. Extensive experiments on hard exploration tasks from continuous control, discrete-action games and realistic environments have demonstrated the effectiveness and scalability of our method.

References

Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Joshua Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research , 39:20 - 3, 2018.

Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations , 2019.

Babak Badnava, Mona Esmaeili, Nasser Mozayani, and Payman Zarkesh-Ha. A new potentialbased reward shaping for reinforcement learning agent. In 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC) , pages 01-06. IEEE, 2023.

Chenjia Bai, Peng Liu, Kaiyu Liu, Lingxiao Wang, Yingnan Zhao, Lei Han, and Zhaoran Wang. Variational dynamic for self-supervised exploration in deep reinforcement learning. IEEE Transactions on neural networks and learning systems , 34(8):4776-4790, 2021.

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Bernadette Bucher, Anton Arapin, Ramanan Sekar, Marc Badger, Feifei Duan, Oleh Rybkin, and Kostas Daniilidis. Perception-driven curiosity with bayesian surprise. 2019.

Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355 , 2018.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations , 2018.

Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006.12122 , 2020.

Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 10069-10076, 2020.

Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-based state similarity for markov decision processes. Advances in Neural Information Processing Systems , 34:30113-30126, 2021.

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV) , 2017.

Jianda Chen and Sinno Pan. Learning representations via a robust behavioral metric for deep reinforcement learning. Advances in Neural Information Processing Systems , 35:36654-36666, 2022.

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR , abs/2306.13831, 2023.

C Colas, T Karch, O Sigaud, and PY Oudeyer. Intrinsically motivated goal-conditioned reinforcement learning: A short survey. arxiv 2020. arXiv preprint arXiv:2012.09830 .

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems , 33:13049-13061, 2020.

Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Proceedings of the 11th international conference on autonomous agents and multiagent systems , pages 433-440. IFAAMAS, 2012.

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning , pages 1329-1338. PMLR, 2016.

Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 , 2019.

Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. Advances in neural information processing systems , 32, 2019.

Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI , volume 4, pages 162-169, 2004.

Norman Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. arXiv preprint arXiv:1207.4114 , 2012.

Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In UAI , pages 210-219, 2014.

Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In UAI , pages 210-219, 2014.

Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversarially guided actor-critic. arXiv preprint arXiv:2102.04376 , 2021.

Yang Gao and Francesca Toni. Potential based reward shaping for hierarchical reinforcement learning. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015.

Marek Grzes and Daniel Kudenko. Learning potential for reward shaping in reinforcement learning with tile coding. In Proceedings AAMAS 2008 Workshop on Adaptive and Learning Agents and Multi-Agent Systems (ALAMAS-ALAg 2008) , pages 17-23, 2008.

Anna Harutyunyan, Sam Devlin, Peter Vrancx, and Ann Nowé. Expressing arbitrary reward functions as potential-based advice. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems , 35:37631-37646, 2022.

Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems , 29, 2016.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning , pages 4940-4950. PMLR, 2021.

Mete Kemertas and Tristan Aumentado-Armstrong. Towards robust bisimulation metric learning. Advances in Neural Information Processing Systems , 34:4764-4777, 2021.

Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. arXiv preprint arXiv:1810.01176 , 2018.

Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32:1238 - 1274, 2013.

Weichen Li, Rati Devidze, and Sophie Fellenz. Potential-based reward shaping for learning to play text-based adventure games. arXiv preprint arXiv:2302.10720 , 2023.

Zhixuan Lin, Pierluca D'Oro, Evgenii Nikishin, and Aaron Courville. The curse of diversity in ensemble-based exploration. arXiv preprint arXiv:2405.04342 , 2024.

Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5125-5133, 2020.

Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based exploration in feature space for reinforcement learning. arXiv preprint arXiv:1706.08090 , 2017.

Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-driven exploration via latent bayesian surprise. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 7752-7760, 2022.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518:529-533, 2015.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml , volume 99, pages 278-287. Morgan Kaufmann, 1999.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning , pages 2721-2730. PMLR, 2017.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 27782787. PMLR, 2017.

Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning , pages 5062-5071. PMLR, 2019.

Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning , pages 835-853. PMLR, 2020.

Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint arXiv:1909.12892 , 2019.

Roberta Raileanu and Tim Rocktäschel. RIDE: rewarding impact-driven exploration for procedurally-generated environments. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274 , 2018.

Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. 1991.

Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning , pages 9443-9454. PMLR, 2021.

Christopher Stanton and Jeff Clune. Curiosity search: producing generalists by encouraging individuals to continually explore and acquire skills throughout their lifetime. PloS one , 11(9):e0162235, 2016.

Christopher Stanton and Jeff Clune. Deep curiosity search: Intra-life exploration improves performance on challenging deep reinforcement learning problems. arXiv preprint arXiv:1806.00553 , 2018.

Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences , 74(8):1309-1331, 2008.

Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407 , 2017.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems , 30, 2017.

Ruo Yu Tao, Vincent François-Lavet, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances in Neural Information Processing Systems , 33:8114-8126, 2020.

Yiming Wang, Ming Yang, Renzhi Dong, Binbin Sun, Furui Liu, et al. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. Advances in Neural Information Processing Systems , 36, 2024.

Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforcement learning agents. In Proceedings of the 20th international conference on machine learning (ICML-03) , pages 792-799, 2003.

Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J. Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin, Piyush Khandelwal, Varun Kompella, HaoChih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi, Leon Barrett, Rory Douglas, Dion Whitehead, Peter Dürr, Peter Stone, Michael Spranger, and Hiroaki Kitano. Outracing champion gran turismo drivers with deep reinforcement learning. Nature , 602:223 228, 2022.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning , pages 11920-11931. PMLR, 2021.

Daochen Zha, Wenye Ma, Lei Yuan, Xia Hu, and Ji Liu. Rank the episodes: A simple approach for exploration in procedurally-generated environments. arXiv preprint arXiv:2101.08152 , 2021.

Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742 , 2020.

Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems , 34:25217-25230, 2021.

Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293 , 2020.

A Broader Impact Statement

This work introduces a reinforcement learning (RL) exploration method by introducing an effective metric-based exploration bonus, a versatile framework applicable to a wide array of decision-making problems, which can be applicable to diverse scenarios, including autonomous driving, household robotics, video gaming, online recommendation or advertisement optimization and etc. Similar to other RL exploration algorithms, our method is designed to facilitate the learning of a policy that maximizes a reward function defined by the designer. The implementation of such a policy, depending on the objectives set by the reward function designer, could lead to either positive or negative outcomes. Additionally, our approach significantly reduces the burden of reward tuning for researchers and algorithm engineers, and concurrently offers insights into the quality of the designed rewards through the shaping weights derived from our methods.

B Proofs

Proposition 1 (Relaxation Divergence) . Relaxing the W 1 metric to W 2 breaks the theoretical integrity of the inverse dynamic bisimulation metric [63] in scenarios where the transition dynamic model P ( s, a ) or the policy π is stochastic.

Proof. First, let's recall the proof of the existence of a unique fixed-point in the inverse dynamic bisimulation metric:

Proof. This proof mimics the proof of Theorem 4 . 5 from [24]. We make use of the same pointwise ordering on met : d ≤ d ′ iff d ( s, t ) ≤ d ′ ( s, t ) for all s, t ∈ S , which gives us an ω -cpo with bottom ⊥ , which is the everywhere-zero metric. Since Lemma 4 . 4 from [24] (Wasserstein metric W is continuous) also applies in our definition, it only remains to show that H ( d, ˆ π ) is continuous:

The rest of the proof follows in the same way as in [24].

In equation (11), the existence of a unique fixed-point in the bisimulation metric requires the continuity and monotonicity of W 1 with respect to d . The properties of continuity and monotonicity do not hold with W 2 . Therefore there is no more guarantee about the fixed-point existence in LIBERTY and other approximation-based methods [68] except that both the dynamics model and the policy π are deterministic, in which case W 2 ( d ) degenerates to d [24] and Banach's fixed-point exists [35]. As for scenarios when the transition model and policy are stochastic, the theoretical guarantee of LIBERTY is broken.

Proposition 2 (Shifted LIBERTY Distance) . The relaxation of expectation during learning process shifts the original LIBERTY distance and introduces a looser value difference bound.

Proof. The proof is based on top of [14, 16].

Lemma 2.

Proof. Since r a i s i and r a j s j are rewards which are scalars, we have | r a i s i -r a j s j | ≥ r a i s i -r a j s j and | r a j s j -r a i s i | ≥ r a j s j -r a i s i . By taking the expectation over a i and a j , we can have

and

By combining Equation (13) and Equation (14), we can have:

Similarly, we can have:

So we can get

Definition 3 (Shifted LIBERTY Distance) . The shifted LIBERTY distance function ˆ H π is defined as

Define the MDP for RL by a tuple ⟨S , A , R , P , γ ⟩ . We consider a lifted MDP constructed by a tuple ⟨ ˆ S , ˆ A , ˆ R , ˆ P , γ ⟩ , where state space ˆ S = S × S , action space ˆ A = A×A , transition distribution ˆ P ( a i ,a j ) ( s i ,s j ) = P a i s i P a j s j , and reward function ˆ R (( s i , s j )) = | E a i ∼ π r a i s i -E a j ∼ π r a j s j | . The Bellman operator T π under policy π ( a i , a j | s i , s j ) = π ( a i | s i ) π ( a j | s j ) is:

So we can construct the lifted MDP of shift LIBERTY by a tuple ⟨ ˆ S , ˆ A , ˜ R , ˆ P , γ ⟩ which is the same as the lifted MDP in Definition 3 except for the reward function ˜ R (( s i , s j ) , ( a i , a j )) = | r a i s i -r a j s j | and shifted metric ˜ d . The Bellman operator ˜ T ˆ π under policy ˆ π is:

As proven in LIBERTY [63], for the inverse dynamic bisimulation metric d , we can get

As proved in [63, 24, 27], d π is the value function of lifted MDP ⟨ ˆ S , ˆ A , ˆ R , ˆ P , γ ⟩ . d π can be expanded as the sum of discounted future rewards,

˜ d π can be expanded as,

with s (0) i = s i , s (0) j = s j , The difference between d and ˜ d can be regarded as

As proved in Lemma 2, we can get:

Combined with Equation (21), we can have:

So the shifted LIBERTY distance introduces a looser value bound.

Proposition 3. Let r s be a random variable over the action distribution defined by p ( r s = r a s ) = π ( a | s ) , var ( r s i ) denote the variance of variable r s i , we can have:

Proof. The proof is based on [14, 16] by expanding the difference between E a i ∼ π,a j ∼ π [ ( r a i s i -r a j s j ) 2 ] and ( E a i ∼ π r a i s i -E a j ∼ π r a j s j ) 2 .

r s i and r s j are independent variables, we can get cov ( r s i , r s j ) = 0 , so we can have

Theorem 1. The EME distance function F ( d E , π ) : met → met has a unique fixed-point ˆ d E .

Proof. Let d E , d ′ E ∈ M . We have

Therefore, F is a contraction mapping w.r.t. the L ∞ norm and there exists a unique fixed-point for F due to Banach's fixed-point theorem. This completes the proof.

Theorem 2. (Guaranteed Value difference bound) Given the EME metric d E , states s i and s j , and a policy π , we have

Proof. The proof mimics [14]. We follow the assumption that ∑ s ′ P a s ( s ′ ) V π ( s ′ ) = V π ( E s ′ ∼ P a s [ s ′ ]) . We will first show that if ∀ s i , s j ∈ S , | V π ( s i ) -V π ( s j ) | ≤ d ( s i , s j ) , then | V π ( s i ) -V π ( s j ) | ≤ F ( d E , π )( s i , s j ) . Suppose | V π ( s i ) -V π ( s j ) | ≤ d ( s i , s j ) holds, we can have:

Similarly, V π ( s j ) -V π ( s i ) ≤ F ( d, π ( s j , s i )) , so we can have

Assuming that we have an initial distance d 0 which | V π ( s i ) -V π ( s j ) | ≤ d 0 ( s i , s j ) , and base on Theorem 1, F ( d, π ) is contraction mapping on d . By repeatedly applying F ( d, π ) on d , d will eventually converge to the fixed-point d E , the fixed point d E satisfies:

C Additional Experiments

C.1 Additional Ablation Experiments

To isolate the impact of different hyper-parameters, we carry additional experiments on the ensemble size of reward models and the max reward scaling M . First, M sets an upper limit on the bonus. We set M = 10 as the default setting. As we can see from Figure 6, if M = 1 , the scaling factor is fixed to 1, resulting in a significant performance decline. Higher M encourages more extensive exploration. The performance with M = 5 slightly lags behind the default setting. The performance with M = 20 and M = 40 is comparable and almost the same, indicating that the performance stabilizes as M increases. Practically, the value of M can be adjusted depending on the specific task and environment to determine the intensity of exploration.

Door Opening

Table Wiping

Pick-and-Place

1.0

1.0

0'T

EME (M=5)

EME (M=10)

Rate

0.8

0.8

0.8

EME (M=20)

Success

0.6

0.6

0.6

EME (M=40)

EME-Static (M=1)

Mean

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

200

400

600

008

0.0

200

400

Epochs

600

800

0.0

0

200

400

600

800

DoorOpening

TableWiping

Pick-and-Place

1.0

1.0

1.0

EME (ES=3)

EME (ES=6)

Rate

0.8

0.8

0.8

EME (ES=9)

EME (ES=12)

Success

0.6

0.6

0.6

BESTBASELINE

Mean

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

200

400

600

008

0'0

200

400

Epochs

600

800

0'0

0

200

400

600

800

Door Opening

TableWiping

Pick-and-Place

1.0

1.0

1.0

EME

EME-Random

Rate

0.8

0.8

0.8

EME-IDM

Success

0.6

0.6

0.6

EME-BM

Mean

0.4

0.4

0.4

0.2

20

0.2

0.0

200

400

600

800

0.0

0

200

400

Epochs

600

800

0.0

0

200

400

600

800

As shown in Figure 7, we observe that as the ensemble size (ES) increases, the performance of ES = 6 and ES = 9 surpasses that of ES = 3. The performance of ES = 6 and ES = 9 is comparable, with no significant difference. However, when the size is further increased to ES = 12, there is a decline in performance, particularly in the Table Wiping and Pick and Place tasks. The performance change can be analyzed from two perspectives. First, the number of reward models is related to the accuracy of the reward variance prediction used to calculate the loss function. As the ensemble size increases, the overall prediction error decreases because the models can average out individual errors more effectively. This leads to a more accurate approximation of the variance ζ ( s t ) . Second, with a larger ensemble size, the variance of reward predictions decreases, resulting in a lower scale of the exploration bonus. Thus, there is a balance or trade-off between estimation accuracy and exploration, explaining why the performance of ES = 12 lags behind ES = 9. The optimal number of ensemble models may vary depending on the specific tasks and environments. Regarding computational cost, it increases with the ensemble size. Therefore, we set ES = 6 as our default setting, where the performance is nearly the same as ES = 9 but with lower computational overhead. It is also noteworthy that even with an ensemble size of 3, EME still outperforms the best baseline methods, further demonstrating the robustness of EME.

C.2 EMECombined with Feature Encoder

Our method EME can be integrated with any state representation derived from different state encoders Φ( · ) , expressed as d E (Φ( s t ) , Φ( s t +1 )) . The encoder may include the inverse dynamic model [47, 51, 63] which isolates environmental factors that do not affect the agent's behavior, the bisimulation-based encoder [68, 35, 14, 13] which learns a compact state representation grouped by the bisimulation metric, and random embeddings captured by random embedded states. We denote the EME variants as follows: EME with an inverse dynamic model-based encoder is referred to as EME-IDM; with a bisimulation metric-based encoder as EME-BM; and with a random embedding as EME-Random. The results can be found in Figure 8. EME's performance declines due to the rapid reduction of the

exploration bonus. The performance of EME under the inverse dynamic encoder and bisimulationbased encoder exhibits a decline, because the exploration bonus under these encoders gets close to zero quickly since their representations are very compact and every state looks more similar with the convergence of the encoder, which harms the efficacy of exploration. With respect to random embedding, the variance is larger, resulting in more unstable performance.

(a) EME

(b) EME-Static

(c) EME-EP

(d) ICM

(e) RND

(f) RIDE

(g) NovelD

(h) E3B

(i) LIBERTY

CloseCab

OpenFridge

Rearrange

100

EME

ICM

Mean Success Rate(%)

80

RND

RIDE

60

NovelD

E3B

LIBERTY

40

20

C.3 Reward-free Exploration

We examine reward-free exploration by assessing the performance of all baselines within Habitat using only exploration bonuses. During training, agents are initialized in varied environments for each episode and are subsequently tested on a set of environments not encountered during training. Figure 9 displays the trajectories of all baselines on one of the test maps, highlighting that EME explores a substantial portion of the space, unlike other methods. Specifically, ICM and RND agents remain confined to a single room, while RIDE, LIBERTY, and NovelD agents explore two rooms before becoming stuck, distant from the goal. E3B exhibits the second-best exploration coverage of the map. Quantitative results from other reward-free tasks are shown in Figure 10, where EME consistently demonstrates robust performance and achieves the best outcomes. In contrast, other methods experience significant declines, underscoring the efficacy of our approach in a reward-free setting.

MultiRoom-N7-S8

KeyCorridor-S4-R3

ObstructedMaze-2D-1h

2100

0.0200

Exploration Bonus

0.0175

Exploration Bonus

0.010

Exploration Bonus

0.05

0.0150

0.04

0.0125

0.008

0.0100

0.03

009

0.0075

0.02

0.0050

0.01

0.0025

0.002

00000

00'0

0.0

15

2.0

3.0

le7

0.0

1.0

st sdas

2.0

3.0

3e7

0.0

0.2

0.6

0.#

1.0

C.4 Delayed Reward Setting of Mujoco

We present results from the delayed reward setting in the MuJoCo environment [21], following the methodology outlined in [63], where accumulated rewards are delayed every 10, 20, and 30 steps. The experiments assess six tasks: HalfCheetah, Hopper, Walker2d, Ant, Swimmer, and Humanoid. As shown in Table 4, our method achieves the best or second-best performance in 15 out of 18 delayed reward tasks. This indicates that EME can facilitate effective exploration and maintain high performance even under sparse reward conditions. Notably, as rewards become sparser, EME's performance improves and becomes more robust. In contrast, other metric-based exploration bonus methods such as RIDE and NovelD struggle due to their less expressive metrics. Curiosity-driven methods like ICM and RND also become less effective as rewards grow sparser. These results provide further evidence of EME's ability to promote efficient exploration effectively, even in environments characterized by delayed rewards.

Door Opening

TableWiping

Pick-and-Place

1.0

1.0

1.0

EME

Rate

0.8

0.8

0.8

EME-EP

EME-Static

Success

0.6

0.6

0.6

Mean

0.4

0.4

0.4

0.2

0.2

0.2

0'0

200

400

600

800

0.0

0

200

400

600

800

0'0

200

400

600

800

Epochs

C.5 Ablation Study on Scaling Factor

To demonstrate the effectiveness of our proposed scaling factor, we conducted an ablation study focusing on this component. We compared EME with two variants: 'EME-EP,' which incorporates episodic counts, and 'EME-Static,' which uses a static scaling factor. The results from continuous tasks in the Robosuite and MiniGrid environments are illustrated in Figure 12, Figure 9 respectively. Without the diversity-enhanced scaling factor, EME shows a noticeable decline in performance. 'EME-EP' performs better in MiniGrid environments compared to continuous tasks in Robosuite, highlighting the efficacy of episodic state visitation counts in grid-based games but demonstrating limited effectiveness in environments with high-dimensional states. Additionally, we visualize the exploration bonus of all MiniGrid environments in Figure 11.

D Implementation Details

D.1 Environments Settings

D.1.1 Robosuite Manipulation Tasks

We choose the three manipulation tasks in the Robosuite platform [70].

Table Wiping. The environment consists of a table with a whiteboard surface and some markings is placed in front of a single robot arm, which has a whiteboard eraser mounted on its hand. The shape of the dirty region is correlated to the position of the cube. When the dirty region is diagonal, the cube is on the right-hand side of the robot arm. The goal of the agent is to learn to wipe the whiteboard surface and clean all of the markings. The whiteboard markings are randomized at the beginning of each episode.

Door Opening. The environment consists of a door with a handle mounted in free space in front of a single robot arm. The height of the handle and the position of the door are correlated. When the door is closed to the robot arm, the handle is in a low position. When the door is far from the robot arm,

the handle is in a high position. The goal of the agent is to learn to turn the handle and open the door. The initial state distribution of the door location is randomized at the beginning of each episode.

Pick and Place. The environment consists of four objects placed in a bin in front of a single robot arm. There are four containers next to the bin. The goal of the agent is to place each object into its corresponding container. This task also has easier single-object variants. The initial state distribution of object locations is randomized at the beginning of each episode.

D.1.2 Habitat

The Habitat platform relies on several key abstractions that model the domain of embodied agents and the tasks they can perform in three-dimensional indoor simulation environments.

This platform consists of a virtually embodied agent, such as a robot, equipped with a suite of sensors that can observe the environment and take actions to alter the agent's state or the environment's state. Each sensor is associated with a specific agent and can return observation data from the environment at a specified frequency. The 3D environment includes a scene mesh, objects, agents, and sensors, organized into regions and objects through a hierarchical representation called the Scene, which can be programmatically manipulated. All components of the Scene are present on

unexploredarea

exploredarea

the SceneGraph. Additionally, a simulator backend instance can update the state of agents and SceneGraphs based on given actions for a set of configured agents and SceneGraphs, and it provides observations for all active sensors possessed by the agents. We use the Matterport3D (MP3D) dataset [15], which consists of high-quality renderings of indoor scenes. To measure exploration coverage, we compute the area revealed by the agent's line of sight using the function provided by the Habitat codebase [32]. As depicted in Figure 13, exploration is measured as the proportion of the environment revealed by the agent's line of sight over the course of the episode.

D.1.3 MiniGrid Environments

MiniGrid [17] is a set of procedurally-generated grid-worlds. In MiniGrid, the world is a partially observable grid of size N × N . Each tile in the grid contains exactly zero or one object. The possible object types are wall, door, key, ball, box, and goal. Each object in MiniGrid has an associated discrete color, which can be one of red, green, blue, purple, yellow, or grey. By default, walls are always grey and goal squares are always green. Rewards are sparse for all MiniGrid environments. There are seven actions in MiniGrid: turn left, turn right, move forward, pick up an object, drop an object, toggle and done. The agent can use the turn left and turn right action to rotate and face one of 4 possible directions (north, south, east, west).

The move forward action makes the agent move from its current tile onto the tile in the direction it is currently facing, provided there is nothing on that tile, or that the tile contains an open door. The agent can open doors if they are right in front of it by using the toggle action. Observations in MiniGrid are partial and egocentric. By default, the agent sees a square of 7 × 7 tiles in the direction it is facing. These include the tile the agent is standing on. The agent cannot see through walls or closed doors. The observations are provided as a tensor of shape 7 × 7 × 3 . However, note that these are not RGB images. Each tile is encoded using 3 integer values: one describing the type of object contained in the cell, one describing its color, and a flag indicating whether doors are open or closed. This compact encoding was chosen for space efficiency and to enable faster training. For all

中

tasks, the agent gets an egocentric view of its surroundings, consisting of 3 × 3 pixels. A neural network parameterized as a CNN is used to process the visual observation.

MultiRoom. The MultiRoomNXSY environment consists of X rooms, with a size at most Y, connected in random orientations. The agent is placed in the first room and must navigate to a green

goal square in the most distant room from the agent. The agent receives an egocentric view of its surroundings, consisting of 3 × 3 pixels. The task increases in difficulty with X and Y. Episodes finish with a positive reward when the agent reaches the green goal square. Otherwise, episodes are terminated with zero reward after a maximum of 20 × N steps.

KeyCorridor. the agent has to pick up an object which is behind a locked door. The key is hidden in another room, and the agent has to explore the environment to find it. Episodes finish with a positive reward when the agent picks up the ball behind the locked door or after a maximum of 270 steps.

ObstructedMaze. The agent has to pick up a box which is placed in a corner of a maze. The doors are locked, the keys are hidden in boxes and the doors are obstructed by balls. Episodes finish with a positive reward when the agent picks up the ball behind the door or after a maximum of 576 steps.

D.2 Algorithm Details

D.3 Computation Details

D.4 RL Hyperparameters

Table 8: The hyperparameters of the tested benchmark algorithms in the Atari experiment

D.5 Codebases Used

Our codebase was built atop the following codebases:

The official NovelD codebase: https://github.com/tianjunz/NovelD (Creative Commons Attribution-NonCommercial 4.0 license) for NovelID, RND, RIDE and count-based baselines (this codebase is built atop the official RIDE codebase below)

The official RIDE codebase: https://github.com/facebookresearch/impact-driven-exploration (Creative Commons Attribution-NonCommercial 4.0 license)

The official LIBERTY codebase: https://github.com/Mingle0228/liberty (MIT License)

The official E3B codebase: https://github.com/facebookresearch/e3b(Creative Commons Attribution-NonCommercial 4.0 license)

E Limitation

While our method has shown success in several challenging exploration tasks, it does not consistently achieve top performance in environments with less sparse rewards, as observed in the MuJoCo continuous control benchmarks. Additionally, our method has not been tested in reinforcement learning domains characterized by large action spaces. Addressing these limitations and developing a more generalized solution for diverse RL environments remains a goal for future work.

F Related Work

Exploration in RL. Exploration remains a long-standing problem in RL. Common approaches include ϵ -greedy [59], count-based exploration [7, 46, 61, 41, 40, 57], ensemble-based exploration [45, 39, 48] and curiosity-based exploration [53, 55, 56, 10]. Several exploration strategies use a dynamics model to provide intrinsic rewards [47, 11, 33, 48, 36, 2]. Latent variable dynamics have also been studied for exploration [4, 9, 62]. Maximum entropy in the state representation has also been used for exploration [54, 66]. Other intrinsic motivation methods have recently been developed for exploration in context MDPs [67, 28, 51, 69, 32], which automatically generate curricula over variations of the MDP to encourage efficient learning, effectively performing a form of curiositydriven exploration in the context space, including goal-conditioned [23, 50, 12, 18, 22] and goal-free variants [58, 49, 34, 19]. Reward shaping refers to modifying the original reward function with a shaping reward function which incorporates domain knowledge. Considering the most general form, namely the additive form, of reward shaping. The first approach to guarantee policy invariance is potential-based reward shaping (PBRS) [44], which defines the shaping reward function as the difference between values assessed through the potential function based on prior knowledge. There are lots of variants of PBRS [20, 38, 31, 3, 64, 30, 29, 63] which have shifted their focus to different areas within the field of reinforcement learning.

Metric-based Exploration Bonus. Metric space is widely used in encoding state representations [68, 14, 13, 16, 35]. Metric-based exploration bonus is based on the evaluation metric that quantifies the degree of difference or distance between two projected states for the measure of novelty. RIDE [51] evaluates the novelty as between two successive state representations under the L 2 norm. NovelD [69] uses the disparity in RND bonuses between adjacent states under the L 1 distance as the exploration bonus. LIBERTY [63] uses the difference between potential functions of adjacent states under the bisimulation metric space. Note that we propose an effective metric that measures the distribution distance between dynamics models by computing the distance between sampled subsequent states. Additionally, we integrate the Kullback-Leibler (KL) divergence between policy distributions to more robustly model the ' behavioral similarity ' between states, which is more effective and scalable across diverse environments with different observations, especially in addressing the critical Noisy-TV problem during exploration.

NeurIPS Paper Checklist

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?[Yes] In the introduction section (page 1).

Limitations Question: Does the paper discuss the limitations of the work performed by the authors?[Yes] In Appendix E.

Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?[Yes] In the Appendix B.

Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?[Yes] In Appendix D.

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?[Yes] In the anonymous link https://anonymous.4open.science/r/EME-4F8B/README.md.

Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?[Yes] In Appendix D.4.

Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?[Yes] In experiment section (page 5) and Appendix C.

Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of computing workers, memory, time of execution) needed to reproduce the experiments?[Yes] In Appendix D.3.

Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/ EthicsGuidelines ?[Yes]

Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?[Yes] In Appendix A.

Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?[NA]

Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited, and are the license and terms of use explicitly mentioned and properly respected? [Yes] In Appendix D.5.

New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?[NA]

Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? [NA]

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?[NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Previous exploration algorithms measure state discrepancy strictly on $L_p$ norms which is limiting since for environments where state differences are minimal in the observations the intrinsic reward is not useful. The authors propose an exploration bonus which is centred aroudn the variance of the predicted reward for a ensemble of reward models. A scaling factor for the exploration bonus is also introduced using the ensemble of reward models, where the variance in the prediction of the ground-truth reward is taken as the scaling factor. This scaling factor is higher in novel spaces since the prediction variance increases for states that have not been explored sufficiently.. The authors conduct extensive experiments on 2D and 3D environments shows superior performance to other known exploration methods.\n\nSTRENGTHS:\n- The paper is well written and structured, although a rough overview of the components of the exploration bonus at the beginning would have further helped my understanding.\n- The approach to consider the metric in which state discrepancy is measured is an interesting idea and seems fitting and quite novel. \n- The intrinsic reward is based on the discrepancy of an ensemble of reward models instead of the observations which is usually done, which I find an interesting idea.\n- The code is available and of high quality which should allow for reproducibility of the results\n- The authors motivate their method very well, highlighting why previous methods perform worse.\n\nWEAKNESSES:\n- To my knowledge other methods that try to aggregate states in high dimensions are not considered. Wouldn’t this mitigate or solve the issue with unique counts in high-dimensional spaces? \n- The maximum reward scaling $M$ seems to be an important parameter, however I do not see an ablation for it. The ablations in the appendix ablate the nature of the scaling factor, but not $M$\n- To my knowledge there is no talk about limitations of the method. Could you provide insight on where possible limitations to your method lie?\n\nQUESTIONS:\n- In practice, how do you train the ensemble of reward models in the reward free setting? Is there any relation to successor representations? What do you do in settings where the reward is not a good indicator for the environment structure? I'm a bit surprised that the method works well given there is no reward to learn the ensemble models. Does the KL-divergence of the policies take the overhand here?\n\nLIMITATIONS:\nThe authors have discussed limitations of previous methods but have not to my knowledge discussed the limitations of their own method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new exploration algorithm in reinforcement learning (RL), particularly in environments with sparse rewards. The authors critique existing exploration bonus methods using state discrepancy, highlighting their limitations in scalability and theoretical guarantees. They propose a novel method called Effective Metric-based Exploration-bonus (EME), which features a robust metric for evaluating state discrepancy, grounded in theoretical assurances. EME avoids approximation gaps found in previous bisimulation metric-based approaches and eliminates the reliance on episodic count scaling factors.  Furthermore, they introduce a diversity-enhanced scaling factor, dynamically adjusted by the variance of predictions from an ensemble of reward models, to improve exploration in tasks with subtle state differences. Experiments on Robosuite, Atari games, MiniGrid, and Habitat environments demonstrate the effectiveness and scalability of EME.\n\nSTRENGTHS:\n*Originality*: The EME metric presents a novel approach for measuring state discrepancies. The paper adequately cites related work and differentiates its contributions from existing methods.\n\n*Quality*: The paper provides theoretical analysis to support the claims regarding the limitations of current methods and the guarantees offered by the EME metric. The experimental evaluation is conducted across different domains, supporting the effectiveness of the proposed method.\n\n*Clarity*: The paper is generally well-written and organized. The motivation and background information are clearly presented, and the proposed method is explained in detail.\n\n*Significance*: The results shows good performance improvement over existing algorithms and show that EME could be become the default exploration algorithm for most problems.\n\nWEAKNESSES:\nWhile the presented results look impressive a more thorough empirical evaluation would be more helpful to assess the performance of EME:\n\n- On Atari the paper only evaluates on hard exploration games though it has been shown that evaluating on more Atari games is usually necessary to properly assess the performance of an algorithm. [1]\n- The paper does not include a ablation, because of that it is difficult to know if the improvement comes from the better exploration bonus and using multiple reward predictions\n\n[1] On Bonus-Based Exploration Methods in the Arcade Learning Environment, Ali Taiga et al. ICLR 2020\n\nQUESTIONS:\n- The explanation in lines 99-102 regarding the breaking of theoretical assurances is unclear. Can the authors elaborate on what specifically breaks down and how EME overcomes this issue?\n\n- On L130, the authors mention that existing methods might already be performing a similar function.  Can the authors clarify this statement and clearly differentiate EME from such methods?\n\n- Why is the Habitat environment considered a more realistic scenario given the variety of task one could attempt to solve with reinforcement learning?\n\n- The paper lacks citations for using an ensemble of models to drive exploration (L261), see for example [1, 2]\n\n- Is the feature encoder jointly learned with the EME metric? I couldn't find the information, though results in the appendix show that even a random feature encoder is providing good results\n\n- How does the number of reward models in the ensemble affect EME's performance and computational cost? Is there an optimal number, or are there diminishing returns as the ensemble size increases?\n\n\n[1] Deep Exploration via Bootstrapped DQN, Osband et al. \n\n[2] The Curse of Diversity in Ensemble-Based Exploration, Lin et al. ICLR 2024\n\nLIMITATIONS:\nThe authors acknowledged limitations regarding the performance in environments with less sparse rewards and untested performance in RL domains with large action spaces.\nAn ablation study of the different component of the algorithm as well as a more thorough evaluation on diverse environments would be helpful to properly evaluation the contribution presented in this paper.\nThe potential negative societal impact is also addressed in the broader impact statement", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Effective Metric-based Exploration-bonus (EME), a novel approach for enhancing exploration in reinforcement learning (RL) tasks. The paper identifies key limitations in existing metric-based exploration methods, such as their reliance on count-based scaling factors and approximation gaps in bisimulation metric learning. To address these issues, the paper proposes a more robust metric for evaluating state discrepancy that provides theoretical guarantees on the value difference bound. Additionally, they introduce a diversity-enhanced scaling factor based on the variance of predictions from an ensemble of reward models to improve exploration effectiveness, especially in hard exploration scenarios. The authors conduct extensive experiments across various challenging environments, including continuous control tasks, discrete-action games, and realistic indoor scenarios, demonstrating the superior performance and scalability of their method compared to prior approaches.\n\nSTRENGTHS:\n1. This paper includes a nicely-written and comprehensive analysis of the limitations in existing metric-based exploration methods; this really helps setting up the motivation for EME.\n\n2. Extensive experimental evaluation across a diverse set of environments, including continuous control, discrete-action games, and realistic indoor scenarios, demonstrating the effectiveness and scalability of EME.\n\n3. Introduction of a theoretically grounded metric for evaluating state discrepancy, backed by rigorous proofs and guarantees on the value difference bound.\n\n4. The reward-free exploration experiment is very nice because in many real-world scenarios, reward function may be hard to write down. \n\nOverall, this is a solid paper with good methodological contributions and convincing experiment results.\n\nWEAKNESSES:\n1. The performance of EME with feature encoder is mixed; this may hamper the scalability of EME to realistic scenarios, where pre-trained feature encoder is commonly used.\n\n2. Ablations are limited. It would be good to know how sensitive EME is to the choice of RL algorithm, the value of M, and other hyperparameters. Only scaling factor is studied right now. \n\n3. On the Robosuite environments, EME do not seem much better than the baselines. How do we characterize when we expect EME to do much better than existing methods?\n\nQUESTIONS:\nMy questions are stated above.\n\nLIMITATIONS:\nYes, the paper has adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the Effective Metric-based Exploration-bonus (EME) to address the limitations of existing state discrepancy methods by proposing a robust metric for evaluating state differences and a diversity-enhanced scaling factor for exploration bonuses. Extensive experiments demonstrate that EME outperforms other methods in various environments, including Atari games, Minigrid, Robosuite, and Habitat, highlighting its scalability and applicability to diverse scenarios.\n\nSTRENGTHS:\nOriginality:\n\nEME has not been proposed before. It appears to be effective.\n\nQuality:\n\nThe paper is supported by thorough theoretical underpinnings and extensive experimental validation across many environments.\n\nClarity:\n\nThe paper is well-written and clearly structured.\n\nSignificance:\n\nExploration is a core challenge in RL. This paper makes a significant contribution to it.\n\nWEAKNESSES:\nThe ultimate method seems incredibly complex, making it difficult for future practitioners to reproduce. Emphasizing Algorithm 1 (potentially moving it from the Appendix into the main paper) could help with this.\n\nThe paper does not provide a thorough discussion of the computational efficiency and potential overheads introduced by the EME method. It would be useful to provide a section discussing this.\n\nThe paper introduces several novel components, such as the diversity-enhanced scaling factor, but lacks detailed ablation studies that isolate the contributions of these individual components.\n\nThe authors repeatedly refer to \"a real-life indoor environment\". While somewhat true (as it is later revealed to be the Habitat environment), it is rather misleading, as the experiments are still entirely done in the computer. (ie there is no real-world robot involved).\n\nQUESTIONS:\n1. Can you provide more details on the computational complexity and runtime performance of the EME method compared to existing approaches? Specifically, how does EME scale with the size of the state space and the complexity of the environment?\n\n2. Could you provide detailed ablation studies to isolate the impact of the diversity-enhanced scaling factor and other key components of EME?\n\n3. Some parts of the theoretical analysis are dense. Could you provide more intuitive explanations or examples to help bridge the gap between the theory and its practical implications?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-21"}