Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus

Authors: yiming  wang
kaiyan  zhao
furui  liu
leong  hou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:23:46

Abstract: Enhancing exploration in reinforcement learning (RL) through the incorporation of intrinsic rewards, specifically by leveraging *state discrepancy* measures within various metric spaces as exploration bonuses, has emerged as a prevalent strategy to encourage agents to visit novel states. The critical factor lies in how to quantify the difference between adjacent states as *novelty* for promoting effective exploration.
Nonetheless, existing methods that evaluate state discrepancy in the latent space under $L_1$ or $L_2$ norm often depend on count-based episodic terms as scaling factors for exploration bonuses, significantly limiting their scalability. Additionally, methods that utilize the bisimulation metric for evaluating state discrepancies face a theory-practice gap due to improper approximations in metric learning, particularly struggling with *hard exploration* tasks. To overcome these challenges, we introduce the **E**ffective **M**etric-based **E**xploration-bonus (EME). EME critically examines and addresses the inherent limitations and approximation inaccuracies of current metric-based state discrepancy methods for exploration, proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to be dynamically adjusted by the variance of prediction from an ensemble of reward models, thereby enhancing exploration effectiveness in particularly challenging scenarios. 
Extensive experiments are conducted on hard exploration tasks within Atari games, Minigrid, Robosuite, and Habitat, which illustrate our method's scalability to various scenarios. The project website can be found at https://sites.google.com/view/effective-metric-exploration.

Main paper: 
 introduction: 
 Reinforcement learning (RL) has made significant strides, yielding breakthroughs across various domains, including video gaming [43] , autonomous driving [65] , and robotic control [1, 37] . However, for many real-world tasks, defining a dense reward function is non-trivial, yet a sparse reward function based on success or failure is directly available, which makes learning effective policies difficult, as they demand efficient exploration of the state space, highlighting exploration in sparse-reward environments as a core challenge in RL [59] .Methods that quantify the state discrepancy between adjacent steps using specific measures within different metric spaces have shown remarkable success in tasks characterized by sparse rewards. These methods, by leveraging a measure of state discrepancy as an exploration bonus, facilitate agents in discovering novel states. For instance, RIDE [51] employs the difference between two consecutive state embeddings, measured under L 2 norm, as an exploration bonus. Similarly, NovelD [69] introduces a bonus based on the state discrepancy, as represented by the RND [11] bonus, but under the L 1 norm. However, the effectiveness of these L p norms-based methods is contingent on the scaling factor expressed as the episodic count term N ep (s), which is the number of times state s has been visited during the current episode. The episodic term becomes ineffective when each state is unique and cannot be counted, posting a significant limitation when it comes to more complex, dynamic, and noisy environments [32] . LIBERTY [63] aims to surmount this hurdle by evaluating state discrepancy through the bisimulation metric [25] , which links state differences to value differences within the bisimulation metric space, thus incentivizing exploration of novel states with greater value differences and substantially enhancing learning efficiency. Nevertheless, due to the high computational cost or infeasibility of accurately computing bisimulation metrics, the approximation, and relaxations over the metric [13, 68, 63] have been used to optimize efficiency. Our analysis reveals that approximation gap in LIBERTY might break the theoretical guarantees of the bisimulation metric, potentially undermining exploration performance. Furthermore, the efficacy of bonuses based on state discrepancies declines in hard exploration tasks or scenarios where state differences are minimal, such as in the "Noisy-TV" [11] problem and vision-based real indoor environments [60] , posing a significant constraint on the scalability of these methods. To address this aforementioned limitations, we introduce the Effective Metric-based Explorationbonus (EME) for exploration. Our method features a more resilient metric for evaluating state differences, simultaneously eliminating approximation gap with a theoretical guarantee. EME significantly boosts learning efficiency by forging a closer connection to the value differences between states, while maintaining fidelity to its theoretical framework. Furthermore, we have developed a diversity-enhanced scaling factor to augment exploration efficacy in hard exploration tasks, where state differences are notably subtle. Our scaling factor dynamically adjusts the exploration bonus based on the variance of predictions from an ensemble of reward models, which is higher when agents encounter novel state space, thereby encouraging more effective exploration. As illustrated in Figure 1 , we present the trajectories of policies trained using count-based state discrepancy bonus, bisimulation-based exploration bonus, and our method within a real-life indoor environment [15] (Full results in Figure 10 of Appendix C.3). Our method successfully explores a significantly larger portion of the space compared to the others, demonstrating the superior effectiveness of EME.The main contributions of this paper are as follows. Firstly, we conduct a comprehensive analysis of the limitations inherent in current metric-based state discrepancy methods for exploration. Based on the analysis, we introduce an effective metric for evaluating the behavioral similarity between states supported by theoretical assurances. Secondly, we propose a diversity-enhanced scaling factor for exploration bonuses based on the variance of predictions from an ensemble of reward models, which is scalable and effective in hard exploration tasks. Lastly, extensive experiments are conducted in Robosuite, Atari games, MiniGrid, and Habitat. The results demonstrate that our algorithm can effectively enhance exploration and accelerate training across various environments. 
background: 
 We focus on exploration bonuses to incentivize exploration in reinforcement learning (RL). To foster the reader's understanding, we first introduce standard notation and common practices.Markov Decision Processes. We assume the underlying environment is a Markov decision process (MDP), defined by the tuple M = (S, A, P, R, γ), where S is the state space, A is the action space, P (s ′ | s, a) is state transition function from state s ∈ S to state s ′ ∈ S, R is the reward function and γ ∈ [0, 1) is the discount factor. Generally, the policy of an agent in an MDP is a mapping π : S × A → [0, 1]. An agent chooses actions a ∈ A according to a policy function a ∼ π(s), which updates the system state s ′ ∼ P (s, a) yielding a reward r = R(s, a). In this paper, we denote a policy by π θ , where θ is the parameter of the policy function. The goal of the agent is to optimize the parameter θ for maximizing the expected accumulative rewards,J(π θ ) = E π θ [ ∞ t=0 γ t R (s t , a t )].State Discrepancy as Exploration Bonuses. Addressing the sparse reward challenge prevalent in many environments necessitates augmenting the external reward function, r e , with an intrinsic reward bonus, b, resulting in a combined reward: r = r e + b. A number of intrinsic bonuses2 that encourage exploration have been proposed. More recent methodologies employ an exploration bonus that varies based on the novelty of the current state relative to previously visited states, evaluated within different metric spaces, which can be generally defined as:EQUATIONwhere SD denotes the state discrepancy, SF denotes the scaling factor, d : S × S → R represents the evaluation metric that quantifies the degree of difference or distance between two projected states, and Φ : S → Z is the projection function which maps the state to a representation space Z.Algorithm Objective = State Discrepancy * Scaling Factor Episodic Approximation Gap Scalability RIDE [51] ∥ϕ(s ′ ) -ϕ(s)∥2 * Nep(s ′ ) -1 NovelD [69] ∥r RND (s ′ ) -α • r RND (s)∥+ * I[Nep(s ′ ) = 1] LIBERTY [63] [γdinv(s ′ , s0) -dinv(s, s0)] * λ EME (ours)dE(s ′ , s) * min{max{ζ(rs), 1}, M } Table 1 : Comparison of exploration methods using different measures (marked in blue) of state discrepancy as exploration bonus. ϕ = f (s) : features encoder; Nep(s) : episodic (pseudo) count of visits to state s; r RND (s): the RND [11] bonus; α: normalized coefficient; dinv: inverse dynamic bisimulation metric used in [63] ; s0: initial state; λ: the scaling hyper-parameter; dE: our proposed effective metric for state discrepancy evaluation; ζ(rs): variance of predictions from an ensemble of reward models: M : maximum reward scaling.The comparison of recent metric-based exploration methods is summarized in Table 1 These bonuses reward high dissimilarity between adjacent states, encouraging exploration by leveraging measures such as the L 2 distance for state embeddings (as used by RIDE [51] ) and the L 1 distance for the disparity in RND bonuses between adjacent states (as utilized by NovelD [69] ). However, these approaches, grounded in L p norms and dependent on episodic state visit counts, face significant scalability challenges when exploring various environments. On the other hand, LIBERTY [63] proposes the inverse dynamic bisimulation metric to evaluate the state difference, which links state and value differences to enhance learning efficiency. Yet, the approximation inaccuracies inherent in bisimulation metric learning compromise theoretical assurances, breaking theoretical integrity of bisimulation metric. In the following sections, we detail the shortcomings of these methods. 
limitations of existing metric-based exploration bonus: 
 3.1 Latent Vector Space under L p Norms L p norms such as L 1 and L 2 norms are widely used in measuring the "distance" between states or embeddings. For example, considering a continuous map between the states of an MDP and a latent vector space: Φ : S → Z ⊆ R n . We can equip this vector space with a Euclidean norm to obtain a Euclidean metric space (Z, ∥ • ∥ 2 ), and the state discrepancy is measured asSD = ∥Φ(s) -Φ(s ′ )∥ 2 .Reliance on Count-based Scaling Factor. The state-difference based exploration bonus employed by RIDE [51] and Noveld [69] can be regarded as the difference between adjacent states in the L p norm space. The efficacy of the exploration bonus heavily lies in the episodic scaling factor (SF) measured by the count of visited states N ep (s) presented in Table 1 . However, the count-based episodic bonus suffers from a fundamental limitation, which is similar to that faced by count-based approaches [5] in general: if each state is unique, then N ep (s t ) will always be 1 and the episodic bonus is no longer meaningful, which is the case for many real-world applications. For example, a household robot's state as recorded by its camera might include moving trees outside the window, clocks showing the time, or images on a television screen which are not relevant to its tasks, but nevertheless make each state unique. As shown in Table 2 , without the episodic count term (denoted as "w/o EP"), the performance of RIDE and Noveld exhibits a significant decline in all different types of environments.Limited Expressiveness of Novelty. As indicated in Table 2 , the performance of RIDE and NovelD significantly degrades in environments with high-dimensional visual observations or continuous control tasks, where the success rate falls below 0.5. The significant decline highlights the inadequacies of utilizing state differences within vector spaces, computed using the L p norm, to effectively express novelty. Therefore, a more robust and effective metric is essential to accurately assess and characterize the novelty among visited states.The bisimulation metric [25, 26] defines a pseudometric d : S × S → R to measure the similarity between two states. The recently proposed variant, π-bisimulation metric [13] , focuses on behaviors relative to a particular policy π, which consists of a reward difference term and a Wasserstein distance in dynamics models between states. Definition 1 (π-bisimulation metric). Given a fixed policy π, the following π-bisimulation metric exists and is unique:EQUATIONwhereE ai∼π r ai si = E ai∼π(•|si) r(s i , a i ), P π (• | s i ) = E a∼π(•|si) P (• | s i , a) and W 1 is the 1- Wasserstein distance.Approximation Gap. LIBERTY [63] proposes exploration bonuses based on state discrepancy within the inverse dynamic bisimulation metric space, which adds the difference between action output by the inverse dynamic model based on the bisimulation metric. The difference between states evaluated under the bisimulation metric is directly associated with the value difference (see Theorem 3.3 in [26] and Theorem 2 in [63] for detailed proof). The loss function of LIBERTY metric learning is:EQUATION(3) where s denotes state with stop gradients, P π si (η) indicates probabilistic dynamics model parameterized with η and I si+1 si (θ) is the inverse dynamic model parameterized with θ. As the 1-Wasserstein distance is usually difficult to estimate, LIBERTY and prior methods [68] propose to use 2-Wasserstein distance W 2 to replace W 1 , as W 2 has a convenient closed-form of a Gaussian distribution with respect to the L 2 distance. Proposition 1 (Relaxation Divergence). Relaxing the W 1 metric to W 2 breaks the theoretical integrity of the inverse dynamic bisimulation metric [63] in scenarios where the transition dynamic model P (s, a) or the policy π is stochastic.Proof in Appendix B. Based on Proposition 1, we can see that the relaxation divergence of 1-Wasserstein distance between dynamic models can only be ignored when both the transition dynamics model and policy π are deterministic. However, this assumption may be too strong to hold in practice.Besides, the metric learning process encounters a theory-practice gap due to the relaxation of reward expectations. Specifically, computing the expected reward differences (first term of Equation ( 2)), |E ai∼π r ai si -E aj ∼π r aj sj |, proves to be computationally daunting and challenging to accurately estimate, even with sampling techniques. Following other bisimulation metric-based methods [13, 68] , LIBERTY [63] shifts the expectation operator outside the absolute value of reward differences (second term of Equation ( 3)) and differences between action outputs predicted by inverse dynamic models, as in E ai∼π,aj ∼π [|r ai si -raj sj | + |I si+1 si -I sj+1 sj|], which facilitates more efficient sampling by avoiding the direct estimation of reward and action expectations. Proposition 2 (Shifted LIBERTY Distance). The relaxation of expectation during learning process shifts the original LIBERTY distance and introduces a looser value difference bound.We extend the concept of the shifted MICo distance [14] introduced in [35, 16] . The proof is in appendix B. Based on Proposition 2, as the shifted LIBERTY distance has a looser value difference bound, it may be less relevant to the value function. As a result, the learned metric may not be able to capture the state similarity within bisimulation metric space accurately. , as the shifted LIBERTY distance has a looser value difference bound, it may be less relevant to the value function. As a result, the learned metric may not be able to capture the state similarity within bisimulation metric space accurately.Scalability Constraints. As shown in Table 2 , LIBERTY's performance significantly drops in hard exploration tasks within MiniGrid and in the realistic scenarios of the Habitat environment [60] . Specifically, in Habitat, agents must navigate through photorealistic simulations of actual indoor spaces. In these settings, the subtle variations between states diminish the impact of the exploration bonus, significantly lowering the efficiency of exploration. This limitation becomes even more apparent in the procedurally-generated MiniGrid environment, where the inherent stochasticity further impedes LIBERTY's performance. Those findings highlight the scalability challenges that bisimulation metric-based methods encounter in complex environments. 
effective metric-based exploration bonus: 
 In this section, we introduce the Effective Metric-based Exploration-bonus (EME), conceived to overcome the limitations identified in existing methods that use metric-based state discrepancy as exploration bonuses. Our development is driven by two primary objectives. Firstly, we aim to refine the metric learning process to achieve a more effective exploration without incurring the approximation gap, thereby upholding the theoretical integrity of value function bound. Secondly, we seek to obviate the reliance on scaling factor of episodic counts, ensuring effective exploration across different environments. To this end, we propose a novel metric that more precisely evaluates the behavioral similarity between states, establishing a more effective exploration strategy compared with previous approaches. Furthermore, we introduce an innovative approach that utilizes the variance between outputs of an ensemble of reward models in metric learning as a dynamically-adjusted scaling factor, which significantly enhances exploration efficiency, especially in hard exploration tasks, thereby improving scalability. The next two sections describe our method in detail.In order to avoid the approximation gap introduced by relaxation of Wasserstein distance and reward expectations mentioned in Proposition 1 and 2, along with the reliance on the episodic count, we eliminate the calculation of the Wasserstein distance, and propose the EME metric: Definition 2 (EME Distance Function). Let met be the space of bounded pseudo-metrics on state space S, the EME metric d E : S × S → R, the EME distance function F(d E , π) : met → met is defined as:F(d E , π)(s i , s j ) = |E ai∼π r ai si -E aj ∼π r aj sj | + γE ai∼π aj ∼π d E (s ′ i , s ′ j ) + γD KL (π(•|s i )∥π(•|s j )) (4)whereD KL (•∥•) represents the Kullback-Leibler (KL) divergence.Behavioral Similarity Between States. We strictly calculate the expectation of reward difference without relaxation. The EME metric measures the distribution distance between dynamics models by computing the distance between sampled subsequent states following representation learning method [14] to avoid the computation of the Wasserstein distance. Additionally, we integrate the Kullback-Leibler (KL) divergence between policy distributions to more robustly model the "behavioral similarity" between states, which is more effective and scalable across diverse environments with different observations, especially in addressing the critical "Noisy-TV" problem [11] during exploration. The "Noisy-TV" problem started as a thought experiment in [11] and is commonly discussed in exploration literature [51, 52, 42] . Imagine that an RL agent is rewarded with seeking the novel experience, a TV with unpredictable random noise outputs would be able to attract the agent's attention forever. The agent obtains new rewards from state discrepancy caused by noisy TV consistently, but it fails to make any meaningful action. To counter this, we utilize differences in policy distributions across states, promoting the exploration of diverse actions rather than repetitive behavior.Theorem 1. The EME distance function F(d E , π) : met → met has a unique fixed-point dE .Proof in Appendix B. Theorem 1 provides a convergence guarantee for the EME distance that by iterating F(d E , π), distance will converge to the fixed-point dE .Theorem 2. (Guaranteed Value difference bound) Given the EME metric d E , states s i and s j , and a policy π, we haveEQUATIONImproved Exploration Efficiency. The proof can be found in Appendix B. Theorem 2 demonstrates that the EME distance between states provides an upper bound on the difference in their state values.Let's recap the definition of one-step TD error:δ t = r t+1 + γV (s t+1 ) -V (s t ).Intuitively, if there is a large value difference between states s t and s t+1 , the corresponding exploration bonus b which is calculated using d E (s t , s t+1 ) will also be large. Consequently, the total reward for the next step, r t+1 = r e + b (where r e represents the external environment reward), increases accordingly. The increase of reward leads to a larger TD error when there is a significant value difference between adjacent states, incentivizing the agent to prioritize transitions with large TD errors, which not only enhances the agent's exploration capabilities but also significantly boosts training efficiency.Based on Equation ( 4), between any pair of states s i and s j , we can define the loss function of the EME metric learning as:EQUATIONwhere d ϕ E is the EME metric encoder parameterized by ϕ. The last three terms in the loss function ( 6) can be regarded as the regression of target metric, which approximates the difference between rewards, distances of next states and distances of policy distributions, respectively. However, the reward expectations E ai∼π r ai si and E aj ∼π r aj sj are computationally intractable and also difficult to estimate even based on sampling [13] . And the relaxation of reward expectations E ai∼π aj ∼π|r ai si -r aj sj |used by previous methods [63, 68, 13] will lead to the learned metric having a looser value difference bound than the original bisimulation metric as proved in Proposition 2. Inspired by metric-based representation learning methods [35, 16] , we propose to use an ensemble of reward models to approximate the reward difference more accurately.Proposition 3. Let r s be a random variable over the action distribution defined by p(r s = r a s ) = π(a|s), var(r si ) denote the variance of variable r si , we can have:EQUATIONProof. (Sketch) The proof is based on [14, 16] by expanding E ai∼π aj ∼π[|r ai si -r aj sj | 2 ] -|E ai∼π r ai si - E aj ∼π r aj sj | 2 . See detailed proof in Appendix B.Ensemble of Reward Models. We train an ensemble of reward models {g(η 1 ), . . . , g(η k )} to predict the reward from the state-action pairs sampling from the buffer D τ : g(s, a; η) : S × A → R by minimizing the prediction error ∥g(s t , a t ; η) -r t+1 ∥ 2 . Consequently, the variance of reward across the output of different models in the ensemble is defined as follows:EQUATIONBased on Equation ( 6), Equation (7) and Equation ( 8), the tractable EME loss without approximation gap can be defined as:EQUATIONDiversity-Enhanced Scaling Factor. The scaling factor of previous methods is either not scalable [51, 69, 2] or hand-crafted hyper-parameter [63] , in light of this, we propose a scalable diversity-enhanced scaling factor for the intrinsic reward to further improve the efficacy of exploration. Let's recap the ensemble of reward models {g(η 1 ), . . . , g(η k )} used in metric training (9) . To maintain the diversity across the individual models g(η), we initialize each model's parameters differently and train each of them on a subset of data randomly sampled with replacement. Each model in our ensemble is trained to predict the ground truth reward. Hence, the parts of reward obtained within the state space that have been well explored by the agent will have gathered enough data to train all models, resulting in a low variance of reward predictions from the models, when generalizing to unseen but similar parts of the reward from unvisited state-space, the areas which are novel and unexplored would still have high prediction error for all models as none of them are yet trained on such examples, resulting in higher variance of the reward prediction. Therefore, we use the variance of reward predictions as the scaling factor of intrinsic reward to encourage exploration on unvisited state space. With the maximum reward scaling M , the exploration bonus is defined as:EQUATION) As a result, the agent receives greater rewards when encountering novel states during training, which facilitates more effective exploration, the ablation study on scaling factor and visualization of exploration bonus is in Appendix C.5. See Algorithm 1 of Appendix D.2 for detailed description. 
experiments: 
 The overall objective of our experiment is to evaluate the performance of EME in comparison to other baselines, we conduct comprehensive experiments on various settings of continuous control tasks, discrete-action hard exploration games, and real indoor environments 3 to assess the effectiveness and scalability of our algorithm. The implementation details can be found in Appendix D.Baselines. We compare it against following competitive baseline methods. ICM [47] : a famous curiosity-driven method. RND [11] : intrinsic rewards are the prediction errors of the distillation network. E3B [32] : proposing count-based episodic bonuses under continuous state spaces. RIDE [51] : using state difference under latent space as exploration bonuses. NovelD [69] : using the state difference measured by RND bonus as intrinsic rewards. LIBERTY [63] : using the state discrepancy evaluated under the bisimulation metric space as shaping reward. In our continuous control experiments conducted on the Robosuite platform [70] , we assess the exploration capabilities of agents across various challenging tasks, which include Door Opening, )). The encoder may include the inverse dynamic model [47, 51, 63] which isolates environmental factors that do not affect the agent's behavior, the bisimulation-based encoder [68, 35, 14, 13] , and random embeddings. The results can be found in Figure 8 and 9 in Appendix C.2, the performance of EME under the inverse dynamic encoder and bisimulation-based encoder exhibits a decline, because the exploration bonus under these encoders get close to zero quickly since their representations are very compact and every state looks more similar with the convergence of the encoder, which harms the efficacy of exploration.With respect to random embedding, the variance is larger, resulting in more unstable performance. Atari Games. To assess EME in scenarios involving pixel-based observations and discrete actions, we evaluate our method on the hard exploration games identified by [6, 69, 63] within the common Atari benchmark [8] , including Asterix, Alien, Breakout, BeamRider, Mspacman, PrivateEye, Pong, SpaceInvaders, Qbert, and UpNDown. The results are presented in Figure 3 . Our method consistently achieves similar or higher average returns compared to other baselines across all tested tasks. Notably, EME demonstrates the fastest convergence, particularly in Asterix and UpNDown, highlighting our method's superior learning efficiency. While LIBERTY and NovelD exhibit competitive performance in 6 games, their effectiveness significantly diminishes in the challenging games, PrivateEye and UpNDown. As for other methods like RIDE, E3B, ICM, and RND struggle across all tasks, primarily due to inadequate exploration capabilities. To isolate the impact of our proposed EME metric, we also include the performance of RIDE and LIBERTY with Diversity-enhanced Scaling Factor, denoted as RIDE and LIBERTY with DSF. EME still achieves the best performance which demostrate the superiority of EME metric.MiniGrid Environments. MiniGrid [17] presents a series of procedurally-generated, challenging environments. We focus on three settings from MiniGrid: Multi-Room (MR), Key Corridor (KC), and Obstructed Maze (OM). For example, MRN12S10-NT stands for MultiRoom-N12-S10-NoisyTV. Details on the specific environmental settings are available in Appendix D.1.2. Table 3 presents the testing performance of EME and all baseline methods across five different seeds. EME successfully solves all hard-level exploration environments within MiniGrid, achieving the best performance in 10 out of 12 settings. While Multi-Room environments are relatively easy, all baselines except for ICM and RND, demonstrate competency. However, as room size and number increase, the complexity of the tasks also rises, with EME consistently outperforming other baselines, a trend that extends to the Key-Corridor environments. In the particularly challenging Obstructed Maze environments, where obstructions block doors, our agent also excels by learning to remove these obstructions, further demonstrating the effectiveness and scalability of our approach.Noisy TV Problem. In addition to standard MiniGrid tasks, we also tested the model's ability to deal with stochasticity in the environment by adding a manually-made Noisy TV setting introduced in [51] , where some blocks change color at every time step. As illustrated in Table 3 , EME maintains strong performance even under these conditions. Notably, EME is the only method that successfully completes the KCS5R3-NT task, underscoring the robustness of our method. To test our model's applicability to real-world environments, we investigate hard exploration tasks in Habitat [60] . Habitat is a platform for embodied AI research which provides an interface Reward-free Exploration. we investigate reward-free exploration by evaluating the performance of all baselines using exploration bonus-only. Figure 10 displays the trajectories for all baselines on one of the test maps, clearly demonstrating that EME explores a significantly larger portion of the space compared to other methods. The full experiment results can be found in Appendix C.3. 
conclusion: 
 In this work, we identify the limitations of existing state discrepancy-based exploration methods: the reliance on episodic count-based scaling factor and theory-practice approximation gap, which leads to limited scalability especially for hard exploration tasks within realistic environments. To address the issues, we propose the Effective Metric-based Exploration-bonus (EME), which addresses the inherent limitations by proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to enhance exploration effectiveness in particularly challenging scenarios. Extensive experiments on hard exploration tasks from continuous control, discrete-action games and realistic environments have demonstrated the effectiveness and scalability of our method. 
 
 Appendix: This work introduces a reinforcement learning (RL) exploration method by introducing an effective metric-based exploration bonus, a versatile framework applicable to a wide array of decision-making problems, which can be applicable to diverse scenarios, including autonomous driving, household robotics, video gaming, online recommendation or advertisement optimization and etc. Similar to other RL exploration algorithms, our method is designed to facilitate the learning of a policy that maximizes a reward function defined by the designer. The implementation of such a policy, depending on the objectives set by the reward function designer, could lead to either positive or negative outcomes. Additionally, our approach significantly reduces the burden of reward tuning for researchers and algorithm engineers, and concurrently offers insights into the quality of the designed rewards through the shaping weights derived from our methods.Proposition 1 (Relaxation Divergence). Relaxing the W 1 metric to W 2 breaks the theoretical integrity of the inverse dynamic bisimulation metric [63] in scenarios where the transition dynamic model P (s, a) or the policy π is stochastic.Proof. First, let's recall the proof of the existence of a unique fixed-point in the inverse dynamic bisimulation metric:Lemma 1 ([63]). Given a fixed policy π. Define H π : met → met by H(d, π)(s i , s j ) = |r π si -r π sj | + γW (d)(P π si , P π sj ) + ∥I π (• | s i , s i+1 ) -I π (• | s j , s j+1 )∥ 1 , then H π has a least fixed point d π .Proof. This proof mimics the proof of Theorem 4.5 from [24] . We make use of the same pointwise ordering on met: d ≤ d ′ iff d(s, t) ≤ d ′ (s, t) for all s, t ∈ S, which gives us an ω-cpo with bottom ⊥, which is the everywhere-zero metric. Since Lemma 4.4 from [24] (Wasserstein metric W is continuous) also applies in our definition, it only remains to show that H(d, π) is continuous:EQUATIONThe rest of the proof follows in the same way as in [24] .In equation (11) , the existence of a unique fixed-point in the bisimulation metric requires the continuity and monotonicity of W 1 with respect to d. The properties of continuity and monotonicity do not hold with W 2 . Therefore there is no more guarantee about the fixed-point existence in LIBERTY and other approximation-based methods [68] except that both the dynamics model and the policy π are deterministic, in which case W 2 (d) degenerates to d [24] and Banach's fixed-point exists [35] . As for scenarios when the transition model and policy are stochastic, the theoretical guarantee of LIBERTY is broken.Proposition 2 (Shifted LIBERTY Distance). The relaxation of expectation during learning process shifts the original LIBERTY distance and introduces a looser value difference bound.Proof. The proof is based on top of [14, 16] . LemmaBy combining Equation ( 13) and Equation ( 14), we can have: E ai∼π,EQUATIONDefine the MDP for RL by a tuple ⟨S, A, R, P, γ⟩. We consider a lifted MDP constructed by a tuple ⟨ Ŝ, Â, R, P, γ⟩, where state space Ŝ = S × S, action space Â = A × A, transition distribution P(ai,aj) (si,sj ) = P ai si P aj sj , and reward function R((s i , s j )) = |E ai∼π r ai si -E aj ∼π r aj sj |. The Bellman operator T π under policy π(a i , a j |s i , s j ) = π(a i |s i )π(a j |s j ) is:EQUATION)So we can construct the lifted MDP of shift LIBERTY by a tuple ⟨ Ŝ, Â, R, P, γ⟩ which is the same as the lifted MDP in Definition 3 except for the reward function R((s i , s j ), (a i , a j )) = |r ai si -r aj sj | and shifted metric d. The Bellman operator T π under policy π is:EQUATIONAs proven in LIBERTY [63] , for the inverse dynamic bisimulation metric d, we can getEQUATIONAs proved in [63, 24, 27] , d π is the value function of lifted MDP ⟨ Ŝ, Â, R, P, γ⟩. d π can be expanded as the sum of discounted future rewards,EQUATION)dπ can be expanded as,EQUATION)with sEQUATIONj = s j , The difference between d and d can be regarded asEQUATION) As proved in Lemma 2, we can get:EQUATION)Combined with Equation ( 21), we can have:EQUATIONSo the shifted LIBERTY distance introduces a looser value bound.Proposition 3. Let r s be a random variable over the action distribution defined by p(r s = r a s ) = π(a|s), var(r si ) denote the variance of variable r si , we can have:EQUATIONProof. The proof is based on [14, 16] by expanding the difference between E ai∼π,aj ∼π r ai si -r aj sj2 and E ai∼π r ai si -E aj ∼π r aj sj2 .E ai∼π,aj ∼π r ai si -r aj Proof. Let d E , d ′ E ∈ M. We haveEQUATIONTherefore, F is a contraction mapping w.r.t. the L ∞ norm and there exists a unique fixed-point for F due to Banach's fixed-point theorem. This completes the proof.Theorem 2. (Guaranteed Value difference bound) Given the EME metric d E , states s i and s j , and a policy π, we haveEQUATION)Proof. The proof mimics [14] . We follow the assumption thatEQUATIONAssuming that we have an initial distance d 0 which |V π (s i ) -V π (s j )| ≤ d 0 (s i , s j ), and base on Theorem 1, F(d, π) is contraction mapping on d. By repeatedly applying F(d, π) on d, d will eventually converge to the fixed-point d E , the fixed point d E satisfies:EQUATIONTo isolate the impact of different hyper-parameters, we carry additional experiments on the ensemble size of reward models and the max reward scaling M . First, M sets an upper limit on the bonus. We set M = 10as the default setting. As we can see from Figure 6 , if M = 1, the scaling factor is fixed to 1, resulting in a significant performance decline. Higher M encourages more extensive exploration.The performance with M = 5 slightly lags behind the default setting. The performance with M = 20 and M = 40 is comparable and almost the same, indicating that the performance stabilizes as M increases. Practically, the value of M can be adjusted depending on the specific task and environment to determine the intensity of exploration. As shown in Figure 7 , we observe that as the ensemble size (ES) increases, the performance of ES = 6 and ES = 9 surpasses that of ES = 3. The performance of ES = 6 and ES = 9 is comparable, with no significant difference. However, when the size is further increased to ES = 12, there is a decline in performance, particularly in the Table Wiping and Pick and Place tasks. The performance change can be analyzed from two perspectives. First, the number of reward models is related to the accuracy of the reward variance prediction used to calculate the loss function. As the ensemble size increases, the overall prediction error decreases because the models can average out individual errors more effectively. This leads to a more accurate approximation of the variance ζ(s t ). Second, with a larger ensemble size, the variance of reward predictions decreases, resulting in a lower scale of the exploration bonus. Thus, there is a balance or trade-off between estimation accuracy and exploration, explaining why the performance of ES = 12 lags behind ES = 9. The optimal number of ensemble models may vary depending on the specific tasks and environments. Regarding computational cost, it increases with the ensemble size. Therefore, we set ES = 6 as our default setting, where the performance is nearly the same as ES = 9 but with lower computational overhead. It is also noteworthy that even with an ensemble size of 3, EME still outperforms the best baseline methods, further demonstrating the robustness of EME.Our method EME can be integrated with any state representation derived from different state encoders Φ(•), expressed as d E (Φ(s t ), Φ(s t+1 )). The encoder may include the inverse dynamic model [47, 51, 63] which isolates environmental factors that do not affect the agent's behavior, the bisimulation-based encoder [68, 35, 14, 13] which learns a compact state representation grouped by the bisimulation metric, and random embeddings captured by random embedded states. We denote the EME variants as follows: EME with an inverse dynamic model-based encoder is referred to as EME-IDM; with a bisimulation metric-based encoder as EME-BM; and with a random embedding as EME-Random. The results of continuous control tasks and hard exploration atari games can be found in Figure 8 and Figure 9 , respectively. EME's performance declines due to the rapid reduction of the exploration bonus. The performance of EME under the inverse dynamic encoder and bisimulation-based encoder exhibits a decline, because the exploration bonus under these encoders gets close to zero quickly since their representations are very compact and every state looks more similar with the convergence of the encoder, which harms the efficacy of exploration. With respect to random embedding, the variance is larger, resulting in more unstable performance. We examine reward-free exploration by assessing the performance of all baselines within Habitat using only exploration bonuses. During training, agents are initialized in varied environments for each episode and are subsequently tested on a set of environments not encountered during training. Figure 10 displays the trajectories of all baselines on one of the test maps, highlighting that EME explores a substantial portion of the space, unlike other methods. Specifically, ICM and RND agents remain confined to a single room, while RIDE, LIBERTY, and NovelD agents explore two rooms before becoming stuck, distant from the goal. E3B exhibits the second-best exploration coverage of the map. Quantitative results from other reward-free tasks are shown in Figure 11 , where EME consistently demonstrates robust performance and achieves the best outcomes. In contrast, other methods experience significant declines, underscoring the efficacy of our approach in a reward-free setting. We present results from the delayed reward setting in the MuJoCo environment [21] , following the methodology outlined in [63] , where accumulated rewards are delayed every 10, 20, and 30 steps. The experiments assess six tasks: HalfCheetah, Hopper, Walker2d, Ant, Swimmer, and Humanoid. As shown in Table 4 , our method achieves the best or second-best performance in 15 out of 18 delayed reward tasks. This indicates that EME can facilitate effective exploration and maintain high performance even under sparse reward conditions. Notably, as rewards become sparser, EME's performance improves and becomes more robust. In contrast, other metric-based exploration bonus methods such as RIDE and NovelD struggle due to their less expressive metrics. Curiosity-driven methods like ICM and RND also become less effective as rewards grow sparser. These results provide further evidence of EME's ability to promote efficient exploration effectively, even in environments characterized by delayed rewards. To demonstrate the effectiveness of our proposed scaling factor, we conducted an ablation study focusing on this component. We compared EME with two variants: 'EME-EP,' which incorporates episodic counts, and 'EME-Static,' which uses a static scaling factor. The results from continuous tasks in the Robosuite and MiniGrid environments are illustrated in Figure 13 , Figure 10 and summarized in Table 5 , respectively. Without the diversity-enhanced scaling factor, EME shows a noticeable decline in performance. 'EME-EP' performs better in MiniGrid environments compared to continuous tasks in Robosuite, highlighting the efficacy of episodic state visitation counts in grid-based games but demonstrating limited effectiveness in environments with high-dimensional states. Additionally, we visualize the exploration bonus of all MiniGrid environments in Figure 12 . We choose the three manipulation tasks in the Robosuite platform [70] .Table Wiping. The environment consists of a table with a whiteboard surface and some markings is placed in front of a single robot arm, which has a whiteboard eraser mounted on its hand. The shape of the dirty region is correlated to the position of the cube. When the dirty region is diagonal, the cube is on the right-hand side of the robot arm. The goal of the agent is to learn to wipe the whiteboard surface and clean all of the markings. The whiteboard markings are randomized at the beginning of each episode.Door Opening. The environment consists of a door with a handle mounted in free space in front of a single robot arm. The height of the handle and the position of the door are correlated. When the door is closed to the robot arm, the handle is in a low position. When the door is far from the robot arm, the handle is in a high position. The goal of the agent is to learn to turn the handle and open the door. The initial state distribution of the door location is randomized at the beginning of each episode.Pick and Place. The environment consists of four objects placed in a bin in front of a single robot arm. There are four containers next to the bin. The goal of the agent is to place each object into its corresponding container. This task also has easier single-object variants. The initial state distribution of object locations is randomized at the beginning of each episode.MiniGrid [17] is a set of procedurally-generated grid-worlds. In MiniGrid, the world is a partially observable grid of size N × N . Each tile in the grid contains exactly zero or one object. The possible object types are wall, door, key, ball, box, and goal. Each object in Mini-Grid has an associated discrete color, which can be one of red, green, blue, purple, yellow, or grey. By default, walls are always grey and goal squares are always green. Rewards are sparse for all MiniGrid environments. There are seven actions in MiniGrid: turn left, turn right, move forward, pick up an object, drop an object, toggle and done. The agent can use the turn left and turn right action to rotate and face one of 4 possible directions (north, south, east, west). The move forward action makes the agent move from its current tile onto the tile in the direction it is currently facing, provided there is nothing on that tile, or that the tile contains an open door. The agent can open doors if they are right in front of it by using the toggle action. Observations in MiniGrid are partial and egocentric. By default, the agent sees a square of 7 × 7 tiles in the direction it is facing. These include the tile the agent is standing on. The agent cannot see through walls or closed doors. The observations are provided as a tensor of shape 7 × 7 × 3. However, note that these are not RGB images. Each tile is encoded using 3 integer values: one describing the type of object contained in the cell, one describing its color, and a flag indicating whether doors are open or closed. This compact encoding was chosen for space efficiency and to enable faster training. For all tasks, the agent gets an egocentric view of its surroundings, consisting of 3 × 3 pixels. A neural network parameterized as a CNN is used to process the visual observation.MultiRoom. The MultiRoomNXSY environment consists of X rooms, with a size at most Y, connected in random orientations. The agent is placed in the first room and must navigate to a green goal square in the most distant room from the agent. The agent receives an egocentric view of its surroundings, consisting of 3 × 3 pixels. The task increases in difficulty with X and Y. Episodes finish with a positive reward when the agent reaches the green goal square. Otherwise, episodes are terminated with zero reward after a maximum of 20 × N steps.KeyCorridor. the agent has to pick up an object which is behind a locked door. The key is hidden in another room, and the agent has to explore the environment to find it. Episodes finish with a positive reward when the agent picks up the ball behind the locked door or after a maximum of 270 steps.ObstructedMaze. The agent has to pick up a box which is placed in a corner of a maze. The doors are locked, the keys are hidden in boxes and the doors are obstructed by balls. Episodes finish with a positive reward when the agent picks up the ball behind the door or after a maximum of 576 steps. The Habitat platform relies on several key abstractions that model the domain of embodied agents and the tasks they can perform in three-dimensional indoor simulation environments. This platform consists of a virtually embodied agent, such as a robot, equipped with a suite of sensors that can observe the environment and take actions to alter the agent's state or the environment's state. Each sensor is associated with a specific agent and can return observation data from the environment at a specified frequency. The 3D environment includes a scene mesh, objects, agents, and sensors, organized into regions and objects through a hierarchical representation called the Scene, which can be programmatically manipulated. All components of the Scene are present on the SceneGraph. Additionally, a simulator backend instance can update the state of agents and SceneGraphs based on given actions for a set of configured agents and SceneGraphs, and it provides observations for all active sensors possessed by the agents. We use the Matterport3D (MP3D) dataset [15] , which consists of high-quality renderings of indoor scenes. To measure exploration coverage, we compute the area revealed by the agent's line of sight using the function provided by the Habitat codebase [32] . As depicted in Figure 16 , exploration is measured as the proportion of the environment revealed by the agent's line of sight over the course of the episode.Algorithm 1 Effective Metric-based Exploration-bonus (EME)1: Initialize parameters of policy π(θ), metric d ϕ E , and reward models {g(η 1 ), . . . , g(η k )} 2: while not converged do Step environment s t+1 ∼ p (s t+1 | s t , a t )6:Record transition in the buffer: D τ ← D τ ∪ {s t , a t , s t+1 , r t+1 } Train policy π(θ) via policy gradient 12:Update the metric d ϕ E : E B [J(ϕ)] ▷Equation (Update reward models by minimizing the prediction error: E Dτ ∥g(s t , a t ; η) -r t+1 ∥ 2 While our method has shown success in several challenging exploration tasks, it does not consistently achieve top performance in environments with less sparse rewards, as observed in the MuJoCo continuous control benchmarks. Additionally, our method has not been tested in reinforcement learning domains characterized by large action spaces. Addressing these limitations and developing a more generalized solution for diverse RL environments remains a goal for future work.Exploration in RL. Exploration remains a long-standing problem in RL. Common approaches include ϵ-greedy [59] , count-based exploration [7, 46, 61, 41, 40, 57] , ensemble-based exploration [45, 39, 48] and curiosity-based exploration [53, 55, 56, 10] . Several exploration strategies use a dynamics model to provide intrinsic rewards [47, 11, 33, 48, 36, 2] . Latent variable dynamics have also been studied for exploration [4, 9, 62] . Maximum entropy in the state representation has also been used for exploration [54, 66] . Other intrinsic motivation methods have recently been developed for exploration in context MDPs [67, 28, 51, 69, 32] , which automatically generate curricula over variations of the MDP to encourage efficient learning, effectively performing a form of curiositydriven exploration in the context space, including goal-conditioned [23, 50, 12, 18, 22] and goal-free variants [58, 49, 34, 19] . Reward shaping refers to modifying the original reward function with a shaping reward function which incorporates domain knowledge. Considering the most general form, namely the additive form, of reward shaping. The first approach to guarantee policy invariance is potential-based reward shaping (PBRS) [44] , which defines the shaping reward function as the difference between values assessed through the potential function based on prior knowledge. There are lots of variants of PBRS [20, 38, 31, 3, 64, 30, 29, 63] which have shifted their focus to different areas within the field of reinforcement learning.Metric-based Exploration Bonus. Metric space is widely used in encoding state representations [68, 14, 13, 16, 35] . Metric-based exploration bonus is based on the evaluation metric that quantifies the degree of difference or distance between two projected states for the measure of novelty. RIDE [51] evaluates the novelty as between two successive state representations under the L 2 norm. NovelD [69] uses the disparity in RND bonuses between adjacent states under the L 1 distance as the exploration bonus. LIBERTY [63] uses the difference between potential functions of adjacent states under the bisimulation metric space. Note that we propose an effective metric that measures the distribution distance between dynamics models by computing the distance between sampled subsequent states. Additionally, we integrate the Kullback-Leibler (KL) divergence between policy distributions to more robustly model the "behavioral similarity" between states, which is more effective and scalable across diverse environments with different observations, especially in addressing the critical Noisy-TV problem during exploration.The detailed discussion of related work can be found in Appendix FDetails of the environments are provided in Appendix D.1, and additional experiments are described in Appendix C. 


Decision: All four reviewers agree that this is a solid paper, making significant contributions on an important topic, presented with clarity and extensive analysis.

Reviews: review: paper_summary: paper_summary: The authors introduce the Effective Metric-based Exploration-bonus (EME) to address the limitations of existing state discrepancy methods by proposing a robust metric for evaluating state differences and a diversity-enhanced scaling factor for exploration bonuses. Extensive experiments demonstrate that EME outperforms other methods in various environments, including Atari games, Minigrid, Robosuite, and Habitat, highlighting its scalability and applicability to diverse scenarios.
strength_weakness: strength_weakness: The ultimate method seems incredibly complex, making it difficult for future practitioners to reproduce. Emphasizing Algorithm 1 (potentially moving it from the Appendix into the main paper) could help with this.

The paper does not provide a thorough discussion of the computational efficiency and potential overheads introduced by the EME method. It would be useful to provide a section discussing this.

The paper introduces several novel components, such as the diversity-enhanced scaling factor, but lacks detailed ablation studies that isolate the contributions of these individual components.

The authors repeatedly refer to "a real-life indoor environment". While somewhat true (as it is later revealed to be the Habitat environment), it is rather misleading, as the experiments are still entirely done in the computer. (ie there is no real-world robot involved).
questions: questions: 1. Can you provide more details on the computational complexity and runtime performance of the EME method compared to existing approaches? Specifically, how does EME scale with the size of the state space and the complexity of the environment?

2. Could you provide detailed ablation studies to isolate the impact of the diversity-enhanced scaling factor and other key components of EME?

3. Some parts of the theoretical analysis are dense. Could you provide more intuitive explanations or examples to help bridge the gap between the theory and its practical implications?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces the Effective Metric-based Exploration-bonus (EME), a novel approach for enhancing exploration in reinforcement learning (RL) tasks. The paper identifies key limitations in existing metric-based exploration methods, such as their reliance on count-based scaling factors and approximation gaps in bisimulation metric learning. To address these issues, the paper proposes a more robust metric for evaluating state discrepancy that provides theoretical guarantees on the value difference bound. Additionally, they introduce a diversity-enhanced scaling factor based on the variance of predictions from an ensemble of reward models to improve exploration effectiveness, especially in hard exploration scenarios. The authors conduct extensive experiments across various challenging environments, including continuous control tasks, discrete-action games, and realistic indoor scenarios, demonstrating the superior performance and scalability of their method compared to prior approaches.
strength_weakness: strength_weakness: 1. The performance of EME with feature encoder is mixed; this may hamper the scalability of EME to realistic scenarios, where pre-trained feature encoder is commonly used.

2. Ablations are limited. It would be good to know how sensitive EME is to the choice of RL algorithm, the value of M, and other hyperparameters. Only scaling factor is studied right now. 

3. On the Robosuite environments, EME do not seem much better than the baselines. How do we characterize when we expect EME to do much better than existing methods?
questions: questions: My questions are stated above.
limitations: limitations: Yes, the paper has adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes a new exploration algorithm in reinforcement learning (RL), particularly in environments with sparse rewards. The authors critique existing exploration bonus methods using state discrepancy, highlighting their limitations in scalability and theoretical guarantees. They propose a novel method called Effective Metric-based Exploration-bonus (EME), which features a robust metric for evaluating state discrepancy, grounded in theoretical assurances. EME avoids approximation gaps found in previous bisimulation metric-based approaches and eliminates the reliance on episodic count scaling factors.  Furthermore, they introduce a diversity-enhanced scaling factor, dynamically adjusted by the variance of predictions from an ensemble of reward models, to improve exploration in tasks with subtle state differences. Experiments on Robosuite, Atari games, MiniGrid, and Habitat environments demonstrate the effectiveness and scalability of EME.
strength_weakness: strength_weakness: While the presented results look impressive a more thorough empirical evaluation would be more helpful to assess the performance of EME:

- On Atari the paper only evaluates on hard exploration games though it has been shown that evaluating on more Atari games is usually necessary to properly assess the performance of an algorithm. [1]
- The paper does not include a ablation, because of that it is difficult to know if the improvement comes from the better exploration bonus and using multiple reward predictions

[1] On Bonus-Based Exploration Methods in the Arcade Learning Environment, Ali Taiga et al. ICLR 2020
questions: questions: - The explanation in lines 99-102 regarding the breaking of theoretical assurances is unclear. Can the authors elaborate on what specifically breaks down and how EME overcomes this issue?

- On L130, the authors mention that existing methods might already be performing a similar function.  Can the authors clarify this statement and clearly differentiate EME from such methods?

- Why is the Habitat environment considered a more realistic scenario given the variety of task one could attempt to solve with reinforcement learning?

- The paper lacks citations for using an ensemble of models to drive exploration (L261), see for example [1, 2]

- Is the feature encoder jointly learned with the EME metric? I couldn't find the information, though results in the appendix show that even a random feature encoder is providing good results

- How does the number of reward models in the ensemble affect EME's performance and computational cost? Is there an optimal number, or are there diminishing returns as the ensemble size increases?


[1] Deep Exploration via Bootstrapped DQN, Osband et al. 

[2] The Curse of Diversity in Ensemble-Based Exploration, Lin et al. ICLR 2024
limitations: limitations: The authors acknowledged limitations regarding the performance in environments with less sparse rewards and untested performance in RL domains with large action spaces.
An ablation study of the different component of the algorithm as well as a more thorough evaluation on diverse environments would be helpful to properly evaluation the contribution presented in this paper.
The potential negative societal impact is also addressed in the broader impact statement
paper_summary: paper_summary: The authors propose a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Previous exploration algorithms measure state discrepancy strictly on $L_p$ norms which is limiting since for environments where state differences are minimal in the observations the intrinsic reward is not useful. The authors propose an exploration bonus which is centred aroudn the variance of the predicted reward for a ensemble of reward models. A scaling factor for the exploration bonus is also introduced using the ensemble of reward models, where the variance in the prediction of the ground-truth reward is taken as the scaling factor. This scaling factor is higher in novel spaces since the prediction variance increases for states that have not been explored sufficiently.. The authors conduct extensive experiments on 2D and 3D environments shows superior performance to other known exploration methods.
strength_weakness: strength_weakness: - To my knowledge other methods that try to aggregate states in high dimensions are not considered. Wouldn’t this mitigate or solve the issue with unique counts in high-dimensional spaces? 
- The maximum reward scaling $M$ seems to be an important parameter, however I do not see an ablation for it. The ablations in the appendix ablate the nature of the scaling factor, but not $M$
- To my knowledge there is no talk about limitations of the method. Could you provide insight on where possible limitations to your method lie?
questions: questions: - In practice, how do you train the ensemble of reward models in the reward free setting? Is there any relation to successor representations? What do you do in settings where the reward is not a good indicator for the environment structure? I'm a bit surprised that the method works well given there is no reward to learn the ensemble models. Does the KL-divergence of the policies take the overhand here?
limitations: limitations: The authors have discussed limitations of previous methods but have not to my knowledge discussed the limitations of their own method.

Comments: comment: Thank you once again for recognizing our work and investing your valuable time in providing feedback on the paper, we will work on more atari games, and your valuable suggestions will definitely improve the quality of our work.
Thank you for your response and properly addressing my comments. I have increased my score to 7.
If you have enough time/compute I would encourage you to add the remaining atari games to your results.
Dear reviewers,

Thank you once again for investing your valuable time in providing feedback on our paper. We had provided additional experiments and other responses based on your insightful suggestions. We highly appreciate the effort to provide in-depth reviews that helped us to improve our work.  Since the discussion period between the author and reviewers is rapidly approaching its end, we kindly request you to review our responses to ensure that all concerns are addressed and look forward to possibly receiving reply from you.

Best,

Authors
Thank you once again for recognizing our work and investing your valuable time in providing feedback on the paper, your valuable suggestions will definitely improve the quality of our work.
Thank you so much for your response and added work to the ablations!

I feel you have addressed all my concerns and also have a better understanding as to why the scaling factor and EME metric work in a reward free setting. 

I think this is a good paper with good experiments.