Episodic Novelty Through Temporal Distance
Authors: Yuhua Jiang
Qihan Liu
Yiqin Yang
Xiaoteng Ma
Dianyu Zhong
Hao Hu
Jun Yang
Bin Liang
Bo XU
Chongjie Zhang
Qianchuan Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Primary area: reinforcement learning

Abstract: Exploration in sparse reward environments remains a significant challenge in reinforcement learning, particularly in Contextual Markov Decision Processes (CMDPs), where environments differ across episodes. Existing episodic intrinsic motivation methods for CMDPs primarily rely on count-based approaches, which are ineffective in large state spaces, or on similarity-based methods that lack appropriate metrics for state comparison. To address these shortcomings, we propose Episodic Novelty Through Temporal Distance (ETD), a novel approach that introduces temporal distance as a robust metric for state similarity and intrinsic reward computation. By employing contrastive learning, ETD accurately estimates temporal distances and derives intrinsic rewards based on the novelty of states within the current episode. Extensive experiments on various benchmark tasks demonstrate that ETD significantly outperforms state-of-the-art methods, highlighting its effectiveness in enhancing exploration in sparse reward CMDPs.

Published as a conference paper at ICLR 2025
EPISODIC NOVELTY THROUGH TEMPORAL DISTANCE
Yuhua Jiang': Qihan Liu'r, Yiqin Y Xiaoteng Ma', Dianyu Zhong', Hao Hul Jun Yang'! Ç Bin Liang', Bo Xu², Chongjie Zhang', Qianchuan Zhaolf
TTsinghua University
"hhe Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences
3 Washington University in St. Louis
(jiangyh22, 1qh20 (ch20)mmails @mailss tsinghuun........ edt.huuaaeddc..om
ABSTRACT
Exploration in sparse reward environments remains significant challenge in reinforcement learning, particularly in Contextual Markov Decision Processes (CMDPs), where environments differ across episodes. Existing episodic intrin- sic motivation methods for CMDPs primarily rely on count-based approaches, which are ineffective in large state spaces, or on similarity-based methods that lack appropriate metrics for state comparison. To address these shortcomings, we propose Episodic Novelty Through Temporal Distance (ETD), a novel approach that introduces temporal distance robust metric for state similarity and in- trinsic reward computation. By employing contrastive learning, ETD accurately estimates temporal distances and derives intrinsic rewards based on the novelty of states within the current episode. Extensive experiments on various benchmark tasks demonstrate that ETD significantly outperforms state-of-the-ard methods, highlighting its effectiveness in enhancing exploration in sparse reward CMDPs.
INTRODUCTION
Exploration in sparse reward environments remains significant challenge in reinforcement learning (RL). Recent approaches have introduced the concept of intrinsic motivation (Meyer & Wilson| 1991;
Oudeyer et al.
2007) to encourage agents to explore novel states, yielding promising results in sparse
reward Markov Decision Processes (MDPs) (Bellemare et al.) 2016) Pathak al., 2017 Burda et al.,
2018 Machado et al. 2020). Most existing methods grounded in intrinsic motivation derive rewards from the agent's ss cumulative experiences across all episodes. While these methods are effective in singleton MDPs, where agents are spawned in the same environment for each episode, they exhibit limited generalization across environments (Henaff et al., 2022). Real-world applications are often more suitably represented by Contextual MDPs (CMDPs) (Henaff et al., 2023), where different episodes correspond to different environments that nevertheless share certain characteristics, such as procedurally-geneeee
environments Sneevalier-Boisved et al., 2023 Cobbe et al.) 2020 Küttler et al. 2020) or embodied AI tasks requiring generalization across diverse spaces (Savva et al. [019]] Li al.] 202T Gan al. 2020 Xiang et al., 2020). In CMDPs, the uniqueness of each episode indicates that experiences from one episode may offer limited insights into the novelty of states in another episode, thereby necessitating the development of more effective intrinsic motivation mechanisms.
To address the challenges of exploration in CMDPs, where episodes differ significantly, several works have introduced episodic bonuses (Henaff et al. 2023). These bonuses are derived from experiences
within the current episode, avoiding the generalization limitations of cross-episode rewards. These
approaches typically be divided into two lines: count-based (Raileanu & Rocktäschel 2020
Flet-Berliac et al. [202]] Zha et al. 202T: Zhang et al. 20216 Parisi et 24.]]2]]] et al.
202
Mu et al.] 2022: Ramesh et al. (022) and similarity based (Savinov et al. 2018 Badia al. 2020
Henaff et al. 22022 Wan et al. 2023). Count- based methods rely on an episodic count term to generate
positive bonuses once encountering new state but struggle in large or continuous state spaces (Lobel
"qqual contribution. Code is availabe at http://gihuuu.co//cccoorr///
Corresponding authors: yiqin.yang(@ii @ia.ac.en ac., (yangjun6033, zhanqe @tsinghua.cdu.cmm

---

Published as a conference paper at ICLR 2025
et al. 2023 where each state is unique and episodic bonuses remain uniform across all states. Meanwhile, similarity- -based methods require appropriate measurements between pairs of states, which used to be assessed via Euclidean distance (Badia et al., 2020; Henaff et al. 2022) or reachable likelihood (Savinov et al. 2018 Wan et al., (023) in some latent spaces. However, these similarity measurements used by existing methods do provide suitable metric for capturing the novelty of states, as illustrated in Figure This inadequacy undermines the credibility of subsequent intrinsic reward calculations and limits the effectivvnesss of these methods in complex CMDP environments. Our work addresses this gap by introducing new metric-tempee captures novelty CMDPs by considering the expected number of steps between states.
distance that more effectively
In this work, we introduce Episodic Novelty Through Temporal Distance (ETD), a novel approach designed to encourage agents to explore states that are temporally distant from their episodic history. The critical innovation of ETD lies in its use of temporal distance-the the expected number of envi- ronment steps required to transition between two states- as a robust metric for state similarity in intrinsic reward computation. Unlike existing similarity metries, temporal distance is invariant to state representations, which mitigates issues like the "noisy-TV" problem (Burda et al. 2018) and ensures the applicability of ETD in pixel-based environments. We employ contrastive learning with special- ized parameterization to accurately estimate the temporal distances between states. The intrinsic reward is computed based on the aggregated temporal distances between new state and each in the episodic memory. Through extensive experiments on various CMDP benchmark tasks, including Min- iGrid (Chevalie-Boiivertt
al. 2023), Crafter (Hafner 2022), and MiniWorld (Chevalier-Boisee iet al. 2023). we show that ETD significantly outperforms state-of-the-ar methods, improving exploration efficiency.
BACKGROUND
We consider a contextual Markov Decision Process (CMDP) defined by (8, A, P,T, ps, ?). where S is the state space, A is the action space, C is the context space, S x A x + A(S) is the transition function, r(sz, at $141) the reward function and typically sparse, As is the initial state distribution conditioned on the context, JlC is the context distribution, and 7 E (0, 1) is the reward discount factor. At start at each episode, a context e sampled from μcc, followed by an initial state so sampled from (-((-c)) and subsequent states are sampled from 8t41 P(- a, c). The goal is to optimize a policy π. S + A(A) so that the the expected accumulated reward across over all contexts Be-pe,torps (-1c) |Σ, 7'r(st, d,, St+1) ]] is maximized.
Examples of CMDPs include procedurally generated environments (Chevalier-Boisver et al. 2023 Cobbe et al., 2020: Küttler et al., 2020 HHffner (022), where each context serves random seed for environment generation. Similarly, Embodied Al environments (Chevalier (Chevalier-Boisvert et al. 2023 Savva et al. 2019 Gan et al., 2020), where agents navigate various simulated homes, also examples of CMDPs. Notably, singleton MDPs (IC) = 1) represent a special case of CMDPs. We primarily focus on CMDPs with |C|=
To address the sparse reward challenges, we augment the reward function r by adding an intrinsic reward bonus. The modified equation is r(st, ae, 8e4 rf + B b. where rī represents the sparse extrinsic reward and by denotes the intrinsic reward at each timestep t. The hyperparameter 3 controls the influence of the intrinsic reward.
LIMITATIONS OF CURRENT EPISODIC BONUSES
Recent intrinsic motivation methods (Andres 2022 Henaff et al. 2022 2023 have demonstrated that incorporating an episodic bonus is crucial for sol solging sparse reward CMDP problems. For instance, gigh-performing methods like NovelD et al. 2021b) often depend on an episodic count term to be effective in CMDPs. However, these count-based methods face challenges in large or noisy state spaces. When each state is unique, the episodic bonus becomes less meaningful as assigns the same value to all states. This issue is evident in the "noisy-TV"" problem (Burda et al.) 2018), where random noise disrupts the state. We validated this observation using the MiniGird- DoorKeyl6x experiment, shown in Figure [ When no noise was present in the environment, NovelD performed well. However, when Gaussian noise with a mean of 0 and variance of 0.1 was added the state input, NovelD failed completely, indicated by the corresponding curve
2

---

Published as a conference paper at ICLR 2025
Average Episode Reward
NovelD
ETD(ours)
NovelD-noise
ETD ETD-naise(ours)
(a) Inverse dynamic
(b) Likelihood
(c) Ours
Figure Distance from m all other states in a 17x17 SpiralMaze. Darker colors indicate greater distance. (Left) Euclidean distance of embeddings trained by inverse dynamics. (Center) Likelihood estimation of easy transitions (EC). (Right) The learned temporal distance (Ours).
Time Steps
Figure 1: Training curves in Hinigrid-DDoKey-166 (w/w.o. noise).
(NovelDD-noise). contrast, our proposed method, ETD, maintained strong performance even with the added noise, demonstrating its robustness where NovelD proved ineffective.
Potential alternatives include computing episodic novelty based on the similarity of the current state to previously encountered states stored in episodic memory. This can be achieved using metrics such as Euclidean distance in an embedding space learned through inverse dynamics, as seen in NGU (Badia et al. 2020) and E3B (Henaff al. 2022). Another approach is to estimate the likelihood of easy transitions between states, as done in EC (Savinov et al. 2018) and DEIR (Wan et al., (023). However, we found that these methods do not accurately measure the similarity between states. We conducted an experiment using the SpiralMaze environment, as shown in the Figure(2 We collected 100 trajectories, each consisting of 50 time steps, as training data. We then evaluated the similarity between the red square state and all other states, As the spiral deepens, the states become increasingly distant from the red square, indicating lower similarity. Figure 2(a) shows the inverse dynamics method, where embeddings are trained to predict actions based on the current and next states, with the Euclidean distance between embeddings serving as the similarity metric. However, we found that the Inverse Dynamics method struggles to provide a smooth estimation of the similarity between states. Figure 2(b) presents the EC approach, where a classifier predicts the likelihood that two states are within K time steps, using this likelihood as the similarity measure. We observed that this likelihood measure effectively captures local proximity but fails to distinguish the similarity between most states due its nature non-distance metric. In contrast, our method, which learns temporal distances, as shown in Figure 2(c) accurately captures the similarity between the red square and each state.
METHODS
In this section, we introduce Episodic Novelty through Temporal Distance (ETD), an algorithm designed to enhance exploration in CMDPs. The core innovation ETD using temporal distance quasimetric to measure state similarity, encouraging the agent to explore states that temporally distant from its episodic memory. As summarized Figure we employ a specially parameterized contrastive learning method to learn this temporal distance and then identify the minimum temporal distance between the current state and all previously encountered states in the episodic memory. This minimum distance serves the intrinsic reward for the current state. The next two sections will elaborate on the details.
4.1 TEMPORAL DISTANCE LEARNING
Temporal distance can be intuitively understood through the transition probability between states, where lower probability indicates a larger distance. For given policy we define p (sx y|so a:) as the probability of reaching state y at time step when starting from r. The transition probability can be described using discounted state occupancy measure, which equals geometrically weighted average the probabilities:
pr, (s, = y/50 = x)) = (1 -7)
p (sk = -ys0 = x).
(1)
k=
3

---

Published as a conference paper at ICLR 2025
Contrastive Learning Estimates Temporal Distance
Episodic Intrinsic Bonus Design
50
Sampling positive pairs (x
Online data collecting (5p,
J0s y1)
Dx 89
Sted 21
(0b,55)
Feature Extractor
Rollout Bufferr
Specially parameterized energy function
(x.,)) = 4,6) de (x.
(0m.ss) 25) (0x
Episodic memory
Sg S
9
92
faettin iataiaa fusetion contawtrrs furetior 5x5-a
3. Classify
from
for each and column
(5.5))
de y) temporal distance from x to
b
min dg (s) (=
Figure 3: Overview of ETD. ETD encourages visits to temporally distant states from episodic memory. Temporal distance can be learned through contrastive learning when positive samples consist a state and its geometrically distributed future state, with the erengy function parameterized network minus a quasimetric network. The intrinsic reward is then derived from the minimum temporal distance between the current state and the states stored in episodic memory.
potential
To ensure the temporal distance behaves as quasimetric (a metric that relaxes the symmetry assumption), we use the successor distance (Myers et al.) 2024). Given policy the successor distance is defined as the difference between the logarithms the probabílities of reaching y from (self-loop) and reaching y from 2:
p. [3]ss 505o 9)
d (z. v) = log
(2)
p.7 (s)
s0 =
This formulation satisfies the triangle inequality and other quasimetric properties (Myers et al. 2024 even in stochastic MDPs.
Consequently, the successor distance can be reliably used measure of similarity between states.
we present how to estimate the successor distance dăp (z, y) defined in Equation 2 based on the con- trastive learning. Given the joint distribution P (s = y so 2) over two state [zz y). define p. (z) as the marginal state distribution, and Ps, (v) J P. (x) (sy - y so = x) as the corresponding marginal distribution over future states. Intuitively, we can define an energy function fo(a, y) that assigns larger values to (x. y) tuples sampled from the joint distribution and smaller values to (x. tuples sampled independently from their marginal distributions. Give a batch of tuples {x,, y4 ) we use the symmetrized infoNCE loss function (Oord al. 2018) to learn fo(æ, y):
exp(fo(Ii, vi)) log Σ exp([oo(
+ log exp(fo(T;, Σn exp(ff((,,
Lo =
Based on Equation we can use the unique solution of the energy function fe- Ma & Collins (I, 2018 Poole et al. 2019) to recover the successor distance d5p(x,)) y) in Equation 2 Following prior work (Myers et al. 2024), we decompose the energy function fe=(o.w) () (xr, y) as the difference
between potential network Co (y) S I. and quasimetric network dg (z, y) S
IR:
fe= (x. y) = (v) ds(x, y)-
Then, have dZp(a, v) do- (2, v). As result, discard co(g) after contrastive learning and directly use (z. as temporal distance. For further details, see Appendix B
To demonstrate the learned temporal distance, we present results from the SpiralMaze 17x17 task, as shown in Figure We collected 100 random episodes (each with 50-time steps) and minimized the loss function following the process above. The resulting temporal distance d (m,)) is visualized with colormap (darker color indicates larger distances), showing strong alignment with ground-truth.
4

---

Published as a conference paper at ICLR 2025
Algorithm Episodic Novelty through Temporal Distance
Initialize policy म, quasimetric de, potential ey and f(a.v) Co. dø.
while not converged do Sample context ~ μ.c and initial state su ~ ps((lc)
for t =0.... do
or ~ (-(8r)
Sample action
81+1, P(-|st, Ct
# Step through environment
be+1 min dø(skr S6+1) # Compute bonus ke[U,++1]
T++1 = rit + 36++1
end for
Sample pair of states (. p = y:1s0 = (x;))
# Practically, T; St, Y/ Srtj. ~ Geom(1 -7).
Update f( to minimize the loss:
B
log exp(f(a.v) 9:))
(xp(f(ar))2
((N
log
21=1 exp(S (c;, y5))
Perform PPO update on using rewards T1, .... TT. end while
4.2 TEMPORAL DISTANCE AS EPISODIC BONUS
Our approach maximizes the temporal distance between newly visited and previously encountered states within the current episode. each time step I, we assign larger intrinsic reward to states that temporally distant from the episodic memory. Formally, the episodic temporal distance bonus is defined as:
(EID min (s)s).5t). ke(o.t)
(5)
where (da(sk, sx) s represents the learned temporal distances between the current state s, and all previous states {sk [= in the episodic memory. The minimum distance used the episodic intrinsic reward. In terms of computational efficiency, storing CNN-extracted embeddings in episodic memory minimizes memory overhead. Additionally, concatenating memory states allow all temporal distances to be computed in single neural network inference, ensuring high time efficiency.
Connections previous intrinsic motivation methods. Many previous episodic intrinsic reward methods, such as DEIR (Wan et al.] 2023). NGU (Badia et al. 2020), GoBI (Fu et al.) 2023). and EC (Savinov et al., 2018). also rely on episodic memory and past states to calculate rewards. Compared to these methods, reachability to assess state similarity, which is similar to our approach. However, EC struggles to learn temporal distance accurately, shown in Figure 2(b) Meanwhile, GoBI depends world model's lookahead rollout to estimate temporal distance, which results in high computational complexity.
reward formulation is notably simpler. Both EC and GoBI use
4.3 FULL ETD ALGORITHMS
We implement our quasimetric network de using MRN (Liu et al.) 2023). which allows us to generate an asymmetric distance. Detailed structural information about the network is provided in the Appendix E.1.2] The potential network cu a multi-layer perceptron (MLP) that shares the same CNN backbone as the MRN, The complete algorithm is outlined in Algorithm
5 EXPERIMENTS
To evaluate the capabilities of existing methods and assess ETD, we aim to identify CMDP environ- ments that present challenges typical of realistic scenarios, such as sparse rewards, noisy irrelevant features, and large state spaces. We consider three domains, including the Minigrid (Chevalier Boisvert et al. 2023) and its noisy variants, well high-dimensional pixel-based Crafter (Hafner
5

---

Published as a conference paper at ICLR 2025
2022 and Miniworld Chesseier-Boisvertt al. 2023). For all the experiments, PPO the base RL algorithm and add intrinsic rewards specified by various methods to encourage exploration.
9459
(a) MiniGrid
(b) Crafter
(c) MiniWorld
(d) MiniWorld Top
Figure 4: Rendering of the environments used in this work.
We compare ETD against previous methods, including DEIR (Wan et al., (023), NovelD (Zhang al. 2021b), E3B Heeaaff et al., 2022), EC (Savinov al., 2018). NGU (Badia et al.) 2020 RND Burda et al. 2018). We also included a pure episodic count-based method named Count, which recent studies (Henaff al., 2023) have shown be a strong bascline CMDPs. We carefully tuned the hyperparameters of each method through grid search to ensure optimal performance.
5.1 MINIGRID ENVIRONMENTS
MiniGrid (Chevalier- -heisvlie-Booivertt et 2023) features procedurally generated 2D environments tailored for challenging exploration tasks. In these environments, agents interact with objects such as keys, balls, doors, and boxes while navigating multiple rooms to locate a randomly placed goal. The agents receive single sparse reward upon completing each episode. We chose four particularly challeng- ing environments: MultiRoom, DoorKey, KeyCorridor, and ObstructedMMaze. In the MultiRoom environment, the agent's task is relatively straightforward, requiring navigating through series of interconnected rooms to reach the goal. DoorKey presents an increased difficulty, as the agent must first find and pick up key and then open door before reaching the goal. KeyCorridor even more demanding, requiring the agent to open multiple doors, locate a key, and then use it to unlock another door access the goal. ObstructedMMaze is the most complex of all: the key is hidden within box. ball obstructs the door, and the agent must find the hidden key, move the ball, open the door, and finally reach the goal. Further details on these tasks can found in the Appendix.
Figure S illustrates the learning curves of ETD and other state-of-the the-arr art exploration baselines across eight challenging MiniGrid navigation tasks. Notably, MultiRoom-N6, DoorKey- 16, and KeyCorridor-SoRX are the most challenging settings for their respective environments, while ObstructedMaze offers multiple difficulty levels, from 2DIh to the most challenging Full version. ETD consistently outperforms all other methods, especially in the ObstructedMaze- Full environment, where it reaches near-optimal performance within 20 million steps, demonstrating double the sample efficiency compared NovelL), the strongest baseline. Our implementation of NovelD achieved the highest reported performance the literature, highlighting the careful tuning of our baseline.
observe that PPO fails to learn effectively without intrinsic reward. Addi- tionally, the global intrinsic reward method RND is almost ineffective, which might be due to the ineffectiveness of global exploration experience in CMDP. We also notice that NGU with episodic intrinsic reward performs poorly, possibly because NGU is primarily designed for Atari tasks, with its episodic intrinsic reward tailored for that domain. NovelD, on the other hand, performs quite well, largely due to its episodic count mechanism, and the global bonus in the current NovelD also contributes to its performance. Meanwhile, Count, which only uses the episodic count of Novell) as the reward, achieves relatively good results on certain maps, such Obstructed-2DIIt but it still significantly lags behind NovelD. Pure episodic intrinsic reward methods like DEIR, E3B, and EC all perform relatively well, but their similarity measurements lack precision. In contrast, our ETD method, which leverages temporal distance, significantly improves exploration learning efficiency. We did not compare it with GoBI because it requires pretraining world model, and the rollout costs of the world model are pretty high. Moreover, we observed that the convergence rate of our learning curves is already significantly faster than what demonstrated in the GoBI paper.
From Figure 5]

---

Published as a conference paper at ICLR 2025
ETD]ours)
DEIR
E38
EC
Count
MGU
NovelD
PPD
RND
Obeinitee/Meee-222 20th
Averagee Epleade teroan Arrrage
12
DbstructedWaam-22
ChstructedMaa-11C
Obitr SbstructedParo-22
ChetructedPaama Pull
13
Figure 5: Training performance of ETD and the baselines 8 most challenging Minigrid environ- ments. The x-axis represents the environment steps. All the results are averaged across seeds.
ETDloursi
E38
NovelD
HutiPccom-ii
Douttoy 15x15
CtstracieePuse12 12
SlatrudesMaccsBBtt
Jverag
12
14
Figure 6: Training performance on Minigrid with noise environments. The x-axis represents the environment steps. All results are averaged across seeds.
5.2 MINIGRID ENVIRONMENTS WITH NOISE
To better simulate realistic scenarios, we introduced noise into the states of MiniGrid, resulting in stochastic dynamics and ensuring that noise with a mean of and a variance of 0.1, which is then directly added to the states. We compared the ETD method with three effective methods for MiniGrid: DEIR, NovelD, and E3B, The results presented in Figure
states identical. The noise is generated Gaussian
Our results indicate that NovelD, a count- based method, completely failed to effectively guide explo- ration, as the episodic rewards based on counts no longer provided useful information. In contrast, similarity-basee methods such as E3B and DEIR continued to perform reasonably well. However, our approach provided more accurate assessment of state similarity by utilizing temporal distance. Even in the presence of noise, temporal distance effectively represented the similarity between two states, while the inverse dynamics representation learning used in E3B and the discriminative representation learning used DEIR could not perfectly measure the distance between states, allowing our method to outperform both E3B and DEIR.
5.3 ABLATIONS OF ETD
Representation Learning To further illustrate the effectiveness of temporal distance as an intrinsic re- ward, we compare the ETD with the Euclidean dis- tance within both inverse dynamics and discriminator representation learning contexts. Discriminator rep- resentation learning, introduced in DEIR, resembles contrastive learning and predicts whether two states and action are part of truly observed transition. While all these techniques utilize ETD as a form of intrinsic reward. they differ in evaluating similarities between states. The results of comparisons are illustrated in Figure 7 the Doorkey-16x16 task, the performance difference is not significant. However, in the ObstructedMaze-10 task, where the state
Biscriminttor
Invens
ETDioun)
Averap: Epcose
Figure 7: Ablation of representation learning.

---

Published as a conference paper at ICLR 2025
Davg
quarkllel0
min
Symmetrc
Asymmetri
Doortay-1536
Deontay
Avenge
Figure 8: Ablation of aggregate function
Figure 9: Ablation of Asymmetric / Symmmericc
considerably richer, ETD outperforms both the inverse dynamic and discriminator methods. This finding indicates that a more accurate distance measurement contributes significantly to exploration efficiency.
Aggregate function For the intrinsic reward formulation, we consider not only the minimum but also other functions, such as the 10% quantile (quantile10), the 10th nearest neighbor (knnl0), and the average of the 1tt to 10th nearest neighbors Kcnn1Oavg), The comparisons are presented in Figure 8 We observe that the minimum consistently outperforms the other functions. This because the minimum provides the most aggressive reward signal. For example, if state in the episodic memory matches the current state, the minimum yields reward signal of zero. This aggressive reward discourages the agent from revisiting similar states, thus enhancing exploration efficiency.
Symmetric Our ETD method uses quasimetric distance function, which is inherently asymmetric. However, symmetric alternatives can also be considered. For instance, by removing the asymmetric components from the MRN, we can obtain symmetric distance function. The comparison results are shown in Figure 9 Interestingly, the performances of both the asymmetric and symmetric versions nearly identical. Given that most environment transitions exhibit more symmetry than asymmetry, employing a symmetric distance function reasonable. Nevertheless, to retain the generality of our approach, choose asymmetric distance function as the default.
5.4 PAXEL-BASED CRAFTER AND MINIWORLD MAZE
To evaluate the scalability of our method to continuous high-dimensional pixel-based observations, we conducted experiments on two pixel-based CMDP benchmarks: Crafter and MiniWorld.
Crafter (Hafner 2022) is a 2D environment with randomly generated worlds and pixel-based observa- tions (64x64x3), where players complete tasks such foraging for food and water, building shelters and tools, and defending against monsters to unlock 22 achievements. The reward system is sparse, granting +1 for each unique achievement unlocked per episode and -0.1/+0.1 reward based on life points. With budget of 1 million environmental steps, Crafter suggests evaluating performance using both the success rate of 22 achievements and a geometric mean score, which we adopt as our performance metric. Additionally, hindered learning efficiency.
conducted experiments without life rewards, as they often
MiniWorld Chevalier-Boisverr et al. 2023) is procedurally generated 3D environment simulator that offers a first first-person, partially observable view as the primary form of observation. We focused where the agent must navigate through procedurally generated maze.
on the Mini MiniWorld-Muze Exploration this environment is particularly challenging due to the 3D first-person perspective and the limited field of view. Additionally, reward is given if the agent fails to reach the goal within the time limit, further increasing the difficulty.
We compared ETD against DEIR, NovelD, and PPO without intrinsic rewards. As illustrated in Figure 10 and Figure [1 ETD consistently outperformed or matched the baseline algorithms, demonstrating its superior ability address CMDP challenges with high-dimensional pixel-based observations.
6 RELATED WORK
Intrinsic Motivation in RL Exploration driven by intrinsic motivation has long been key focus in the RL community (Oudeyer al. 2007 Oudeyer & Kaplan 2007). Various methods that
8

---

Published as a conference paper at ICLR 2025
ETD(ours)
DEIR
NovelD
PPO
Average Episode Reward
PuilMaze53
FullMaze54
FullMaze55
03
02
140
Figure 10: Training performance of ETD and the baselines on MiniWorld Maze with different sizes.
9.5
8.5
8.0
Gafeet
8.0
7.5
7,0
ETOtours)
DReR
Novel0
PPo
ETOtours)
DBe
NevelD
PPO
(a) Crafter
(b) Crafter w.o life reward
Figure 11: Evaluating ETD and the baselines on Crafter.
combine deep RL agents with exploration bonuses have been developed. Notable examples include
ICM (Pathak et al. 2017). RND (Burda et al. 2018). and pseudocounts (Bellemare et al., [016]] Martin et al. 2017 Ostrovski et al., 2017; Machado et al. (020), which have demonstrated success in challenging tasks like Montezuma's Revenge (Bellemare al., 2013). These methods, often
categorized as global bonus approaches, were primarily designed for singleton MDPs, presenting
limitations in CMDPs, where environments vary across episodes (Savva et al. 2019 al. 202
Gan et al. 2020 Xiang et al., 2020; Zisselman et al., 2023 Jiang et al. 2024))
To address this limitation, recent works have proposed episodic bonuses (Henaff al. 2023) relying on episodic memory (Blundell et al. 2016; [Pritzel et |l.] (201)), where intrinsic rewards are derived from experiences within the current episode. These methods can be roughly grouped into two categories: count-based (Raileanu & Rocktäschel |020 Flet-Berliac et al. 202 Zha et al. 2021
Zhang et al. 202 lb Parisi et al. [222 1,21.000
/ Zhang et | 202 Mu et al. al.] 2022 Ramesh et al. 2022
and similarity- based (Savinov et al.) 2018 Badia al. [2200; Henaff et al.) 2022 Wan et al. 2023
Combining global and episodic bonuses and effectively utilizing both remains an open challenge.
Approaches like AGAC (Flet-Berliac
al., 2021), RIDE (Raileanu & Rocktäschell 2020). and
CMDPs (Parisi et al. 2021 Zhang et al. 2021a| Mu et al. (202 Ramesh al. (022). However, other methods, such as EC (Savinov et al. 2018) and E3B (Henaff et al. 2022)). focus solely on episodic bonuses and have also succeeded in CMDPs. Our approach belongs to the latter category, leveraging episodic bonuses
NovelD (Zhang et al., 2021b) utilize both, yielding better performance
to enhance performance in CMDPs. Table I compares recent intrinsic motivation approaches and highlights our method.
Our approach, which employs temporal distance as an intrinsic reward, shares similar ideas with EC (Savinov et al., 2018) and GoBI (Fu et al. 2023). EC also utilizes contrastive learning to assess the temporal proximity of states. However, while EC only predicts the probability that two states are temporally close, our method defines temporal distance as a theoretically quasimetric measure. GoBI uses learned world model and extensive random rollouts to simulate reachable states, rewarding uniqueness. However, GoBI requires world model pretraining and incurs substantial computational costs. In contrast, our method achieves comparable performance while maintaining lower computational overhead.
Temporal Distance RL Temporal distance has been extensively applied in imitation learn- ing (Sermanet et al. 2018), unsupervised reinforcement learning (Park al. 2024 Hartikainen et al.)

---

Published as a conference paper at ICLR 2025
Method
Intrinsic Bonus: bMethed sc)
Episodic Bonus Category
ICM
[1δ(s)) o(s))
/
RND
11f(se) f(s))11
/
AGAC
DKL (IT (-1 sc) aa (- [()) + B- (Nalom))
Count
RIDE
11 ($t+1) - (st)ll2 (No(s))
Count
NovelD
(BRND($E+1)) - (RND I[N((s)) =
Count
NGU
(RND(
Similarity
(Viocce K(o(a))
E3B
① (8t) ( 21 (81)
Similarity
EC
a0B3 - P(C(si, 81 8t)Ne)))))
Similarity
DEIR
minge |1 (e)) (e
Similarity
ETD(ours)
min 8t)
Similarity
Table
Summary recent intrinsic motivation methods. We marked the episodic bonus as Blue.
2019
Klissarov & Machado 2023), and goal-conditioned reinforcement learning (Kaelbling 1993
Durugkar et 2221] Eysenbach et al. 2022 Wang et al.] (023)). Common methods for learning
temporal distance include Laplacian- based representations (Wu al. 2019 Wang et al. 2021 2022
which use spectral decomposition to capture the geometry of the state space; constrained optimiza- 2023), which maintains distance threshold between adjacent
tion (Park al. 2024; Wang
states while dispersing others; and temporal contrastive learning (Sermanet et al. 2018 Eysenbach
al. 2022). which brings temporally close states together in representation space while pushing apart negative samples. Each approach, however, has its limitations: Laplacian-basee representations
can be unstable during training (Gomez al. 2024). ) constrained optimization highly depends on deterministic environments (Wang et al. 2023 and temporal contrastive learning often violates the triangle inequality (Wang et al. 2023). a key property of metrics.
Recently, CMD (Myers et al.) (024) proposes the successor distance, which theoretically guarantees a quasimetric temporal distance by using a specific parameterization of temporal contrastive learning. While CMD is limited to goal-conditioned tasks, we extend this method to sparse reward CMDPs.
7 CONCLUSION
In this work, we introduce ETD, novel episodic intrinsic motivation method for CMDPs. ETD leverages temporal distance as a measure of state similarity, which is more robust and accurate than previous methods. This allows for more effective calculation of intrinsic rewards, guiding agents to explore environments with sparse rewards. We demonstrate that ETD significantly outperforms exist- ing episodic intrinsic motivation methods in sample efficiency across various challenging domains, establishing the sate-of-the-are RL approach for sparse reward CMDPs.
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation China under Grant No. 62192751, in part by the National Science and Technology Innovation 2030 Major Project (Grant No. 2022ZD0208804), in part by Key R&D Project of China under Grant No. 2017YFC0704 the 111 International Collaboration Program of China (No. NNo.B25027) and in part by the BNRist Program under Grant No. BNR2019TD01009 in part by the InnoHK Initiative, The Government of HKSAR; and in part by the Laboratory for Al-Powered Financial Technologies.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alain Andres, Esther Villar-Rodrigus and Javier Del Ser. An evaluation study of intrinsic mo- tivation techniques applied to reinforcement learning over hard exploration environments. In Andreas Holzinger, Peter Kieseberg, A. Min Tjoa, and Edgar Weippl (eds.), Machine Learning and Knowledge Extraction, pp. 201-220, Cham, 2022. Springer International Publishing. ISBN
978-3-031-14463-9
Adrià Puigdomênech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020.
Junik Bae, Kwanyoung Park, and Youngwoon Lee. TLDR: Unsupervised goal-conditioned RL via temporal distance-aware representations. In 8th Annual Conference Robot Learning. 2024. URL http:///wggnnneeeee.ee////ooo///
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013, ISSN 1076-9757. doi: 10.1613/jair.9912.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.04440 2016.
Michal Bortkiewicz, Wladek Palucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arezewski, Lukasz Kucinski, and Benjamin Eysenbach. Accelerating goal-conditioned algorithms and research, 2024. URL hhtps//aarivaar////2244..1111
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint mxXiv:8//0. /2894, 2018.
Maxime Chevalier- Boisverr, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lueas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, 138/2306.13811, 2023.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning, 2020. URL http://s/aaaxi.oog//////9/1111//
Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for
reinforcement learning. Advances in Neural Information Processing Systems, 34:8622-8636, 2021.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Ruslan Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL http:////peeeeeeee//////mm/////a
Yannis Flet- Ber-Bee Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversarially guided actor-critic. arXiv preprint wx2v:2/022004376 2021.
Yao Fu, Run Peng, and Honglak Lee. Go beyond imagination: maximizing episodic reachability with world models. In International Conference Machine Learning, pp. 10405-10420. PMLR, 2023.
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowea, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworid: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.0499 2020.
Diego Gomez, Michael Bowling, and Marlos C. Machado. Proper laplacian representation learning, 2024. U httppps//aaaii.orgg/////1......0

---

Published as a conference paper at ICLR 2025
Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference Learning Representations, 2022. htt.hs/s//ppeaeeeeeeeeeooomommm/ 1W009996EFOO00
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprint arXiv: 2019.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631-37646, 2022.
Mikael Henaff, Minqi Jiang, and Roberta Raileanu. A study of global and episodic bonuses for exploration in contextual mdps. In International Conference on Machine Learning, pp. 12972- 12999. PMLR, 2023.
Jeffrey J Hunter. Stationary distributions and mean first passage times of perturbed markov chains. Linear Algebra and its Applications, 410:217-243, 2005.
ictibones. Logarithmic triangle inequality. faackeechange.com/quuutttts/////
Math Stack Exchange, 2017. URL https://math
Yuhua Jiang, Qihan Liu, Xiaoteng Ma, Chenghao Li. Yiqin Yang, Jun Yang, Bin Liang, and Qianchuan Zhao. Learning diverse risk preferences in population- based self-play. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 12910-12918, 2024.
Leslie Pack Kaelbling. Learning achieve goals. In IJCAI, volume 2, pp. 1094-8. Citeseer, 1993.
Martin Klissarov and Marlos C. Machado. Deep Laplacian- Options for Vemporally-Extendee Exploration. Proceedings of the 40th International Conference on Machine Learning, pp.
17198- 17217. PMLR, July 2023.
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment, 2020. URL https://arxiv
Nug/abs/2006.1376/
Chengshu Li, Fei Xia, Roberto Martín Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv: 00.00072.222 2021.
Bo Liu, Yihao Feng. Qiang Liu, and Peter Stone. Metric residual network for sample efficient goal-conditioned reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8799-8806, 2023.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34:18459-1 18473, 2021.
Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins to estimate pscudocounts for exploration in reinforcement learning. In International Conference Machine Learning, pp. 22594-22613. PMLR, 2023.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional
models: Consistency and statistical efficiency. arXiv preprint arXiv: htt:::/80000/12. 2018.
Marlos C Machado, Mare G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5125-5133, 2020.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Mareus Hutter. Count- exploration in feature space for reinforcement learning. arXiv preprint arXiv: Tttv:/7060000000 080900, 2017.
Ccan-Arcady Meyer and Stewart W. Wilson. A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers, pp. 222-227. 1991.
Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. Advances in Neural Information Processing Systems, 35:33947-33960, 2022.
12

---

Published as a conference paper at ICLR 2025
Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision- Making, In International Conference on Machine Learning. 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXivv1807.0377 03748,, 2018.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Pierre-Yve Oudeyer and Frederic Kaplan. What is intrinsic motivation? ttpoooogy of computational approaches. Frontiers in neurorobotics, 1:108, 2007.
Pierre-Yo Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286, 2007.
Simone Parisi, Victoria Dean, Deepak Pathak, and Abhinav Gupta. Interesting object, curious agent: Learning task-agnostic exploration. Advances in Neural Information Processing Systems, 34: 20516-20530, 2021.
Seohong Park, Oleh Rybkin, and Sergey Levine. METRA:
Scalable unsupervised RL with metric- aware abstraction. In The Twelfth International Conference on Learning Representations, 2024.
URL https://ppeneevee.ee///oonnmnn
LOSoay
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2877.
PMLR, 2017.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180.
PMLR, 2019.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International conference on machine learning, pp. 2827-2836. PMLR, 2017.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driver exploration for procedurally- generated environments. urXiv preprint arXiv:2002. 12292, 2020.
Aditya Ramesh, Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Exploring through random curiosity with general value functions. Advances in Neural Information Processing Systems, 15:18733-18748, 2022.
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timothy Lilli- crap, and Sylvain Gelly. Episodic curiosity through reachability, arXiv preprint arXiv:1810.0227:: 2018.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research, 2019. URL https:::/aaaiiioo//aabbaa//////a 19040/00aaaa99900211001
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. International Conference on Machine Learning, pp.
PMLR, 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. cime-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141 IEEE, 2018.
13

---

Published as a conference paper at ICLR 2025
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint DrXXv:1801.00690 2018.
Shanchuan Wan, Yujin Tang, Yingtao Tian, and Tomoyuki Kaneko. Deir: efficient and ro- bust exploration through discriminative-moode- arXiv:2304.10770. 2023.
episodic intrinsic rewards. arXiv preprini
Kaixin Wang, Kuangqi Zhou, Qixin Zhang, Jie Shao, Bryan Hooi, and Jiashi Feng. Towards better laplacian representation in reinforcement learning with generalized graph drawing. In International Conference on Machine Learning, pp. 11003-11012. PMLR, 2021.
Kaixin Wang. Kuangqi Zhou, Jiashi Feng, Bryan Hooi, and Xinchao Wang. Reachability-wwar laplacian representation in reinforcement learning. arXiv preprint arXiv:2210.13153, 2022.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforce- ment learning via quasimetric learning. In International Conference on Machine Learning, pp. 3641 133440. PMLR, 2023.
Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in RL:: Learning representations with efficient approximations, In International Conference on Learning Representations, 2019. URL
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, al. Sapien: simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
11097-11107, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Daochen Zha, Wenye Ma, Lei Yuan, Xia Hu, and Ji Liu. Rank the episodes: A simple approach for exploration in procedurally-generan environments. arXiv preprini arXiv:2101.08//2, 2021.
Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E Gonzalez, and Stuart Russell. Made: Exploration via maximizing deviation from explored regions. Advances in Neural Information Processing Systems, 34:9663-9688 96800, 2021a.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, $4:25217-25233 2021b.
Ev Zisselman, Itai Lavie, Daniel Soudry, and Aviv Tamar. Explore to generalize in zero-shot
RL. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
Using reward bonuses, whether within an episode or globally, violates the MDP assumption where the reward ra depends only on the immediately preceding state 8; and action at. Even if the original RL task is an MDP, introducing a reward bonus that depends on the episode history implicitly transforms the task into POMDP setting. This transformation problematic because the value function based on the Markovian state might be biased. Although reward bonuses have proven highly effective for exploration in sparse reward settings, further research is needed to mitigate the impact of the POMDP transformation. Additionally, the successor distance used may be infinite in non-ergodic settings, implying an assumption that we're re dealing with ergodic MDP problems.
Another limitation of this work is that, our intrinsic reward is purely episodic and doesn't 't incorporate global bonuses from all experiences, limiting its application in singleton MDPs where global bonuses essential. We believe designing combined approach using temporal distance for both global and episodic bonuses is a promising direction. A potential method could involve designing a global bonus through maximum state entropy (Seo et al. (202T); Liu & Abbeel (202T): Bae et al. (2024) and integrating it with ETD. We leave this exploration for future work.
B THEORETICAL PROPERTIES OF SUCCESSOR DISTANCE
Here we list the most relevant properties of successor distance (Myers et al. 2024). which we used in the this paper as the temporal distance. The definition of successor distance (Myers et al. (024) is independent of æ,, whereas our successor distance is dependent of π and learned in on-policy manner. We also provide proof that our on-policy form of successor distance still satisfies the quasimetric property, which differs from (Myers al. 2024).
Proposition 1. For all π E II, x. y E S, define the random variable HJ Hz( y) as the smallest transit time from x to y. i.e., the hitting time of y from x.
d5 (r, y) = - log E ( (2.0))
Proof. Starting from state x and given H*(a,, v) h). let (0t y1so x, H (x,,))
h)
denotes the probability of reaching state at the time 1, we have
p¹ = y1sa = x. Ha (z. y) h) = {f 0
ift < h ift 2 h
= (s: y1sh= =0)
(6)
And thus,
D (89 = | = x) = =(1 (1- 11-7? 7)
Y'D" (st 1 10 =0=0=)
1=0
= =(1 Σ
1'D (ot = y 80 80 H² (I, v) = h) P (H y) h)
t=0 h=0
= (1 (1-7) 7) Σ
p (11 = h) Σ
P (s: = | = x,, H1 (z, y) = h)
h=0
(1 7)
Σ p (H'(H"( (z.z) h) Σ
YPp = (s s
h=0
8
P( (H.
(1- 7)
1- h- pd (s: = y so v)
t=h
8
= "P (I,z) :0))==1 hh) (1 (1-7)
7 p" (s: = y 30 = 4)
h=
t=
( (eN0))
= Eur(es)
po (st = y 50 = y)
15

---

Published as a conference paper at ICLR 2025
Therefore,
[s = 3/30 = y) (s, = yysu = 2)) =-lg y (x
(2D(r,)) log
(8)
Corollary 1. Assume H (Ir v) is deterministic value,
ds v) c. (x, y), where e a free value.
Proof. Following Proposition
d (I. --log ( 11 (x, log
Proposition 2. dsp a quasimetric over satisfying the Positivity, Identity and triangle inequality.
Proof. A distance function d S R is called quasimetric if satisfies the following for any y. S.
Positivity: d(z,y) v) 2
2. Identity: d(z, y) = 0 + y
3. Triangle inequality: d(x, d(x,v) d(y,2)
Positivity:
From Proposition 1 we have dsn - log EH- [7 (2.0) 2 0.
Identity:
=: d y) if and only p., (s y/su x) = pT su y)) which occurs
when x = y. For x + y. H" (xx, y) 2 1, so by Proposition 1 d (x, v) 2 log 1
<: When y. (s = y|so = x) = p., (8 = y|so = y). thus dgp( v) = 0.
Triangle Inequality: According to (Hunter 2005) (Lemma 4)1). the hitting time H (xx, 2)
isfies the triangle inequality, that is, H= y) < (x, + H (M z). Let f(H*(x, (x,,)) v)) log (x..v)) log
is a convex function, and thus f is a concave function. Fur- By the property of concave functions (ictibones) (017). f subadditive, i.c. f(a + b) < f(a) + f(b) for all and b. As desired, f f(H v)) S F(H* (I, + (y, <
thermore, f(0)
f(H" (x, 2)) + f(H (y. 2)) and thus (x. f(H (I, satisfying the triangle inequality.
Proposition 3. For * y, the unique solution to the the loss function in Equation with the parametrization in Equation is
pT y030 = =y) 50 x)
dg- (z, v) log
p.,
Proof. If the batch size large enough, the optimal energy function in Equation satisfy
(s) y/so =
fā (x log
where C a free value.
(10)
C. P.
16

---

Published as a conference paper at ICLR 2025
We can further decompose the optimal function into potential function that depends solely on the future state minus the successor distance function,
(s y = 1)
P5 = y 80 = y)
S (2r, v) - -log
log
(1)
Poy (3)
(5 8080 = 2)
(v(
de(w.))
P5 slso y)
log
(u, 9) (2 x)
1 (85 = y180 x
= C.( - dz(u, v) - (c6(() -dz(2,)) v))
= (x)
17

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTS
C.1 EXPERIMENTS IN CONTINUOUS ACTION SPACE
We further conduct experiments DeepMind Control (Tassa et 2018 (DMC) and MetaWorld Yuu
et al. 2020) and HalfCheetah VeliChheattaVelSarseee
We selected three tasks with relatively sparse rewards from the DMC environment: Acrobot- swingup sparse, Hopper-hop. and Caripole-swingup sparse. As shown in Fig. 12 results demon- strate that ETD consistently performs well, showing significant improvements over PPO. We also reproduced two baselines that excelled in discrete tasks tasks-NovelD NovelD and E3B B3t-but found they couldn't maintain consistent performance across all three continuous control tasks.
DMC
Meta MetaWorld Meta-World environments typically use dense rewards. We modified these provide rewards only when tasks succeed. We tested on Reach, Hammer, and Button Press tasks, finding that PPO without intrinsic rewards performed well. This suggests that Meta- World may be ideal benchmarks for exploration problems. To our knowledge, intrinsic motivation methods haven' It been extensively tested on Meta-World, making less suitable for comparing exploration-base methods.
HalfCheetah VelSSparss We modified the Mujoco HalfCheetah environment's forward reward function to provide rewards only when the target velocity is reached. Our experiments revealed that PPO already performs well in this task. Adding intrinsic motivation methods didn't significantly improve performance, likely due to the low exploration difficulty. These environments require relatively low exploration capabilities, do necessitate intrinsic motivation methods, and are unsuitable as benchmarks for comparing exploration methods.
NovelD
Average Episode Reward
Acrobot Acrebot-siingu swingup sparse
Hoppan-hop
Cartpole wwwngup sparse
140
Figure 12: DMC Experiments. All results averaged over 5 seeds.
ETD(ours)
Reach
Hommer
Button Press
Success Rate
62
14
144
1.
24
Figure 13: MetaWorld Experiments. All results averaged over 5 seeds.
P911
Average Episod Reward
HalfChestal Velipasse
-25
Figure Halfcheetah VelSSaee Experiments. All results averaged over seeds.
18

---

Published as a conference paper at ICLR 2025
C.2 ABLATIONS ENERGY FUNCTION AND CONTRASITIVE LOSS
Previous work in contrastive RL (Bortkiewicz et al., 2024 has utilized various energy functions and contrastive loss functions for goal coal-condttooned tasks. We followed their methodology. examining the impact of different energy functions and contrastive losses on temporal distance and performance.
Specifically, we considered four energy functions: Cosine, L.2, MRN, and MRN-POT (which used in this study), defined as follows.
[0.000 v) = (o(x), o(v))
lo(x))lall(v)) 2
fo (2, y) = -Il((z) (1))l2. f0MMRN(E y) = -do v),
(12)
fo. fo..MRN=+PT y) = vly) ( y).
For contrastive loss functions, we evaluated InfoNCE Forward, InfoNCE Backward, InfoNCE Symmetric (which we employed in this work), described below.
e (a).)0))
LInfaNCT: forward (B:
Σ log
j= el(o)))))
B
(13)
LnnfoNCE-bbckwaad (B; =
Σ
log
Σ j=1 efto)
1m
LinteNCEEssymmennd (B; - IntoNCCE-oowwad
(B: bateeCCt (B; f).
Our experiments conducted noisy version of the 17x17 SprialMaze, which is similar the Disco Maze in (Badia et al.) 2020). This version introduced random colors to the walls the maze, making representation learning challenging.
COS
MRN
MRN+POT
Forward
Backward
Symmetric
Figure 15: Ablations of Energy Function and Contrastive Loss in 17x17 SpiralMaze-Noisy.
Fig. 15 illustrates the temporal distances learned for each energy function and contrastive loss function. The results indicate that the cosine energy function performed poorly distinguishing distant states. The 1.2 and MRN functions produced similar results, while MRN-POT exhibited the
19

---

Published as a conference paper at ICLR 2025
best performance, consistent with theoretical expectations. For contrastive loss functions, the overall differences were minimal.
Additionally, conducted exploration tasks in the SamnGGrid-Obstuuteeehhe IQ environment, as shown in [6 The conclusions from these tasks aligned with the results from the SprialMaze: improved learning of temporal distance correlates with higher exploration efficiency.
Fooward
Symmentt:
Pveraae Episode Reward
Average Episode
Time Steps
Time Steps
(a) Ablation of Energy Function
(b) Ablation of Contrastive Loss
Figure 16: Ablations of Energy Function and Contrastive Loss in MiniCriid-OOstrnteedMne-ee-- IQ
C.3 MORE COMPLEX EXAMPLES
We add an example of more complex maze demonstrate that our temporal distance method can effectively capture the similarity between states. As shown in the Fig. [7] our learned temporal distance remains very close to the ground truth, while inverse dynamics and likelihood methods fail.
(a) Inverse dynamic
(b) Likelihood
(c) Ours
(d) Ground truth
Figure 17: More Complex Examples
20

---

Published as a conference paper at ICLR 2025
D ENVIRONMENT DETAILS
We describe each task environment utilized in our experiments.
D.1 MINIGRID
(a) MultiRoom-N6
(b) DoorKey-16x16
(c) KeyCoridorS6R3
(d) Obstructeddaze-Full
Figure 18: Rendering of MiniGrid Environments used in this work.
MultiRoom-N6: An agent initiates from the initial room and navigates towards the goal located in the final room. The rooms interconnected via closed yet unlocked door. The variable 'N" denotes the number of rooms.
1oorKey-16116: The agent is tasked with locating key in the first room, unlocking and opening a door, and reaching the goal in the second room. 16 refers to the maze size.
KeyCorridorS6R3: The environment comprises corridor and six rooms separated by walls and doors. The agent, starting from the central corridor, must find key in unlocked room and then reach the goal behind another locked door. The prefix 'S' denotes the number of rooms, while R signifies the size of each room. 'S6R3' represents the most complex scenario within the KeyCorridor series of MiniGrid.
The agent's objective to retrieve box situated in corner of 3x3 maze. Doors are locked, keys are hidden in boxes, and doors are obstructed. The ObstructedMaze is the most challenging environment within MiniGrid, featuring multiple configurations. The notation "NDI" specifies the quantity of locked doors. The presence of a hidden key within box denoted by "h," and an obstructed door by a ball is indicated by "b." " "Q" signifies the number of quarters that will contain doors and keys out of the nine quarters the map inherently possesses.
ObstructedMaze:
ObstructedMaaz-22DD0 Features 2 locked doors with the key concealed in a box.
ObstrreteddMazze22Dlbbb Includes 2 locked doors, with the key hidden in box, and ball obstructing a door.
ObstructedMazze-I Similar to ObstructedMaze-2DDD
but with larger maximum
number steps, and the map consists of quarter of a 3x3 maze.
ObstructedMazze-20: The map encompasses half of the 3x3 maze.
ObstructedMaaze-Fulll The full 3x3 maze.
D.2 MINIWORLDD
In this work, we focus on MiniWorld-Maze with different maze sizes, where MazeS3 means the map of maze contains
tiles. Figure 19 shows the first-person view observation and the top view of the mazes. In each episode, the agent and goal are initialized at both ends of the diameter of the maze map, which ensures that they are far away enough so that the agent can not easily see the goal without exploration. A random maze will be generated in each episode. There are three possible actions: (1) move forward, (2) turn left, and (3) turn right. The agent will move 0.2 x tile ileelength moving forward, where tile length is the length of one side of a tile. The agent will rotate 90 degrees turning right/left. The time budget of each episode is x tile num. The agent will not receive any positive reward if it can not navigate the goal under the time budget.
21

---

Published as a conference paper at ICLR 2025
(a) First-person view observation
MazeS3
(c) MazeS4
(d) MazeS5
Figure 19: Rendering of MiniWorld Environments in this work.
E IMPLEMENTATION DETAILS
In the experiments, all methods are implemented based on PPO. We primarily follow the implementa- tion of DEIR] which is based Stable Baselines 3 (version 1.1.0).
E.1 NETWORK STRUCTURES
E.1.1 POLICY AND VALUE NETWORKS
For the policy and value networks, we follow the definitions of DEIR, All the methods shares the same policy and value network structures.
MiniGrid
CNN
Conv2d(in=3, out=32, kemnl=2,, stride=1, pad=0), Conv2d out=64, kernel=2, stride=l, pad=O), Conv2d(in=64, out=64, kemel=2,, stride=1, pad=0). FC(in=1024, out=64).
RNN
GRU(in664, out=644.
MLP (value network)
FC(in=64, out=128), FC(in=128, out=1).
MLP (policy network)
FC(in=64, out=128),
FC(in=128, out=number of actions).
FC stands for the fully connected linear layer, and Com22d refers to the 2-dimensional convolutional layer, GRU is the gated recurrent units.
Crafter & MiniWorld
CNN
Conv2d(in=3, out=32, kemne=8,, stride=4, pad=0), Conv2d(in=32, out=64, kernel=4, stride=2, pad=0), Conv2d(in=64. out=64, kernel=4, stride=1, pad=0). FC(in=576, out=64).
RNN
GRU(in=64, out=64).
22

---

Published as a conference paper at ICLR 2025
MLP (value network)
FC(in=64, out=256),
FC(in=256, out=1).
MLP (policy network)
FC(in=64, out=256), FC(in=256, out=number of actions).
E.1. QUASIMETRIC NETWORK
Our quasimetric network is based on the MRN (Liu et al. 2023). It consists of both a symmetric and an asymmetric component, which together determine the distance between two states. The structure of this network is illustrated in Figure 20 Our potential network shares the same CNN as quasimetric network, followed by an MLP that outputs a scalar value.
d(x,y) (a) (2(1)11 max(h2(a) hg(b))_18)
h1
h2
CNN
CNN
y
Figure 20: MRN Network
E.2 HYPERPARAMETERS
We found that applying batch normalization all RoN- layers could significantly boost the learning speed, especially in environments with stable observations, a finding also noted in the DEIR paper. We use Adam optimizer with e = le 5, 31 0.9, 32 = 0.999. We normalized intrinsic rewards for all methods by subtracting the mean and dividing by the standard deviation.
For ETD, hyperparameters were initially tuned on DoorKey-8x8 and refined using KeyCorridorSGR3 and ObsturctedMaze results. For DEIR, we adopted their original hyperparameters but couldn" 't fully performance. Despite this, ETD still outperforms original DEIR
replicate their ObsturctedMaze-FFFi performance by factor of two in sample efficiency in ObsturetedMaze-Funt Full. Our NovelD implemen- tation achieves the best performance reported in the literature. For the Count implementation, we use the episodic form (s,) 1] and found superior to 1 A/VNN(s,) The hyperparameters for each method are summarized in following tables. Unless otherwise specified, the hyperparameters are consistent with those used for ETD.
23

---

Published as a conference paper at ICLR 2025
MultiRoom
Hyperparameter
DoorKey KeyCorridor
ObsturctedMaze Candidate Values
0.99
0.99
PPO ÄGAE
0.95
0.95
PPO rollout steps
512
512
PPO workers
16
16
PPO clip range
0.2
0.2
PPO training epochs
4
PPO learning rate
3e-4
3e-4
model training epochs
8
4
1,3,4,
mini-batch size
512
512
entropy loss coef advantage normalization
5e-4
le-2
5c-4, le-2
yes
yes
model learning rate normalization for layers
3e-4
3e-4
3e-4, le-4, 5e-5, 1e-5, 5e-6
Batch Norm
Layer Norm
Batch Norm, Layer Norm, None 1,
extrinsic reward coef
1.0
10.0
intrinsic reward coef
le-2
le-2
le-2, 1e-3, 5e-3, 1e-4
Table Hyperparameters for ETD in MiniGrid.
MultiRoom
Hyperparameter
DoorKey KeyCorridor
ObsturetedMaze Candidate Values
PPO learning rate model training epochs
3e-4
le-4
3e-4, 1e44 1, 6, 8
4
3
mini-batch size entropy loss coef model learning rate normalization for layers extrinsic reward coef
512
512
le-2
1e-2
5e-4, 1e-2
3e-4
3e-4
3e-4, Ic-4, 5e-5, le-5, 5e-6 Batch Norm, Layer Norm, None 1, 10
Batch Norm
Layer Norm
1.0
10.0
intrinsic reward coef
3e-2
3e-3
le-2, le-3, 5e-3, le-4
0.5
0.5
/
0
O
/
Table Hyperparameters for NovelD MiniGrid.
MultiRoom DoorKey KeyCorridor
Hyperparameter
ObsturctedMaze Candidate Values
PPO rollout steps
512
512
256, 512
PPO workers
16
64
16, 64
PPO learning rate
3e-4
1e-4
model training epochs
4
3
/
mini-batch size
512
512
512, 2048
entropy loss coef model learning rate normalization for layers
le-2
5e-4
/
3e-4
3e-4
Batch Norm
Layer Norm
Batch Norm, Layer Norm, None
extrinsic reward coef intrinsic reward coef
1.0
10.0
le-2
le-3
/
observation queue size
le5
le5
/
Table 4: Hyperparameters for DEIR in MiniGrid.
24

---

Published as a conference paper at ICLR 2025
MultiRoom DoorKey KeyCorridor
Hyperparameter
ObsturctedMaze Candidate Values
PPO learning rate mini-batch size entropy loss coef normalization for layers extrinsic reward coef intrinsic reward coef
3e-4
3e-4
/
512
512
/
le-2
5e-4
/
Batch Norm
Layer Norm
Batch Norm, Layer Norm, None
1.0
10.0
le-2
le-3
/
Table Hyperparameters for Count in MiniGrid.
MultiRoom DoorKey
Hyperparameter
ObsturctedMaze Candidate Values
KeyCorridor
PPO learning rate mini-batch size entropy loss coef normalization for layers extrinsic reward coef intrinsic reward coef
3e-4
3e-4
/
512
512
/
le-2
le-2
Batch Norm
Layer Norm
Batch Norm, Layer Norm, None
1.0
10.0
/
le-2
le-2
/
1
/
1
F
90-th percentil
/
Table Hyperparameters for EC MiniGrid.
MultiRoom DoorKey
Hyperparameter
ObsturetedMaze Candidate Values
KeyCorridor
PPO learning rate mini-batch size entropy loss coef normalization for layers extrinsic reward coef intrinsic reward coef
3e-4
3e-4
/
512
512
/
le-2
le-2
Batch Norm
Layer Norm
Batch Norm, Layer Norm, None
1.0
10.0
/
le-2
le-2
/
0.1
0.1
/
Table 7: Hyperparameters for E3B in MiniGrid.
MultiRoom DoorKey
Hyperparameter
ObsturctedMaze Candidate Values
KeyCorrdoor
PPO learning rate mini-batch size entropy loss coef normalization for layers extrinsic reward coef intrinsic reward coef
3e-4
3e-4
/
512
512
/
le-2
le-2
/
Batch Norm
Layer Norm
Batch Norm, Layer Norm, None
1.0
10.0
/
3e-3
1e-2
/
Table 8: Hyperparameters for RND in MiniGrid.
25

---

Published as a conference paper at ICLR 2025
Hyperparameter
ETD
NovelD
DEIR
0,99
0.99
0.99
PPO ÁGAE
0.95
0.95
0.95
PPO rollout steps
512
512
512
PPO workers
16
16
16
PPO clip range
0,2
0.2
0.2
PPO training epochs
4
4
4
PPO learning rate
3e-4
3e-4
3e-4
model training epochs
4
4
4
mini-bater size
512
512
512
entropy loss coef advantage normalization
le-2
le-2
le-2
yes
yes
yes
model learning rate normalization for layers
le-4
lc-4
le-4
Layer Norm
Layer Norm
Layer Norm
extrinsic reward coef intrinsic reward coef
1.0
1.0
1.0
le-2
le-2
le-2
0.5
observation queue size
le5
Table 9: Hyperparameters for ETD, NovelD and DEIR in Crafter.
Hyperparameter
ETD
Novell)
DEIR
0.99
0.99
0.99
PPO ÀGAE
0.95
0.95
0,95
PPO rollout steps
512
512
512
PPO workers
16
16
PPO clip range
0.2
0.2
0.2
PPO training epochs
4
4
PPO learning rate
3e-4
3e-4
3e-4
model training epochs
16
4
4
mini-batch size
512
512
512
entropy loss coef advantage normalization
le-2
1e-2
le-2
yes
yes
yes
model learning rate normalization for layers extrinsic reward coef intrinsic reward coef
le-4
le-4
le-4
Layer Norm
Layer Norm
Layer Norm
10.0
1.0
1.0
le-2
le-2
le-2
0.5
0
observation queue size
le5
Table 10: Hyperparameters for ETD, NovelD and DEIR in World.
26

---

Published as a conference paper at ICLR 2025
Hyperparameter
ETD
NovelD
E3B
0.99
0.99
0.99
PPO ÀGAE
0.95
0.95
0.95
PPO rollout steps
512
512
512
PPO workers
16
16
16
PPO clip range
0.2
0.2
0.2
PPO training epochs
4
4
4
PPO learning rate
3e-4
3e-4
3e-4
model training epochs
4
4
4
mini-batch size
512
512
512
entropy loss coef advantage normalization
1e-2
1e-2
le-2
yes
yes
yes
model learning rate normalization for layers
3e-4
3e-4
3e-4
Layer Norm
Layer Norm
Layer Norm
extrinsic reward coef
1.0
1.0
1.0
intrinsic reward coef
le-2
le-2
le-2
0.5
AgAR
0.1
Table 11: Hyperparameters for ETD, NovelD and E3B in DMC, MetaWorld and HalfCheetah Vec- Sparse.
27


Reviews:
[{"review_id": "8amZlhXeoQ", "reviewer": "Reviewer_uqHy", "summary": "This paper introduces Episodic Novelty through Temporal Distance (ETD), a new approach for exploration in Contextual Markov Decision Processes (CMDPs) with sparse rewards. ETD uses temporal distance as a metric for state similarity to compute intrinsic rewards, encouraging agents to explore states that are temporally distant from their episodic history. The authors employ contrastive learning to design a quasimetric that estimates episodic temporal distances. They evaluate  ETD when combined with PPO on various CMDP benchmarks, i.e., MiniGrid, and Crafter (pixel-based observations), and MiniWorld (pixel-based observations).", "questions": "1. Could the authors shed light on why PPO, an on-policy algorithm, was chosen to be combined with ETD instead of off-policy algorithms such as DQN or SAC? Prior work has used off-policy algorithms extensively for sparse-reward tasks [2] [3].\n2. Is ETD straightforward to be extended to continuous action spaces? If so, did the authors run any experiments? If not, what could be possible bottlenecks?\n3. Did the authors consider combining temporal distance with reward-based similarity, or explore weighting the temporal distance based on extrinsic rewards? Are there any potential trade-offs or challenges in incorporating extrinsic rewards into the similarity metric that the authors anticipate?\n\n[2] Andrychowicz, Marcin, et al. \"Hindsight experience replay.\" Advances in neural information processing systems 30 (2017).\n\n[3] Co-Reyes, John D., et al. \"Evolving reinforcement learning algorithms.\" arXiv preprint arXiv:2101.03958 (2021).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel method to use temporal distance as a (quasi)metric for state similarity.\n2. The paper conducts extensive experiments across multiple CMDP environments, comparing ETD to several baseline methods.\n3. ETD + PPO demonstrates robust performance improvements, especially in challenging sparse reward scenarios.\n4. Results on extensive experiments across multiple CMDP environments, comparing ETD to several baseline methods have been reported.\n5. The paper is well-structured, with clear explanations of the motivation, methods, and results.", "weaknesses": "1. The proposed ETD doesn’t take into consideration extrinsic rewards to compute similarity. Intuitively, states with similar rewards could be considered similar in terms of the task objective [1].\n2. The approach has been primarily tested on discrete action spaces, and its effectiveness in continuous action domains such as MuJoCo [2], DeepMind Control Suite [3], or Fetch [4] environments remains unexplored.\n\n[1] Agarwal, Rishabh, et. al. “Contrastive behavioral similarity embeddings for generalization in reinforcement learning.” arXiv preprint arXiv:2101.05265 (2021).\n\n[2] https://gymnasium.farama.org/environments/mujoco/\n\n[3] Tassa, Yuval, et al. \"Deepmind control suite.\" arXiv preprint arXiv:1801.00690 (2018).\n\n[4] Plappert, Matthias, et al. \"Multi-goal reinforcement learning: Challenging robotics environments and request for research.\" arXiv preprint arXiv:1802.09464 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uy0iZdYNTX", "reviewer": "Reviewer_gFtv", "summary": "This paper presents a novel reinforcement learning approach to measure similarities between visited states, using the resulting state distances as a reward bonus to encourage exploration in tasks with sparse rewards. The methodology is technically sound, and the results are promising.", "questions": "Please address the issues listed in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, with a clear and cohesive narrative. Most technical details are effectively conveyed through illustrative figures and results from intuitive toy tasks.\n- The experimental tasks are appropriately chosen, providing sufficient complexity to evaluate the approach.\n- The experimental results, along with ablation studies, clearly demonstrate the advantages of the proposed method over other baselines.\n- The paper offers a comprehensive analysis and comparison of different technical approaches used by the proposed method and baseline methods, providing valuable insights for readers.", "weaknesses": "## MDP assumption\nThe definition of the intrinsic bonus reward violates the MDP assumption. In section 2, the total reward $r(s_t, a_t, s_{t+1})$ is a decomposed as the environment reward $r_t^e$ plus the weighted bonus $\\beta b_t$. However, $b_t$ is a function that depends on the visited states within the episode, which disrupts the definition of total reward and violates the MDP assumption. In this case, the visited states influences the intrinsic reward, potentially harming the policy built upon MDP, as the actions are solely selected by $\\pi(a_t|s_t)$ without knowledge of the states' visiting history. Consequently, the same action $a_t$ may yield completely different rewards $r_{t+1}$, depending on whether the $s_{t+1}$ has been visited in the history or not. I believe this is a critical technical issue in this paper, and should be explained thoroughly. At the same time, do other related works using reward bonuses also encounter the same issue?\n\n## Unclear content\nThe content between lines 186 and 208 is challenging to follow. I suggest the authors restructure this section, providing additional explanations and intuitions. Specifically:\n- The meaning of the sentence, \"Contrastive learning can estimate the successor distance when positive samples are drawn from the discounted state occupancy measure,\" is unclear. What does it mean to draw samples from a measure?\n- The motivation for using the energy function and its decomposition into the potential net and quasimetric net in practice is unclear. What is this energy function, and why is it applied here?\n- The rationale behind using the infoNCE loss also needs further explanation.\n\n## Minor issues\n- The SpiralMaze in Figure 2 appears to have a linear structure, which may not effectively demonstrate ETD's advantage in distance learning. A more complex example could better illustrate the method's strengths.\n- What is the best practice of choosing the $\\beta$? I think it depends on the specific task and is therefore a HP.\n- Line 188, there should be a $ds$ in the integral.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAUQKJEKI3", "reviewer": "Reviewer_TiSQ", "summary": "The paper presents a new intrinsic reward for novelty-based reinforcement learning. The intrinsic reward is based on the estimated temporal distance between two states, which estimates the probability that one state can be reached from another state in multiple time steps. Here, discounting is used to give preference for shorter transitions. The temporal distance is learned by a contrastive loss based on the infoNCE loss. The algorithm is compared against several baseline methods on a wide variety of discrete action exploration tasks and shows promising performance.", "questions": "- a more detailed discussion on the successor distance and its assumptions would be insightful. E.g., what happens if the MDP does not allow for loops (which is the case we have to deal with real dynamics, e.g. velocities). Then this distance would be infinite, wouldnt it?\n- More insights on the parametrization of the energy function. Why do we need separate networks for  the potential and the assymetric distance? Could we also ablate removing the potential here?\n- Why do we need to \"symmetrize\" the infoNCE loss? Does this imply that transitions are \"invertible\"? Also here, a ablation and more discussion would be nice.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A well motivated new intrinsic reward signal for RL\n- The concept is simple and provides a strong performance\n- Exhaustive evaluations, good comparisons the baseline methods as well as nice ablations\n- Good insights why baseline methods fail", "weaknesses": "the paper is already of high quality and I could not find major weaknesses. Minor weaknesses are given in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KqptbRMQRt", "reviewer": "Reviewer_akpb", "summary": "This paper addresses the challenge of exploration in sparse reward environments within Contextual Markov Decision Processes (CMDPs), where environments vary between episodes. The authors propose Episodic Novelty Through Temporal Distance (ETD), a method that leverages temporal distance as a metric for state similarity and intrinsic reward computation. Using contrastive learning, ETD estimates temporal distances to assign intrinsic rewards based on the novelty of states within an episode. Experimental results on diverse benchmark tasks show that ETD outperforms state-of-the-art methods, enhancing exploration in sparse reward CMDPs. The paper is well-structured and detailed, and the ETD method is well-described.", "questions": "1. Please test the performance of ETD on continuous control tasks.\nTo evaluate the ETD method in continuous control tasks, the authors could consider tasks in environments such as Meta-World (or Panda-Gym) and Mujoco. Examples of basic tasks include \"Push,\" \"Reach,\" and \"Pick-Place\" in Meta-World (or Panda-Gym), which offer high environmental complexity and sparse reward settings, making them suitable for assessing ETD's exploratory capabilities. Additionally, Mujoco environments like \"Cheetah-Vel-Sparse,\" which have high-dimensional state spaces, could further validate ETD’s performance in high-dimensional continuous spaces. By plotting and comparing learning curves across environments—such as average episode rewards or success rates over training steps—the authors could demonstrate whether ETD converges faster or more stably in sparse reward continuous control tasks compared to other methods.\n\n2. Please provide code.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper compares the proposed method with various existing approaches (including DEIR, NovelD, E3B, EC, NGU, RND, and the Count-based method), covering many classical and common algorithms in the exploration field. By testing these methods across multiple benchmark environments, the paper demonstrates the advantages of the ETD method in sparse reward and high-dimensional pixel environments, validating its applicability across different exploration demands and tasks.", "weaknesses": "1. The core approach of ETD primarily builds upon the main idea of \"Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making,\" using contrastive learning to estimate temporal distance. It lacks substantial algorithmic innovation.\nThe innovation of the ETD (Episodic Temporal Distance) method primarily lies in its approach to learning temporal distances. However, this method closely follows key aspects of the referenced work, \"Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making,\" in several crucial areas:\n1) Discounted State Occupancy Measure: In the ETD method, the discounted state occupancy measure p^\\pi_\\gamma(s_f = y | s_0 = x) is used to estimate the transition probability from state x to y . The formula is as follows: p^\\pi_\\gamma(s_f = y | s_0 = x) = (1 - \\gamma) \\sum_{k=0}^{\\infty} \\gamma^k p^\\pi(s_k = y | s_0 = x) This formula originates from \"Learning Temporal Distances,\" where the discounted state occupancy measure p^{\\pi}_{\\gamma} is defined as the discounted distribution over future states s^{+} : p^{\\pi}_{\\gamma}(s^{+} = s' | s_0 = s) = (1 - \\gamma) \\sum_{k=0}^{\\infty} \\gamma^k p^{\\pi}(s_k = s' | s_0 = s)\n2) Successor Distance: The successor distance formula used in ETD is also directly derived from the referenced work, \"Learning Temporal Distances.\"\n3) Loss Function: Both works adopt a symmetrized InfoNCE loss function to train the model through contrastive learning, which is a critical component of the approach.\n4) Energy Function: The energy function utilized in ETD, f_{\\theta = (\\phi, \\psi)}(x, y) = c_{\\psi}(y) - d_{\\phi}(x, y) , is also sourced from \"Learning Temporal Distances,\" where it is presented as: f_{\\phi, \\psi}(s, a, g) = c_{\\psi}(g) - d_{\\phi}(s, a, g)\n\n\n2. Experiments are conducted solely on discrete control tasks, with no tests on continuous control tasks (e.g., robot control tasks in Meta-World and Mujoco). State spaces in discrete control tasks are relatively small, allowing agents to explore enough states within a shorter timeframe. In contrast, continuous control tasks present larger state spaces and greater exploration challenges, making it essential to validate the method’s effectiveness in these scenarios. Given that many real-world tasks involve continuous control, this omission limits the method's persuasiveness for broader application scenarios.\n3. The absence of publicly available code restricts the ability to independently verify the method's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)