Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards
Authors: Faisal Mohamed
Catherine Ji
Benjamin Eysenbach
Glen Berseth
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
exploration
intrinsic motivation
surprise
empowerment
contrastive learning
Primary area: reinforcement learning
TL;DR: We propose an exploration method based on temporal contrastive representation, our method maximizes state coverage as perceived through the lens of these learned representations.

Abstract: Effective exploration in reinforcement learning requires not only tracking where an agent has been, but also understanding how the agent perceives and represents the world. To learn powerful representations, an agent should actively explore states that contribute to its knowledge of the environment. Temporal representations can capture the information necessary to solve a wide range of potential tasks while avoiding the computational cost associated with full state reconstruction. In this paper, we propose an exploration method that leverages temporal contrastive representations to guide exploration, prioritizing states with unpredictable future outcomes. We demonstrate that such representations can enable the learning of complex exploratory behaviors in locomotion, manipulation, and embodied-AI tasks, revealing capabilities and behaviors that traditionally require extrinsic rewards. Unlike approaches that rely on explicit distance learning or episodic memory mechanisms (e.g., quasimetric-based methods), our method builds directly on temporal similarities, yielding a simpler yet effective strategy for exploration.

Published as a conference paper at ICLR 2026
TEMPORAL REPRESENTATIONS FOR EXPLORATION: LEARNING COMPLEX EXPLORATORY BEHAVIOR WITH- OUT EXTRINSIC REWARDS
Faisal Mohamed Catherine Ji
Benjamin Eysenbach
Glen Berseth 1.2,*
'Mila-Quebe' Al Institute, Université de Montréal
Princeton University
faisal www.mehhhaaiiiiiccmcmmmmcc la. hew.eeee..i.ueueeceeeoooommmee cj7280@princeton.ccm
ABSTRACT
Effective exploration reinforcement learning requires not only tracking where an agent has been, but also understanding how the agent perceives and represents the world. To learn powerful representations, an agent should actively explore states that contribute its knowledge of the environment. Temporal representations can capture the information necessary to solve a wide range of potential tasks while avoiding the computational cost associated with full state reconstruction. In this paper, we propose an exploration method that leverages temporal contrastive representations to guide exploration, prioritizing states with unpredictable future outcomes. We demonstrate that such representations can enable the learning of complex exploratory in locomotion, manipulation, and embodied-Al tasks, revealing capabilities and behaviors that traditionally require extrinsic rewards. Unlike approaches that rely on explicit distance learning or episodic memory mechanisms (e.g., quasimetric-bar methods), our method builds directly on temporal similarities, yielding simpler yet effective strategy for exploration.
INTRODUCTION
Exploration remains a key challenge in reinforcement learning (RL), especially in tasks that de- mand reasoning over increasingly long horizons (Thrun, 1992) or with -ishesnmennional observa- tions (Stadie et al., 2015; Burda et al., 2019b; Pathak et al., 2017).
Effective exploration in high-dimensional settings requires that agents (futilely) do not attempt to visit every last state, but only visit those states where they have something to learn. But how can an RL agent recognize such states? One direction to leverage representation learning compress the observations into meaningful space where the agent can measure some sense of "usefulness," to drive and guide exploration. This raises question Which representations should be used drive exploration?
We start by observing that the RL problem is fundamentally about time, so representations that reflect temporal structure should be more useful than those that additionally include all bits required to reconstruct the input. We therefore adopt representations acquired by temporal contrastive learning. Theoretically, such representations function (Mazoure et al., 2023) (they are effectively a kernelized successor representation (Dayan, 1993; Barreto al., 2017)). Computationally, these representations avoid the computational costs associated with world models and reconstruction (Achiam & Sastry, 201 7; Stadie et al., 2015; Sekar al., 2020; Bai et al., 2020). Indeed, prior work has shown that such representations are useful for learning policies (Myers et al., 2025) and value functions (Laskin et al., 2020). In our method, we use these representations reward the agent for visiting states with unpredictable futures.
appealing because they sufficient statisties for any Q
Our work is closely related to Jiang et al. (2025), which uses contrastive learning to estimate similarity metric for exploration via quasimetric learning and constructs a reward signal using an episodic memory. Our method differs by (1) avoiding quasimetric learning and (2) avoiding episodic
"quual advising. Project website: https: : http:p://eeppppee http://eerrccotraeeeeeeeeeoeeeee rasti ve-expl w.w.gitt..u.........

---

Published as a conference paper at ICLR 2026
Figure 1: Curiosity-Driven Exploration via Tempo- ral Contrastive Learning. We learn temporal rep- resentations so that the representation of (8o, 4o) is more similar to ($..3.1....). We reward the agent for visiting future states that seem unpredictable. For
Nearby have higher temporal similarity
Visiting states far in the future more rewarding
(53)
($1)
((50, ao)
// (844)
($2)
trajectory
S
S
S2
$3
$4
example, from state S0, 81 should confer lower reward than the state $4-
a
a1
a2
d3
memory, which makes our method more amenable to off-policy RL algorithms. A graphical summary of our method is shown in Fig. 1.
The main contribution of this work a new objective for exploration based on the prediction error of temporal representations. We demonstrate approach maximizing these intrinsic rewards with PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018c). Our approach achieves state-of-the- art state coverage across navigation (Ant and Humanoid mazes), manipulation, and open-world environments (Craftax Searaffax-Classsio
RELATED WORK
Exploration. Prior work exploration (Schmidhuber, 2010; sch, 1991; Sorg et al., 2012; Brafman TTnnenholtz, 2002; Kearns & Singh, 2002) in reinforcement learning has proposed a variety of task-agnostic methods for encouraging agents to acquire diverse behaviors without relying on external rewards(also konwn as unsupervised RL) (Laskin et al., 2021). A central line of research focuses on intrinsic motivation, where agents seek novelty by maximizing state coverage surprise. In low-dimensional and/or discrete enviromments, count-basee exploration methods (Gardeux et al., 2016; Bellemare et al., 2016; Tang et al., 2017: Ostrovski et al., 2017; Martin et al., 2017; Xu et al., 2017; Machado et al., 2020) have demonstrated effective performance, particularly in Atari games. However, these methods often struggle in high-dimensional or continuous state spaces. In such settings, prediction-ereee ceceriion-errrr--ee error-based approaches (Pathak et al., 2017; Burda et al., 2019a;b; Lee et al., 2019) have been more successful, showing effectiveness both in video game environments and continuous control tasks. Another direction leverages representation learning: compact features extracted from raw inputs, and entropy estimators are applied to these representations to quantify novelty (Liu & Abbeel, 2021; Laskin et al., 2022).
Beyond novelty-driven exploration, another class of methods emphasizes the agent's ability to influence regulate its environment. Empowerment-basse approaches maximize mutual information between states and actions, encouraging agents to discover actions that yield significant control over future states (Klyubin al., 2005b;a; Bichl al., 2015; Zhao et al., 2021; Mohamed & Jimenez Rezende, 2015; Karl et al., 2019; Hayashi & Takahashi, 2025; Levy et al., 2024; Jung al., 2011; Du al., 2020; Myers al., 2024). While conceptually appealing, solving the full empowerment objective remains intractable. A complementary perspective is surprise minimization, where agents reduce prediction uncertainty to maintain stability or create structured niches in the environment (Friston, 2010; Berseth et al., 2021; Rhinehart et al., 2021; Hugessen et al., 2024). These approaches demonstrate how regulating predictability can give rise to complex behaviors both fully and partially observed domains.
Representation learning for RL. Prior work on representation learning for RL focuses on self- supervised methods to improve the data efficiency of RL agents. A notable approach in this category involves the use of unsupervised auxiliary tasks, where a pseudo-reward is added to the task reward to shape the learned representations and provide additional training signal. Examples of this approach include (Jaderberg et al., 2017; Farebrother et al., 2023; Oord et al., 2018; Laskin et al., 2020; Schwarzer al., 2021). Another line of work focuses forvard-backward representations (Touati &
2

---

Published as a conference paper at ICLR 2026
Ollivier, 2021; Touati et al., 2023), which aim to capture the dynamics under all optimal policies and have been shown to exhibit zero-shot generalization capabilities. Moreover, contrastive learning has been applied in various exploration settings, including goal-conditioned learning (Eysenbach et al., 2022; Liu et al., 2025), skill discovery (Laskin et al., 2022; Yang et al., 2023; Zheng et al., 2025), and state coverage curiosity (Liu & Abheel, 2021; Du et al., 2021; Yarats et al., 2021). In the context of curiosity-driven exploration, (Du et al., 2021; Yarats et al., 2021) employ contrastive learning to learn visual representations in image-based environments, where the RL agent is trained to maximize the error of the representation learner (similar in spirit to prediction-erve approaches), Our work is similar to these method as it also uses contrastive learning, but for learning representations that capture the temporal structure of the policy and environment dynamics, without explicit world-mod,,ing, or skill learning. Jiang et al. (2025) uses a special parametrization of contrastive learning to learn temporal distances via quasimetric learning. It then constructs an aggregated intrinsic reward to maximize the minimum temporal distance between the state at the current time step and the states from previous time steps, which are stored in episodic memory. Our work is closely related to Jiang et al. (2025), which likewise uses contrastive learning to estimate a similarity metric for exploration. Our method differs by (1) avoiding the quasimetric parametrization and (2) avoiding episodic memory, which makes our method more amenable to the off-policy setting. We compare with ETD (Jiang et al., 2025) in the experiments.
3 BACKGROUND
We consider controlled Markov process defined by time-indexed states 8, and actions d, The initial state is sampled from Po())), and subsequent states are sampled from the Markovian dynamics p(5(+1 $t: az). Actions are selected by stochastic, parameterized policy <( st). Without loss of generality, we assume that episodes have an infinite horizon; the finite-horizon problem can be incorporated by augmenting the dynamics with an absorbing state. The key to C-TeC is to a self-supervised, or intrinsic reward, built on temporal contrastive representations. We detail the necessary preliminaries below.
Discounted state occupancy measure. Formally, we define the 7-discounted state occupancy measure of policy conditioned on a state and action (Ho & Ermon, 2016; Eysenbach et al., 2021; 2022) as
Pa(sy | 8z, ( (1-7)
(1
Σ Pr = 8ts Qt),
A=U
where pr (sy s, is the probability of being at future state sy conditioned on 8t, dr and following policy т. In continuous settings, the future state distribution p=(5f 8, a) is a probability density.
Traditionally, the discounted state occupancy measure is defined with respect to a policy as (s 8t, a.). However, in this work, the intrinsic reward Fintr is defined using a discounted state occupancy measure over the trajectory buffer T, which contains trajectories collected from history of policies:
PT (85 88,a,) a.) A (1 77)
Σ A P = =85 81. 8t.
A=0
To sample from the trajectory buffer distribution (T(s) St. ot)), we first sample an offset A GEOM(1 - 7)), then set the future state 8 = 81+4- Here, future state 8 8 is the state reached from (st, 0z) after executing n-number of actions within a sampled stored trajectory.
Contrastive learning. Contrastive representation learning methods (Chopra et al., 2005; Oord et al., 2018; Chen et al., 2020) train a critic function Co that takes as input pairs of positive and negative examples, and learn representations so that positive pairs have similar representations and negative pairs have dissimilar representations. To estimate the discounted state occupancy, positive examples sampled from a joint distribution pr (($t, at), 8y) = PT PT aq (s) st, at). while the negative examples are sampled from the product of marginal distributions PT (St, as (r)PP((s), Here, pr (s)) is the marginal discounted state occupancy pr(s))
Jf P-(s; st, a4 (p (st, az) ds, day- We use the InfoNCE loss to train the contrastive learning model (Oord et al., 2018). Let B =
(i) ( ( ) be the sampled batch, where 8f is the positive example and {s (2(2:K) are the K negatives sampled independently from (5 a(()). In addition the standard InfoNCE
{(s()) at .8
3

---

Published as a conference paper at ICLR 2026
objective, prior work has shown that a LogSumExp regularizer is necessary for control (Eysenbach et al., 2021). The full contrastive reinforcement learning (CRL) loss is as follows:
Coll(era)):a)))) )/T
(a)
E (e) (s),a))
log
(2)
K Σ Collssrae) F=1
o uPt (syls),as)
(p(k)~pp(s))
where 7 is a temperature parameter. The optimal critic C' ("s(ss, az), 8y) corresponds to a log probabil- ity ratio (Ma & Collins, 2018), C* ((st. sy).sy) logPr(s; 8t, at) (T where we use the negative E and e distances as the critic function (see Appendix G.3) Conceptually, the critic Co gives temporal similarity score between state-action pairs (St, az) and future states sy via learned representation @9 and 18g.
EXPLORATION VIA TEMPORAL CONTRASTIVE LEARNING
To improve exploration, we learn representations that encode the agent's future state occupancy using temporal contrastive learning. We begin by deseribing how contrastive representation learning can be used to estimate state occupancy by learning a similarity function that assigns high scores to frequently visited future states and low scores rarely visited ones (Eysenbach al., 2022; Oord et al., 20. We then explain how this similarity score can be leveraged to derive an intrinsic reward signal for exploration.
4.1 TRAINING THE CONTRASTIVE MODEL
The contrastive model Co(se, Ot, 8y) is trained on batches B of (St, oe, sy) tuples, where each sy is sampled from the discounted future state distribution (Section 3). We use two parameterized encoders to define the contrastive model: Φy (S ae) for state-action pairs and (ha(s)) for future states. A batch of state-action pairs {(s()) aj (2))
} is passed through óg, while the corresponding batch of future
states { Ji=1 is passed through qlg. The resulting representations are then normalized have unit norm. To compute the similarity between representations in practice, we found that using either the negative L or 6 norm was effective, depending on the environment. The contrastive encoder is trained minimize the InfoNCE loss (Equation (2)) (Oord et al., 2018). . For each batch sample, the positive examples of other samples are treated negatives, following common practice (Chen et al., 2020). The temperature parameter is learned during training. The details of the implementation are provided in Appendix E.
4.2 EXTRACTING AN EXPLORATION SIGNAL FROM THE CONTRASTIVE MODEL
Given the contrastive model, a useful intrinsic reward can be constructed. The aim is to reach unexpected but meaningful states. This is in contrast surprise maximization or similar objectives which may prioritize unexpected but meaningless (i.e. random) states as those observed in the Noisy TV problem (see Figure 21) (Gruaz et al., 2024).
The contrastive model produces a similarity score between state-action pairs (st, az) and future states 8,, Negating this similarity score results in our exploration signal Timtr encouraging the agent to visit states that are not predictive of future states in the same trajectory in the eyes of the representations. The expression for the expectation of Time is as follows:
F[rimr(#8 ):) (pp(eele (-Co at), 84)] Epr(erlenn)) [lloo(se, (8,, ar) tho (st)
where the norm can be taken be L or L2 (See Section 6). Here, Fintr rewards the agent for exploring states that provide the least amount of information about future states. The reward captures both temporal distance and possible inconsistencies in the model, where the representations assign erroneously low relative likelihoods to future states (see Section 5.2 for analysis).
We use PPO (Schulman al., 2017) and SAC (Haarnoja et al., 201 (018b;;;c) for policy training (pscudocode Algorithm 1). In practice, we found that using a single sample future state to approximate the expectation in Equation (3) works well, except in Praftax-Classic where used
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 Curiosity-Driven Exploration via Temporal Contrastive Learning
1: Initialize: Qp, Vo,trajectory buffer T
2: for each iteration do
3: for each environment step 1 S < do
4:
de ~ T(a st), S141 ~ p(5++1 8c. at).
5:
Ty Tj (s 01,
6:
T TUG,, f
7:
Sample ((5), a;) T
Sample a batch of state,action pairs
8:
Sample A; ~ GEOM -1) vi E [1, .... [B]
> Sample geometric offsets
9:
Set 8 S44, vi 21.,...., [B]]
Set the future state sy; according A;
10:
Compute intrinsic rewards: r; = CC a)),
>> Equation (3)
11:
Update representations: 0 t 1/ CIaIONCC = {(s), ai st))
& Equation (2)
12:
RL update using {(st, af. r
Update the policy using PPO/SAC
Monte Carlo estimate. Additional details are provided in Appendix E.4.1, furthermore, we ablate the future state sampling strategy. We observe that sampling from the discounted occupancy measure yields good performance across environments and we stick to this strategy in our experiments. We also show the performance differences between these sampling strategies in Appendix G.6.
INTERPRETATION OF C-TEC
In the below sections, we provide intuition for how the catpeeettatioo--pprrmmtterzzeeei intrinsic reward may drive effective exploration behavior. Sec. 5.1 details an info-theoretic interpretation of C-TeC, ignoring learned representations to help build intuition and compare with other common objectives (see Appendix I). Scction 5.2, we discuss the importance of the learned contrastive representations to C-TèC performance. For example, contrastive representations enable C-TeC's performance to remain the same with or without noise in the Noisy TV environment (Figure 21).
5.1 INFORMATION-THEORRER EXPRESSION OF C-TEC
The intrinsic reward has an information-theoret interpretation. We consider the limit where repre- sentations perfectly capture the underlying point-wise mutual information (MI). In this regime, the intrinsic reward evaluates the negative of the KL-divergence between the conditional future-state distribution pr( 04) and the marginal future-state distribution Pr (sy):
E(rint (81, at)] = PRpressleeeeee log PT(sy St a4)
PT($))
- (P 86, at) | (sy)]
This intrinsic reward describes mode-seeking behavior (Murphy, 2022, Section 6.2.6): the conditional should only have support where the marginal (T has support. This optimization is distinct from minimizing the forward KL-divergence [xt.pp (8y) ||PT(s | St, at)]. which instead prioritizes mean-seeking behavior over regions of the state space where the marginal may have support.
This mode-seeking reward can be interpreted as prioritizing (s, a) that are minimally informative about reached future states:
E[ (St, a+)] = - -Dx DKLLPPT(8) 8t, at) (s))
H[S st, ail + Epr logpr(s)))
surprise
"famillirity" term
where Sy denotes the future state random variable and 85 PT(ss with spread-out trajectories ("surprise") over states that have actually been seen ("familiarity") have higher reward i.e., high reward should be given to states that have been visited but temporally distant, rather than giving a high reward to unvisited states.. States encountered during roll-out are then added the marginal, and the process repeats.
az). State-action pairs
5

---

Published as a conference paper at ICLR 2026
The success of the intrinsic reward cannot solely be attributed to "fitting" pr(sy s, a, to pr(s)) via the policy rollout distribution. To test the hypothesis that the mode-seeking behavior is important, experiments where the intrinsic reward is the forward mean-secking KL (Equation (20)). Appendix G.5 shows that the objective succeeds because it is minimizing this mode-seeking formula- tion of the KI. rather than fitting the conditional future states to broad marginal. A linear stability analysis on the fixed points of C-TeC is in Appendix J.1; we simplify the problem setting for analysis and find that there are no easily-achievable stable fixed points for general nontrivial MDPs, meaning that the distribution over reached states continually evolves with iteration time.
5.2 REPRESENTATIONS ARE NECESSARY FOR C-TEC TO SUCCEED
The representations not only capture a raw info-theoretic exploration signal but also a form of representation prediction error. All of the analysis in Section 5.1 assumes fully-expressive critic that perfectly captures the point-wise MI. However, the true learned representations only approximate the point-wise MI. The full expected intrinsic reward the following
Po.w(s5 8t; 0e)
E om v (st, az)l -pp log
(o,)(s))
where P describe probability ratios under the learned contrastive representations e and tb. Thus, the reward prioritizes exploration in areas where state-actions are not informative of future states according to contrastive model. In other words, the method rewards state-actions with low predictability of future states. Learned representations that fail to capture features necessary for the classification task lead to higher intrinsic reward.
When the representations do capture features important for temporal classification, the resulting reward is invariant to pessificatiio-irrelewae perturbations such as spurious noise. This is a highly useful property: in the Noisy TV environment, C-TeC performance is strong (see Fig. 21 results) and unaffected by noise. Noise randomly sampled from the same distribution every timestep does not lead to stronger classifier performance, and, thus, the intrinsic reward is invariant these distractors.
Finally, the contrastive representations are crucial C-TeC's performance. Experimental results in Appendix G.4 show that the method is not robust the usage of monolithic critic (8,,a,0) indicating that the representation parameterization of the critic is key.
6 EXPERIMENTS
Our experiments show that contrastive representations can be used reward the agent for visiting eecs-icd or distant future states. We then use the C-TeC reward func-
tion for exploration in robotic environments and Craftax- Classic. We mainly study the following questions: (Q3)
How well does C-TeC compare to ETD Jiang et al. (2025)?
(Q2) How well does C-TeC reward capture the agent's 's fu- ture state distribution? (Q3) How effectively does C-TeC explore in locomotion, manipulation, and Craftax environ- ments compared to prior work?
Environments We use environments from the JaxGCRL codebase (Bortkiewicz et al., 2025). Specifically, maze,
we evaluate C-TeC on the ant humanoid f u u_maze, and arm wxm_binpickk___ environments, which require solving long directed plans to reach goal states. In the maze-based environments, the agent's objective specified at the start of each episode. Exploration in these settings corresponds to maze coverage: an agent that visits more unique positions in the maze demonstrates better exploration capabilities. In the arm binpick hard environment, which differs from the more navigation theemed tasks used in prior work, the agent must pick up cube from blue bin and place it at specified target location in
Figure 2: Environments. Maze cover- age, robotic manipulation, and the sur- vival game Craftax.
to reach a designated goal

---

Published as a conference paper at ICLR 2026
red bin. This represents challenging exploration task, as the agent must locate the cube, grasp it, and successfully place it at the correct target location.
Our experiments with the ant and humanoid d agents assess the method's ability to achieve broad state coverage using two complex embodiments. Mcanwhile, the arm_ binpick hard task evaluates the method's effectiveness at exploration in object manipulation setting. We also run C- TeC on "raftax-Classit (Matthews et al., 2024), a challenging open- open-world survival game resembling 2D Minecraft. The agent's goal is to survive by crafting tools, maintaining food and shelter, and defeating enemies
In the locomotion and manipulation environments, compare C-TeC common prior methods for exploration: Random Network Distillation (RND) (Burda et al., 2019b) and Intrinsic Curiosity Module (ICM) (Pathak et al., 2017), both of which are popular intrinsic motivation methods for ex- ploration. Actíve Pre-training (APT) (Liu & Abbeel, 2021): APT learns observation representations using contrastive learning, where positives are augmentations of the same observation and negatives are different observations. It uses the KNN distance between state representations as exploration signal, which correlates with state entropy. Unlike C-TeC, APT does not learn representations that are predictive of the future.
In Craftax, we compare against RND, ICM, and Exploration via Elliptical Episodic Bonuses (E3B) (Henaff' et al., 2022), count-based exploration method. We found that using the negative L1 distance (Equation (16)) as the critic function works best in the robotics environments, while the negative L2 distance (Equation (17)) performs best in Craftax. A comparison of different critic functions can be found in the appendix. We also compare our method to ETD (Jiang et al., 2025). While C-TeC is implemented in JAX, ensure fair comparison we re-implemented it on top of the ETD codebase and ran the experiments on the same robotic environments as well as on Crafter (Hafner, 2022). The comparison results are presented in Section 6.1.
6.1 COMPARISON TO ETD (Q1)
In this section, we compare C-TeC to ETD (Jiang et al., 2025), a recent method that uses contrastive learning learn quasimetric that encodes temporal distances between states via metric residual networks (MRN) (Liu et al., 2023; Myers et al., 2024). ETD extracts an exploration signal by measuring the minimum temporal distance between the current state and previous states stored in an episodic memory. By contrast, C-TeC is simpler: it does not require episodic memory or an explicit quasimetric, and works in both on-policy and off-policy settings.
For a fair comparison, we implemented C-TeC on top of the ETD codebase". We ran C-TeC across multiple hyperparameter configurations (primarily ablating different contrastive similarity functions) and selected the configuration with the best overall performance. We used the same contrastive encoder architecture as ETD, and for ETD we adopted the hyperparameters reported Appendix E of Jiang et al. (2025). Our experiments use only intrinsic rewards, as our goal is to understand C-TeC's behavior in the absence of any task rewards.
Figure 3 shows the results on the robotic environments and Crafter. All experiments are conducted in the intrinsic exploration setting (without providing the task reward). Both methods perform similarly on ant_ aat_hhardee hww.estrrt........ _aaeee____eeee__ee C-TeC outperforms ETD in humano d ______mm maze, albeit with higher variance, while ETD performs better in arm, hamm__iinncc_____ significantly outperforms ETD.
In Crafter, however, C-TeC
We speculate that this improvement stems from a difference in the exploration signals: ETD encour- ages novelty by maximizing the minimum temporal distance from past states (backward-lookingg while C-TeC prioritizes states that can lead to a larger set of possible future states (forward-looking) This forward-looking perspective may better capture the long-term exploratory value of states, which could explain C-TeC 's stronger performance in Crafter(We We refer the reader to Appendix K and J.2 for a didactic example that illustrate the differnce between C-TeC and ETD rewards). Moreover, as mentioned, ETD requires a more constrained architecture, specifically the MRN (Liu et al., 2023), to learn quasimetric representations that encode temporal distance. Our findings show that such architectural constraints are not necessary for effective exploration. Learning representations that
Phttps://gt http://tt.......coooooooo/eee/ae
7

---

Published as a conference paper at ICLR 2026
Ant hardest maze
Humanold u maze
Arm binpick
Crafter
* VVsited States
40000
150
Crafter Score
2000
3000
100
20000
1000
1000
50
0
0.0
0.5
1.0
0.0
0.5
1.b
1.5
0.0
0.5
1.b
ETD
C-TeC
Environment Steps
Environment Steps
547
Environment Steps
C-TeC (Ours)
ETD
Figure 3: C-TeC Performance compared ETD (Jiang et al., 2025) C-TeC competitive to ETD in terms of state coverage in continuous control environments, and outperform ETD in Crafter.
capture the temporal structure of policy behavior and environment dynamics seems sufficient to achieve meaningful exploration without the MRN architecture temporal distances.
The key takeaway is that, while C-TeC is conceptually similar to ETD, C-TeC achieves comparable or stronger performance while also reducing algorithmic complexity.
6.2 LEVERAGING THE FUTURE STATE DISTRIBUTION FOR EXPLORATION (Q2, Q3)
In this section we demonstrate that the C-TeC reward captures the future state distribution and it can be used incentivize the agent to visit less cesspicc and more distant future states. We visualize the C-TeC reward at different stages of training in the ant suttharrdest maze environment. The contrastive critic defined as the negative L1 distance (Equation (16)), and the policy is trained to maximize the intrinsic reward defined in Equation (3). Figure 4 shows the reward values in a section of the maze. Following the qualitative results in Figure 4, we evaluate C-TeC in the and frrm_binpick paapicckhardd environments. We run two variants of the experiment: (1) using the complete state vector as the future state, which is common in exploration tasks where the agent is encouraged explore the entire state space; and (2) incorporating prior knowledge by narrowing the future state to specific components of the state vector. The latter allows us assess whether C-TeC can flexibly explore subspaces of the state space, which is often useful in practice. In ant_large_mave
ant_large_maze
jumanoid_u_maae
we define the future state as the future the ant's torso. In humanoid m__zee we use the future (x, y, z) position of
(x, y) position the humanoid's torso. Finally, in arn_binpick_harr hard, we define the future state as the future position of the cube.
As evaluation metric, count the number of unique discretized states covered by each agent. In ant _____arree_____ maae_maaeee._ we count the number of unique (x, y) positions in the maze visited by each agent. Similarly, in humanoid in arm_bt rem_binpii ck_ hardd we count the number of unique cube positions. We compare C-TeC to RND, ICM, APT, and a uniformly random policy. Figure 5 shows the learning curve when using the complete future state vector while Figure 10 shows the performance when we incorporate prior knowledge by restricting the future state to specific components of the state vector. Each agent is run with 5 random seeds, and we plot the mean and standard deviation (Patterson et al., 2024).
ma ze, count the number of visited (x, y, 2) positions, and
0,2
3M
50M
400M
500M
0.0
Environment Steps
Figure 4: Evolution of the C-TeC reward during training. This figure shows how the intrinsic reward changes over the course of training based on future state visitation. The black circle in the lower-left corner represents the starting state. C-TcC reward captures the agent's future state density and rewards the agent for visiting states faraway in the future.
8

---

Published as a conference paper at ICLR 2026
Ant hardest maze
Humanoid u maze
Arm binpick hard
24000
Visited States
240
2400
180000
1600
160
120000
80
0.0
1.0
Environment Step
Environment Step
Environment Step
C-TeC (ours)
APT
RND
ICM
Random Policy
Figure 5: C-TeC explores states than prior methods. We compare the state coverage of C-TeC APT (Liu Abbeel, 2021), RND (Burda et al., 2019b) and ICM (Pathak et al., 2017). We include uniform random policy as well.
Our agent outperforms the baselines both variants the experiment and learns interesting behaviors in the challenging humanoid d_ u__aze environment. Figure 6 shows screenshots of C-TeC behavior: More visuals are provided Appendix M. This improvement can be result C-TeC's consistent reward properties. Methods like RND and ICM will eventually tend to reward as the state distribution is covered. A nice property of C-TeC is that it does not have zero reward the limit.
6.3 LEARNING COMPLEX BEHAVIOR CRAFTAX-CLASE
Can an RL policy learn complex behavior in Craftax-Clain without any task reward? To answer this question, we run C-TeC Craftax- Classic (Matthews al., 2024), a complex sur- vival game where the agent's goal is survive
Craftax Classic
6
Achievements
(Ours)
4
3B
by crafting tools, maintaining food and shelter, and defeating enemies.
Randaa Policy
2
RND
In this experiment, we use the same PPO imple- mentation as used in the baselines in the Craftax paper (Matthews et al., 2024), adding the C-TeC reward on top of it.
0.0
0.2
Environment Steps
1eg
We compare against RND, ICM, E3B, and uniform random policy. We found that using PPO with memory (PPO-RNN) yields the best performance. The results are presented in Figure The y-axis represents the sum of the achievements success rate, which measures how many capabilities and useful objects the agent has discovered. C-TeC outperforms the baselines and unlocks more achievements. Figure visualizes some of the achievements of the C-TeC agent during evaluation episode.
Figure 7: C-TeC achieves higher achievements in the Craftax survival game.
7 CONCLUSION
This work has shown how learn and leverage temporal contrastive representations for intrinsic exploration. With these representations, we construct a reward function that seeks out states with unpredictable future outcomes. We find that C-TeC simple method that yields strong performance on state visitation metrics. These results hold across different RL algorithms and environments. Future work includes further investigating the role of temporal representations in effective exploration, combining the C-TeC reward with task rewards, and adapting the method to observed settings.
and partially
Ageett videos: httpp:://tenpp ve-expl exw.r.githuu.ie... thub.iu..

---

Published as a conference paper at ICLR 2026
Figure 6: C-TeC behavior in humanoid-u-men mazze. C-TeC agent learns escape the u-maze by jumping over the wall. None of the alternative exploration methods discovered this kind of unexpected behavior*.
REPRODUCIBILITY STATEMENT
For reproducing the paper's results, we provide the algorithm codehase in the supplementary material and in the GitHub link https: /tt::/gittti gight..t.bb...... com Fa i www.el0/ www...eee..oo.. git training details and hyperparameters are in Appendix E.
Acknowledgments. We thank Marco Jiralerspong and Daniel Lawson for feedback the draft of the paper. We thank Daniel Lawson, Roger Creus Castanyer, Siddarth Venkatraman, Raj Ghugare, Mahsa Bastankhah, and Grace Liu on discussions throughout the project. We thank Liv d' Aliberti for their plotting code and format for Figure 28. We thank the anonymous reviewers for helpful comments and feedback that improved the paper. We want to acknowledge funding support from Natural Sciences and Engineering Research Council of Canada, Samsung AI Lab, Google Research, Fonds de recherche du Québec, The Canadian Institute for Advanced Research (CIFAR), and IVADO. We acknowledge compute support from Digital Research Alliance of Canada, Mila IDT, and NVIDIA.
REFERENCES
possibility implementing curiosity horedom in model-building neural controllers. Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
Joshua Achiam and Shankar Sastry. Surprise-basek intrinsic motivation for deep reinforcement lcarning. arXiv preprint arXiv:1703.011 2017.
Chenjia Bai, Peng Liu, Zhaoran Wang, Kaiyu Liu, Lingxiao Wang, and Yingnan Zhao. Variational dynamic for self-supervised exploration in deep reinforcement learning. IEEE Transactions Neural Networks and Learning Systems, 34:4776-4791 2020. URL https:///// http://appp..aaaiieeeeeseeaaaeaa 2247046000 6066.
semant ietemantiisson icscholar. org CozpusID:
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 2017.
Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count- based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Glen Berseth, Daniel Geng, Coline Manon Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. irl] Surprise minimizing reinforcement learning in unstable environments. In Interna- tional Conference on Learning Representations, 2021. URL https: ttttts::/oppnreeeeee h/tpp/pppeeiieeeeeeeeeeeeeeeeeee net/ forum? cF cFZOyoDlcx1.
Martin Biehl, Christian Guckelsberger, Christoph Salge, C Smith, and Daniel Polani. Free energy, empowerment, and predictive information compared. Technical report, Technical report, University of Hertfordshire, URL: https://www. mis. mpg .... 2015.
Michat Bortkiewicz, Wladystaw Palucki, Vivek Myers, Tadcusz Dziarmaga, Tomasz Arezewski, Lukasz Kuciński, and Benjamin Eysenbach. Accelerating goal-conditioned reinforcement learning algorithms and research. In The Thirteenth International Conference on Learning Representations, 2025. URL htttps:/cppett wpwpreeveeeeeeeeeeeeeeeeeee net fow.mmidd-ggyysepe
Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(0e)):213-231, 2002.
10

---

Published as a conference paper at ICLR 2026
Yuri Burda, Harri Edwards. Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity- drivvn learning In International Conference on Learning Representations, 2019a. URL https: /cppnrvveeicc.cccoooooo net Eorum? JNW JAQYX.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation.
In International Conference on Learning Representations, 2019b. URL httLt:t///p/peeeeeee net/ Eorum? diammidddiiieee 1JJRR5mMm
Junya Chen, Zhe Gan, Xuan Li, Qing Guo, Liqun Chen, Shuyang Gao, Tagyoung Chung, Xu, Belinda Zeng. Wenlian Lu, et al. Simpler, faster, stronger: Breaking the log-k curse on contrastive learners with flatnce. CoRR, 2021.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607, PmL., 2020.
S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (EVPR'05), volume pp. 539-546 vol. 1, 2005, doi: 10.1 10.1109/CVPR.2005500
Thomas M. Cover and Joy A. Thomas. Elements of information theory. Wiley series in telecommunications. Wiley, New York, NY [u.a.], 1991. httpht/::/ www. loc. ca html.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 5(4)6613 624, 1993.
Yilun Du, Chuang Gan, and Phillip Isola. Curious representation learning for embodied intelligence. iece. In CVF International Conference on Computer Vision, pp. 10388-10397, 2021.
Yuqing Du, Stas Tromkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave: Assistance via empowerment. Advances in Neural Information Processing Systems, 33:4560-445 15714, 2020.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification, In International Conference on Learning Representations, 2021. URL https: //apenrevieeeeeeeeeee ew wtree//ooruriii/oe de SoB-C.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Ruslan Salakhutdinov. Contrastive learning as goal- conditioned reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: http:////w/peeeeeeeeeee.em net/ forum? forummid==ven Ld=vecuUssqueee
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. In The Eleventh International Conference on Learning Representations, 2023. URL net/ / forum? www.froirooooerroeo dimmmndd=doottSteeemmm 9Jr2i.
Karl Friston. The free-energy principle: unified brain theory? Nature reviews neuroscience, 11(2):12--38.. 2010.
Vincent Gardeux, Fabrice David, Adrian Shajkofci, Petra C. Schwalie, and Deplancke. Asap: a web-based platform for the analysis and interactive visualization single-cell ma-seq data. Bioinformatics, 33:3123- 3125, 2016. URL. https: /api semant cschola .rrgorgg.ooooo.ooooooo..... Borg/Coo /Corpus ID: 2237 186.
Lucas Gruaz, Alireza Modirshanechi, Sophia Becker, and Johanni Brea. Merits of curiosity: a simulation study. PsyArXiv, 2024.
Tuomas Haamoja, Schoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning walk via deep reinforcement learning. arXiv preprint arXiv: 1812. 11/22.11100 1000 2018a
Tuomas Haarnoja, Aurick Zhou, Pieter Abheel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018b. URL https::/ http://///ccedigss. mlr. press//vil haw.aaaaaaooaaaaaaaaaaaaaaaaaaaa B....bttt..... html.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker. Sehoon Ha, Jie Tan, Vikash Kumar, Henry 7hu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 181 1812v 1812.05905 2018c.
11

---

Published as a conference paper at ICLR 2026
Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022. URL https: wwwpeeeewww.ee//oooooieieee wwwp:/ummeeooottteeeeee/// 1www::wwww00m 6MF
Yusuke Hayashi and Koichi Takahashi. Universal ai maximizes variational empowerment. arXiv preprint arXiv:2502. /5820, 2025.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Informa- tion Processing Systems, 2022. URL https: www./rppereeeeeeeeeeee exx www.reeeeeweeeeeeeeeeeeeeeeeeeee forum? 2 d-xii-x------- yzas QqJQ.
Devon Hjelm, Alex Fedorov, Samuel and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations. 2019.
Karan Grewal, Phil Bachman, Adam Trischler,
Jonathan and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information
processing systems, 29, 2016.
Adriana Hugessen, Roger Creus Castanyer, Faisal Mohamed, and Glen Berseth. Surprise-adaptive intrinsic motivation for unsupervised reinforcement learning. Reinforcement Learning Journal, 2:547-562, 2024.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czameeki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. International Conference on Learning Representations, 2017. URL https: : http:s://oppeee /ttp///ppe/eeeeeeee//e/eeeeeeeee net worem//ormmmmm/ SewpummddddSJGySyeeeee
Yuhua Jiang, Qihan Liu, Yiqin Yang. Xiaoteng Ma, Dianyu Zhong. Hu, Jun Yang. Bin Liang, Bo XU, Chongjie Zhang. and Qianchuan Zhao. Episodic novelty through temporal distance. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URL https::t/// http:///pppeeeeeeeeeeeeeeeeeeeee net. forum?
Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for contimuous agent-enviromment systems. Adaptive Behavior, 19-1) 2011.
Maximilian Karl, Philip Becker- Maximilian Soelch, Djalel Benbouzid, Patrick van der Smagt, and Justin Bayer. Unsupervised real-time control through variational empowerment. In The International Symposium of Robotics Research, pp. 158-173. Springer, 2019.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning. 292 2002.
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. All else being equal be empowered. In European Conference on Artificial Life, pp. 753- Springer, 2005a.
A.S. Klyubin, D. Polani, and C.L. Nehaniv. Empowerment: a universal agent-centric measure of control. In 2005 IEEE Congress Evolutionary Computation, volume 1, pp. 128 135 1ol.1, 2005b. doi: 10.1 109/CEC. 1506.1554676.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp. 5639-5650. PMLR, 2020.
Michael Laskin, Denis Yarats. Hao Liu, Kimin Lee. Albert Zhan, Kevin Lu. Catherine Cang, Lerrel Pinto. and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httpst // /ppenrevii ew net forum? wwwp://wwwwwwwwwwwwpemm _is.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URI. https:
//openrevis eww net. fwrm//ormmmt// dam.miid--iiii 9OWAAH 2xF t.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. ArXiv, abs/1906.05274, 2019. URL https: //api.
semanticschclar. wtwg//oopueeeeeoooooooo
Andrew Levy, Alessandro Allievi, and George Konidaris. preeitt-e empowerment: Measuring empower- ment without simulator. arXiv preprint arXiv:2410.1115: 2024.
Bo Liu, Yihao Feng, Qiang Liu, and Peter Stone. Metric residual network for sample efficient coll-conditioned reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 8799-8806. 2023.
12

---

Published as a conference paper at ICLR 2026
Grace Liu, Michael Tang, and Benjamin Eysenbach. A single goal is all you need: Skills and exploration emerge from contrastive RL without rewards, demonstrations, subgoals. In The Thirteenth International Conference on Learning Representations, 2025. URL httpctp://ppeeeeeee
net/ /wt//eetetoeeeeeeeee forum? 1 1d=xckg24XXxXX
Hao Liu and Pieter Ahheel. Behavior from the void: Unsupervised active pre-training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan Advances in Neural Information Processing Systems, 2021. URL https: wwwp//wwwwww..e.epoo//o/eeoooeep net / f orumm
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 3669--3777, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. (0.18653311/D1S 1111///88114...- URL https::::: http:///ccaaaaaaaaaao ogy. otggb18--40055
Marlos C Machado, Mare G Bellemare, and Michael Bowling. Count- exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5125-5133, 2020.
Jarryd Martin, Suraj Narayaman Sasikumar, Tom Everitt, and Marcus Hutter. Count-based exploration in feature space for reinforcement learning. arXiv preprint arXiv: Tttv:/7000 001.10000000000 080000 2017.
Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Thomas Jackson, Samuel Coward, and Jakob Nicolaus Foerster. Craftax: A lightning-fstt benchmark for open-ended reinforcement learning. In Forty-first International Conference on Machine Learning, 2024. URL net. net. forum? id=hg/wX1rQ
Bogdan Mazoure, Benjamin Eysenbach, Ofir Nachum, and Jonathan Tompson. Contrastive value learning: Implicit models for simple offline rl. In Conference on Robot Learning, pp. 1257-1267. PMLR, 2023.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems, 28, 2015.
Kevin P Murphy. Probabilistic machine learning: an introduction. 2022.
Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, and Anca Dragan. Learning to assist humans without inferring rewards. In The Thihtt-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: openreviee ew net/ forum? i d-WCn.mb7cv1,
Vivek Myers, Bill Chunyuan Zheng, Anca Dragan, Kuan Fang, and Sergey Levine. Temporal representation alignment: Successor features enable emergent compositionality in robot instruction following. arXiv preprint arXiv:2502.05455 2025.
Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinnfrreement learning environments in jax. Advances in Neural Information Processing Systems, 37:43809-438855 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1XXivv807700375 2018
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on muchine learning, pp. 2721-2730 PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. Curiosity- driven exploration by self- supervised prediction. In International conference on machine learning, pp. 2778 2787. PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement, 2019. URL https:/ //wpiv/aaavvvvv////////eaaeaiiva org/ abs/ 1 1906..440. 04161. 61.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement leam- ing. Journal of Machine Learning Research, 25(318):1 63, 2024. URL http: //mprrr/////////////////eeeerr org/ /apprs//////////// v25/23- 018 83, html.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. 2020. URL https:t/t//ttc.t httss///xxxvvvoo//ee /aaabb/// 2207. 02832.
Nicholas Rhinehart, Jenny Wang. Glen Berseth, John D Co- Co-Reyss, Danijar Hafner, Chelsea Finn, and Sergey Levine. Information is power: Intrinsic control via information capture. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL. https: //pprr/eeeeeeeeeeeeeeme net forum? 7t. 9RI.
13

---

Published as a conference paper at ICLR 2026
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IELE Transactions on Autonomous Mental Development, 2(3):230- 247, 2010. doi:
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.06347, 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data- efficient reinforcement learning with self- predictie representations. In International Conference on Learning Representations, 2021. URL http:: http///pprrviiieeeeeeeeee net/ forum? Pemmmiid-uuee 1i= www.uuueeEeeaaa..u.oooom
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel. Danijar Hafner, and Deepak Pathak. Planning to explore via self-ssup world models, In International conference on machine learning, pp. 5583-8592. PMLR, 2020.
Jonathan Sorg, Satinder Singh, and Richard L Lewis. Variance sareance-based rewards for approximate bayesian reinforcement learning. arXiv preprint arXiv:1003.3516 2012.
Bradly C Stadic, Sergey Levine, and Pieter Abbeel. Incentivizing exploration reinforcement learning with deep predictive models. arXiv preprint 150.000844, 2015.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Sebastian B Thrun. Efficient exploration in reinforcement learning. Camegie Mellon University. 1992.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: wwwpreewew....cc//o/pppp. net www.mmodddddaa fBf7ddrg.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Represeniations, 2023. URL
net/ forum? wim.miiddaaap................... ddddYEEp_OcQ2
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on hypersphere. International conference on machine learning, pp. 9929 9939. PMLR. 2020.
Zhi Zhi-Xiong Xu, Xi-Liang Chen, Lei Cao, and Chen-Xi Li. study of count-based exploration and bonus for reinforcement learning. In 2017 IEEE 2nd International Conference on Cloud Computing and Big Data Analysis (ICCCBDA), 425 429. IEEE, 2017.
Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Behavior contrastive lcarning for unsupervised skill discovery. In International conference on machine learning, pp- 29183-39204 PMLR, 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference Machine Learning, pp. 1920-11931.. PMLR, 2021.
Ruihan Zhao, Kevin Lu, Pieter Abbeel, and Stas Tiomkin. Efficient empowerment estimation for unsu- pervised stabilization. In International Conference on Learning Representations, 2021. URL https: //openrevies. wet.:/ooumi.ddu2200PPP...
Chongyi Zheng, Jens Tuyls, Joanne Peng, and Benjamin Fysenbach. Can a MISI. fly? analysis and ingredients for mutual information skill learning. In The Thirteenth International Conference on Learning Representations, 2025. https: / www.reveeevve.. net wwt/ffouume forum? l-xo
Alicja Ziarko, Michal Bortkiewicz, Michal Zawalski, Benjamin Eysenbach, and Piotr Miloć. Contrastive representations for temporal reasoning. In The Thitt Annual Conference on Neural Information Processing Systems, 2025. URL https:: : / /oppnreviivv ew, net. forum? kcH5Cork.
14

---

Published as a conference paper at ICLR 2026
A USAGE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs as grammar and spelling correction tool. We provide each section in the prompt and ask the LLM to correct any obvious grammatical spelling mistakes; however, prevent the LLM from changing the style or introducing any new claims.
B SAMPLE EFFICIENCY OF C-TEC
In this section, we show the performance of C-TeC with different amounts of environment steps. the main experiments, we used 500M environment steps. In Table I, we present the results of running C-TeC with significantly fewer environment steps: 50M (10x less) and 30M (16x less) than the main experiments. Our results demonstrate that C-TeC can explore effectively with fewer environment interactions, and they also highlight C-TeC 's scalability with respect to the number of environment interactions, an important property for a pure exploration method. We also visualize the state coverage with the C-TeC reward heatmap in the ant-hardest-maze by plotting the x.y positions that the agent covered during training (Figure 8).
Environment Ant-hardest-mant
500M steps
50M steps
30M steps
2500 + 300
1916 + 430
1119 + 304
Humanoid Humanoid-u-maze u-maze
230 40
143 34
102 11
Arm-binpick-han
135000 + 10000
40000 + 14000
31150 + 3156
Table 1: Sample Efficiency of C-TeC
Coverage
steps)
Coverage stops)
1.0
00
07
05
05
02
Figure 8: C-TeC State Coverage The Black circle in the lower left corner is the starting state. The figure the left shows the state coverage when C-TeC with 50M environment steps in ant hnt-hha ht--hhardsstaaaaa On the right, we show state coverage when we run C-TeC with 500M environment steps.
C DOES C-TEC'S CONTRASTIVE MODEL SUFFER FROM REPRESENTATION COLLAPSE?
In this section, we investigate the representations of C-TeC's contrastive model. Specifically, do the contrastive representations suffer from mode collapse? This is common issue in contrastive learning, as one possible local optimum to output a constant vector the negative examples are not chosen carefully. As measure for collapse, we plot the variance of the contrastive representations in the robotic environments during training in Figure 9, Our plot shows that C-TeC does not suffer from mode collapse, and the variance steady during most of the training time.
15

---

Published as a conference paper at ICLR 2026
hardest
Humanoid
binpick
Representatiin Vartanen
241
C-TaC
03
15
Envirenment Step
Environment Step
Environment Step
Figure 9: Representations Variance C-TeC's contrastive model does not suffer from mode collapse.
D INCORPORATING PRIOR KNOWLEDGE
Figure 10 shows the performance when incorporate prior knowledge on our method by restricting the future state to specific components of the state vector. Each agent is run with random seeds, and we plot the mean and standard deviation (Patterson al., 2024).
Ant hardest
Humanold
Arm binpick hard
# Visited Stabes
cesp ausa
2400
2000
303
polcy
polic
polcy
Agent
Agent
Agent
Figure 10: State coverage when leveraging prior knowledge C-TeC outperforms prior methods APT, RND,
ICM, and can explore effectively when leveraging prior knowledge. This shows the improved flexibility of C-TeC in incorporating prior knowledge by narrowing the exploration space compared prior work.
E TRAINING DETAILS
We summarize the hyperparameters and model architectures for all experiments. Appendix E. provide the training details for the locomotion and manipulation experiments. In Appendix E.2, we provide the details of the Craftax experiments. In Appendix E.3, we provide the details of each environment.
Finally, in Appendix G, we include the ablation experiments.
E. ROBOTICS ENVIRONMENTS
In the robotics environments, used SAC as the RL algorithm. Table 2 shows the hyperparameters that are shared across all methods. Table 3 and Table 4 show the segorithh-sspciffe hyperparameters for C-TeC and the baselines, respectively.
16

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
num jim_timesteps
500,000,000
nax_replay_size
10,000
min_replay_size
1,000
episode_length
1,000
discounting
0.99
num_envs
1024 (256 for humanoid_u_mazze
batch_size
1024 (256 for humanoid__u_mazze
action_repeat
1
unroll_length
62
policy_ir
3e-4
critic_lr
3e-4
hidden layers (for both actor and critic)
[256,256]
Table 2: Hyperparameters for all methods in robotics environments
Hyperparameter
Value
contrastive_ir
3e-4
InfoNCE L1
similaritty_funtiion
Joosumexp_penate
0.1
hidden layers (for both encoders)
10024
representation dimension
64
Table 3: Hyperparameters for C-TeC in robotics environments
Hyperparameter
Value
rnd encoder 1r rnd embedding dim rnd encoder hidden layers
3e-4
512
[256, 256]
icm encoder (forward and inverse models)
3e-4
iem embeddings_dias icm encoders hidden layers
512
[1024, 1024]
icm weight forward loss
0.2
apt contrastive ir apt similarity function apt contrastive hidden layers apt representation dimension Augmentation type
3e-4
L1
[1024, 1024]
64
NO 0.5)
Table 4: Hyperparameters for baselines in robotics environments
E.2 CRAFTAX
In Craftax, we used PPO as the RL algorithm'. Table shows the hyperparameters shared across all methods. Table and Table 7 show the algorithm-speccfic hyperparameters for C-TeC and the baselines, respectively.
hhttps://gi httpgggtthbbcooo.oo.oooo chael http://hhhhetttteeeeee/aaaaaaaaa hes
17

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
num cam_timestepa
1, 1,000,000,000
num_step
64
learning_rate
2e-4
Saneal_learningg_ee
True 4
update_epon _etcees
discounting
0.99
gae_lambda
0.8
clip_epsilon
0.2
ent_coes
0.01
max_ max_grad_norn
1.0
activation
tanh
action_repeat
1
RNN_layers (GRU)
[512 (embedding dim) ,512 (hidden dim)
hidden layers (both actor and value)
[512, 512]
Table 5: Hyperparameters for all methods in robotics environments
Hyperparameter
Value
contrastive_ls
3e-4
IoNCE
similarity_funct.e
L,
Discounting
0.3
logsumexp_penalty
0.0
hidden layers (for both encoders)
[1024,1024.102]
representation dimension
64
Table 6: Hyperparameters for C-TeC in Craftax
Hyperparameter
Value
rnd encoder ir rnd embedding dim rnd encoder hidden layers
3e-4
512
[256, 256]
icm encoder ir (forward and inverse models)
3e-4
icm embeddings_dis
512
icm encoders hidden layers i icm weight on forward loss e3b (icm) lambda
[256, 256]
1.0
0.1
Table 7: Hyperparameters for baselines in Craftax
E.3
ENVIRONMENT DETAILS
Ant-hardest-maze The observation space of this environment has 29 dimensions, consisting of joint angles, angular velocities, and the x.y position of the ant's torso. The action space is 7-dimensional, representing the torque applied to each joint.
Humanoid-u-man The observation space of this environment has 268 dimensions, con- sisting of joint angles, angular velocities, and the x.y position of the humanoid's torso. The action space is 17-dimensional, representing the torque applied each joint.
arm-binpick-hard The observation space of this environment has 18 dimensions, consisting of joint angles, angular velocities, the cube position, and the end-effector position and offset. The action space 5-dimensional, representing the displacement of the end-effector.
18

---

Published as a conference paper at ICLR 2026
Souftax-Classie The observation space is one-hot encoding of size 1345, capturing player information (inventory, health, hunger, attributes, etc.) as well as the types of blocks and creatures within the player's visual field. The action space is discrete and consists of 17 actions.
E.4 DETAILS ON C-TRC REWARD
One important detail is that the policy's objective is slightly different from the (negative) representa- tion objective (Equation (2)) because it omits the log-sum-exp term. This can be seen by rewriting the reward function as follows:
Tintr (8, a) (s,a)
(e. (l) (1) + log Σ
log Σ E 1b
(10(0.0) -
6f
(neg) contrastive loss
(6)
To further gain intuition for what this is doing, we note that (in practice) the Φ(s, a) representations are quite similar to the (s) representation evaluated at the same state. Thus, we can approximate this second term as
log Σe (10(s)-+(6/))) log](s),
which we identify as kernel density estimate of the marginal likelihood of state s under the replay buffer distribution Pr(s). This observation helps explain why including the tog-sum-exp term in the reward would degrade performance - it effectively corresponds to minimizing state entropy, which can often hinder exploration, especially environments without much noise (Zheng et al., 2025). One additional consideration here is that, because the likelihood is measured using learned representations, is sensitive to the policy's 's understanding of the environment. While ordinarily maximizing state entropy lead to degencrate solutions (like the noisy TV), our approach mitigates this problem because the contrastive representations will only learn features that are predictive of future states (hence, they would ignore noisy TV).
E.4.1 VARIANCE REDUCTION IN THE REWARD ESTIMATE
We can decrease the variance in our estimate of the expectation in Equation (3) by looking at all future states sy = St41, $142, and weighting each summand by 7":
T: (s) (st (a
sy)]
(8)
Ep(sylan [lo [l) a) - [b2
(9)
1 1-
"1(o(s (ss))ll2
(10)
The (unbiased) approximation comes because we only look future states that occur in one trajectory, and other trajectories might visit different future states. The ugly fraction is the normalizing constant for a truncated geometric series. In the last line. note that the summation Σ/= be quickly computed for every Te by starting at T = H and decrementing t, updating sum = (s.) + 71//uum- This is the same trick that's usually used for computing the empirical future returns in RÉINFORCE, and decreases compute from O 22)) to O We use this estimator in Craftax- Classic but we found that omitting the normalization term results in much better performance.
(v((s)) can
F COMPUTE RESOURCES
In all experiments, we use CPUs, a single GPU, and GB of RAM. The specific GPU type varies depending the job scheduling system, but most experiments NVIDIA RTX 8000 V100 GPUs. Training in the robotics environments takes approximately 24 hours on average, while Craftax experiments require around 30 hours.
19

---

Published as a conference paper at ICLR 2026
G
ABLATION STUDY
To understand the contribution of each component to the overall performance C-TeC, we conduct ablation study several key elements of the algorithm, illustrated in the following section.
G.1 REPRESENTATION NORMALIZATION
Is important normalize the contrastive representations when computing the intrinsic reward? To answer this question, we compare the exploration performance of C-TeC across all environments, keeping all hyperparameters fixed except for the normalization of the representations.
Ant hardest
Humanoid
Arm binpick hard
Craftax Classic
# Viateed States
12000
1.00
33
806
Environment Step
Environment Step
Environment Step
Step
With Representation Normulization
Without Representation Normalization
Figure 11: Normalizing the contrastive representations. Normalizing the representations is crucial for effective exploration-using unnormalized representations significantly degrades exploration performance.
G.2 CONTRASTIVE LOSSES
We compare the performance of C-TeC using different contrastive loss functions. Specifically, we evaluate InfoNCE, symmetric InfoNCE, NCE (Hjelm et al., 2019), FlatNCE (Chen et al., 2021), I), and Monte-Carlo version of the forward-backward (FB) (Touati & Ollivier, 2021) loss, defined in [Equation (11) (11)-Equation (15)). Figure 12 presents the results. Overall, NCE leads to poorer exploration, particularly in Craftax. InfoNCE and symmetric InfoNCE exhibit similar performance across all environments. In general, the method is reasonably robust to the choice contrastive loss.
Co((snaa):))))
LinfoNCr:0) =
Σ log 1
(11)
e Collor,ai)
Col(sraa))a)))
Collsnaa):)))
Lsymmetric__Iffnn InfoNCE:
=-
log
+ log
(12)
Col(sna())a()))
Collsgias)io)))
1=1
K Σ
K
(Bimny(NCC))()) =-
log (a ac),
Σ
log (1 - az),
(x)
J=2
(13)
K
ΣK e (f
LHINNCE()) = Σ log
(14)
detach E Collsiazs ( - Coosrrann)))))
i=1
(r) Σ (c(os,a),)))) + 2(K 1)
(15)
14
20

---

Published as a conference paper at ICLR 2026
hardest
binpick hard
Visited States
1.44
6009
20
Environment Step
Environment Step
Environciett Step
Environment Step
InfoNCE
Symmetric InfoNCE
NCE (Binary)
FlatNCE
FB
Figure 12: Comparison of Different Contrastive Losses. Overall, C-TeC is robust to the choice of contrastive loss. A notable exception the Binary NCE loss in Craftax, where performs relatively poorly.
G.3 CONTRASTIVE CRITIC FUNCTIONS
We compare four critic similarity functions shown below:
Col(se: at), 8T)11 - [10bo a.) 4/0
(16)
Col(se, 88) 85)1. -11cb( at) (0((8))l2
(17)
Col(se, 8/2 1/1.2//////ae0 w/o-sqrt sgrt = -1lo0(s1, a.) ( Co( at). 8y) (0, (
(18)
(19)
Figure 13 shows the results. In general, using the L1 distance yiclds the best performance across the robotic environments, while L2 performs better in Craftax. This highlights the importance of this design choice and suggests that some tuning may be required select the most effective critic function.
Ant ha rt mazze
Humanoid maze
Arm binpick hard
Craftax Classic
Visited Stattes
1206
Ach
800
25089
Environment Step
14
Enciromment Step
Environment Step
Step
W/O sqrt
Dot product
Figure 13: Comparison of Critic function. Overall,the L, distance yields the best performance across the robotic environments, while La performs better in Craftax.
G.4 CONTRASTIVE CRITIC ARCHITECTURE
In this ablation compare two architectures of the contrastive critic, the separable architecture (Øe(Se: at), the (sy)), which the one we use in all our experiment, and the monolithic critic fa ie., single model that takes in triplet f(s, a, 85).
binpick
Vieibad States
100
3400
240007
450
3802
150002
300
1200
150
0.5
0.5
1.0
155
2.0
2.5
Environment Step
Environment Step
Environment Stop
Contrastive Critie
Monolithic Critic
Figure 14: Critic parameterization Using monolithic critic results in poor exploration performance, while using the separable architecture results in much better exploration. This shows the importance of the critic parameterization distance function between two representations.
21

---

Published as a conference paper at ICLR 2026
Importantly, these experiments show that the factorized representation parameterization is necessary (relative to the monolithic critic) condition for effective exploration. We discuss the possible failure mode of using the monolithic critic in Section 5.2. These experiments do not demonstrate sufficiency, and we claim that the coomatiion-thorrete interpretation for critic that fully captures the point-wise MI is still useful for analysis.
G.5 FORWARD VS. REVERSE KL
As mentioned in Section 5.1, we hypothesized that the reverse C-TeC reward is important for exploration. As it encourages mode-seeking behavior (prioritizing unfamiliar states), to test this hypothesis we run C-TeC with the negative-forward KI. reward (Equation (20)), the results shown in Appendix G.5 indicates that using the reverse is necessary for exploration.
Efrimtr (St, at)] - (pr( log Pr(8))
(20)
pr(os 8e, a.)
- Dx..k.TTsss (1pT(3) 82, 0z)) S O.
(Drr. is always non-negative.)
Ant hardest maze
Humanoid u maze
Arm binpick hard
Visitedd States
1200
150
800
Environment Step
Environment Step
Environment Step
Reverse KL (Mode- (edde-Seeking)
Forward KL (Mean-Seeking)
Figure 15: Forward vs Reverse KL C-TeC with the reverse KL reward promotes mode-seeking behavior which encourages the agent to prioritize visiting unfamiliar states resulting in much better exploration.
G.6 FUTURE STATE SAMPLING STRATEGY
In this experiment (Figure 16), we investigate the sensitivity of C-TeC to the future state sampling strategy. Specifically, we consider two variants in addition to the geometric sampling. The first is uniformly sampling from the future. Unlike geometric sampling, uniform sampling does not prefer states that sooner in the future over later ones. The second geometric sampling with an increasing value. The intuition behind this strategy is that exploring nearby states is easier for the agent the start of training, and as the agent becomes better at exploring them, can progressively explore farther states in the future. We refer to this strategy the "-schedule, and experiment with two different starting values of
y: one ranging from = 0.9 to 7 = 0.99, and another from 7 = 0.1 to y 0.99. The results are shown in Figure 16. Regardless of the future state sampling strategy, the contrastive method explores better than the baselines all three environments and appears robust.
Ant bardest maze
Humanoid maze
Arm binpick hard
# Visited States
20:00
450
3600
300
190
50000
0.5
0.5
1.0
1.5
2.0
2.5
Environment Step
Environment Step
Environment Step
Geametric Sampling
Uniform Sampling
Schodule from 0.1 to 0.99
Schedule from 0.9 to 0.99
Sensitivity to future state sampling strategy. We compare variants of C-TeC with different future state sampling strategies, the method is robust to the choice of the sampling strategy and all the variants outperform the baselines.
Figure 16:
22

---

Published as a conference paper at ICLR 2026
G.7 EFFECT OF THE CONTRASTIVE MODEL DISCOUNT FACTOR
We study the effect of the discount factor in Equation (1), which defines the sampling window of future states; this discount is distinct from the discount factor 7 used in the underlying RL algorithm. We found that, in general, a discount value of Tet = 0.99 yields good exploration performance; however, we suspect that adjusting the discount might result better performance depending on the environment. Figure 17 shows the results in the robotic environments, In zumanoid-u-man smaller values "et result in better performance. 0.9leads to faster exploration; however, by the end of training, performance is similar. Finally, in arm binpick hard, a discount of % 0.99 achieves the best performance.
ant-hardest.
ze, a discount of ?et =
Ant hardest maze
Humanoid u maze
Arm binpick hard
30880
# Visited States
201e
1.00
100033
-4
Enviroument Step
Environment Step
Environment Step
C.TcC (Yo 0.99)
TeC 0..9
C-TbC (yo
C-TeC (yu 0..5)
C-TcC (yo 0.3)
Figure 17: Sensitivity of C-TeC to the discount factor Different environments require different discount factors to obtain the best exploration performance.
G.8 EFFECT OF THE TEMPERATURE PARAMETER 7
In practice, (Equation (2)) is learned during training additional learnable parameter. However, we study the effect different fixed values of r. Intuitively, we can think of as a weight on the alignment and uniformity terms in the contrastive loss (Wang & Isola, 2020): smaller values of put more weight on the alignment term, while larger values put more weight on the uniformity term. Figure 18 illustrates that temperature value of often results in good performance, indicating the importance of both alignment and uniformity in learning representations for exploration.
Ant hardest
Humanoid
Arm binpick hard
Viatzed States
2014
1.00
20
25
Environment Step
Environment Step
Step
C.TeC (== 10)
(r=5)
C-TeC (T=
C-TeC (+50
C-TeC (r=0.1)
Figure 18: Sensitivity of C-TeC to the temperature r in the contrastive loss (Equation (2)) C-TeC sensitive to the temperature parameter, however, in general, a value = yiclds large state coverage across environments.
G.9 IN-EPISODE VS ACROSS EPISODE NEGATIVE SAMPLING
Contrastive learning requires negative sampling to prevent representations from collapsing, In practice, for each batch sample, the positive examples of other samples are treated as negatives, following common practice (Chen et al., 2020). However, it might be desirable to sample negatives from the same trajectory, and results from Ziarko et al. (2025) suggest that doing so leads to learning better temporal structures compared standard practice. To sample negatives in-episode, utilize the
23

---

Published as a conference paper at ICLR 2026
method from Ziarko et al. (2025), where control the amount of in-episode negatives by duplicating the same trajectory in the sampled batch. We adjust the amount duplication using a repetition factor, where repetition factor of is equivalent to sampling negatives across episodes, and larger values indicate more in-episode samples. Figure 19 shows that in-episode sampling can result in better exploration performance, partieularly in humanoi ---aanaadd--aze d-u-maze and am-bii ch-ckrd.
Ant hardest maze
Humanoid u maze
Arm binpick hard
2 Visitad States
201e
0.00
137
-00
38633
11
23
Environment Step
Environment Step
Environment Step
Across-episode negulives
In-episode negatives (RF=2)
In-episede negatives (RF=1)
Figure 19: In-episode vs Across-episode negative sampling Sampling in-episode negatives results in additional performance boost in humano id-- u-maze and a a trm-biinicc--enn
H EXPLORATION IN NOISY TV SETTING
We investigate C-TeC performance in the presence of noisy TV state, run this experiment modified grid environment from minigrid (Nikulin et al., 2024) size 256 256 Figure 21 with noisy TV region. We did not observe any evidence of worse exploration performance namely the agent has covered all the states in the grid world, Appendix shows the state coverage of C-TeC compared to the maximum coverage.
State Coverage in Noisy TV minigrid
60000
45000
30000
15000
C-TeC
Maximum Coverage
Figure 20: C-TeC Coverage in noisy TV setting C-TeC can effectively explore in the presence of noisy states
I COMPARISON WITH PREVIOUS METHODS
At a high level, C-TeC is related to other intrinsic exploration objectives that reward uncertainty. Objectives such as RND (Burda al., 2019b) and Disagreement (Pathak et al., 2019) explore unfamiliar states, presumably leading to these states becoming more familiar in future rounds. A related method, CURL (Du et al., 2021), also relies using negative contrastive similarity score
24

---

Published as a conference paper at ICLR 2026
Figure 21: Xland-Mingg Mand-Miiigrid (Nikulin al., 2024) with noisy TV states indicated by the random colors.
for exploration like C-TeC. CURL prioritizes exploration over states with high error/low similarity scores with augmented states; however, the contrastive features learned CURL are not temporal and can be concretely related to prediction error.
The key difference between prior methods and C-TeC lies in the usage of temporal contrastive features. Our method drives the agent to explore areas where fulure outcomes have been seen but appear improbable. Taken together, our analysis and results show that temporal contrastive representations simple yet powerful frameworks for intrinsic motivation.
J INTRINSIC REWARD INTERPRETATION
interpretation of reward. The intrinsic reward with representations rewards (s, a) pairs that result in the largest additional number of bits needed encode the represen- tation induced P ($y st, a.) with a code optimized for the marginal pa.el (sy). In other words, prioritizes exploration in areas where the representation encoding schemes is highly inefficient.
On information-theome
On c interpretation of objective assuming perfect estimation of point-wise MI. We assume that representations perfectly capture point-wise MI. Taking an additional expectation of the roll-out occu-accy reveals that the PPO/SAC objective is minimization of MI
J (p:a)) PT (sylo,a) (a)laa (Se, at) = I[Sy:Se. A=]
(21)
where P.r is the policy induced discounted state-occupancy measure (see Eq. 1).
On C-TeC as a Two-Player Game
In addition to quantifying temporal similarity, the converged InfoNCE loss CŌRL. provides lower bound on the mutual information (MI) (Oord et al., 2018; Eysenbach et al., 2021):
I(Sy; S., Ar) log K LiRI (B;
Contrastive learning finds representations that maximize a lower bound on the MI between current states and actions and future state distributions.
Thus, we can view C-TeC as two-player game over an ever-expanding buffer. Namely, the CL step learns minimize LCRL. Meanwhile, the policy objective learns approximately maximize LCRL. when state-actie aatten pairs are strictly drawn from the roll-out policy (as opposed to the entire buffer), and the conditional and marginal future-state distributions are still defined over the buffer.
J.1 No (ACHIEVABLE) TRIVIAL FIXED POINTS
Does C-TeC have stable fixed points? Without additional simplifications, this problem is intractable. Notably, standard analysis would fail to prove convergence due to the con-commexityycoocavtty objectives. While the zero- gradient condition for the InfoNCE objective is clear, the zero gradient condition for the objective is not obvious due to the complex relationship between v and the state occupancy measure.
the
25

---

Published as a conference paper at ICLR 2026
A more aggressive simplification that can simplify analysis of the global optimum is (1) assume that the policy optimization is done directly over S and A,, and (2) assume the representations perfectly capture the point-wise MI. Furthermore, we assume that future states are exclusively sampled from the one-step transition dynamics and are deterministic.
In practice, these assumptions are very unrealistic; however, such simplifications have been used in prior work on unsupervised RL to give a conceptual picture of exploration methods (Pitis et al., 2020). Throughout, we assume fully expressive representations that capture the point-wise MI thus, are strictly analyzing fixed points and fixed point-stabliiyyaccievaaiil without taking into account representations.
Though the following analysis assumes a discrete setting (summations vs. integrals, Kronecker Deltas vs. Dirac Deltas), we do directly invoke the assumption of discreteness. The conclusions should continue to hold in the continuous case assuming all relevant probability distributions are bounded and smooth.
With these simplifications, the InfoNCE objective reduces to:
max [log K - Ls. (ZT, ((T,,FT) FT)] K (C-too,,nfiiiiee expressive reps 0,50
(ST,, Sy).
Because the "policy" optimization is fixed in p(ss s,a), the MI I(S Sz, Ar) (see Eq. 21) is concave in a) and Pa (sy) (Cover & Thomas, 1991). Our objective has now reduced to a constrained optimization problem with conditions Σ., Pa (s, a) and (=(s, 2 for all
(s, a) E A.
Consider the fixed point conditions given by the Lagrangian that is Lipschitz-continoous over the probability simplex As. Let and juss, a) denote the Lagrange multipliers for the normalization and non-negativity conditions respectively. Then, the full Lagrangian LLagrangian is follows:
Liagrangian (P X, T(S,, ST) + Ex
11) Σ u(s,a)p==s,00
Note that by complementary slackness, we have plss, a)p(s, a) O. Taking the functional derivative of Lagrangian with respect to distribution p(s, a) yields the KL-diieegenne:
8 Lagganggian
[s, a] = DKLLPr(s) s.a))lpr(s)) - 1 + p(s,a).
6px
By the complementary slackness, the distribution Ps (s, a) is a fixed point if the KL-divergence DEL DKL[PT(s, (sy 8, (s})) is constant for any (s,a) where (s a) has support. Any deviation would lead to a non-zero gradient at the point (s, a). In other words, all conditional trajectory future state distributions look equally "far" from the marginal.
Stationarity over iterations of C-TeC requires an additional condition: that the DxL remains constant over all (s, a) after updating buffer pT (8() with states encountered during the roll-out. A model of this is reweighing the marginal with the rollout probability distribution Pa (s = Pa Px
(s,
DKLLPT(S) a))a) (s))) = DKL(PT(s)) s, a))((1 (1) pp(s)) + a Pa
(22)
where 0 < 1. Again, we assume deterministic dynamics for simplicity and that 8 is always the next state (i.e. small discount factor like the Craftax setting) so the conditional distribution does not change; otherwise, we have no easy way of determining the change in (s s, a) after roll-out,
We drop subscripts on transitions and simplify:
p(3, 1s
PT())) Σp(o, p(ss 8, s,a) (1 (1-a) a) PT + ta pa(ss)
LHS Σ p(sy 8, a) log pT(8))
*5
*5
Pr(sf)
Dxulpr(ss ( s,aa)lpr(ss)) Ep(sylaaa log (1 (1-a) a--a.pP(T)) (s() + px (3))
pr(ss)
Cold Ep(sylea)
log (1-a) (s5) +a a (s()
26

---

Published as a conference paper at ICLR 2026
where Cola is the old constant DEL across (s, a). Thus, for the LHS to also be constant across (s, a). the difference must also be constant. We assume that transitions are nontrivial (as in, p(s/ 8, a) 7 p(sy)). This implies that the updated D remains constant iff
(1 (1-a) .T((s)) + P=(ss) = pTy) + pr(ss) = pny))
Under the assumptions of one-step, deterministic transitions and the a-reweighing of the buffer distribution, the distribution p. (s, a) remains a fixed point iff the roll-out future distribution and buffer future distribution are identical.
What is the stability of these fixed points? We can do linear fixed-pone point stability analysis by calculating the Jacobian of the update, where prime (*) denotes the next-step (sy). The update of (5x(s)) is as follows:
6p'r(s)) 6px((s) mppss/ s, s,a)[( (n (( Ar: 6px (s.a)
(23)
6pp((8) - 1 ( (o (S Ari Sy) 5p (18)
(change of vars.)
(1- (-))) (-,))
( (S Ari ST) 6p (s5),
(24)
We can similarly calculate the update for åpT (st):
(5))(8)) = a V (2, I( Azi Sy: (p-(8))
(weight new traj.)
+ - (1p-0))p788))
(down-weight old traj.)
Thus, the equation relating (6pm(s)), (pT(s) and (p)) is
6pp((65)
I I-nH 6p4(33)) (I-IHH) (1-a) (pT(s))
0
6pp (Op((1))
to first order in iteration time r, where H is the Hessian of the MI with respect p> (8y). Because the MI is concave in Pa (8)), the Hessian H is negative semi-definite; note that if H has any negative eigenvalues at the fixed point, the Jacobian would have at least one eigenvalue non-vertex fixed points in the product of two probability simplices As x As (where py = (s))) the Jacobian,
1. Thus, the
are either unstable, where at least one direction corresponds to an eigenvalue or semi-stable fixed points, where the MI is locally flat at the fixed point. Finally, fixed points at the vertices of the probability simplex (Delta functions) are uninteresting and are not observed in practice.
For an arbitrary MDP, we note that semi-stable fixed points are generally hard to achieve: a nontrivial, non-constant transition function, random roll-outs at initialization, the mixture of policies in the buffer, and newly encountered states prevent such semi-stable states from being easily accessible. Particularly, the random roll-outs help prevent no-op from being trivial fixed point.
This analysis shows that there are no casily-obtainabee stable fixed points for standard MDPs even under aggressive simplifications, implying constantly evolving probability distributions. Future work remains to investigate the existence of dynamical steady-states and whether the reached probability distributions cover large region of the probability simplex.
J.2 DIDACTIC TOY EXAMPLE
We present didactic example that clarifies the mode-seeking interpretation of C-TeC reward and the difference between our reward and the ETD reward by Jiang et al. (2025). The ETD reward (at convergence of contrastive representations) can be written as follows:
p(#t 82)
FEID
min log
(25)
k=(o,t)
p(s: sket)
ETD rewards states s, that are improbable from prior episodic states 8₁ (low p(s+ = 8, St. st))
relative to p(8+ St 84 se), where the reward is computed in the worst over the episodic
27

---

Published as a conference paper at ICLR 2026
memory. The overall ETD objective is to maximize the discounted sum of worst-case temporal distances. The C-TeC reward can be expressed as
p(ss) p(s 8, 8,a)
B|rc.tc]
log
(26)
C-TeC rewards states that are that are improbable (have low p(5 f 8, a)) relative to the overall marginal p(sy). Thus, C-TeC rewards states present in the buffer ("familiarity") that are tough to reach from the current state-action.
We show the difference in optimal agent behavior from maximizing Equation (25) and Equation (26) in a simple MDP (Figure 22). The MDP consists of root node connected to a right and a left branch. All trajectories begin from this root node. The left branch contains fast dynamics: the agent deterministically moves down the branch to the leaf node, progressing one level per timestep.
The right branch contains sticky dynamics. With 90% probability, the agent will remain stuck the state for a given timestep. With 10% probability, the agent will progress down the tree. The dynamics of the agent are independent of the policy after choosing the branch. Thus, this problem 2-armed bandit where the agent chooses left or right branch. We consider episodes of length 30 with 7 = 0.99 for both the discount factor and future state sampling in C-TeC.
only action: choose left right branch no stay action
dynamic left side:
cannot stay at current state, transitions deterministically node
slow/sticky right side: stay at current node with 90% probability go to next node with 10% probability
Figure 22: From the root, trajectories go either left or right. The left branch has fast, deterministic dynamics: the agent moves one level per timestep the leaf. The right branch has slow, sticky dy- namics: the agent stays in place with some probability and progresses with the remaining probability. After choosing a branch, the dynamics are pplicy-independen. reducing the problem a two-armed bandit.
An agent that maximizes the C-TeC reward is incentivized to match the marginal p(sy) ) and p(s) a). We visualize the future state distributions for different nodes in Figure 23. Clearly, the left side of the MDP leads to p(sy | s, Q) that much more readily matches the marginal p(sy): the state visitation has mode at leaf node with very little probability Meanwhile, the right branch assigns more visitation probability to nodes and 4 from the slow dynamics: the distributions p(s/ s, a) are less aligned with the marginal. Correspondingly, the CTEC agent chooses the left, fast-moving branch, reflecting our modc-seeking interpretation, and the ETD agent chooses the right, slow-moving mow-mmoing//stiky branch where nodes are temporally distant.
nodes 0 and in the buffer.
To visualize the mode-seeking nature of C-TeC, Figure 24 shows the discounted future state distribution for C-TeC on the preferred (left) side of the MDP and ETD on the preferred (right) side of the MDP. Even though the graphical MDP structure on the left and right are identical, the difference the transition kernel splits the methods' behaviors. The left side of MDP more readily enables p(8+ 8) to seek modes in p(8+)
28

---

Published as a conference paper at ICLR 2026
Learned Policies (Arrows) and Last 1000 Episode State Visitations (Normalized)
Tabular CTEC
Tabular ETD (Episodic Novelty)
0.
8.4
06 State Visitation (normalized)
0.8
1.0
Figure 23: State visitation of C-TeC and ETD, C-TeC distribution has mode on the most left node, while ETD prefers to stay in any state in the right branch
Discounted Future State Distributions
CTEC: Left Branch (Deterministic)
FTD: Right Branch (Sticky)
0.5
66-
015
03
66-
Future State s. (Left Branch Index)
Future State s, (Right Branch Index)
Figure 24: Future state distribution in the toy MDP in Figure 22. C-TeC prefers the states in the left branch with fast, deterministic dynamics while ETD prefers the right branch with slow, sticky dynamics.
K FORWARD-LOOKING VS BACKWARD-LOOKING REWARDS
In this section, we illustrate when hewward-looking reward like C-TeC may lead to divergent behavior when compared with ETD-type wokwwddd-ookinn reward (Jiang et al., 2025). We work in the tabular setting to isolate our results from any function approximation error.
We claim that forward-looking rewards may be beneficial when there are environment transitions with arbitrarily long waiting times. Consider an MDP in which a subgraph is a tree with actions A {left child, right child, stay]. Actions are fully deterministic from the root, with the exception of a 10% chance of a random action at any node. At depth of the tree, there is a 90% probability that taking left child right child action will simply lead the agent staying in place. Episodes are of maximum length of 10 with = 0.9 for both the discount and future geometric sampling parameter Figure 25).
29

---

Published as a conference paper at ICLR 2026
actions all nodes (left child, right child, stay)
near deterministically (~90%) transition from to take left child, to take right child.
taking action child or right child only succeeds <10% of the time; otherwise, stays stationary -90% probability
Figure 25: Toy Tree MDP. At the root, transitions are near deterministic. At level 1, taking the left child or right child action works only 10% of the time, while action stay works 90% of the time.
Learned Policies (Arrows)
Tabular CTEC, 0-9.9
Tabular ETD, 0=0.9
State Visitation (oormalized)
Figure 26: forward-looking (C-TeC) vs backward-looking (ETD) rewards As indicated by the large state visitation C-TeC prefers to stay at node since it can reach multiple distinct future states, while ETD prefers to stay node 3, the deepest node in the tree.
By definition, temporally distant states take a long time to reach. Packward-looking agent will be rewarded for visiting these states even if it has already visited them before, which can be problem, as the agent might spend valuable training time repeatedly visiting those regions. In the aforementioned toy MDP, we expect that maximizing the wockward-looking reward will incentivize the agent to continually push down the tree, and this is indeed the result in practice. On the other hand, we expect that maximizing the boward-looking reward will incentivize the agent to stay, for example, state or at the root, as the agent is rewarded for being in state that can reach distinct future states.
To test this hypothesis, we ran C-TeC and ETD on the aforementioned MDP. We show the results in Figure 26, where arrows display the optimal learned policy after 20k episodes and color intensity denotes normalized state visitation the last 1K episodes training. Our results reflect the behavior of forward-lookse (C-TeC) and backward-lookke (ETD) policies: C-TeC prefers to stay at decision- making node since can reach diverse future states from it, while ETD tries to stay at the deepest node.
30

---

Published as a conference paper at ICLR 2026
L COMPARISON TO MODEL-BASED BASELINES
We compare C-TeC to a model-based based on Stadie et al. (2015) and we show the results ?? C-TeC outperforms model-based exploration and explore more states.
Environment
C-TeC
MBRL (Stadie et al., 2015)
Ant- hardest-maze
2500 = 300
849 = 63 31 -
Humanoid-u-mi mmzze
230 = 40
Arm-binpick-hard
135000 10000
35000 + 3170
Table 8: Sample Efficiency of C-TeC
M EMERGENT EXPLORATION BEHAVIOR
Figure shows some of the learned behaviors of C-TeC in the humanoid-u-maze where the agent learns to jump over the wall to escape the maze.
Figure 27: Emergent Exploration Behavior in humanoid-u-maze. C-TeC exhibits interesting emergent behaviors; for example, the humanoi d-u-maze environment, the agent learns to jump the maze walls to escape the maze. Each represents independent evaluation epsidoe.
31

---

Published as a conference paper at ICLR 2026
C-TeC (Ours)
APT
RND
Figure 28: Qualitative Comparison in bumanoid-u-mazze
C-TeC Achievements in Craftax-Classic
Wake up
5
4-
Achievements ments
3
Collect wood
2
1
Collect sapling
125
Environment Step
Figure 29: C-TeC Achievements. C-TeC unlocks interesting achievements in Craftax-Classic the
plot shows a subset the unlocked achievements during an evaluation episode.
32


Reviews:
[{"review_id": "X7GOR0LNaS", "reviewer": "Reviewer_6M1w", "summary": "This article proposes C-TeC (Curiosity via Temporal Contrastive Learning), a novel exploration method that relies on intrinsic rewards and contrastive learning to reach unexpected but meaningful states. This technique differentiates itself from prior methods by its simplicity, avoiding the use of quasimetric learning. The method is also compatible with classic algorithms such as Soft Actor-Critic and Proximal Policy Optimization. Finally, empirical results show strong exploration performance on locomotion, manipulation, and open-world tasks.", "questions": "-", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presented empirical results show generally strong performance across various tasks, providing solid evidence of state-of-the-art performance.\n\nThe article presents sufficient theoretical background to justify the proposed method, going beyond simple empirical results.\n\nThe proposed approach is conceptually simple yet broadly applicable, removing complex components such as episodic memory or explicit distance metrics while maintaining competitive results. This simplicity makes C-TeC easier to integrate with standard off-policy algorithms like SAC and potentially more stable and efficient in training compared to previous exploration methods.", "weaknesses": "While the authors present extensive ablation studies, the paper does not include ablations related to sample efficiency or the method's sensitivity to hyperparameters.\n\nAdditional benchmark comparisons with recent state-of-the-art world-model-based algorithms, such as DreamerV3, would provide valuable insight into the trade-offs between their higher computational cost and their performance compared to the proposed method.\n\nWhile the authors recognize that this will be tackled in future work, understanding the performance of the proposed method in partially observable environments is crucial to demonstrate the method’s broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbTWbV0aNC", "reviewer": "Reviewer_xkVH", "summary": "This paper focuses on the problem of using intrinsic motivation to promote exploration in reinforcement learning, primarily by designing a new intrinsic reward. This new intrinsic reward directly follows the previous work \"Episodic Novelty Through Temporal Distance (ETD).\" The main motivation is to simplify the ETD method, with the following improvements:\n\n- ETD learns the temporal distance between states.  Distance meaning it should at least satisfy the properties of a quasi-metric. C-TEC demonstrates through experimental results that learning a quasi-metric is unnecessary; it is sufficient as long as the representation can reflect the temporal relationship between states.\n- ETD calculates the intrinsic reward by computing the distance between the current state and every state in the episodic memory, taking the minimum value as the intrinsic reward (backward-looking). In contrast, C-TEC samples a future state according to a geometric distribution and calculates the distance between the current state and that future state as the intrinsic reward (forward-looking).", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The algorithm uses the idea of temporal distance as an intrinsic reward. It has some contributions in simplifying ETD's algorithmic modules, and it was found on continuous control tasks and the Crafter task that simplifying these modules does not lead to performance degradation. I believe this contribution is sufficient.\n\nChanging ETD's intrinsic reward from backward-looking to forward-looking is interesting.", "weaknesses": "- The comparison results between ETD and C-TEC on Crafter are very strange. In the original ETD paper, the results for the method and its baselines were all in the (7.0 ~ 9.0) range. However, here the result for ETD is less than 0.5, and even C-TEC is only 2.0.\n\n- The paper's title, \"Discovering Diverse Behaviors via Temporal Contrastive Learning,\" is somewhat strange. I believe \"Discovering Diverse Behaviors\" corresponds to learning a set of policies, each with a different behavior. However, the paper is clearly not doing this. This title is ambiguous.\n\n- In line 256, the paper divides the intrinsic reward into a \"surprise\" term and a \"familiarity\" term. The \"surprise\" term can be understood as promoting exploration, but I do not understand the role of the \"familiarity\" term. I also do not quite understand the role of \"mode-seeking\" in exploration. Does the author mean that a high reward should be given to states that have been visited but are temporally distant, rather than giving a high reward to unvisited states?\n\n- C-TEC is (forward-looking) while ETD is (backward-looking). Besides the fact that the forward-looking approach does not require an episodic memory, are there any other advantages compared to the backward-looking approach? I find the current Crafter results unconvincing, and I believe that results from some toy examples would be more illustrative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ljdoJhoAFU", "reviewer": "Reviewer_aRUG", "summary": "C-TeC uses a temporal contrastive critic to give intrinsic reward as the negative similarity between (s_t,a_t) and discounted future states—favoring states with diverse, hard-to-predict futures, without episodic memory and friendly to off-policy RL. It plugs into PPO/SAC, can target specific future-state subspaces, improves coverage in Ant/Humanoid mazes, and beats ETD on Crafter. Contributions: a simple contrastive exploration objective, its info-theoretic view, and consistent empirical gains across domains.", "questions": "- You connect the objective to $−D_{KL}​(p_T​(s_f​∣s_t​,a_t​)∥p_T​(s_f​))$ or $I((s_t,a_t);s_f)$, under what assumptions (temperature τ, negative sampling, batch size) does the empirical contrastive loss yield a consistent estimator of this quantity? Can you give a finite-sample bound relating representation error 𝜖 to the bias in $r_{int}$?\n\n- How do you choose the distribution over offsets k∼p(k) for the “discounted future”? Did you try adaptive or curriculum schedules for k (e.g., increase max k over training)? Please add sensitivity/ablation for p(k) and the discount used inside the similarity.\n\n- Contrastive objectives can be biased by replay imbalance (e.g., many near-duplicate futures from recent policies). Do you stratify negatives by time/episode, or reweight by importance? Ablate (i) in-episode vs. cross-episode negatives, (ii) balanced vs. raw replay sampling, and report stability/sample-efficiency impacts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Provides an information-theoretic interpretation: in the idealized limit, the intrinsic reward equals $−D_{KL}​(p_T​(s_f​∣s_t​,a_t​)∥p_T​(s_f​))$, linking the objective to mode-seeking over future-state distributions.\n\n- Clear algorithmic specification (sampling of future offsets, InfoNCE update, PPO/SAC integration) that is implementable and off-policy friendly.\n\n- Demonstrates flexible targeting of exploration to task-relevant subspaces, a practical advantage for complex embodiments and object-centric tasks", "weaknesses": "- The “future-diversity” view overlaps conceptually with recent contrastive or mutual-information–based exploration (e.g., APT/CIC/Plan2Explore/APS). The paper doesn’t clearly isolate what is new beyond replacing episodic memory with a temporal contrastive critic\n\n- The intrinsic reward hinges on design knobs that may materially affect behavior: future horizon/window sampling, temperature in InfoNCE, negative-sampling strategy, and choice of future subspace (e.g., positions vs. full features)\n\n- Using negative similarity as reward risks mode chasing or collapse if the representation is not sufficiently regularized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6bsGnHFyPM", "reviewer": "Reviewer_yEZ5", "summary": "Based on the work of Jiang et al. (2025), this paper proposes a new reward objective that avoids quasi-metric learning and episodic memory. The experiments are conducted in navigation, manipulation, and open-world environments.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces a reward objective that is easier to compute.\n2. The experiments demonstrate that the proposed method leads to the agent visiting more states.", "weaknesses": "1. Temporal Similarity as Intrinsic Reward:\n\nThe use of temporal similarity as an intrinsic reward is explored in [1]. The novelty of this approach needs further clarification.\n\n2. Advantages Over [1]: \n\nCompared to [1], the proposed method (1) avoids quasi-metric learning and (2) eliminates the need for episodic memory. What are the specific benefits of these two design choices? A more detailed explanation would be helpful.\n\n3.  Experimental Results and Task Rewards: \n\nThe experimental results show that the visited states are more diverse. However, how does this diversity benefit RL training with task rewards?\nI believe this is not a significant burden on the authors, as they could simply incorporate the extrinsic reward into the current intrinsic reward framework. The absence of this experiment makes it difficult for readers to fully believe in the method's adaptability.\n\n4. Title Clarification: \n\nThe paper is titled \"DIVERSE BEHAVIORS,\" which typically implies that the agent can generate diverse trajectories, not just more diverse visited states. This could be somewhat confusing.\n\n\n[1] Yuhua Jiang, Qihan Liu, Yiqin Yang, Xiaoteng Ma, Dianyu Zhong, Hao Hu, Jun Yang, Bin Liang, Bo XU, Chongjie Zhang, and Qianchuan Zhao. Episodic novelty through temporal distance. In The Thirteenth International Conference on Learning Representations, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)