Demystifying The Mechanisms Behind Emergent Exploration in Goal-Conditioned RL
Authors: Mahsa Bastankhah
Grace Liu
Dilip Arumugam
Thomas L. Griffiths
Benjamin Eysenbach
Venue: ICLR 2026 Poster (2026)
Keywords: Goal-Conditioned RL
Contrastive RL
Emergent exploration
Cognitive interpretability
Primary area: reinforcement learning

Abstract: In this work, we take a first step toward elucidating the mechanisms behind emergent exploration in unsupervised reinforcement learning. We study Single-Goal Contrastive Reinforcement Learning (SGCRL) (Liu et al., 2025), a self-supervised algorithm capable of solving challenging long-horizon goal-reaching tasks without external rewards or curricula. We combine theoretical analysis of the algorithm’s
objective function with controlled experiments to understand what drives its exploration. We show that SGCRL maximizes implicit rewards shaped by its learned representations. These representations automatically modify the reward landscape to promote exploration before reaching the goal and exploitation thereafter. Our experiments also demonstrate that these exploration dynamics arise from learning low-rank representations of the state space rather than from neural network function approximation. Our improved understanding enables us to adapt SGCRL to perform safety-aware exploration.

Published as a conference paper at ICLR 2026
DEMYSTIFYING THE MECHANISMS BEHIND EMERGENT EXPLORATION IN GOAL-CONDITIONED RL
Mahsa Bastankhah
Grace Liu"
Department of Computer Science Princeton University
Machine Learning Department
Camegie Mellon University
645 @riincttonn..... edu
gliu2@andres cmu edu
Dilip Arumugam', Thomas L. Griffiths!.? Benjamin Eysenbach*
Department of Computer Science
Department of Psychology Princeton University
ABSTRACT
In this work, we take a first step toward elucidating the mechanisms behind emer- gent exploration unsupervised reinforcement learning. We study Single-Goal Contrastive Reinforcement Learning (SGCRL) (Liu et al., 2025), a self-supervised algorithm capable of solving challenging long-horizon goal-reaching tasks without external rewards or curricula. We combine theoretical analysis of the algorithm's ' objective function with controlled experiments to understand what drives its explo- ration. We show that SGCRL maximizes implicit rewards shaped by its learned representations. These representations automatically modify the reward landscape promote exploration before reaching the goal and exploitation thereafter. Our experiments also demonstrate that these exploration dynamics arise from learning low-rank representations of the state space rather than from neural network fune- tion approximation. Our improved understanding enables us to adapt SGCRL. to perform safety-aware exploration.
INTRODUCTION
Recent breakthroughs in deep reinforcement learning (RL) have revealed emergent behaviors: agents that develop skills without explicit rewards (Liu al., 2025), learn to plan without world models (Bush et al., 2025; Simmons-Fode
et al., 2025), and exhibit sophisticated exploration in open-ended environments (Team al., 2021). For example, Single-Goal Goal Contrastive Reinforcement Learning (SGCRL), learns several manipulation skills before receiving any rewards and without any explicit skill-learning objectives (Liu et al., 2025). Subsequent work has demonstrated this phenomenon in robotic manipulation (Liu et al., 2025), locomotion (Bortkiewicz et al., 2025), and multi-agen agent tasks (Nimonkar et al., 2025) (see Fig. 1), but the underlying mechanism that drives this emergent exploration remains unknown.
Conventional wisdom suggests that emergent properties arise from using large models (Bubeck et al., 2023), but even small neural networks learn feature hierarchies (Krizhevsky al., 2009), and learning word representations that support analogical reasoning is a consequence of the choice of loss function, rather than architecture (Hashimoto et al., 2016; Arora al., 2016). In the context of reinforcement learning, the extent to which emergent behaviors depend on neural network function approximation remains an open research question. Without understanding the drivers of behavior, we cannot reliably predict when, how, or why exploration strategies will emerge, thereby limiting our ability to use these models safely and reliably.
Methodologically, our goal of understanding this phenomenon sits askew to the standard ML toolkit used optimize performance on benchmark tasks. We thus take inspiration from cognitive science,
"Equal Contribution
Project page: https:/ aataaaabbbatakaaahhaaaa . tht.. i 1 /emysstiiiii wwtsstiffiiiggo fying- sing1 winggglgggg e-goal peall-exploratinnn

---

Published as a conference paper at ICLR 2026
where researchers study intelligent behavior with a rich toolkit including rational analysis (Anderson, 1990), intervention experiments (Bower & Clapper, 1989), and cognitive modeling (McClelland. 2009). As case study in how methods from cognitive science can be used to study the properties of Al models, we adapt these methods to understand emergent exploration in SGCRL. Specifically, we (1) theoretically analyze the optimization objective uncover the implicit drivers of agent behavior,
(2) conduct controlled intervention experiments on these behavioral drivers, and (3) build a simple model of the exploration mechanism tabular setting.
The main contribution of this paper is to show that it possible gain insight into the emergent behavior of even relatively complex learning algorithms. We do so by answering a specific question: Why does the SGCRL. algorithm explore effectively in the absence of any obvious intrinsic or extrinsic rewards? We show that exploration is driven by the interplay between the actor and the critic, Although the algorithm trained without external rewards, the actor's objective can be reinterpreted maximizing (implicit) rewards which drive the agent to states that look like the goal, according to the agent's current repre- sentations. The critic shapes this implicit reward landscape by decreasing the representational goal similarity of states along unsuccessful trajectories, pruning them from future exploration. Surprisingly. simplified tabular model of SGCRL reveals that these exploration dynamics arise from learning low-rank representations, rather than from neural network function approximation. Finally, our analysis provides insight into how to adapt SGCRL to perform safety-aware exploration. Our success in analyzing the emergent behavior of this particular algorithm suggests that this approach can be applied productively to other settings where Al systems produce complex and unexpected behaviors.
Figure I: SGCRL exhibits emergent ex- ploration on range of tasks. But why?
RELATED WORK
growing line of work has discovered emergent behaviors in deep RL: agents demonstrate in- creasingly sophisticated skills without explicit programming (Liu et al., 2025; Bush et al., 2025; Simmons- Edler et al., 2025: Team al., 2021). Existing approaches to understanding deep RL agents aim to improve algorithm transparency or explain behavior post-hoc (I (Heuillet et al., 2021; methods include learning low-dimensional, meaningful representations of the state space (Lesort et al., 2018; 2019; Raffin et al., 2019), labeling actions based on targeted reward components (Juozapaitis et al., 2019), and maintaining subgoals with hicrarchical RL (Beyret et al., 2019; Cideron et al., 2020). Although transparency-basee rithmic understanding, they require modified system architectures or auxiliary training tasks (Beyret et al., 2019; Juozapaitis et al., 2019). In contrast, post-hoc methods take the RL algorithm black-box and attempt to distill the representations or predictions into interpretable trees (Bewley & Lawry, 2021; Coppens et al., 2019) maps (Greydanus et al., 2018: Zahavy et al., 2016). Our work differs from prior work in trying to explain behavior from an algorithmic perspective without the overhead of auxiliary training tasks.
Glanois et al., 2024). Transparency-bet
methods improve algo-
Following a recent trend (Hamrick & Mohamed, 2020; Binz & Schulz, 2023; Frank, 2023; Ivanova, 2025; Ku et al., 2025), our methodology is inspired by tools in cognitive science for understanding intelligent behavior, including rational analysis, intervention experiments, and small-scale models. Rational analysis explains the behavior of agents by considering the optimal solutions to the problems that they face (Anderson, 1990). In the context of Al models, this means determining the consequences of maximizing their objective (MeCoy et al., 2024). Intervention experiments are used to test theories about the causal mechanisms underlying behavior (Bower & Clapper, 1989). Building simple cognitive models is a tool that can be used to test whether a small set of principles is sufficient to reproduce that behavior (McClelland, 2009). Accordingly, we draw on rational analysis and controlled interventions to uncover the implicit objectives driving SGCRL's exploration dynamics and use simplified computational model of the algorithm to understand how these dynamics arise.
Exploration in goal-conditioned RL is challenging because rewards are extremely sparse. Many prior methods address this by commanding data collection on a distribution of goals, even when the
2

---

Published as a conference paper at ICLR 2026
downstream task involves only single target, to encourage broad exploration during the training (Chane-Sane et al., 2021; Savinov et al., 2018; Shah et al., 2021; Zhang et al., 2021a; Florensa et al., 2018; Pong et al., 2019; Venkattaramanujam et al., 2019). However, Liu et al. (2025) showed that the contrastive RI. algorithm, when collecting data with a single hard goal, induces strong exploratory behavior and allows the agent acquire useful skills without additional supervision. They further demonstrated that this single-goal data collection approach outperforms methods based on distribution of goals, goal curricula, and waypoint generation. In this work, we explain why single-goal training succeeds. Moreover, we show that collecting data with a single hard goal produces representations that drive more effective exploration than those learned from multi-goal data data-demystiffingg the surprising result of Liu et al. (2025) that single-goal training can outperform subgoal curricula.
PRELIMINARIES
Problem Setup. We formulate sequential decision-making problem Markov decision pro- cess (Bellman, 1957; Puterman, 1994) without an explicit reward function. The state space denoted by S with the initial state is sampled So Po and subsequence states sampled st+1 p(- st, az). The agent is given a single target goal g E S, representing the desired state to be reached. The agent must learn con-icondi policy a₂ x(- $t.
Goal-conditioned RL. For any policy и, we define the y-discounted occupancy measure (Sutton al., 1999a) p, (sy) == (1 - 7) Er= y'p. (st $y), where p. (s, sy) is the probability of being at state 8y at timestep t. Following prior work (Blier et al., 2021; Chane-Sane et al., 2021; Eysenbach al., 2022), the objective is to learn a policy that maximizes the probability of reaching the goal: max, P (g)- This task is challenging because the agent does not receive any feedback from the environment regarding its intermediate progress towards success.
Single-Goal Contrastive RL (SGCRL). We study SGCRL (Liu et al., 2025), an actor-critic frame- work based temporal contrastive learning. In contrast to prior work (Nasiriany et al., 2019; Chane-Sane et al., 2021), the exploration of SGCRL (Liu et al., 2025) is not guided by human- designed curricula of tasks or manually specified reward functions. The critic estimates the likelihood that a state-action pair (s, leads to future state 8f. and is parameterized as Φ(s,a) a) *b(8)), where (ss a) and (sr) are learned embeddings. The critic embeddings are trained with con- trastive loss where, for each state-action pair, (se;0e) positive future states are drawn by looking Geom(1 - 7) steps ahead; meanwhile, negative examples are sampled from the marginal distribution p(sy) == EE (E)(..)) (sy s, a.). In particular et al., 2024; Liu et al., 2025):
the backward InfoNCE loss (Myers
N
exp(o(s), ai) T (o() log
max (a, 0.0
Σ exp(o(s) a)) ((091))
of (p)
where (D(8, a) denotes the empirical data distribution of the replay buffer. The contrastive objective aligns each state-action pair with its true positive future state while discouraging alignments with unrelated states. We normalize all representations by their Eg norm. Once trained, the critic encodes a log-Q value Φ(s, a) {s))) = log p., (s a) log p(sy), (Eysenbach et al., 2022).
The actor aims select actions that maximize the likelihood of reaching the goal:
max Ex-p(s), a~a((1)-)) a) (a]
(()) + T ())()))))))))))))))))))))))))) s..))]
(2)
where T is an entropy-regularrzztion coefficient (Williams & Peng, 1991). In the discrete-action setting, the policy optimizing this objective samples actions from a softmax distribution:
e ((a.0) (o())
ata s,9)
Ea et "(s,a') ((9)
For continuous action spaces, we train a parameterized actor T(- |s, sy). where sy denotes a target state that is not restricted to the final hard goal. The algorithm always collects data conditioned on a single goal g and does not make use of rewards, demonstrations, or subgoals. For a complete description of the method, see Appendix A.
3

---

Published as a conference paper at ICLR 2026
Negntiee exampl (om
Gool eriboddingg
Sattosatinn embedding (In))
Goal
Goal
Gual
*
d (0) (si,
glfoo) (o)))
(00)
(8is a.)
Neguiio comppe (arnverged)
Siatoartion embodating (comsaned))
(a) Single-goal exploration dynamics
(b) Representation geometry
Figure 2: (a) The agent moves toward goal-like states (left), rules out non-goal states by assigning them low (green) ii-similarity (center), and leaves a high (red) gisimilarryt trace along the successful trajectory once the goal is found (right). Solid lines indicate past paths; dashed lines indicate new paths. (b) Negative representations o(s;, aq) and (())7) are pushed away from the common component (g)), enhancing their contrast.
4 SGCRL REPRESENTATIONS INDUCE A CURRICULUM OF REWARDS
In this section, we theoretically characterize the dynamics that drive exploration in SGCRL and make testable hypotheses about its behavior, which we later verify in Section 5. We posit that the actor maximizes discounted sum of implicit rewards shaped by the critic. Even when the algorithm poorly estimates the probability of reaching the goal, the implicit reward function is well-defined and directs exploration (Fig. 2a-left). As the actor optimizes this implicit reward signal, the critic dynamically reshapes the reward landscape. This implicit reward reflects the agent's belief about the goal position, which updates over time as the agent gathers more data. Before the goal is found, the critic decreases the implicit reward for states along unsuccessful trajectories (Fig. 2a-center). However, once the goal is reached, the critic instead increases rewards along the successful path (Fig. 2a-right), shifting the agent's behavior from exploration to exploitation. this way, our analysis highlights a connection to a long line of work on reward shaping and the optimal design of reward functions that best facilitate efficient synthesis of good policy parameters (Ackley & Littman, 1992; Ng et al., 1999; Singh et al., 2009; Sorg et al., 2010;;;; Sorg, 1011; Devlin & Kudenko, 2012).
Moreover, since our analysis revolves exclusively around the associated actor and critic objectives, would suggest that the exploratory behavior of SGCRL does not depend on function approximation.
Section 4. establishes the equivalence between SGCRL and a reward-maximizing agent that max- imizes representation similarity. Section 4.2.1 shows how contrastive learning naturally reduces representation similarity explored states, while Subsection 4.2.2 discusses how representations evolve once the goal is reached. Formal theorem statements and proofs are provided in Appendix B.
4.1 THE ACTOR MAXIMIZES AN IMPLICIT, REPRESENTATION -BASDD REWARD
Our analysis reveals that, although the SGCRL objective is defined with respect to reaching g. simultaneously drives the agent toward states sf that exhibit high representational similarity to the goal, quantified by the inner product between the state and goal representations, metric we denote by di-similarity := vssy) (gg). One interpretation of this similarity function the agent's encoding of beliefs about the goal's 's location in the v-similaritt metric, which is progressively refined through critic updates as more data are collected. This intuitively resembles posterior- sampling approaches, where the agent maintains posterior distribution over the underlying reward and transition functions (Strens, 2000; Osband et al., 2013) guide exploration via epistemic uncertainty (Der Kiureghian & Ditlevsen, 2009).
Our main result this section relies on an alignment property of the InfoNCE objective (Eq. 1), namely that optimization encourages the representations of positive pairs to align in expectation. This assumption is consistent with insights from prior work (Wang & Isola, 2020). However, in Appendix B.2, provide two concrete conditions under which the alignment property provably holds. For example, Appendix B.2.2 shows that alignment is guaranteed when the learned contrastive representations have isotropic Gaussian distribution (an assumption corroborated by previous work
4

---

Published as a conference paper at ICLR 2026
Wang & Isola (2020)). For clarity, state the alignment property here as assumption, with full proofs and technical details deferred to the Appendix.
Assumption (Alignment of positive examples under InfoNCE), When the InfoNCE loss (Eg. 1) is oplimized, the representation of any stule-uction pair (8, a) aligns with the expected representation of its future states:
o(s,a) (ss=pt((slss.a) [bb(s)]
Theorem 1. Given Assumption 1, maximizing the likelihood of reaching the goal in the SGCRL actor objective (Eq. 2) is equivalent to maximizing the return
(((s,a)T((o))
[QT
E s=p((s)
E smp(ss)
amda (-1.9)
a~H((19.1)
where
(s, = (pt(ss)
( ( (1) 80 5,
0g
Theorem implies that, although SGCRL is unsupervised algorithm and does not use any external reward from the environment, it implicitly maximizes an internal reward i.e., vi-similarity. This result suggests that when cisiiiilaytty is well-structured and sufficiently dense, it effectively drive exploration. For instance, initial high ci-siiilarity near the initial state provides the agent with gradient to move away from the start; however, di-similarity must decrease over time in regions without the goal, otherwise the agent risks becoming trapped. Moreover, Theorem yields testable predictions, which we validate in Section 5. In particular, predicts that the agent will be drawn toward regions of high ql-similarity and avoid regions of low v--imiilarity even when these regions are respectively far from or close to the goal (see Section 5.2 for the empirical validation). Moreover, Theorem I yields testable predictions, which we validate in Section 5. In particular, the theorem predicts that the agent's 's behavior guided by similarity representation space: the agent is attracted to states with high vi-similarity to the goal embedding and repelled from states with low sl-similarity, even when those states are respectively far from or close the goal the raw environment. We empirically confirm this prediction in Section 5.2.
Notably, the formulation in Theorem resembles the structure of the successor representation (Dayan, 1993) and successor features (Barreto et al., 2017; Kulkarni et al., 2016), where (s) plays the role of a state feature vector and Φ(s, a) resembles a successor feature prediction. Yet several conceptual differences distinguish our setting from this prior work. First, successor-feature methods explicitly train state features (e.g., via reconstruction losses) and then learn successor features on top of them. In SGCRL., both the features and their successor representations emerge naturally from the InfoNCE objective, without requiring explicit successor-feature training. Second, whereas successor features are typically used for rapid reward adaptation and transfer (Barreto et al., 2017: Kulkarni al., 2016), our focus is on how the learned representations themselves drive exploration. In SGCRL, the state features evolve dynamically during training a way that induces exploratory behavior. Third, although this shaping effect resembles potential-basee based reward shaping (Ng et al., 1999). SGCRL does not implement such shaping explicitly. The "potential" toward the single goal arises implicitly from the contrastive objective rather than from hand-crafted potentials. This contrasts with frameworks such as eigenoptions, which deliberately construct potential- based rewards for different options (Machado et al., 2017).
4.2 REPRESENTATION UPDATES ADVANCE THE IMPLICIT REWARD CURRICULUM
So far, we discussed that the agent is driven by the shaped reward defined through ei-similarity. In this section, we analyze the evolution of gi-similarity as the representations are updated. Specifically, in Section 4.2.1 we show that before the goal is discovered, the gi-similarity values of states that have already been explored gradually decrease, preventing the agent from revisiting them and thereby pruning the search space. In contrast, in Section 4.2.2 argue that once the goal is found, this process reverses: states along the path to the goal acquire higher si-iimilarity after their representations are updated, which encourages the agent to consistently exploit that trajectory.
While the use of intrinsic rewards that exhibit this cettttiioneelloitttionnn behavior has a rich incluig-including novelty bonuses, episodic or count-based bonuses, and prediction-error bonuses
5

---

Published as a conference paper at ICLR 2026
(Mohamed & Jimenez Rezende, 2015; Bellemare et al., 2016; Burda et al., 2019; Raileanu & Rocktäschel, 2020; Pathak et al., 2017a; Henaff et al., 2022; Zhang et al., 20 differs fundamentally from these approaches. In prior methods, intrinsic rewards are heuristically designed, manually added to the task reward, and tuned via hyperparameters to balance exploratory drive with task-directed performance and exploitation. By contrast, the intrinsic reward in SGCRL is not an external design choice but emerges directly from the actor objective; it is principled, requires no additional tuning, and comes with an intuitive theoretical guarantee: it corresponds exactly to maximizing the probability of reaching the goal under the agent's s current knowledge about candidate goal states. In contrast, while other principled mechanisms for incentivizing exploration exist!(A garwal et al., 2020a;b), empirical demonstrations of their practicality and scalability remain limited compared to SGCRL.
4.2.1 EXPLORATION BEFORE GOAL DISCOVERY
We analyze a simplified setting in which (()) is fixed and the agent explores a region of the environment that does not contain the goal: this assumption is realistic when the states are sufficiently far from the goal and the critic network is sufficiently expressive. In this case, the updates affect only the representations of states in that triplet, i.e., (s,a, a, (4,8))) sy), while leaving (b(g) unchanged. The following theorem shows that if states in region initially exhibit consistently high gi-similarity (that is, if their representations share a common component parallel to the goal representation, in addition to independent state-specific noise) then repeated InfoNCE updates will reduce their similarity to (g)) until they become orthogonal, rendering the region unattractive to the actor. Intuitively, since the InfoNCE loss is invariant to shared components (i.e., adding fixed vector to all representations) the component of the representations parallel to (g(g) does not help learn temporal differences. Because normalized representations have limited capacity, they suppress this redundant component to better use their representational budget to minimize the loss. See Figure 2b for an intuitive illustration.
Theorem (Informal). Let D {(ss, as. ) denote the collected dataset, where each triplet consists of a state 8ú, an action as and the corresponding future state 8 observed in the trajectory after taking (8i; 0a). Let vb(g) E Rd be fixed high dimenssimai unit vector representing the goal
such that g & {8 NN
Ti=1 Consider the normalized anchor embeddings (o(8,,a))?* az)] and future
embeddings (v(s) / D that are initialized as follows:
( (s,,a;) = =c((g)= + Co.
(813) = =c((g) + Kis
where Gi, Edd are i.i.d isotropic Gaussian vectors and c E IR is non-zero scalar. Suppose these embeddings are updated using the InfoNCE gradient descent update rule as specified in Appendix B. with sufficiently large batch size N and sufficiently small learning rate n. Then, with high probability over the random initialization, the system converges to an equilibrium that satisfies ((si,a;) (b))) =
(0) (g) vi
Notably, the proof of this theorem relies only on fixed-point analysis of the InfoNCE loss and does not require any neural network function approximation.
In practice, initialization, all & representations are nearly the same (no temporal structure has been learned), so e is high for all states. As training progresses, states along unsuccessful trajectories move away from (gg) in representation space. Consequently, Theorems and 2 reveal a two-player dynamic: the actor seeks regions with high u--imilaritt, while the critic reduces their similarity when the goal is absent. Refer to Appendix B.5 for a discussion on how this mechanism enables efficient exploration even in continuous settings, where the set of states to be ruled out is infinite. Theorem 2 assumes that (gg) remains fixed while other representations are updated-aa an assumption that holds exactly in the tabular setting. With shared neural encoder, updates to distant states can slightly shift (g)),, but the theorem is stable to such drift: if (g)) changes by at most E, orthogonality holds up to error. Moreover, we show in Appendix D. that this orthogonalization effect persists empirically in continuous settings, indicating that the fixed-poim analysis remains a good practical approximation.

---

Published as a conference paper at ICLR 2026
4.2.2 EXPLOITATION AFTER GOAL DISCOVERY
Considering that contrastive learning aligns the representations of positive examples (See Assump- tion 1, Appendix B.2), the representations along successful trajectory to the goal should align with the goal representation (g))), since g appears as a positive example for those states. This leaves "trace" of high uf-similarity states that enables the agent to reliably rediscover the goal. While full alignment cannot be guaranteed theoretically, we find empirically that successful trajectories indeed form high similiiyyyy trace from the start state to the goal. As shown in Section 5. 1, this trace consistently guides the agent to the goal, marking the transition from exploration exploitation once the goal has been discovered.
5 EXPERIMENTS
In this section, we present empirical evidence supporting our theoretical characterization of single- goal exploration (Section 4) in both continuous and tabular settings. Through our experiments, we address the following research questions:
RQ1. How do critic representations evolve during training to facilitate exploration and exploitation? RQ2. How do critic representations influence the actor's data collection strategy?
Subsection 5.1 addresses RQI using both a simplified model of SGCRL in a tabular setting as well as standard SGCRL in the continuous setting. Subsection 5.2 addresses RQ2 in the continuous setting and motivates our findings with preliminary results on how to utilize goal-similarity to improve safety.
Tasks. We study SGCRL on 2D point maze navigation tasks adapted from prior work (Eysenbach et al., 2022; Liu et al., 2025) well as the Tower of Hanoi goal-reaching task. The navigation tasks involve reaching a goal in various maze configurations including the classic Four Rooms domain (Sutton et al., 1999b), an L-shaped wall, and a spiral wall. The Tower of Hanoi task involves moving a stack of disks across three locations with the constraint that larger disks cannot be placed on smaller ones, and has been used extensively in studies of human problem-solving in cognitive science (Simon, 1975). We do not use rewards or subgoals to solve any of the tasks, and success is determined by whether the goal state reached some point during an episode. All training metric curves are averaged over 8 random seeds. All shading denotes one standard error.
Tabular SGCRL. Previous implementations of SGCRL utilize neural network function approxi- mation, raising the question of whether these behaviors are fundamental properties of the SGCRL algorithm or artifacts of neural network dynamics. To isolate the SGCRI. exploration mechanism from neural network generalization properties, we designed a simplified computational model of the algorithm for the tabular FourRooms maze and Tower of Hanoi task. Each state s has an embedding (s) stored in a lookup table and updated via the InfoNCE gradient rule (Appendix B. We assume the environment follows deterministic transition dynamics and, instead of learning ((s, a). further
assume access to the ground-truth dynamics $1+1 p(st, dz) (this latter assumption can be relaxed
by learning in a based fashion). The policy takes actions according to Equation 3. Following the assumptions for Theorem 2, all representations are initialized (p(s)
e(s), with global Gaussian seed x shared across states and small, independent Gaussian noise E(8) per state.
5.1 RQI. HOw DO CRITIC REPRESENTATIONS EVOLVE TO FACILITATE EXPLORATION AND EXPLOITATION?
Our theory suggests that SGCRL automatically develops curriculum of subgoals by updating representations such that unsuccessful paths become less appealing over time (Thm. 2). To observe how representations change in controlled setting, conducted series of experiments with both the simplified tabular SGCRL model and standard SGCRL algorithm, showing that vi-iiiilarity decreases for states along unsuccessful trajectories and increases for states along successful trajectories. These results support the hypothesis that SGCRL benefits from a natural exploration curriculum that progressively pushes the agent toward unexplored regions.
Distinct phases of representation updates emerge. By running a simplified tabular version of SGCRL, we aim to test whether SGCRL's mechanism arise primarily from contrastive learning
7

---

Published as a conference paper at ICLR 2026
of low-rank representations rather than from generalization properties of neural networks. Our characterization of SGCRL's exploration mechanism preseribes that representation updates should be distinct before finding the goal compared after finding the goal. That is, before reaching the goal, we would expect the /isimillartty of frequently visited states to decrease, and after finding the goal, would expect the ui-similarity of frequently visited states to increase. To verify these dynamics, we ran tabular SGCRL in the Tower of Hanoi environment. We measured the correlation between state visitation and goal similarity before and after the agent reached the goal.
We find that, even with the simplified tabular model, the agent demonstrates effective explo- ration and real-iea with distinct phases of behavior. Given uniform initialization of all states close to the goal, the correlation between state visitation and goal similarity starts high. As train- ing progresses, the state visitation count and goal similarity become negatively correlated prior to reaching the goal and positively correlated after reaching the goal (see Fig. 3). We also performed an ablation study where we replaced the vector- ized representations with a |S] x [S] lookup table of Mate-gooll similarity and updated these scalar values with the same contrastive objective. In this setting, the agent fails to explore efficiently, requiring 100x more samples than tabular SGCRL. (Appendix D.3). These results indicate that SGCRL's exploration dynamics arise from contrastive learning with low-rank representations rather than from neural network approximation or from the contrastive learning objective alone.
1.0
Pearson's
0.5
First majority success
0.0
0.5
-1.0
1000
2000
3000
Trial
Figure 3: Rumning SGCRL in tabular Tower of Hanoi environment reveals distinct phases of learning. The 3 disk task shown, but results generalize to 4/5 disks.
In Appendix D.I, we compare the evolution of U-similarity in SGCRL with the value function of R- MAX (Brafman & Tennenholtz, 2002), an optimism-basse exploration method for the tabular setting that is provably-efficient (Strehl et al., 2009). R-MAX and SGCRL share similar exploration dynamics: R-MAX starts with the belief that all states yield the maximum reward (cquivalent to assigning high el- similarity in SGCRL), and exploration progressively corrects these estimates as states are visited and their true rewards are revealed. We also refer the reader to Appendix D.6, which shows that although the exploration dynamics of SGCRL could, in the very worst case, require searching the entire state space to reduce all -i-iiiilaryty values, this does not occur in practice. In continuous settings represented by neural networks, the algorithm avoids exhaustive search. Although our findings suggest that neural network generalization is not the primary reason for SGCRL's exploration mechanism, nevertheless provides useful inductive bias: ti-similarity reductions for unsuccess- ful paths generalize to nearby states, im- proving exploration efficiency.
Triil 40000
'Fial 60000
Trial 120000
20
0.5
Representations along unsuccessful tra- jectories become dissimilar to the goal. To study RQI further, we simulate how representations evolve when goals are un- reachable. Theorem 2 predicts that rep- resentations for states along unsuccessful trajectories should become orthogonal to the goal. To test this prediction,
taal
(a) Representation evolution
y Similarity
an experiment in which we assigned the agent an imaginary goal v(g)) z. with z sampled from a Gaussian distribution. We projected the learned representations into three dimensions using PCA (see Fig 4), with z aligned to the vertical axis. We find that initially the representations cluster near the goal at the top of the unit sphere. Over time, they drift toward the "equator", col- lapsing into the subspace orthogonal to z, while still clustering states from the same
Start
Goal
(b) u--imiilarity evolution
Figure 4: (a) Initially all representations start close to later collapsing into subspace orthogonal to z, while preserving local room-level structure. (b) vi-iiiilayity decreases along traversed, unsuccessful paths.
8

---

Published as a conference paper at ICLR 2026
room together. The drift reflects the agent's visitation pattern: states in the bottom-left room, visited first, are pulled away earliest, while states in the top-right room, reached only in later episodes, collapse last. This experiment highlights a key property of the SGCRL data collection strategy: Because the single-goal conditioned policy consistently drives the agent toward areas with higher vi-imillarty (unvisited states), representations of previously visited states continually pushed further from the goal. These results provide insight into RQI and highlight the effectiveness of single-goal exploration, even in settings without real goal. Moreover, the same exploration dynamic also extends to the multi-goal setting. With a simple modification to the action seliettne mechanism, SGCRL is able to reach distribution of goals (Appendix D.7).
To test whether the same representation trend holds for the continuous setting, we conduct an experiment using standard SGCRI, in which we fixed the data collection trajectories to always move in particular cardinal direction. This experiment allows us to isolate the effect of unsuccessful visitation on representational similarity. We find that initially the s--imillarity of all states is high, but, over the course of training, states along the frequently traversed paths systematically become less similar to the goal (see Fig 4b).
SGCRL data collection structures representations strategically. We also investigate whether the single-goal data collection strategy offers distinct advantages over other data collection strategies for the representation shaping described in the above experiment. For the FourRooms task, we compare single-goal data collection with an alternative strategy that samples goals from a uniform distribution. This strategy fails to push representations of frequently visited states far from the goal, preventing effective exploration (Appendix D.5). These results indicate that the SGCRL actor is not merely a consumer of well-formed representations, but also active contributor. Through data collection, shapes representations in a principled way that decreases vi-similarity for frequently visited states.
5.2 RQ2. HOW DO CRITIC REPRESENTATIONS INFLUENCE THE ACTOR'S 'S DATA COLLECTION STRATEGY?
Next, we investigate how the dynamic representation updates characterized in Section 5.1 influence the actor's data collection strategy. Our theory posits that goal-similarity provides an internal reward (Thm. 1) to direct agent behavior. In this vein, we conduct intervention experiments to study how goal-similarity influences SGCRL's visitation behavior, with applications to safety-aware exploration.
Agent targets states with high d-similarity. In our first experiment, we perturb the initial position of the agent at various training checkpoints to be closer to the goal and observe whether the agent directly targets the goal state or target states that "look like" the goal (high gi-similarity). We find that the agent navigates toward the closest region with high representational goal similarity, even it can reach the goal directly through shorter path (see Fig. 5a). To test this behavior further, we conduct another intervention in which we fixed the representations of patch in the top-right room of the FourRooms environment to match the goal embedding ((g)). As shown in Fig. 5b (top), the agent is strongly attracted to this patch. leading to a substantial increase in visitation of the top-right room compared to the control setup (Fig. 5b, bottom). Notably, even when the agent succeeds in reaching the true goal, it frequently detours into this patch along the way. These results yield an answer to RQ2, indicating that the agent is guided by an implicit reward signal based on representational similarity to the goal. This behavior emerges from the learning objective without any explicit programming to seek goal-like states.
Agent avoids states with low visimilarity. Based on our characterization of SGCRL's behavioral drivers (Sec. 4), we predict that manipulating contrastive representations allows more fine-grained control of agent behavior during both training and deployment. We tested this prediction by setting the representations of the states in one of the rooms in the continuous FourRooms environment to be the negative of the goal representation (1²()) = - -x(g); v/s E R). We find that this intervention leads the agent to systematically avoid that region during both training (Figure Sc - top) and test time (Figure 5c bottom). The agent successfully finds alternate paths to the goal while respecting the imposed constraints (see Appendix D.15 for more figures). These preliminary experiments show that understanding goal representation-diie drivee exploration could lead to improvements in safety and control of goal-reaching tasks, enabling practitioners to guide agent behavior through representation design rather than explicit reward engineering (Ibrahim al., 2024).
9

---

Published as a conference paper at ICLR 2026
5m Berrty
booiig room 49 lasen
Bottoneiat room Sobation
Dentrsl
63
82
6:3g
30006
Perturbations
(b) Interventions
Safety
Figure 5: SGCRL targets states with high s-similarrty (a, b), avoiding states with low sisimilarity (c)
CONCLUSION
Through theoretical analysis and controlled experiments, we have shown that SGCRL implicitly maximizes rewards based on representational goal-similarity, enabling effective exploration without explicit rewards. Our results indicate that these exploration dynamics arise from contrastive learning of low-rank representations rather than from function approximation with neural networks.
Prior work in self supervsup learning has considered contrastively- leameed low-rank representations detrimental to downstream classification tasks due to loss of representational capacity. However, for SGCRL, limited representational capacity not a weakness, but rather a necessary component the method's ' success. More specifically, the mechanism that allows SGCRL to prune the search space relies on the geometric constraints imposed by low-dimensional, normalized embeddings. Before the goal is found, the InfoNCE objective suppresses shared components (e.g. psi(g)) that do not help learn temporal differences, driving the representations of states along unsuccessful trajectories to become orthogonal to the goal embedding and incentivizing exploration towards unvisited states. At a high-level, these dynamics align with classic exploration algorithms - like R-MAX and PSRL that refine set of candidate desirable states during exploration. This characterization not only helps to explain the success of SGCRL in prior work, but also charts a path for how to retain the strong, appealing theoretical properties of R-MAX/PSRL in sigh-dimensional well as long-horizon tasks (Liu et al., 2025; Eysenbach et al., 2022; Zheng et al., 2023).
Beyond analyzing SGCRL specifically, our analysis provides case study for understanding emergent exploration through the lens of algorithmic interpretability inspired by cognitive-scieet science methods. We draw on analysis techniques commonly used in cognitive cedennce-incudeee controlled interventions, and simplified modeling, to construct and test theoretical account of this work
rational analysis.
SGCRL's behavior. The particular instantiation of this framework operationalized successfully yields insight into the behavioral drivers of exploration, enabling us to better control these systems for safer deployment. We anticipate that a similar methodology can be used to gain deeper insight into range of RL algorithms, potentially identifying ways in which those algorithms can be improved through the same implicit, contrastive reward-shaping process (Asmuth et al., 2008).
6.1 LIMITATIONS
Our empirical study, through its focus on SGCRL, exclusively revolved around goal-reaching tasks and did not consider reward maximization more broadly. We analyzed the learning dynamics of SGCRL but have yet to establish formal theoretical guarantees on its sample efficiency. In future work, aim study whether SGCRL achieve polynomial sample complexity in the tabular setting (Kakade, 2003; Strehl et al., 2009) and extend the method empirically to broader space of tasks beyond goal reaching.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have provided the source code for tabular SGCRL here. The experiments using standard SGCRL use the codebase and default parameters given in (Liu et al., 2025). The hyperparameters used for both tabular and standard SGCRL are given in Appendix C. The proofs of all theoretical results are provided in Appendix B.
ETHICS STATEMENT
Our work investigates the exploration dynamics of a self-supervised reinforcement learning algorithm and therefore has no immediate ethical concerns. We also develop variant behaviors relevant to safety-critical applications. However, as with many advances in RL, similar techniques could, in principle, be misused enhance adversarial behavior.
that mitigates certain
ACKNOWLEDGMENTS
Thanks to the members of the Princeton RL lab for feedback on preliminary versions of the work. DA and TG supported by ONR MURI N00014-24-1-2748 and ONR grant N00014-23-1-2510 GL acknowledges support by the National Science Foundation Graduate Research Fellowship under Grant No. DGE2140739, BE and MB acknowledge support from the National Science Foundation under Award No. 2441665. Any opinions, findings and conclusions or recommendations expressed in this material are those the author(s) and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
Yasin Abhasi-Yadkoon and Csaba Szepesvari. Bayesian Optimal Control of Smoothly Parameterized Systems: The Lazy Posterior Sampling Algorithm. arXiv preprint arXiv: 1406,394 2014.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference machine learning, pp. 22-31. PMLR, 2017.
David Ackley and Michael Littman. Interactions Between Learning and Evolution. Artificial Life 1992.
Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pe-pg: Policy cover directed exploration for provable policy gradient learning. Advances in neural information processing systems, 13.13399-13412. 2020a.
Alckh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances neural information processing systems, 33:20095- 20107, 2020b.
Shipra Agrawal and Randy Jia. Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds. In Advances in Neural Information Processing Systems, pp. 1184-1199 1194, 2017.
John Robert Anderson. The Adaptive Character of Thought. Psychology Press, 1990.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, Wojciech Zaremba. Hindsight experience replay. Advances neural information processing systems, 30, 2017.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
John Asmuth, Michael L. Littman, and Robert Zinkov. Potential- -Baed Shaping Model-Based Reinforcement Learning. In AAAI, pp. 604-609, 2008.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2026
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying Count- Based Exploration and Intrinsic Motivation. Advances Neural Information Processing Systems, 29, 2016.
Richard Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics, pp. 679-684, 1957.
Tom Bewley and Jonathan Lawry. Tripletree: A versatile interpretable representation of black box agents and their environments. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11415-11422, 2021.
Benjamin Beyret, Ali Shafti, and A Aldo Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 20/9 IEEF/RSJ International Conference intelligent robots and systems (IROS), pp. 5014-5019, IEEE, 2019.
Marcel Binz and Eric Schulz. Using cognitive psychology understand gpt-3. Proceedings of the National Academy of Sciences, 12066)e2218523120, 2023.
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: mathematical viewpoint. arXiv preprint arXiv:1001.07233, 2021.
Michal Bortkiewicz, Wladyslaw Palucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arczewski, Eukasz Kuciński, and Benjamin Eysenbach. Accelerating goal-conditioned reinforcement learning algorithms and research. In The Thirteenth International Conference on Learning Representations, 2025. URL
https:/ https///ppeeeeeeeeeeeee net/ forum? Saa.ummiddaga---- (8kvX.
Gordon H Bower and John P Clapper. Experimental methods in cognitive science. In M. I, Posner (ed.), Foundations of cognitive science, pp. 245-300. The MIT Press, 1989.
Ronen I Brafman and Moshe Tennenholtz. R-MAX- General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research, 3(0et))213-231, 2002.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lec, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303. 12712, 2023.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL htttss://ppmttt eew. net/ forum? 1amm:iid==ainS-----
Thomas Bush, Stephen Chung, Usman Anwar, Adrià Garriga Alonso, and David Krueger. Interpreting emergent planning model-free free reinforcement learning. arXiv preprint arXiv:2504.01877 2025.
Elliot Chane-Sanc, Cordelia Schmid, and Ivan Laptev. Gnal-conditioned reinforcement learning with imagined subgoals. International conference on machine learning, pp. 1430-1440. PMLR, 2021.
Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Higher: Improving instruction follow- ing with hindsight generation for experience replay. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 225-232. JEEE, 2020.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Danicle Magazzeni. Distilling deep reinforeement learning policies in soft decision trecs. In Proceedings of the IJCAI 2019 workshop explainable artificial intelligence, pp. 1-6, 2019.
Peter Dayan.
Improving Generalization for Temporal Difference Learning: The Successor Representation.
Neural Computation, 3:613-624, 1993.
Armen Der Kiureghian and Ove Ditlevsen. Alcatory Epistemic? Does it Matter? Structural Safety, (1)):: 105-112, 2009.
Sam Michael Devlin and Daniel Kudenko. Dynamic Potential-Based Reward Shaping. In 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), pp. 433-440. IFAAMAS, 2012.
Benjamin Eyseabach. Tianjun Zhang, Sergey Levine, and Russ Salakhutdinov. Contrastive learning as goal- conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603-35620, 2022.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528, PMLR, 2018
12

---

Published as a conference paper at ICLR 2026
Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451-455) 2023.
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tiappei Yang. Jianye Hao, and Wulong Liu, A survey on interpretable reinforcement learning. Machine Learning, 113(8)55847-588 5890, 2024.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International conference on machine learning, pp. 1792-1801. PMLR, 2018.
Jessica Hamrick and Shakir Mohamed. Levels of analysis for machine learning. arXiv preprint arXiv:2004.05100 2020.
Tatsunori B Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola. Word embeddings metric recovery in semantic spaces. Transactions of the Association for Computational Linguisticx, 4:273-286, 2016.
Mikacl Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episadic bonuses. NIPS 22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387100
Alexandre Heuillet, Fabien Couthouis, and Natalia Rodriguzz Explainability in deep reinforcement learning. Knowledge-
Systems, 2 214:106685, 2021.
Sinan Ibrahim, Mostafa Mostafa, Ali Jnadi, Hadi Salloum, and Pavel Osinenko. Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications. IEEE Access, 2024.
Anna A Ivanova. How to evaluate the cognitive abilities of llms. Nature Human Behaviour, 9(2)230-233, 2025.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-Optimal Regret Bounds for Reinforcement Learning. Journal of Machine Learning Research, 11(4), 2010.
Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is Q-Learning Provably Efficient? In Advances Neural Information Processing Systems, volume 31, 2018.
Zoe Juozapaitis, Anurag Koul, Alan Fern. Martin Erwig, and Finale Doshi- Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence, 2019.
Sham Machandranath Kakade. On the Sample Complexity of Reinforcement Learning. PhD thesis, 2003.
Michael Kearns and Satinder Singh. Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49:209 202, 2002.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexander Ku, Declan Campbell, Xucchunzi Bai, Jiayi Geng, Ryan Liu, Raja Marjich, R Thomas McCoy, Andrew Nam, Ili Sucholutsky, Veniamin Veselovsky, al. Using the tools of cognitive science to understand large language models different levels of analysis. arXiv preprint arXiv:2503.1344 2025.
Tejas Kulkamni, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv: lrXiv:660600..333 626.0233, 2016.
Timothée Lesort, Natalia Dodz-Rodríg,ez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392 2018
Timothée Lesort, Mathieu Seurin, Xinrui Li, Natalia Dodz-Rodr,g,ez, and David Filliat. Deep unsupervised state representation learning with robotic priors: robustness analysis. In 2019 International Joint Conference on Neural Networks (IJCN)). pp. 1-8. JEEE, 2019.
Grace Liu, Michael Tang, and Benjamin Eysenbach. A single goal is all you need: Skills and exploration emerge from contrastive RL without rewards, demonstrations, or subgoals. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /pw.revicnveeooooooooooo net / forum? 1 1d=xCkgX4X5u0
Xiuyuan Lu and Benjamin Van Roy. Entemation-Theoree Confidence Bounds for Reinforcement Learning. Advances in Neural Information Processing Systems, 32, 2019.
Marlos C Machado, Mare G Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning In International Conference Machine Learning, pp. 2295-2304, PMLR, 2017.
Doug Markant and Todd Gureckis. Category learning through active sampling. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 32, 2010.
13

---

Published as a conference paper at ICLR 2026
Douglas B Markant and Todd M Gureckis. Is it better to select receive? learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1):94, 2014.
James I. McClelland. The place of modeling in cognitive science. Topics in Cognitive Science, 1(1):11-38 2009.
R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. Embers of autore- gression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41))2232222012,, 2024.
Shakir Mohamed and Danilo Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. Advances in Neural Information Processing Systems, 28, 2015.
Vivek Myers, Chongyi Zheng. Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: contrastive successor features can provide metric structure for decision-making. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, JogR. 2024.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Advances neural information processing systems, 32, 2019.
Andrew Y Ng. Daishi Harada, and Stuart Russell, Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the 16th International Conference Machine Learning. pp. 278, 1999.
Chirayu Nimonkar, Shlok Shah, Catherine Ji, and Benjamin Eysenbach. Self-supervised goal-reaching results in multi-agent cooperation and exploration. arXiv preprint arXiv:2509. 10656, 2025.
Ian Osband and Benjamin Van Roy. Model-Based Reinforcement Learning and the Eluder Dimension. Advances in Neural Information Processing Systems, 27, 2014.
Ian Osband and Benjamin Van Roy. Why is Posterior Sampling Better than Optimism for Reinforcement Learning? In International Conference on Machine Learning, pp. 200222772221 2017.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (More) Efficient Reinforcement Learning via Posterior Sampling, Advunces in Neural Information Processing Systems, 26:3003-3011, 2013.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283-23333 2021.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell, Curiosity-driven exploration by self- supervised prediction. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 2778-2787 .ogg, 2017a.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pp.
PMLR, 2017b.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State- covering self-supervised reinforcement lcarning. arXiv preprint 1rXiv:1903.03000 2019.
Martin I.. Puterman. Markov Decision Processes-Diserette Discrete Stochastic Dynamic Programming. John Wiley Sons, New York, 1994.
Antonin Raffin, Ashley Hill, René Traoré, Timothée Lesort, Natalia Diaz-Rodríguez, and David Filliat. Decou- pling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics. arxiv preprint
2019.
Roberta Raileanu and Tim Rocktäschel. RIDE: Rewarding Impact-Driven Exploration for Procedurally- Generated Environments. In International Conference on Learning Representations, 2020.
Alex Ray, Joshua Achiam. and Dario Amodei. Benchmarking safe exploration deep reinforcement learning.
arXiv preprint /910:/9900//0000 7(1:22 2019.
Daniel Russo and Benjamin Van Roy. Eluder Dimension and the Sample Complexity of Optimistic Exploration. Advances Neural Information Processing Systems, 26, 2013.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi- peramttraeettic topological memory for navigation. arXiv preprint A 006,00653, 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /077:1777 2017.
14

---

Published as a conference paper at ICLR 2026
Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. Rapid exploration for open-world navigation with latent goal models, arXiv preprint arXiv:2 2021.
Riley Simmons-EEdie Ryan P Badman, Felix Baastad Berg, Raymond Chua, John J Vastola. Joshua Lunger, William Qian, and Kanaka Rajan. Deep needs deep behavior analysis: Exploring implicit planning by model-free agents open-ended environments. arXiv preprint wexiv:2506.0698/ 2025.
Herbert A Simon. The functional equivalence of problem solving skills, Cognitive psychology. 7(2):268 288, 1975.
Satinder Singh, Richard Lewis, and Andrew G Barto. Where Do Rewards Come From? In Proceedings of Anmual Conference of the Cognitive Science Society, pp. 2601-2606, Cognitive Science Society, 2009.
Jonathan Sorg, Richard L Lewis, and Satinder Singh. Reward Design via Online Gradient Ascent. Advances Neural Information Processing Systems, 23, 2010a.
Jonathan Sorg, Satinder Singh, and Richard Lewis. Internal Rewards Mitigate Agent Boundedness. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 1007-1014, 2010b.
Jonathan Daniel Sorg. The Optimal Reward Problem: Designing Effective Reward for Bounded Agents. PhD thesis, University of Michigan, 2011.
Alexander L Strehl, Lihong Li, Michael Littman. Reinforcement Learning in Finite MDPs; PAC Analysis. Journal of Machine Learning Research, 10(11), 2009.
Malcolm Strens. A Bayesian Framework for Reinforcement Learning. Proceedings of the Seventeenth International Conference on Machine Learning, pp. 943-950, 2000.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems, 12. 1999a
Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: Framework for Temporal Abstraction in Reinforeement Learning. Artificial Intelligence, 1 12:111) 2-11,,1111111111111111 1999b.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint 1xXi:2100.118000 2021.
Srinivas Venkattaramanujam, Eric Crawford, Thang Doan, and Doina Precup. Self-supervised learning of distance functions for goal-condee conditioi reinforcement learning. arXiv preprint arXiv:1907.02998 2019.
Roman Vershynin. Digeesilimnsi Probability: An Introduction with Applications in Data Science, volume 47.
Cambridge University press, 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International conference on machine learning, pp. 9929-9939. PMLR. 2020.
Ronald J Williams and Jing Peng. Function Optimization Using Connectionist Reinforcement Learning Algorithms. Connection Science, 3(3):241-266 1991.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-workd; A benchmark and evaluation multi-task and meta reinforcement learning. Conference on robot learning, pp. 1094-11 1100, PMLR, 2020.
Tom Zahavy, Nir Ben- Zri-Zmm and Shie Mannor. Graying the black box: Understanding DQNs. International conference on machine learning, pp. 1899-1908. PMLR, 2016.
Tianjun Zhang, Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine, Joseph Gonzalez. C-planning: An automatic curriculum for learning goal- poal-reaching tasks. arXiv preprint arXiv:21 12080, 2021a.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang. Yi Wu, Kurt Keutzer, Joseph Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Prucessing Systems, 34: 25217-25230, 2021b.
Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing contrastive RL: Techniques for robotic goal reaching from offline data, arXiv preprint arXiv:2306.033.. 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A SINGLE-GOAL CONTRASTIVE RL ALGORITHM
In this section, we provide more details about the SGCRL algorithm as presented in (Liu et al., 2025).
Algorithm 1 Single-goal Exploration with Contrastive RL. The difference from most prior methods that exploration done by commanding single difficult goal s", rather than sampling goals with range of difficulties.
Initialize policy ta(a s. replay buffer B, classifier with logits a(s, a) (b()).
2: while not converged do
3:
Collect one trajectory of experience using (a s, s,8y = 8*), add to buffer B.
4: Update representations ((8, a), (byy) and policy x(a 8,8 using contrastive RL.
5: Return policy x(a s, = =*).
SGCRL is simple modification of contrastive RL (Eysenbach et al., 2022): rather than asking the human user to provide training subgoals for exploration, SGCRL always commands the policy to collect data with single hard goal 8" This single hard goal is chosen to be a semantically meaningful state corresponding task completion. The actor and critic objectives presented in Section 3.
B THEORETICAL RESULTS
B. GRADIENT DESCENT UPDATES FOR OPTIMIZING THE INFONCE OBJECTIVE
We derive the gradients the backward InfoNCE loss with respect to the different representation parameters in order to characterize their update dynamics. Consider the batch D {s, a, sfiJN where 8fs is the positive example for state action pair 8i, Aç.
()) = Σ log exp (Ø(s), ax) T (sf()) Σx exp (Ø(Sk: ax) (s)f))))
exp (6(8:) (b8fs))
Pij Σn exp (Ø(sk, ax) (s$1))
(4)
Note that Pip. 1 but E, Pij #1
[ Σ
- -pis) --
(({{))
(5)
Vo(er))L=- Σ (ōis Pis) (2))
E(irs
(6)
(s;, = (11-1) a:) + "n(66s - Pij)) (b(1-1)(sf))
11 (+[s) = ()(0-1) (x1,) + E(ois - Pij) - (82, ai)
(8)
where yy is the learning rate and 6, = 1 = f. The update rules for the forward loss are analogous; the only difference is that, in the denominator of Pij. the summation is taken over (s
When representations are meant to have unit norm, we perform an additional normalization step after every update.
16

---

(8fs)), Vi, Published as a conference paper at ICLR 2026
B.1.1 CONVERGENCE.
Definition (Convergence). The convergence of the system characterized by Equations and is configuration in which the updates cease to alter any of the representations.
In the non-normalized case, this corresponds to vanishing gradients:
V Vole, = Volet))-= Vi.
In the normalized case, convergence arises either when the gradients vanish, or when they are parallel to the representations themselves, i.e.,
V Voloras) I. = c ((88,)))
for some scalars G2 d. In this case, normalization preserves the represenlation since scaling by constant leaves the direction unchanged:
(c, + 1) a.)
o(s;,
[[c] + ())(s;);))
and similarty for (st)-
B.2
SUPPORTING ASSUMPTION
Here provide two illustrative cases that support Assumption 1.
The first scenario considers the setting where representations are normalized, high- -ig-ddmmnssional, and-at unnfooocyy distributed on the unit sphere. The uniformity assumption is well-established fact the literature, with prior work showing that the optimum of contrastive losses indeed yields such distributions (Wang & Isola, 2020).
The second scenario addresses the case where the data trajectories used for training can partitioned into disjoint trajectories. In this setting, show that the e and vi representations along each trajectory fully align-not only in expectation, but exactly-for all state-action pairs and future states sampled along that path. This provides an even stronger version of Assumption 1.
We formalize the first scenario in Lemma 1, and then establish the second case in Lemma 2.
B.2.1 ALIGNMENT OF POSITIVE EXAMPLES UNDER DISJOINT TRAJECTORY ASSUMPTION
Consider the following data collection assumption: we collect data consisting of one anchor pair (8; (;) together with multiple future states {s) each trajectory. The trajectories are disjoint, meaning that anchor pair (s;, ag) or future state s is shared across different trajectories. Under this assumption, can state the following lemma (the proof of the lemma easily extendable to cases where there are multiple anchor per trajectory):
Lemma (Positive examples have fully aligned representations), Let N > 1 be the batch size and 1 be the representation dimension. Consider anchors (St, as) and their corresponding positive states 8f. i i= 1,....
.....,,, N, with initializations
as) =
(sft) =K.
(0, Lla.. Suppose the representations updated using gradient descent update rule of either the backward or forward InfoNCE loss (with normalization) with update rules as characterized in Appendix B.I. Then, with high probability over the initialization, the following holds at equilibrium:
where G. K4 ltd
p(s, a,) = (b(f)). Vi.
Moreover, if each anchor (si, as) has more than one positive example {ssb }k, then at equilibrium
a() (sff)) ki
where k indexes the positive examples.
17

---

Published as a conference paper at ICLR 2026
Proof. We refer the reader to the proof of Theorem 3 for the first part of this proof. There, analyze InfoNCE convergence under the assumption that each anchor (s,, 0;) has only a single future state sf In particular, Claim 2 of Theorem establishes the result in this case.
The more realistic setting, however, is when each anchor is associated with multiple positive examples. To handle this, we slightly adapt the proof of Theorem 3 to again establish full alignment. We use the same notation as in that proof and restrict attention the case where each anchor has K 2 positive examples. The extension any K >> 1 follows identically.
Concretely, we assume batch of size N, with anchor representations denoted [u:)a=1 (correspond- ing (s,,, a:)) and positive representations {v.] (corresponding ()(f))), Each anchor u, is duplicated, i.e., U2i U2i+1, while its two positive examples V2s. V2i+1 are independent. In other words, each U2i has two distinct positive examples.
We establish the following claims by induction, grouping indices (2i, 2i + 1) into bundles:
(a) (+) = (V:2 ((t)) = (V ( does not depend on the choice of and at > 0, 2 1, and (x0) =1.
(1)
(b) g000 (v22 v v21t1) does not depend on the choice of i, and 8at > 0, Vt 2 2, and g(00) 1.
(t)
(c) (4) = (v! v()) (ul) vf()v/") (u
(())
whenever [i/2] Lj/2]; that is.
cross-inner products are zero across bundles.
These properties hold at initialization (t = 0), since independent Gaussian vectors are almost surely orthogonal in high dimensions (see. for instance, Equation 3.14 of Vershynin (2018)). We show that the update dynamics then preserve these index invariant properties We simplify the probability matrix as defined in Equation 4 using the induction assumptions step 1:
p exp(u (1-1) , v ((--1)
if [//2 Li/2],
E. explu
(f-1) vj "y
and
exp(u!" (1-1) v ((-1))
=
Σ. cxp(u (1-1) vj 1111) if [ + 1i/2].
These satisfy the normalization condition 2p + (N - 2)g = 1 and. since N is large, q is small.
Now we write the GD update rule noting that each anchor u, receives two gradient updates per iteration (corresponding to its two positives):
2 ún
(t-1) =u
+1 (1 2p) (v
(v
v
Σ gvl(-1)
1#22.2411
(0)
(0)
(t)
u
U
|
(t)
(1)
21
Vii
(t-1)
+11
(1
2p)
uzi
(t-1)
Σ
1721,2041
(1)
We first prove (c). For simplicity, we denote (t-1) = a and 3(t-1) = B. We also note that by the induction hypothesis, the norms ( l|e and || [0 are independent of the index i and we denote
18

---

Published as a conference paper at ICLR 2026
these norms at time t 1 by Ta Tu- We write the cross inner product and simplify it using the fact that (1-1)) 0, 0 and v) is small and representations at time / 1 are unit norm.
(9) (,)) (u
(0))
| | VTu Tv
(3 +
= 0, whenever [1/2] + 1j/2]. 9200
The proof' for the cross inner products (v; analogous the argument above. Note that the invariance to the index, carry from timestep f to Now we analyze the evolution of (00 and ( which similarly remain invariant to the choice of index at time t, if being invariant to index at time t 1.
and (u, u
(0) when [i/2] * [j/2] are entirely
2
(10
(1-1)
+ (1 (V-22))(211))
(t-1) + v -2 Σ qv((-
U2i
2
1 O17²) + 4n - 2p)a,
(9)
100
V22
(t-1) 1)
17 (1-2p) 2p) u22
((-1)
E qult
2421
1 + 0172) + 2n((1 2p)a.
(10)
8 (v (8)
v20+1)
(B +11 - 2(1 2p))a (0)
T'
B+ 217(1 2p)a
(11)
1 21y(1 - 2p)a
21y(1 - 2p) 3)
B + + 21)(1 2p)er
(12)
Using the induction assumption that (*) for all 2 1, every gradient descent update increases 3(0 starting at 2. At equilibrium, 800 longer increase, which implies that 3(xc) 1.
(
(1) (t) U2i V2i
(a + 17(11 (2 B) + O(7"))
VT
a + 71(1 (2) + 3)
(by the given form of rw)
(13)
1 4n(1 2p)a /1 2n(1 2
a x22 (3)
5et (1-2p)))))
v + 4xa /1 + 2ra
a + x2 3)
(14)
(1+ 2 (1 + xTOO
a + x22 1 ())
n small
(1 3ra)
xx2 + 3 3x²2) (1 + 3xx) a
(15)
19

---

Published as a conference paper at ICLR 2026
Where Equation 13 follows from the approximation VI+ cT ca for small (equivalently, small 77). First, note that by Equation and since 3 2 0 (in- cluding at initialization), we have a > 0 for all t 2 1. This also completes the induction step to show that ((0) > 0 for all 2 2, using the update equation for / (Equation 12). Moreover, as established earlier, at equilibrium we must have 1. Substituting this into Equation 15 further implies that at equilibrium. Hence, we have successfully proved properties (a) and (b) too.
B.2.2 SUPPORT FOR ASSUMPTION VIA UNIFORM REPRESENTATIONS
Contrastive representations, when normalized, have been proven converge uniform distribution on the unit sphere (Wang & Isola, 2020), Building on this result, we adopt the same assumption to provide additional theoretical support for Assumption The specific form prove here does yield exact equality,
[x,a))sss))(((ss)) @(s,a),
but rather a positive proportionality, But since the multiplicative factor is strictly positive, the argmax in the maximization problem of Theorem remains unchanged and this result still useful.
Lemma 2. Assuming both high dimensionality (d > 1) and that, convergence, the contrastive representations (ss) are uniformly distributed on the unit sphere, we have:
(1)1 d exp((2) o(s,a).
Proof. We begin by noting that, in high dimensions, uniform distribution on the unit sphere is equivalent to an isotropic Gaussian distribution N/0,, (a) (see, for instance, Equation 3.15 of Vershynin (2018)):
d 1
((
(2
jd/2
exp
where d is the representation dimension.
From the InfoNCE objective (Equation 1), at convergence the representation satisfies
pr (s ass log
Therefore, for the expectation we have
= / p ((s))),))(((1) dsf
Ep- (09]11006/1((8))1))
/ (5,) p² (1 ((s))
(x1)
/ (5() (ss))) (1s,) dsy.
Now, switching variables to wy == (s)) and substituting the Gaussian density:
Ep (10( J&s (27)a/2 exp(-111v) exp(o(s, a)T w)) w dey.
Completing the square in the exponent:
o(s, a) 112 z
| + ((s,))
bb -
ll0(s, a)l
d
Thus,
/
exp ( -f1lby
a( 12)
E (((())) = (1))(s)))))3)))
by drb..
gg
(2
d
20

---

Published as a conference paper at ICLR 2026
The integral above is simply the expectation of a Gaussian random vector with mean (0, a), which equals ()) Therefore,
1 Zo(s.o).
exp ( (1)())))))(()))
Finally, since in high dimensions the learned ((s, a) is unit norm, we obtain
Bp (1) ā exp 2 o(s,o).
B.3 PROOF OF THEOREM
Proof. From Assumption 1:
a(s,a) Es (s,))(((
Substituting this into the first term of the SGCRL actor objective (Equation 2), we obtain
(00 max E anz(ale))) (b((,))) a) [t)) max Egpp(e) anv(ale.)) (Ba [Pp(esle (())
Expanding the discounted future state distribution yields
(1 (1-7) Σ
y pe (st =ss (s,a)) ( (())
= max Es)p(s),
aa=p())..amaalee)) (al(a)))
==
E Ep Bp (se) T (b))) su s, 0g
max Bs-p(s).
max amm(all))))))) ES
7 (s;)) (T:)ssTTi)) (g)) 50 a0
This is exactly the reinforcement learning reward maximization objective with reward function r(s. a) = (ss) (b)).
Therefore, maximizing the SGCRL objective is equivalent to maximizing the Q-value induced by this reward function.
B.4 A STRONGER VERSION OF THEOREM AND PROOF
We now analyze the equilibrium dynamics of the InfoNCE update rule in the following theorem. Claims and 3 of this theorem directly imply the result stated in Theorem 2. For simplicity of notation, we write u, in place of ((s;, a:) and v; place of (s...)..
Theorem 3 (InfoNCE representations equilibrium). Let E Id be a fixed unit vector, with d >> 1. Let {u, ) and (r C IRd be anchor and future embeddings, initialized as:
up c v
where C n0. Lid. N (2. and e a sealar. Suppose these vectors are updated via gradient descent the backward (or forward) InfoNCE loss with batch size N 1 and step size n > followed by unit-norm normalization. We assume is sufficiently small.
Then, with high probability over the initialization, the dynamics satisfy the following:
21

---

Published as a conference paper at ICLR 2026
1. At every step t, each representation decomposes as uf = c'z Çt vt = c'z =
where Ç R4 1. and ee is the same for all i.
2. At fixed point (i.e., when all the gradients are zero), u, (((0) =v(")) Vi, and (u (0) vj 1/00))
0, +
3. At fixed point: c(xx) = O, i.e., all representations become orthogonal to z as + oc.
Proof. We establish the above results, together with three additional claims regarding the InfoNCE update dynamics at equilibrium, via induction. At iteration t, we decompose each representation as
uf ( 5
c()) + R::
where the residuals ç and nf orthogonal the unit vector z. We define the following quantities and note that they are invariant with respect to the choice of index (or i # where applicable):
(a) (u' z) = (vi z) = c00) for all i.
(b) (*) = (C) K!))
(c) ( (St n²) (5) kf) for all i i and for all t.
(d) (+)) == ||< 1 ||x]1|2 for all
Base case (t = 0): By initialization, , ~ ~V00 i.id., and G kg 1 z. In high dimensions, with probability 1 these vectors are all orthogonal to each other therefore ( ( 0 and e" is the same for all vectors by construction. And ||<_|| = ||x Hence, all four properties a.b.e,d hold at 0.
c² by construction.
Inductive step: Assume the properties hold at time 1. We now prove they also hold at time t.
We first prove it for the that representations normalized and for the backward InfoNCE loss, the proof for the forward InfoNCE loss is exactly the same due to the symmetry at initialization, which, will see later, is maintained through all the updates. The backward InfoNCE updates with normalization are given by:
úi = u +
v 1
(16)
u = úf
similarly for v
(17)
j
(t-1) expl(uf A.v3 1))
where Psj
Σ. exp((ux ,v7 ")
Due to symmetry at time f 1, we have:
Pz (t-1), (t-1)
=: ((-(( for all i,j Pa
Pjj
so the matrix p(t-1) is symmetric, with equal diagonals and exchangeable off-diagonals. For ease of notation p P:; (1-1) and g Pij ((-1) Note that p + (N 1)q=
Property (a), (d): projection onto z and norm From the updates and the fact that all (((-1) are equal, we get:
- -P .z)
(t-1 -1
Σ (t)
vi
22

---

Published as a conference paper at ICLR 2026
Thus,
c(1-1)
(ii,2) = (uu-" cl1-1),
(uf 2)
(0)
(18)
|
and the same holds for v.. In order to prove (a), we need to show that ||u:l and ||v are equal and invariant to the index for any L. (For ease of notation we use at A, r instead of a (t-11) X (t--1) ,((-1),)
cl 1) + -
+ (1-D))) -
871
= (d(t-1)) + 1n22(11 - -pla 2nn(N 1)g
1 + 2n(1 - p) a a+ 011)²
(19)
We used the fact that due normalization (c(t-1) 3 + r 1, we also used 0.
straightforward to verify that the expression for || 1 is independent of the choice of index i. Furthermore, if we expand || we encounter the same number of matched pairwise or cross- term inner products, with only the order of terms being swapped. As result, we obtain the same expression.
We therefore denote this common quantity by
[(t-1)= = || = ||
Therefor, it follows from Equation 18 that (ui, z) is identical for all representations, Since all these representations share the same norm and identical projection onto z (i.e., the parallel component), it must be that their orthogonal components namely, & and xç-also have equal norms. Hence, condition (d) is satisfied too, this also ends the proof for claim 1 of the theorem statement.
Properties (b), (c): Matching and cross inner product We expand:
Ç 1(--1)
5 ct- +
(1-p)n)
Σ
x
Kig + +
(1- (1-p)</- -
Σ qcp
Σach
L((-
j71
Then:
(1) =0(= k()
(IIC-1) [a + 21/(1 - p)r 2n((N - 1)Aq + 0(72)]
(Lit-1))2aa + 21r(1 - p) O
This expression is independent of the choice of index i; this proves statement (b). Similarly, can evaluate (c!" (0), and (c( ((1)) for + j:
Ç 4-1
(1 - p)nt 1-1 Σ
qni
I(-II
5 1)
n
(1 k))
Σ
qny
1x
K/
1)
Ki
1-1 n
(1 (1- p)
Σ
23

---

Published as a conference paper at ICLR 2026
(5)
(L(--1))2 (1 + 2n711 p)) 2n 2) 2nga 017))
N Lurge (Lit-1)32 [A + 2n(1 - p}} -2ng(N - + + +017)
-+(N-1)a=1 (L(--1) [A + 0(72)
(C) ng) (Z0-1) 1A + +-2111 - p) 2n 2) - 2nyor + (1²))
N Large (L(t-1))2 1A + 2n/11 p)} 2ng(NV - 1)
1N11) (T((-11))2 ( 011))
Since N is large and p. q 2 0 with p- (N 1)q follows that q z We observe that these inner products are also independent of the specific choice of i and j, due to the same underlying symmetry. Moreover (c!") niy 1 and (c("), (()) are all zero given that À is zero (induction) hence (c)
is also proved.
This completes the inductive step, i.e. the proof for a. b. c, d.
Now assess how (1)) and p(1) change to prove claims 2, 3 of the theorem statement.
(t 1
(L(-1)) la + -
- p))
(20)
a 2ny(1 -p) T
2
+ 211(1 -p)
2n((1 - -pr (1 x
a+
1+ +121111= p) .
27y(1 p)r
a+ 1 + 2n/1 p)
(21)
27(1 - p)r (1 +
20
(22)
1+ 2n p)
Na
Since the denominator is positive and 1- a/rr 2 0. only stops growing if the equality holds i.c., a/r 1,
(c!
cos(c!" (!t11) , kig (((-11)) 1,
" (1-1)
so as long as cos ( 11 Ri (1-1) < 1, (t) increases, at equilibrium its growth should be stopped and that means the alignment of each positive pair increases strictly, and positive pairs keep aligning until they are fully aligned. This result, along with the fact that (()) 0, Vt, which we proved before, completes the proof for claim 2 of the theorem statement.
Finally, we prove Claim 3 of the theorem by showing that, at equilibrium, r = 1, i.e., the squared norm of the component orthogonal to z. Since each representation is normalized, this immediately implies that the parallel component must vanish.
(1) + 27 p)a
24

---

Published as a conference paper at ICLR 2026
Since ( = 0 and, by Equation 21, (+)) increases strictly (Note that due norm constraint the representations p can never approach 1) until full alignment, it follows that a+ > 0 for all 2 Consequently,
(
(t-1)) for all t 2 2.
Thus, the sequence {r(t)) } strictly increasing until it saturates the unit-norm bound.
B.5 IMPLICATION OF THEOREM 2 IN FUNCTION APPROXIMATION SETTING
Theorem 2 shows that agent trained with contrastive RL can rule out regions previously visited where the goal was not found, as indicated by low v-similarity. At first glance, this mechanism might appear inefficient in continuous settings, where the state space is infinite. However, in the function approximation regime, we expect the critic network to generalize in two important ways: (1) states that are far from the goal in the underlying state space should also be represented far from the goal in the embedding space, making them unattractive to explore; and (2) if a region is ruled not containing the goal (low dd-similarity), nearby states should likewise be ruled out. Thus, with function approximation this mechanism extends naturally to infinitely many states, Indeed, as we have observed in previous work (Liu al., 2025), SGCRL capable of solving long-horizon planning tasks in continuous environments. Also, we refer the reader to Appendix D.6 for two experiments that show SGCRL in continuous setting explores the environment efficiently and avoids non unrellaeee states.
C EXPERIMENTAL DETAILS
Table I: SGCRL Hyperparameters.
hyperparameter
value
Standard SGCRL (Liu al., 2025)
batch size
256
learning rate
3e-4
discount
0.99
actor target entropy
hidden layers sizes (policy, critic)
(256, 256)
initial random data collection
10,000 transitions
replay buffer size
1e6
samples per insert'
256
representation dimension (dim(ø(ss, a)), dim())(s.)) ))
64
actor minimum std dev
le-6
Tabular SGCRL
batch size
128
learning rate
le-2
discount
0.99
initial random data collection
False
replay buffer size
1e3
representation dimension (dim(tb(s,) (s,)))
16
How many times is each transition used for training before being discarded.
D ADDITIONAL EXPERIMENTS
D.1 TABULAR SINGLE GOAL EXPLORATION FALLS WITHIN A CLASS CLASSICAL EXPLORATION ALGORITHMS
Through comparison with tabular exploration methods, we investigate whether SGCRL employs clas- sical exploration strategies explores in unique way. Historically, statistcalll--effccii exploration algorithms broadly employ one two strategies: optimism in the face of uncertainty (Kearns & Singh, 2002; Brafman & Tennenholtz, 2002; Kakade, 2003; Strehl al., 2009; Jaksch et al., 2010; Jin
25

---

Published as a conference paper at ICLR 2026
Initialization
Exploration
Exploitation
Trial 210
Trial 480
50
48
Q-value
+
44
44
R-max
40
40
-40
36
36
u-similarity to goal
Trial 0
1.0
Trial 60
3.0
Trial 620
0.8
9.8
0.6
SGCRL
0.2
0.0
0.0
0.2
-0.4
0.0
Trial 0
Trial 100
Trial 480
25.4
25.2
20
Q-value
PSRL
24.6
12
Figure 6: SGCRL shares characteristics with R-max and PSRL
et al., 2018) or posterior sampling (Osband et al., 2013; Osband & Van Roy, 2014; Abbasi- Yadkori & Szepesvari, 2014; Agrawal Jia, 2017; Osband & Van Roy, 2017). One representative delegate of each is R-MAX (Brafman & Tennenholtz, 2002) and Posterior Sampling for Reinforcement Learn- ing (PSRL) (Strens, 2000; Osband et al., 2013), respectively. Curiously, instances either camp may admit unified regret analysis through the construction of confidence sets (Russo & Van Roy, 2013; Osband & Van Roy, 2014; Lu & Van Roy, 2019), collections that hold the true ground-truth hypothesis with high probability and (by virtue of good exploration strategy) can be shown to have shrinking widths as data accumulates.
Empirically, observe that SGCRL operates according this same unified perspective, maintaining an implicit collection of hypothesized goal states with aptimistically-infaule values and progressively refining this confidence set through targeted exploration. The algorithm's 's goal-conditioned represen- tation learning creates an initial landscape where many states appear promising (similar to the goal). By visiting these candidate states, the algorithm systematically winnows this set until it finds the true goal state (see Fig. 6). Visualizing training checkpoints for all three algorithms, we find that, due to optimistic initialization, SGCRL progressively explores candidate states like R-MAX until it finds the goal, at which point it demonstrates the rapid exploitation of PSRL (upon identifying the true underlying environment).
D.2 SGCRL ACHIEVES MODEST SUCCESS WHEN TRAINED WITH DATA COLLECTED BY ANOTHER ALGORITHM
In this experiment, we investigate the role of the actor data collection algorithm SGCRL perfor- mance. We conduct yoked experiments in which we perform the SGCRL representation updates using data collected by another algorithm (e.g. PSRL, R-MAX) running independently in parallel. We also conduct yoked control experiment in which SGCRL learns using the data collected by another independently initialized SGCRL agent running in parallel. We find that when we yoke SGCRL to PSRL R-MAX, it learns to reach the goal, though not consistently (Fig. 7a, 7b). When we yoke SGCRL to another SGCRL instance, both agents lear to consistently solve the task (Fig. 7c). These results imply that SGCRL's learn somewhat useful representations when trained data
26

---

Published as a conference paper at ICLR 2026
collected by another algorithm, but still attains the best performance with single-goal data collection. Contrary to previous work in both cognitive science (Markant & Gureckis, 2014; 2010) and deep RL (Ostrovski al., 2021), the success of the SGCRL-SGCRL yoked experiment implies that online interaction with the environment is not necessary.
R-MAU soca
Actlen GOCAL SOCRI
Rale
Success
Succoss
0.4
0.4
0.
0.2
0
2003
1502
2010
542
1500
2500
0.0
200
1020
1300
200
Thall
Thial
Trial
SGCRL yoked with PSRL (b) SGCRL yoked with R-MAX (c) SGCRL yoked with SGCRL
Figure 7: (a.b) SGCRL acheives modest success when trained on data collected by PSRL or R-MAX. (c) SGCRL succeeds consistently when trained on data collected by a different SGCRL initialization. Evaluation curves show the performance of the yoked SGCRL agent policy trained on data collected by another agent.
D.3 SGCRL FAILS TO EXPLORE WITHOUT REPRESENTATIONS
In this ablation experiment, investigate whether representations are important for strategic ex- ploration. We ablate the representations by implementing a version of tabular SGCRL without representations. Rather than parameterizing with 16-dimensional vector for each state, simply maintain an [) x |S] table scalar values for the ri-similarity of each state, goal pair. We learn the values in this table using the same InfoNCE updates and the rest of the method is exactly the same as tabular SGCRL. We find that by ablating vectorized representations (Fig. 8), the agent fails explore effectively, requiring on the order of 100x more samples find the goal and converging to low success rate. The results of this experiment imply that SGCRL representations, whether approximated by neural network or vector, are important for strategic and efficient exploration.
1.0
Tabular SGCRL (with representations)
Tabular SGCRL (without representations)
Success Rate
0.8
0.6
0.4
0.2
0.0
5000
10000 Trial
15000
20000
Figure 8: Replacing the vectorized representations in tabular SGCRL with a lookup table results in
slower exploration.
D.4 SINGLE-GOAL EXPLORATION IN ROBOTIC MANIPULATION TASKS.
We find that the characterization SGCRL detailed Section holds in a Sawyer robotic manipula- tion task where the agent must pick up block and place it in bin (Yu et al., 2020). During early stages training, the agent moves the robotic end-effector towards regions of high representational similarity to the goal. Subsequently, the representational goal similarity of these frequently visited regions decreases, and the agent visits new regions (see Fig. 9). These observational results suggest that our characterization of single-goal exploration generalizes beyond 2D navigation tasks.
27

---

Published as a conference paper at ICLR 2026
Figure 9: XY cross section of representational goal similarity for the Sawyer Bin environment. Each row represents checkpoints throughout training for different training seeds. The gray lines show the trajectory of the end-effector across 5 episodes. The agent moves the end-effector towards regions of high el-similarity to the goal, and then those regions subsequently develop low el-similarity, driving continued exploration to new areas.
D.5 SINGLE- GONGLE-GOAL DATA COLLECTION ESSENTIAL FOR EXPLORATION ENCOURAGING EXPLURANI REPRESENTATIONS
In this section, we analyze the role of single-gon exploration in forming representations that promote exploration even before the goal is discovered. We also examine the behavior of a single-goal exploration agent when the goal unreachable. Interestingly, the agent still engages exploration, eventually covering the entire state space. and at convergence, settles into random walk across the maze since the goal is never found. To conduct experiments, we use the simplified tabular model of SGCRL without neural networks introduced in Section 5.
broad
As we saw earlier, the central mechanism that enables SGCRL's effectiveness is precisely its ability to drive the ui-similarity of non-goal regions toward zero. This ensures that previously visited goal-free states are ruled out, forcing the agent to focus on unexplored areas. Theorem establishes this formally, showing that contrastive representations in SGCRL implement exactly this property.
In practice, however, the assumptions of Theorem 2 for instance, symmetric initialization - do not necessarily hold. For instance, in the four-room environment, the agent begins the bottom-left room and gradually visits new ones. Adding states from newly visited rooms to the replay buffer alters the updates of earlier rooms, breaking the symmetry assumption and eliminating the guarantee of orthogonality between room representations. In this section, we address two key questions: 1) Does the phenomenon of decreasing u-similarity persist in more realistic settings? 2) If so, what the underlying dynamics, and are they unique SGCRL's goal-directed data collection, can other exploration strategies achieve the same effect?
Evolving the representations, as shown in Figure 4, demonstrates that even in realistic scenarios the representations visited rooms drift farther from the goal representation at (0, O, 1) on the z-axis. This behavior is intuitive: first, within single room, the shared component aligned with (b(g) essentially wasted energy for contrastive learning. To achieve a stronger contrastive loss among states in the same room, the z-component of their representations is dampened (refer to Theorem 2). Second, as the agent explores more and begins visiting new rooms, the introduction of these new room representations pushes the older room representations even further away from the goal. This
28

---

Published as a conference paper at ICLR 2026
happens because newly visited rooms initialized close z, in order to maintain contrast, older rooms are better off shifting downward and away from it.
This observation naturally leads the following question:
RQ1. What happens if the representations of some areas in the new rooms are not initialized close to z? Do we still see the decreasing u--siillartty trend of visited states which is essential for exploration?
RQ2. Is single-goal exploration data collection necessary for d-similarity reduction in visited states? What does the representation evolution look like if we instead use multi- goal exploration data collection policy that collect data by sampling a new exploratory goal the beginning of every episode?
To explore this, we designed new experiment. In this setting, the representations of small patch in the top-left and bottom-right rooms is initialized orthogonal to z (in order to address RQ1), while the of the state representations are initialized as plus small random noise (see Figure 10). The agent starts in the bottom-left room (marked by the green dot).
98
Samilaam
Figure 10: SGCRL data collection vs random goal data collection. At initialization, all state representations are a noisy version of the imaginary goal representation (z) while the two small patches in the top left room and the bottom right room are initialized with an initialization orthogonal to z.
Apart from representation initialization, the experimental setup is the same the imaginary goal experiment of Section 5.1. i.e., (g))) = z is a random Gaussian vector that does not correspond to the representation of any actual maze state, simulating scenario where many representation updates occur without the agent ever observing the true goal. This experimental setup also allows us to address another interesting side question:
RQQ. How does the agent behave during early training and after convergence when the goal is not feasible in the environment?
start by answering RQI and RQ2 through comparing the following two data-collection strategies:
1. Single-goal exploration actions are selected according to the greedy policy:
1 ( (((p(se
ag
Z(s²) exp
2. Random-goal exploration actions chosen based on:
((((s,)))) ye yo ~ N(O, Ta)
~
Z(se)
exp
Where p(sr, ax) is the environment dynamic that outputs 8 moreover ye is Gaussian- sampled goal embedding drawn anew at each episode. This corresponds to the widely used convention in earlier works Andrychowicz et al. (2017); Eysenbach et al. (2022) where exploration is guided by sampling from distribution of potential goals to learn more effectively about the environment.
29

---

Published as a conference paper at ICLR 2026
Tolicds 9380
tutaado 42000
64
4.2
-2
-07
(a) Random- -gaal (before conver- gence)
(b) Random-gen goall (after conver- gence)
Fenase 9003
Fpitndte 42eco
50
(c) Single-goal (before conver- gence)
(d) Single-goal (after conver- gence)
Figure 11: Comparison of representation evolution under two data collection strategies: (Top row) random goal exploration and (Bottom row) SGCRL. Solid lines demonstrate the agent trajectory across 5 episodes.
Both data collection strategies start from the same initial representations. We then compare how the representations evolve under these two different data collection strategies.
As demonstrated Figure 11, at episode 9000-before the representations have fully converged-tte single-goal strategy naturally avoids the small dark patches. Because these patches have lower ei-similarity compared to their surroundings, the policy does not visit them. Instcad, focuses on areas with higher u-similaaitt. As result, the representations of the most frequently updated states consistently evolve by moving farther away from the goal, allowing them to form stronger contrasts with the newly visited states that are closer the goal in representation.
In contrast, the random goal exploration strategy does not avoid the dark patches (Figure 12a). By visiting these areas, it adds their states the replay buffer, which in turn forces the older room representations to contrast with these new states. This dynamic prevents the older representations from drifting away from the goal. Indeed, even after convergence, the qi-similarity under random goal exploration fail to approach zero, as shown in Figure 12b.
Figure 12 further visualizes this evolution by projecting the representations into three dimensions using PCA. The key takeaway is that single goal greedy data collection provides an implicit structural benefit: by consistently moving toward regions higher sisimilarity, pushes the representations of previously visited areas farther from the goal. In doing it shapes the representations of visited, non-goal regions in a way that naturally encourages further exploration.
Agent behavior when the goal unreachable Finally, we address RQ3 by examining the agent's behavior during both early and late training. The single-goal exploration agent is consistently drawn toward new states, but never toward the dark patch, since the representation of that region lies far from (g)). As training progresses without ever reaching the goal, the representations of all states gradually become orthogonal to the goal, yielding a uniform g-similarity zero across the maze. In this regime, the agent performs unstructured random walk indefinitely (Figure 12d). By contrast, the multi-gon goal exploration agent exhibits more diverse behavior, since changing goals occasionally directs it toward states that initially far from (g)). However, because it fails uniformly reduce
30

---

Published as a conference paper at ICLR 2026
Bettorn
Top Right
-00
2.0
(a) Random goal (before conver- gence)
Random goal (after conver- gence)
PCA gial fouffecro Cateds 05000
300
1775
04
50
0005
04
66
(c) SGCRL (before convergence)
(d) SGCRL (after convergence)
Figure 12: Comparison of representation evolution under two data collection strategies, with rep- resentations projected into 3D using PCA. (Top row) random goal exploration and (Bottom row) SGCRL.
u-similarity across the entire state space, it often becomes trapped in small regions that retain slightly higher similarity values (Figure 12b).
D.6 SINGLE-Gon GOAL EXPLORATION EQUIPPED WITH NEURAL NETWORKS AVOIDS EXHAUSTIVE STATE SPACE SEARCH
Our earlier analysis in Sections and 5 showed that single-goal exploration operates as optimization process: it begins by treating every state as a potential goal and, through contrastive learning, progressively prumes away non-goal states. In Appendix D. demonstrate the similarity of SGCRL to optimism-based and uncertainty-based methods such as R-MAX and PSRL.
However, in the worst case, these classical tabular exploration methods require exhaustive search over the entire state space (SS) which infeasible continuous settings. While this limitation holds for our simplified tabular SGCRL model without neural networks, we conjecture that the standard SGCRL algorithm with neural network approximation (v)(s) = Fø(s)) is less susceptible to exhaustive search. Due generalization, it is plausible that the reduction in v-similarity for visited states also extends to nearby states, thereby pruning not only the states actually visited but potentially additional states that would otherwise lead to unproductive exploration.
To test our conjecture, we designed two experiments that evaluate state coverage in the presence of distracting or irrelevant dimensions. The first setup introduces exogenous noise (noisy TV), while the second introduces controllable but irrelevant dimension. Together, these experiments probe whether SGCRL avoids exhaustive search over parts of the state space that do not contribute to solving the task. Our results suggest that SGCRL focuses its exploration rather than expanding coverage indiscriminately. In contrast, the novelty-drive -diven method PPO+RND, optimism-based novelty bonus, often expends effort exploring irrelevant states. This contrast highlights the practicality of SGCRL continuous domains, where exhaustive search is infeasible and where other optimism-basee approaches may struggle.
task-relevant regions of the state space
which incorporates
31

---

Published as a conference paper at ICLR 2026
Table State coverage (grid) comparison between PPO+RND and SGCRL. Lower coverage in the noisy-TV TV and irelevant-dimension experiments indicates robustness to distraction and irrelevant state dimensions.
Method
Noisy TV +
Irrelevant Dimension
Four-Room (no noise)
PPO+RND 0.91
0.65
0.82
SGCRL
0.36
0.40
0.90
Baseline. As baseline, consider a popular optimism-driven exploration method in continu- ous control: PPO combined with Random Network Distillation (PPO+RND) (Burda et al., 2019; Schulman al., 2017). In addition to the goal-conditioned extrinsic reward 1 g), this method incorporates an intrinsic reward based on the mean squared error between the predictions of two networks, A and B. where B is slowly distilled into A. This prediction error serves measure of novelty, assigning higher rewards to states that have not been frequently visited. The agent is trained using PPO to maximize weighted sum of the extrinsic reward (scale 2) and intrinsic reward (scale 1). We run this baseline for 300k environment steps. All results are averaged over at least 5 seeds of randomness.
Environment.
We consider continuous 11 11 four-room maze. The agent starts in the top-left corner, and the goal is located in the bottom-right comer.
D.6.1 NOISY TV EXPERIMENT
Setup. Following prior work on exploration, we evaluate whether algorithms fall into the so-called noisy 7V trap, scenario where agent may be distracted by uncontrolled stochastic signals rather than making meaningful progress toward the goal (Burda et al., 2019; Pathak et al., 2017b). To simulate this phenomenon, we augment the bottom-left room of the four-room maze with additional stochastic dimension 2, sampled uniformly from [0, 11] at every step. In all other rooms, the z-value fixed zero. This setup creates an exogenous source of noise that the agent cannot influence. The purpose of this experiment is to compare the state coverage achieved by PPO+RND and SGCRL in the presence of such uncontrollable noise.
Results. We compare SGCRL and PPO+RND task success and exploration behavior during training. In the base environment, SGCRL achieves a success rate of 98%, while PPO+RND reaches 90%. In the noisy-TV setting, success rate decreases by about 8% for SGCRL and 11% for PPO+RND, indicating that SGCRL is slightly more robust to the noisty-TV problem. However, task success alone does not reveal exploration efficiency. We additionally measure state coverage of both algorithms by discretizing the continuous maze into grid cells and computing the fraction of visited cells. Table 2 reports results after 300k environment steps.
In the standard setting, which corresponds to a grid of 121 states, both methods achieve broad coverage (0.90 for SGCRL vs. 0.82 for PPO+RND). In the noisy-TV setting, where the grid expands to 371 states due to the added noisy dimension, a clear divergence emerges: PPO + RND attains much higher coverage (0.91) compared to SGCRL (0.36). This difference reflects the tendency of PPO RND to overexplore irrelevant states. The noisy TV introduces additional dimensions with high intrinsic novelty, drawing PPO+RND to explore uninformative states. In contrast, SGCRL. maintains relatively low coverage (0.36), indicating that the additional noisy-TV states do not cause it to overexplore uninformative states.
Moreover, we plot the average fraction of each episode spent in the bottom-left room (the noisy- TV room) in Figure 27, where episode lengths are normalized to 50 steps. The results show that PPO+RND spends roughly 2-4x more time in the noisy-TV room compared SGCRL. This tendency persists even after the agent has successfully discovered the goal. These results further emphasize that SGCRI demonstrates robustness to irrelevant noise compared PPO+RND.
32

---

Published as a conference paper at ICLR 2026
Fraction of episode in noisy room
Fraction episodd
0.4
RND
02-
50k
100k
200k
250k
300k
Actor step
Figure 13: Average fraction of episode time spent in the noisy-TV room.
D.6.2 IRRELEVANT STATE DIMENSION EXPERIMENT
Setup. In this experiment, we again extend the four-room maze with an additional z-axis. Unlike the noisy TV setting, the agent now has full control over this dimension and can move freely along without encountering walls. However, both the initial state and the goal are constrained to lie on the 0 plane, meaning that optimal behavior does not require exploring the z-dimension all. The purpose of this setup is to test whether SGCRL explores the irrelevant dimension (z * 0), or whether efficiently focuses on states relevant reaching the goal.
Results. The results summarized in Table 2. In this setting, the state space contains 1331 grid cells. After 300k steps, SGCRL explores approximately 40% of this space, while PPO+RND covers about 65%. In absolute terms, PPO+RND visits roughly 332 more cells, which is substantial difference given the size of the maze. These results again highlight that SGCRL avoids exploring dimensions of the state space that are irrelevant the goal, in contrast to PPO+RND.
D.7 SGCRL IS CAPABLE OF REACHING MULTIPLE GOALS
To demonstrate that the SGCRL exploration mechanism extends beyond single-goal settings, adapt the policy to handle multiple goals simultaneously. Specifically, we extend the SGCRL policy from targeting a single goal optimizing over distribution of goals, weighted by their relative desirability. To illustrate this, we present empirical results in the point-maze environment using tabular setting without neural networks (the same experimental setup as Section 5). These results show that the single goall exploration framework naturally generalizes to more complex multi-goal tasks through algorithmic modifications, without requiring neural network architectures.
Formally, consider a task defined by set of goals
G {6 ..l..g.....99... IK),
Here, achieving goal 9i yields reward r.. We modify the action-selection mechanism so that the policy chooses action a at state 8, according to the distribution Softmax (1b(p(s;, a,) Veomb) where the combined goal embedding is defined as
Σ rv (())) EE TV (g.))12
Veomb
The transition model Per+: p(s a:) represents the environment dynamics. our setting, this model is provided to the agent; however, in practical applications it could also be learned. Our purpose here is not to propose the most practical solution, but rather to fllustrate-within a simple, tabular model-the the computational capabilities of the algorithm. This formulation enables the agent to act according to weraardweiigted combination of goals, rather than committing to a single target.
We evaluate this algorithm in the four-room environment by selecting two goals located at the top-right sub-room, indicated by stars in Figure 14. We assign equal weights r1 r2 = 0.5. All representations initialized with common component x e(s). where e(s) is depeedennnd Gaussian noise. We visualize i--simlarrtty, ie., (ss) eeomh as a heatmap throughout training in Figure 14. Similar to the single-goal setting, observe that while all representations initially resemble each other (and align
33

---

Published as a conference paper at ICLR 2026
Episode 200
Episode 600
Episode 6600
Start
6.75
Gool
4.75
0.6
6.50
4.50
0.4
4.75
6.5
e-sim simmllantyy
0.2
4.50
2.00
-02
0.0
-0.5
(a) Initially, di-imillarty is high in
(b) d-similarity in explored states
(c) eisimillryty is high only in the
most states.
reduces.
goals and nearby states.
Figure Representation evolution the multi-goal task
Goal reach rates
Average reach rate
1.0
Goal
0.8
Goal
0.6
Both
0.4
0.2
0.0
0
1000
2000
3000
4000
5000
Trial
Figure 15: Multi-goal SGCRL maintains a high per-episode reach rate for both goals. Solid lines are agent trajectories
with the combined representation), the agent explores the environment, they gradually diverge from the common component (reflected by lower u-similarity values). Eventually, the vi-similarity values stabilize, remaining high in the vicinity of the two goals and low throughout the rest of the environment.
We also plot the average success rate of reaching goal 1, goal 2, and both goals per episode in Figure 15,, where the shaded region denotes the standard error across 6 random seeds. The results show that the algorithm manages to reach both goals with nearly equal frequency, and in many episodes it successfully visits both by alternating between them. This experiment demonstrates that SGCRL is capable of handling multi-goal tasks and more complex behaviors, in addition to the simpler single-goal setting. It is worth noting that in the absence of two goals, when only the right hand side goal is chosen, the agent always takes a different path that doesn't 't pass through the left hand side goal; therefore visiting the left hand side goal in this experiment is not accidental.
D.8 ABLATING THE REPRESENTATION DIMENSION
We investigate whether changing the representation dimension affects the orthogonality phenomenon predicted by Theorem Although the theorem assumes a high-dimense dith-iimensional representation space, we assess what representation dimension is sufficient in practice for this orthogonality to emerge. Specifically, we analyze the learned representations a tabular FourRooms environment when the agent explores but fails to find the goal. This scenario is equivalent to selecting random goal embedding that does not correspond to any actual state the environment, as the experimental setup of Figure 4a. We repeat this analysis for range of representation dimensions.
Our results show that representation dimension of 4 (or any dimension larger than 4) sufficient for the representations to become orthogonal when the goal is not found (see Figure 16). However, when we reduce the representation dimension to 2, the orthogonality phenomenon no longer appears (In Figure 16c, the representations lie in 2D plane and don't 't move away from the goal representation
34

---

Published as a conference paper at ICLR 2026
FourRoom10x100 Dim=2
FourRoom10x10, D6m=16
FourRoom10x10 Dim=4
selt Ko Aight Gow
siimillaitt
0.05
up-imilarity
0.8
y-similantty
-.05
-0.05
0.10
-0.10
(a) 16mension=16
(b) Dimension=4
(c) Dimension=2
Figure 16: Any representation dimension of four or higher is sufficient to form orthogonal representa- tions in the absence of the goal.
convergence. As result, the agent collapses into local minimum and repeatedly visits states near its starting point, since the non-orthogonal representations cannot separate unexplored regions.
D.9 ABLATING THE REPRESENTATION NORMALIZATION
To investigate the role of normalization in representation formation and in the overall success of the algorithm, we ablate representation normalization the tabular FourRooms environment.
Theorem 2 suggests that normalization plays crucial role in the exploration dynamics: because the representation norm is constrained, any component (b(s) that was previously parallel to <b(g) (the "common" component) is forced into the orthogonal subspace. Without normalization, however, the representations can continue to encode useful information without suppressing this parallel component.
We evaluate this hypothesis by removing representation normalization in the tabular setting. We find that normalization indeed essential: in the 10 x 10 FourRooms environment with an episode length of 30, the non-normalized representations achieve zero success. It is worth noting that with longer episode length of 100, the normaazal variant can eventually solve the task, but we report the more challenging 30-step setting to evaluate performance under stress conditions.
Tabular FourRoom
Non-normalized
Success Rate
0.8
Normalized
0.4
0.2
0.0
2000
4000
6000
8000
Episode
Figure 17: In the tabular experiment, representation normalization plays very important role in success.
D.10 ABLATING THE CRITIC LOSS
We now consider the case where the critic learned via temporal-difference (TD) learning. In particular, suppose the discounted successor measure admits the low-rank form
(5 8, a, = Φ(8, a) (sy))
To update the representations using TD, minimize the squared TD error
c3p = (p(s,a) ((())))) - [11 - ->)1(s =5 + 7 ad) u(tr)))?
(23)
35

---

Published as a conference paper at ICLR 2026
Episode 500
Episode 39500
Start Goal
0.99
0.990
0,880
P-S miimi Iry
0,960
0,344
0.570
0.965
0.920
0.60
(a) Early training
(b) Late training
Figure 18: TDystyle critic produces the same qualitative pattern CRL: the si-imillartt of visited states decreases over training. However, despite this representational effect, the agent consistently fails to reach the goal.
where sy is sampled at a geometrically distributed time offset from t, (s', a') is the next transition in the replay buffer, and a' |s)).
Let the TD error be
8 = ((s (1s,))((ss))))))))) [[1 [11-1]11) >)1[s = - - oo [(sr))
The gradient-descent updates are then
(s, a) ( - 2n ( (
(st)
- 2n (b(( 8,
followed by normalization of the representations. This TD-based procedure can be viewed as alternative to the contrastive CRL objective for learning a low-rank approximation of the successor representation.
We implemented this TD-leaming variant in the tabular FourRooms environment. While the learned representations exhibit qualitatively similar structure those obtained with the contrastive method-statee states visited frequently during exploration acquire lower u-similaritt, the agent consis- tently fails reach the goal across five random seeds, refer Figure 18.
D.11 ANALYZING THE GOAL SIMILARITY IN THE CONTINUOUS SETTING
In this section, examine whether the orthogonality result of Theorem 2 continues to hold in the continuous setting. The theorem assumes that the representations of non-goal states are updated while (g)) remains fixed, modeling the situation where the agent explores regions of the state space without encountering the goal. This assumption exact in the tabular case, where updates are local. However, in practice, when representations are produced by shared neural encoder, updating states far from the goal can still modify (gg) indirectly.
To test whether the orthogonality effect persists despite this coupling, we SGCRL in the continuous FourRooms environment and choose a goal embedding corresponding to an out-of- coordinate, g (20, 20), ensuring that the agent never observes the true goal state. We then track the evolution of goal similarity throughout training across multiple seeds. See Figure 19, Consistently, we observe the same pattern in the tabular setting: representations initially have high vi-imilarity to the goal but gradually decay toward values near zero, reflecting an approximate orthogonalization. Although the similarity does not reach exactly zero, becomes close, indicating that the theoretical prediction remains good approximation even with shared encoder.
D.12 SAFETY BASELINE EXPERIMENTS
To better contextualize our safety experiments, we evaluate a few baseline safe RL algorithms from the Safe Policy Optimization algorithm benchmark (Ray al., 2019) the FourRooms safety
36

---

Published as a conference paper at ICLR 2026
Checkpoist
similarity Checkpoint
Similarity Checkpoint
(a) Early training
(b) Mid training.
(c) Late training
Figure 19: Even with shared neural encoder, updating representations in regions far from the goal drives the representations visited states become approximately orthogonal the goal embedding. Starting point is at the top left corner and the impossible goal coordinate is (20, 20).
0.6
Visitation Rate
0.3
0.1
100
200
300
5ón
600
Trials
PPO-Lag (multipliar 0.11
CPO
PPO-Lag (multiplier 0.01)
PPO
PPO-Lag (multiplier 1.01
Figure 20: Comparison of safety violation rate for baseline safe RI. algorithms
environment. In this environment, the states the bottom left room are considered "unsafe". We evaluate the following baselines:
PPO:
Proximal Policy Optimization (PPO) (Schulman et al., 2017) serves uncon- strained baseline.
PPO-Lagrangian (Ray et al., 2019) uses an adaptive penalty coefficient on the PPO objective to enforce safety constraints. This method equivalently solves the unconstrained max-min optimization problem
PPO-LAG:
max min L(0, A) = Ag(0), 0
(24)
X20
where f(0) is the objective and g() < 0 is the constraint. We set the cost-limit hyperpa- rameter 1.0 (no tolerance safety violations). We tune over values for the initial penalty coefficient.
CPO: Constrained Policy Optimization (Achiam et al., 2017) enforces constraints by solving trust region optimization problems analytically at each policy update. We set the cost-limit hyperparameter to 1.0 (no tolerance to safety violations).
Figure 20 shows the safety violation rate of these baselines. We do not plot SGCRL in this comparison because standard SGCRL converges at a much slower rate than these lightweight, online baselines (Fig. 5c).
37

---

Published as a conference paper at ICLR 2026
Doint_FourRoomy
1.0
(Single Hard Goal)
Success
0.6
0.4
Actor Steps
couffident
0.001
0.01
0.05
0.1
Figure 21: Role of the logSumExp term in the success rate when using the InfoNCE forward loss.
D.13 ABLATING FORWARD VS BACKWARD INFONCE LOSS AND THE ROLE OF THE LOGSUMEXP TERM
While the main experiments use the InfoNCE backward loss to train the critic (Eq. 1), in this section we include an ablation in which we also evaluate the InfoNCE forward loss (Equation 25), which also contains additional log-sum-exp term.
N
a:) ((s a log Ej=1 exp(o(si, an) 0(8
exp (ss (o)
max E (-1aaa)) N
lo
i= N
(25)
Our goal is to analyze the role of this log-sum-exp factor in shaping the learned representations and determining the agent's success rate. We sweep the forward-loss temperature parameter using a e 0.001, 0.01, 0.05, 0.1 in the tabular four-room environment and observe meaningful differences in representation dynamics. particular, the evolution of sb similarity during exploration-vv when the agent visits states far from the exhb-ixhibit the same orthogonality pattern as with the backward loss.
In the continuous setting, however, varying a does affect the agent's rate of convergence. We include these results in Figure 21. The representation patterns appear similar across difference values of a (Figure 22).
Figure 22: Representations during training for different logSumExp coefficient values
D.14 ADDITIONAL EXPERIMENTS
IMPORTANCE SINGLE-GOAL COLLECTION
To further test the hypothesis that exploration with single hard-goal representation is responsible for the orthogonality effect described above, we conduct an additional experiment that mirrors the setup of Appendix D.5. The only change is in the dala-collection policy: instead of comparing single- goal exploration to a completely random- -oaa policy, we now compare a family multi goal exploration strategies that interpolate between these extremes.
38

---

Published as a conference paper at ICLR 2026
Trial 50000
Trial 50000
Trial 50000
(a) 0.5
(b) 2
(c) 20
Figure 23: As the data-collection goal representation deviates further from (0)) (larger σ), the learned representations are less strongly pushed toward the orthogonal subspace.
Concretely, in each episode we sample a goal embedding
Ye = (g)g) + oe,
N~, Ta),
and use Ye in place of (gg) in the multi-goal exploration policy. The scalar a controls how tightly the sampled goals concentrated around the hard goal representation (g)): small values of o correspond to goals tightly clustered around (g)) (approaching the single- soalle-gaalll regime), while large values of recover widely spread goal distribution similar to the random-goal setting. We study the evolution of the representations for a E {0.5,2 20).
Our expectation is that, when is small, the induced data collection remains close single-gon exploration, leading to "clean" representation formation in which all frequently visited, non-goal regions are driven far from (g(g) and thus enjoy the same orthogonality effect. As increases, the data collection becomes less aligned with the hard goal, and we anticipate that this degrades representation quality: many visited states retain high similarity to ('()), preventing the agent from escaping already explored regions and ultimately reducing exploratory coverage. The representation evolution under these varying data-collection strategies summarized in Figure 23.
Multiple reachable goal data collection While the previous experiment studied representation formation under an unreachable goal, we now consider setting in which the goal is reachable and compare single-goal data collection with multi-goal data collection using two and four goals.
In the two-goal setup, the exploration goal is sampled uniformly at random from {gi (9, 9), .2 (6, })} at the beginning of each episode; And in the four-goal setup, the exploration goal is sampled uniformly from 91. 92 or Ss (0, 9), 94 (9, 0). In all cases, the evaluation goal used to measure success is fixed to the hardest goal. (9, 9).
Figure 24 shows the resulting success rates. The results clearly indicate that exploration with single goal leads to more effective representations and higher goal-reaching performance than distributing exploration across multiple goals. This finding is consistent with the results of Liu al. (2025).
Beyond success rates we also visualize the u-similarity across the three data-collection regimes do not expect the
and the different goals. Note that in this realm since the goal is reachable, representations to become fully orthogonal to the goal because Theorem 2 only predicts exact orthogonality when the goal is unreachable. But we expect states that are far from the goal should exhibit grow low u--imilaarity over time, while states closer to the goal especially those along typical paths to the goal should retain high similarity.
With this intuition in mind, we investigate two key questions:
RQI
When visualizing (s)) (g.) for each of the four goals 9:, i 2,, 4). do we observe the coherent structure in which states far from each goal exhibit low similarity, while states near the goal and along the path it exhibit high similarity for all 4 goals the same time?
39

---

Published as a conference paper at ICLR 2026
Multi-Goal Comparison FourRoom
1.0
Goals
Success Rate
0.8
Goal (Baseline) mw
0.6
0.4
0.2
0.0
2000
4000
6000
8000
10000
Episode
Figure 24: Success rate is the highest when we only collect data with the single hard goal. Tabular experiment in the FourRoom environment.
RQ2 When focusing solely on the hardest goal 91 (9,9), do the heatmaps of ()(s) (91) differ depending on whether the data was collected using only 91, or using two four exploration goals? Specifically, are the representations learned through single-goal data collection superior?
We first address RQ1. The similarity maps (s) (g) for the four different goals in the four four-goal data-collection experiment shown in Figure 25. While do observe the general trend that states far from each goal have low similarity and states closer to the goal have higher similarity for all four goals, the structure is noticeably less precise than when data is collected using single goal.
For instance, Figure 26a shows the representation when data collected using only one goal, where observe clean and smooth gradient of similarity toward the goal. In contrast, in the four-goal setting (Figure 25a), the gradients are much less structured, and the model does not clearly distinguish between (1 and 92-
An even clearer example is shown in Figure 25d, where almost the entire left side of the maze exhibits high similarity with J1. even though many of these states are actually far from 94- This indicates that when exploration is done with multiple goals, the learned representations become less sharply aligned with the true spatial structure the environment.
We now address RQ2. Refer to Figure 26 for comparison of (s) (b(g1) across different data- collection strategies. We again see that, for multi-gos, data collection strategies, although overall states far from the goal have low similarity, there are still many states at distance 3-4 from the goal that exhibit high similarity and those areas are where the agent gets stuck.
We hypothesize that this happens because the path from the starting point to 91 (9, (9,9) and G2 = (6, 6) is partially shared. Since the dataset contains trajectories that follow the same path but sometimes terminate gI and sometimes at g2, the representation of g2 becomes similar to that of zi- As a result, the agent can get "stuck" at J2 even when it commanded to reach 91. the single-gon goal data-collection setting, this failure mode is avoided: the agent is always commanded go to the same goal, so irrelevant states such as 92 do not acquire gi -like representations that would cause the policy stall there.
D.15 ADDITIONAL SAFETY EXPERIMENT RESULTS
Here, we provide additional plots to demonstrate that the safety intervention experiments (Fig. 5c) hold when the intervention is conducted in either the top-right or bottom-left room of the FourRooms environment.
E THE USE OF LARGE LANGUAGE MODELS
We used LLMs while implementing experiments to generate boilerplate code, debug errors, and plot results. We wrote the paper manuscript manually but used LLMs to help edit writing for clarity and grammar.
40

---

Published as a conference paper at ICLR 2026
Episode 10000 Goal 1
Gool
Start
Episode 10000 Goal
Goa
Start
Gual 2
Gow
Gosl
0.800
0.800
0.600
0.600
-0.400
sim Aimilaritt
0.400
0.200
0.200
0.000
-0.200
-0.400
-0.400
00600
-0.600
(a) (s)s), (())))
(b) (v(s), (o2))
Episode 10000 Goal 3
Episode 10000 Goal
Goa
Stert
Gow
0.800
0.750
Goa
0.600
0.500
0.400
y w.similani
0.250
@-Similamii Similantty
0.200
0.000
-0.000
-0.250
00.200
-0.500
-0,000
-0.750
-0.600
(c) (Ma),, ((a)))
(d) (Wb(s, (b()))
Figure 25: Data collection with goals. Heatmaps shows the goal similarities with respect to each of the four goals. The trajectory over episodes is shown, with line colors indicating the commanded goal.
Episode 10000 Goal
Sut
Episode 10000 Goal
Sun Ser12
Episode 10000 Goal
Se11
0.750
50514
4.500
6.250
0.200
6.000
4.250
4,250
4.20
..300
4.400
4,750
0,600
(a) single-goal data collection
(b) 2-goals data collection
4-goals data collection
Figure 26: Comparison of representations with respect to 91 under single-goal, two-goal, and four- goal data collection.
mt
1.00
Giaal
Figure 27: Modifying the representations be dissimilar to the goal improves controllability of the agent's 's behavior.
41


Reviews:
[{"review_id": "xtYlnRIhBj", "reviewer": "Reviewer_8Y6w", "summary": "The paper provides a mechanistic analysis of SGCRL [1], aiming to explain its “emergent exploration.” It formalizes the actor’s behavior as optimizing a shaped return based on similarity metrics, making explicit the intuition that policies are guided by goal-similarity in the learned representation rather than raw state-space distance. Building on this, it analyzes how InfoNCE training shapes the representation such that explored regions with low similarity to the goal are effectively “pruned”. The authors validate these predictions via controlled representation interventions and via a tabular reconstruction to show the exploratory behavior does not come from the use of function approximators. Experiments in grid worlds (e.g., Four Rooms) and toy tasks (e.g., Tower of Hanoi) empirically match the theory.\n\n[1] Liu. et. al. A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals 2025", "questions": "1. Please explicitly scope when ψ(g) can be treated as fixed (tabular lookup, imaginary goal, frozen goal head) and discuss how in practice, the predictions degrade when ψ(g) drifts under shared encoders.\n\n2. Can you provide multi-goal experiments with HER-style baselines (sampling goal from future states) to support the framing?\n\n3. The sentence “drawn to high ψ-similarity even when far from the goal…” is confusing. Please state explicitly that attraction/repulsion is governed by similarity in embedding space, not raw state-space distance.\n\n4. The role of the LogSumExp could potentially also be discussed in the represetation learning as it's so crucial to the contrastive RL. How does that impact the represetation/exploration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Clear mechanistic story linking the actor objective to an implicit ψ-similarity reward. Completing the intuition that SGCRL left as future work.\n\n2. The use of interventions (attract/repel via embedding edits) and tabular models gives a good ablation study and nicely visualizes the behaviors.\n\n3. Empirical trends in grid worlds align with the theory, making the presentation generally clean.", "weaknesses": "1. I'm concerned mostly about the scope mismatch: Framed as broadly “goal-conditioned RL,” but almost all analysis/evidence is for one algorithm (SGCRL) and mostly single-goal settings; multi-goal evidence is limited/toy and lacks HER-style goal-conditioned algorithms.\n\n2. One of the main contributions, Theorem 1 helps with the formalization, but conceptually modest. It largely reframes the intuition as a theorem, but only under the alignment assumption.\n\n3. Assumption of fixed ψ(g) limits the evidence strength. It does not hold using neural encoders (ψ(g) will drift). The paper shows robustness empirically, but the theoretical scope could be clearer.\n\n4. Cognitive-science in the conclusion part is questionable. It reads as methodological borrowing (rational analysis, interventions, small model), not really a real \"cognitive\" experiment. I suggest this should be toned down or made more precise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uwS4ZQrM0O", "reviewer": "Reviewer_GW6a", "summary": "This work analyses the origins of implicit exploration behaviors observed in Single-Goal Contrastive Reinforcement Learning (SGCRL), a model that actively aligns state-action representations with the representations of a single goal. The paper presents a theoretical and experimental analysis of the dynamics of state and goal representations learned by the model. A long list of experiments characterizes the desiderata to make SGCRL induce a form of decaying optimism that boost exploration.\n\nThe paper is well-written and the work provides a sound and novel analysis that helps understand the emergence of implicit exploration behaviors in a RL setting. Thus, I recommend acceptance. \n\n\nI'd be happy to further increase my score:\nSection 4.2 kicks in an open door. Given the infoNCE loss function, it is not surprising that the similarity between phi(g) and phi(s) rules the behavior. The paper decides to focus on Single-Goal RL and shows in Appendix that uniform goal sampling does not work. The experimental section could provide an analysis of how the number of fixed goals, along with their goal-conditioned policy, affect the representations and the exploration behaviors. Single-Goal RL is a limited setting, extending the analysis may bring more general results or significant insights for future work.\n\n\nMinor comments: \n- Figure 5a), please, bring the star forward.\n- The benefits of intervening on the representations rather than with a direct reward is unclear.", "questions": "Please, see above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and the work provides a sound and novel analysis that helps understand the emergence of implicit exploration behaviors in a RL setting.", "weaknesses": "The analysis focuses on a constrained setting, Single-Goal contrastive learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0iF26BgK8J", "reviewer": "Reviewer_2Td2", "summary": "This paper aims to demystify the mechanisms behind emergent exploration in Single-Goal Contrastive Reinforcement Learning (SGCRL). The authors combine theoretical analysis and controlled experiments both in continuous and tabular settings to argue that SGCRL implicitly maximizes a representation-based reward defined by goal similarity. This implicit reward gives rise to an automatic exploration–exploitation curriculum: states along unsuccessful trajectories gradually lose goal similarity (discouraging revisits), while those along successful paths gain similarity (encouraging exploitation). The authors show that these dynamics persist even without neural network function approximation, suggesting they arise from the low-rank structure of contrastive representations rather than network inductive biases. They further propose that understanding these dynamics enables safety-aware exploration by manipulating representations.", "questions": "1. How does the proposed “implicit reward” interpretation differ formally from the successor feature formulation of value functions? \n        \n2. Would the same dynamics appear if the critic used a non-contrastive predictive loss (e.g., temporal difference learning or policy gradient)? One could argue that with the policy gradient once the agent is near convergence a similar patterns appears, i.e., that the described phenomenon in the paper might not be exclusive to the contrastive approach\n        \n3. How sensitive are the observed behaviours to representation normalisation or embedding dimensionality?  I would assume normalisation/compression plays a big role in shaping the geometry of the reward function.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper provides a clear and well-structured theoretical account of why SGCRL exhibits emergent exploration despite lacking explicit rewards.\n* I think the use of tabular environments to isolate the contrastive representation mechanism is elegant and persuasive.\n* The paper is well written, and the figures effectively communicate the representation dynamics.\n- The demonstration that altering representation geometry can control exploration pathways is an intriguing direction for safe RL.", "weaknesses": "- I feel the the proposed interpretation, exploration as maximisation of a representation-based implicit reward, is closely aligned with existing frameworks such as successor representations, potential-based reward shaping, and eigenoptions (e.g. (Machado et al)). The connection to these earlier ideas is not fully acknowledged, giving the impression of a new theory rather than a reinterpretation.\n- The framing suggests spontaneous behaviour arising from complex dynamics, but the mechanism can be straightforwardly understood as representation-induced reward shaping.\n- The intervention experiments are promising but not systematically evaluated or compared to existing safety-aware exploration methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)