Dual Goal Representations
Authors: Seohong Park
Deepinder Mann
Sergey Levine
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
goal-conditioned rl
offline rl
Primary area: reinforcement learning

Abstract: In this work, we introduce dual goal representations for goal-conditioned reinforcement learning (GCRL). A dual goal representation characterizes a state by "the set of temporal distances from all other states"; in other words, it encodes a state through its relations to every other state, measured by temporal distance. This representation provides several appealing theoretical properties. First, it depends only on the intrinsic dynamics of the environment and is invariant to the original state representation. Second, it contains provably sufficient information to recover an optimal goal-reaching policy, while being able to filter out exogenous noise. Based on this concept, we develop a practical goal representation learning method that can be combined with any existing GCRL algorithm. Through diverse experiments on the OGBench task suite, we empirically show that dual goal representations consistently improve offline goal-reaching performance across 20 state- and pixel-based tasks.

Published as a conference paper at ICLR 2026
DUAL GOAL REPRESENTATIONS
Seohong Park"1 Deepinder Mann *1 'University of California, Berkeley (seohong, dmann Sberkeee.. edu
Sergey Levine
ABSTRACT
In this work, introduce dual goal representations for goal-conditioned rein- forcement learning (GCRL). A dual goal representation characterizes a state by "the set of temporal distances from all other states"; in other words, it encodes a state through its relations to every other state, measured by temporal distance. This representation provides several appealing thcoretical properties. First, it de- pends only on the intrinsic dynamics of the environment and is invariant to the original state representation. Second, it contains provably sufficient information recover an optimal goal-reaching policy, while being able to filter out exogenous noise. Based on this concept, we develop a practical goal representation learning method that can be combined with any existing GCRL algorithm. Through di- verse experiments on the OGBench task suite, we empirically show that dual goal representations consistently improve offline goal-reaching performance across 20 state- and pixel-based tasks.
Blog post: https: // seohong me /tlplolo/aaallrrppoostttttttttta
Original Representation
Dual Representation
Dualize
original (g) =
* (g) (s) d' (s):))]
Figure 1: Dual goal representations. A dual goal representation (g) is defined as the set of temporal distances d' (-,9 g) from all other states. This representation has number of appealing properties: it only depends on intrinsic dynamics of the environment, contains sufficient information to express an optimal goal-reaching policy, and able to discard exogenous noise. In continuous environments, we approximate the dual representation sp (g((g) by the goal emhedding (gg) in a parameterized temporal distance function, d' (s.g) = (s)) v(g).
INTRODUCTION
Representation learning lies the heart of modern machine learning. The successes of large-scale unsupervised pre-training (Brown et al., 2020; Ramesh et al., 2022) have shown that rich represen- tations can enable both efficient adaptation and strong generalization across tasks. In this work, we study representation learning for an important class of reinforcement learning (RL) problems, goal- conditioned reinforcement learning (GCRL) (Kaelbling, 1993; Park et al., 2025a). The objective of GCRL is to train a multi-task policy capable of reaching any target state from any starting state in minimal time. Many real-world tasks, including navigation, object manipulation, and even games, naturally fit into this goal-reaching framework.
important question in GCRL how to represent goals. While many works simply use original state observations provided by the environment to represent goal states (Kaelbling, 1993;
Equull contribution.

---

Published as a conference paper at ICLR 2026
Park et al., 2025a), these observations not necessarily best structured to describe goals (i.e., tasks), and often contain superfluous information, such as uncontrollable noise. On the other hand, good, well-designed goal representation accelerate policy learning during training, improve goal-reaching performance at test time, and even potentially enable robust generalization to out-of- distribution goals.
To achieve these benefits, we define two desiderata for an ideal goal representation. First, it must be able to capture sufficient information represent an optimal goal-conditioned policy (sufficiency). Second, it must be able to discard irrelevant information, such as background noise, that does not affect goal-reaching (noise invariance).
Our key idea to achieve these desiderata is to represent goal as the set of temporal distances from all other states. More specifically, we represent goal a functional (i.e., a function from the state space to IR) that takes a state as input and outputs the corresponding temporal distance. We call this a dual goal representation. We theoretically show that this dual goal representation satisfies both desiderata (sufficiency and noise invariance): i.e., we can represent an optimal policy solely based on the dual representation, and the representation is invariant to exogenous noise.
Building on this theoretical insight, we propose a new practical goal representation learning recipe for offline goal-conditioned RI.. Our method first trains a parameterized goal-conditioned value function with an offline value learning algorithm to compute temporal distances. Then, take the goal embedding of the learned value function as an approximate representation of the functional, which is then used goal representation for the policy. We demonstrate that this goal representa- tion learning recipe consistently improves the performance and generalizability of goal-conditioned policies, across diverse robotic locomotion and manipulation environments.
Contributions. Our main contributions are the theoretical formulation and empirical verification of dual goal representations. First, we theoretically show that dual goal representations learn sufficient and noise-invariant representation for optimal goal reaching. Second, we demonstrate that our empirical recipe based on this principle improves performance and generalization capabilities in practice, outperforming previous goal representation learning approaches on 20 diverse goal- reaching tasks in robotic locomotion and manipulation.
PRELIMINARIES
Controlled Markov processes and goal-conditioned RI. Let S be state space, A be an action space, and p(s' a,a) S x A + (S) be a transition dynamics kernel, where ( denotes the set of valid probability distributions over space X' and placeholder variables are denoted in gray. For theoretical results, we assume that S and A are discrete spaces. This is only for the sake of simplicity in discussion, and our results can be extended to continuous state spaces with appropriate modifications.
A controlled Markov process (CMP) is defined by the tuple M = pS. Given goal- conditioned policy Tia 9,9) S x S - A(A), and a goal-conditioned reward function Eyrpp Ei 2 (xt.))))) and Q (s)
r(5,9) S RR, define V V*( Ey (7/0)=8.0==a, D 7'r(st, g), where y E (0, denotes discount factor and p" de- notes the trajectory distribution induced by the policy and transition dynamics. Unless otherwise mentioned, we define r(s.g) I[(s 9). where I is the 0-1 indicator function, and assume that the agent enters absorbing state once it reaches the goal (i.e., it can get reward of at most once). We define the optimal value functions as V*'(x = max,.(x8--((A) V* g) and Q" a,
max QT a, g). We also define the temporal distance function as d' (s, .) = log, V*(s,g). (). Note that, in deterministic environments, d' (s,.g) corresponds to the length of shortest path from 8 to
The objective of offline goal-conditioned RL is to find goal-conditioned policy æ that maximizes (= for all 8,9 E S. from dataset D [ J consisting of state-action trajectories T T= (So, an, $1, $1,..., ST) with no environment interactions.
Block CMPs. Some theoretical results in this work are built upon the block CMP (BCMP) frame- work (Du et al., 2019; Efroni et al., 2022; Lamb et al., 2022). A BCMP is defined by a tuple (S, Z, A. p. pf, pd)) where S is an observation space, Z is latent state space, A is action space,
2

---

Published as a conference paper at ICLR 2026
p(2' 22 A 2 is latent transition dynamics kernel, (s
A(S) is observa- tion emission distribution, and (s() S + 2 is a latent mapping function, such that p (s)s = 2 for all E Z and (' 2). Intuitively, BCMP can be thought of as latent CMP (2, p) aug- mented with noisy observations, such that the observation distributions from different latent states always have disjoint supports (i.e., every observation maps unique latent state).
In a BCMP, we define a goal-conditioned reward function as r'(s.)) g) = (sp' = =pp p')g)) Note that this only depends on the latent states. Given goal-conditioned policy, <(a 9, S x S ((A), we define the value and Q functions and temporal distance function similarly as before, but with the modified reward function r². The BCMP framework encompasses many real-world scenarios (Efroni et al., 2022; Lamb et al., 2022), including image-based robotic control (where latent states correspond to the ground-truth poses of the robot arm and objects, and the emission function corresponds to noisy yet fully observable rendering of the current state).
3 DUAL GOAL REPRESENTATIONS
The main aim of this work is find an effective goal representation ((()) : S - W for a goal- conditioned policy π (a | 8, ((()) parameterized by y,, where W denotes a latent representation space. good goal representation can offer several benefits. During training, it can reduce the learning complexity of the policy network by providing structured information about the goal state. During evaluation, it can help the agent generalize to unseen, potentially out-of of-iiiiiiiittionnn goals.
What constitutes an ideal goal representation? We define two desiderata. First, it must be able to capture enough information about the state to represent an optimal policy (sufficiency). Second, must be able to discard information irrelevant to goal-reaching (noise invariance). In other words, an ideal goal representation should be able to maintain only a necessary and sufficient amount of information to perform optimal goal-reaching. This will make the agent focus only the control- lable part of the state, enabling faster policy learning and better generalization.
3.1 THE IDEA
Our main idea to achieve these desiderata is to represent goal the set of the temporal dis- tances from all other states. For example, in discrete state space S = {81. $2..,,...,,5 this corresponds to the vector consisting of all temporal distances to that goal; i.e., (0g) (d' (81.g) g), d (#2, (82.9),... g)...., .d' d'(sx, (*x:9)] g)] In the general case, this representation corresponds to a func- tion from S to R (such function is called a functional).
8k
Formally, for a CMP M = (S,A,p) p), we define vp S + (S IR)
(g) = (s d" (s,
for 9.9 E S. In other words, ( (s) d' (s (s.g). g). We call yor the dual goal representation (or simply dual representation) function of the CMP M.
Intuitively, dual goal representations replace a goal state with its relations to all other states, mea- sured by temporal distance. This has several benefits. First, the resulting representation is imvari- anf to the original state representation given by the environment, as it depends only on the intrinsic temporal dynamics of the environment. Second, it can be shown that this representation retains suf- ficient information to represent an optimal goal-reaching policy, while being able to discard exoge- nous noise, as we formalize in the following section.
3.2 THEORETICAL PROPERTIES
We now state two important theoretical properties of dual goal representations. The first property is sufficiency:
Theorem 3.1 (Sufficiency of Dual Goal Representations), Let M = (S, A, be a CMP and & be its dual goal representation function. Then, there exists a deterministic policy TV :Sx x SV + A that takes dual goal representation input, such that its induced policy के S x S + A defined as 7 (s. = n (s, 4p (g)) satisfies V* (*(s)) = (* g) for all s, E S.
The proof can be found in Section A. Intuitively, the above theorem states that the dual goal repre- sentation function &" retains enough information so that one can recover an optimal policy that de- pends only on dual goal representations. The second property is noise invariance:
3

---

Published as a conference paper at ICLR 2026
Digressive remark: Connections to other concepts in mathematics and machine learning
The dual goal representation based on the philosophy of "relative point of view" (Ze., un- derstanding object via its relation to all other objects). This is analogous to various concepts mathematics and machine learning, and we highlight a few examples. In category theory, the identify an object through its morphisms to all other objects
Yoneda lemma implies that in the same category (Riehl, 2017). In kernel machines and functional analysis, the Riesz rep- resentation theorem implies that can uniquely represent point in Hilbert space as linear functional in its dual space (Folland, 1999; Wainwright, 2019). The dual goal representation is named after this analogy.
Theorem 3.2 (Noise Invariance of Dual Goal Representations), Let M M= (S, p.p.", .p' be BCMP and sp be its dual goal representation function. Let , E S be two goal observations from the same latent state; i.e. p¹ (g1))= = r (g2). Then, they have the same dual goal representation; i.e. (91)) = (g)
Again, the proof can be found in Section A. Intuitively, this theorem states that, under some technical assumptions (based on the BCMP framework described in Section 2), dual goal representations are invariant under exogenous noise in the observation space. Combining Theorems 3.1 and 3.2, we conclude that dual goal representations satisfy both desiderata described at the beginning of Section 3.
PRACTICAL INSTANTIATION
There are two challenges with implementing a practical learning algorithm for dual goal represen- tations. First, the functional form of dual goal representations cannot be directly implemented in practice, unless the state space is small and discrete (in which case we can represent function as fmite-dimensional vector). Second, we need know the temporal distance function d".
Approximating functionals. We address the first challenge by employing parameterized temporal distance function. Namely, we model the temporal distance function follows:
d' (s. f(v((s,, ((g)).
where (s) + R and p(g) f TRN x RRN + R is a (simple) aggregation function that combines the information from both embeddings to compute the temporal distance. Then, we treat the output of the goal embedding, (gg), as practical finiteddimensional representation of the functional dual goal representation, sp" (g). Intuitively, this makes capture temporal information about the goal, in the sense that we compute the temporal distance by pairing with any other state via (+)(ss p(g)).
R.V are respectively the state and goal embeddings, and
For the aggregation function f. we use the following inner product parameterization:
((b(s), p())) = (x)) ((())
(2)
We note that this inner product form is universal (Park al., 2024b), meaning that it is expressive enough to approximate any two-variable function (on compact set) up to an arbitrary accuracy.
Approximating temporal distance. Next, we approximate the temporal distance function d' in practice, using an existing offline goal-conditioned value learning algorithm. In particular, we em- ploy goal-conditioned IQL (Kostrikov al., 2022; Park et al., 2025a) to approximate d' (to be pre- cise, transformed variant of d' via V*, be instantiated with any other goal-conditioned value learning (or temporal distance learning) algo- rithm, we find that goal-conditioned IQL generally leads to the best performance while remaining simple.
explain below). While dual goal representations can
Specifically, goal-conditioned IQL trains a parameterized goal-conditioned value function V(s. f()(s), (())): S x S + IR and a goal-condittoned Q function Q(+, ((s,a,g) : S Ax with the following losses (Kostrikov al., 2022):
L(,h u) = E. [ ((((())) p(g)) ((s,
(3)
L(Q) = Es,,,'' anD (Q(s,,, r(8.9) (...)))(((((()))
(4)
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 Offline Goal-Conditioned RL with Dual Goal Representations
b Dual goal representation learning
Initialize state representation v)()), goal representation ((g), Q function Q(+, (,9)
while not converged do
Sample batch {(s, a, s', ())))) from D
D Train parameterized value function f())(s), ((g)) with goal-conditioned IQL.
Train 1h, yp by minimizing R/C.. (f(()))) ((g)) Q(s (.2.,,
Train Q by minimizing [[(((s, a. g. r(s.g) --(((((())))))))))))))))
Update Q using exponential moving averaging
Downstream offline GCRL with dual goal representation (can be in parallel with above) Initialize policy xaa s, (9))
(If necessary) initialize representatiion-comiiiinone value functions V.CRL (8, (g))) CCCRL (s,a, ((())
while not converged do
Train m, GGRR, QCCRI with any offline GCRL algorithm (e.g., GCBC. GCIVL, CRL)
return π [a s, sp())
where e denotes the expectile loss with expectile (Kostrikov et al., 2022) and Q denotes the target Q network (Mnih et al., 2013).
If the reward function is given as the 0-1 indicator function, r(s,g) = I(s = g), we have the relation (*(sgg) = (*.() (Section 2). Hence, when the IQI. losses converge, ((+(s), (())) =
(g) approximates this transformed temporal distance function, (*
7dd (+.0). In
practice, we use the - 1-0 reward function (i.e., r(s, g) = Il(s = g) - 1) instead of the 0-1 indicator function, following Park et al. (2025a). While this leads to slightly different relation (*(s,
- xf (s.s)) (), we found this version to work better in practice. Note that regardless of the choice of reward function, f(((s), ((g)) still approximates a (transformed) temporal distance function.
Downstream offfine goal-conditioned RL. After obtaining a dual goal representation yo we train a parameterized goal-conditioned policy T(a s. s.)()) (and optionally coal-conditioned value func- tions, VGCRL (x, p(g)) and QGCRL (s. o, (p())) with an existing offline GCRL algorithm to per- form downstream goal-conditioned policy learning. Note that the choice of this downstream GCRL algorithm is orthogonal to the one used to approximate the temporal distance in the previous para- graph, and thus our method can be applied to any general GCRL algorithm. In this work, we apply dual goal representations to three different offline GCRL algorithms, goal-conditioned IVL (GCIVL) (Kostrikov et al., 2022; Park et al., 2025a), contrastive RL (CRL) (Eysenbach al., 2022), and goal-conditioned flow BC (GCFBC) (Park et al., 2025b). We describe the full goal-conditioned RL procedure in Algorithm I, where e denotes the expectile loss with expectile k (Kostrikov al., 2022) and Q denotes the target Q network (Mnih et al., 2013).
5 RELATED WORK
Offline goal-conditioned RL. At the intersection of offline RL (Lange et al., 2012; Levine et al., 2020) and goal-conditioned RL (Kaelbling, 1993), offline goal-conditioned RL aims to train goal- reaching policy from an unlabeled (i.e., reward-free) dataset. Offline GCRL provides a principled way to pre-train policies, representations, and value functions in a self-supervised manner, which can later be adapted to downstream tasks (Ma et al., 2023; Ghosh et al., 2023; Kim et al., 2024; Park al., 2025a). Prior works in offline condiiondi RL have proposed diverse approaches based on implicit value learning (Park et al., 2023), contrastive learning (Fysenbach et al., 2022; Zheng et al., 2024), metric learning (Wang et al., 2023; Myers et al., 2024), and planning (Savinov et al., 2018; Eysenbach et al., 2019; Wang et al., 2024). In this work, we aim to improve the performance and generalizability of goal-reaching agents by learning an effective goal representation (as with prior goal representation learning methods discussed below), which can be orthogonally combined with any existing offline GCRL algorithm.
Representation learning for RL. Representation learning has long been studied in RL (Echchahed & Castro, 2025). By learning structured representations for states, actions, or other components in the given environment, prior work aims to facilitate learning and improve the generalization of RI.
5

---

Published as a conference paper at ICLR 2026
agents. To this end, variety of representation learning techniques (Li et al., 2006; Castro, 2019; Wu et al., 2019; Laskin et al., 2020; Schwarzer et al., 2021) have been proposed, and we refer to Echchahed & Castro (2025) for comprehensive literature review. Among representation learning algorithms, our practical method has a resemblance to the successor representations and successor features (Dayan, 1993; Barreto al., 2017; Touati & Ollivier, 2021; Touati al., 2023). Unlike these works, we approximate optimal temporal distances (or equivalendy optimal goal-conditioned values), which do not correspond to any successor features or representations with respect to fixed policy. This enables us to directly learn representation for optimal goal-reaching.
Representation learning for GCRL. Our work is most closely related to previous studies that pro- pose representation learning techniques based on conditiod RL (Sermanet et al., 2018; Ey- senbach et al., 2022; Steccanella Jonsson, 2022; Ma et al., 2023; Park et al., 2024a; Myers al,, 2025; Lawson et al., 2025). While the concept of dual goal representations and our theoretical results (Section 3) are, to our knowledge, novel, our practical learning algorithm for dual goal representa- tions (Section 4) bears similarity to several existing algorithms. Some previous works train temporal distance representations y using the metric parameterization [|<p(>) ((0)ll2 learning algorithms (Sermanet al., 2018; Steccanella & Jonsson, 2022; Ma al., 2023; Park et al., 2024a), which can potentially be viewed as variants of dual goal representations with a different ag- gregation function. Our method differs from these works in two ways. First, use yp for goal rep- resentations (as our theory in Section 3 suggests), whereas these works use şp for metric-based skill learning (Park al., 2024a), state representations (Ma al., 2023), reward shaping (Sermanet al., 2018; Steccanella & Jonsson, 2022; Ma et al., 2023), and planning (Sermanet et al., 2018; Stecca- nella & Jonsson, 2022; Ma et al., 2023; Park et al., 2024a). Second, unlike these works, we employ the more general inner product parameterization (instead of the metric one), which we show leads to better performance likely due its universality. Similar our work, TRA (Myers al., 2025) em- ploys the inner product parameterization to learn goal representation. However, this method trains the representation with behavioral temporal contrastive learning, which does not aim to approximate d"; in contrast, we train representation based on the optimal value function (as our theory suggests), and we empirically show that our method leads to substantially better performance in Section 6.
2|2 with different metric
EXPERIMENTS
In this section, empirically verify the effectiveness of dual goal representations through series of experiments. In Section 6.1, I, we evaluate the performance of "ideal" dual representations (Section 3) on discrete-space tasks as a proof of concept. Next, we evaluate our practical goal representation learning algorithm (Section 4) on a standard offline goal-conditioned RL benchmark (Park et al., 2025a), comparing with various previous representation learning approaches. Then, we provide ablation studies of our method in Section 6.3 to understand the importance of cach component.
In our experiments, we use 8 random seeds for the main result table (Table 1) and 4 seeds for other tables and plots, unless otherwise stated. We report standard deviations in tables and 95% confidence intervals in plots. In tables, we highlight numbers that are and above 95% of the best performance, following Park et al. (2025a).
6.1 RESULTS WITH "IDEAL" DUAL REPRESENTATIONS
We begin our experiments by demonstrating how our original concept of dual representations (i.e., the "ideal" dual representation described in Section not the ap- proximated one in Section 4) can potentially improve goal-reaching capabilities. To this end, we consider a dis- crete puzzle environment where we can analytically pre- compute the temporal distances d' Specifically, we em- ploy the discrete "Lights Out" puzzle (Park et al., 2025a). The aim of this puzzle is to reach a given goal configura- tion from an initial state, where pressing button toggles the colors of that button and the adjacent ones (Figure 2). The dataset is collected by a uniform random policy, and we evaluate agents with five pre dre-eef state-goal pairs two different puzzle sizes, 4x5 and 4x6 (Park et al., 2025a).'
Press
Figure 2: The "Lights Out" puzzle.
Note that these puzzle tasks tabular variants of the ones in OGBench (Park al., 2025a).

---

Published as a conference paper at ICLR 2026
On these tasks, we compare the performance of the original and dual representations. They are both trained with goal-conditioned DQN (Mnih et al., 2013). For dual representations, to reduce dimen- sionality, we randomly sample 64 states {s; (s (i.e., 64 random board configurations), and use [d ()...)...... d" (964.9)) In this experiment, we use the dual representation for both states and goals. We present the comparison result in Figure 3. The result suggests that our dual representations substantially improve training speed and goal-reaching performance. Note that even though the dual representation yp is bijective in this case, provides structured (processed) information about the temporal dynamics, which makes pol- icy learning significantly casier. This highlights the benefits of using structured representation.
puzzle-4x5
puzzle-4x0
Sucooes Race
100
100
50
50
the representation of
Stapa
Staps
Original
Dual
Figure 3: Original vs. dual representations.
Disclaimer: In this experiment, our goal is to show how dual representations improve performance in the ideal case (as proof of concept), and use privileged information about ground-truth temporal distances. In the following section, we demonstrate the performance of practical dual goal representations without such privileged information.
6.2 RESULTS ON OGBENCH
Next, we evaluate practical algorithm for dual goal representations (Algorithm 1) OG- Bench (Park et al., 2025a), a benchmark suite for offline goal-conditioned RL algorithms.
Tasks and datasets. We employ 13 state-based and 7 pixel-based tasks from OGBench across robotic navigation and manipulation (Figure 5). In navigation tasks ((point, ant, humanoid maze)) the aim is control robot body (from single point to complex humanoid robot with 21 degrees of freedom) to reach a desired goal location in a maze. In antsoccer, the agent must control quadrupedal robot dribble a soccer ball to desired location. In manipulation tasks (cube, scene, puzzle), the agent must control a robotic arm to manipulate cubes, re-arrange everyday objects, or solve combinatorial puzzle. In pixel- pixel-based environments, the agent must per- form robotic manipulation solely from 64 x 64 x 3-sized image observations. We use the standard navigate or play datasets for these tasks, where the datasets are collected by task-agnostic "play"- style (Lynch et al., 2019) demonstrations that randomly perform diverse atomic behaviors (e.g., con- tinuously reaching random locations or performing random button presses).
Previous goal representation learning algorithms. We compare dual goal representations with five previous goal representation learning methods: Original, VIB (Shah et al., 2021; Park et al., 2023), VIP (Ma et al., 2023), TRA (Myers et al., 2025), and BYOL-y (Lawson et al., 2025). Original ("Orig")) ") is a baseline that does not use any representations (i.e., ( = g). VIB (Shah et al., 2021; Park et al., 2023) trains goal representations via variational information bottleneck (VIB) within goal-conditioned value function or policy. VIP (Ma et al., 2023) trains metric-based goal representa- tions with goal-conditioned value learning algorithm. TRA (Myers al., 2025) trains contrastive goal representations with behavioral contrastive learning (Eysenbach et al., 2022). BYOL-y (Law- son et al., 2025) trains self-predictive goal representations with temporal self-supervised learning. We refer to Section D for the full implementation details about these methods.
Downstream offline GCRL algorithms. We measure the effectiveness of goal representations by training a parameterized goal-conditioned policy saa s, (p())) with downstream offline GCRL al- gorithms, as described in Algorithm 1. To extensively assess the performance of goal representa- tions, we consider three downstream algorithms across diverse categories: goal-conditioned implicit V-learning (GCIVL) (Kostrikov al., 2022; Park al., 2025a), contrastive RL. (CRL) (Eysenbach et al., 2022), and goal-conditioned flow behavioral cloning (GCFBC) (Park al., 2025b). We refer to Section for full details.
6.2.1 RESULTS ON STATE-BASED TASKS
We report the performance of six goal representation learning methods combined with three down- stream GCRL algorithms (18 combinations in total) 13 state-based OGBench environments in Table above and Tables and 7 in Section C. These results suggest that our dual goal represen- tations achieve the best average performance in all three settings, outperforming both the previous

---

Published as a conference paper at ICLR 2026
Table 1: Results on state-based tasks with GCIVL downstream algorithm. Our dual goal represen- tation mostly achieves the best performance among goal representation learning methods.
Environment
Orig
VIB
VIP
TRA
BYOL->
Dual
pointaazze-eeduunnnaiiii--e
78 is
69 #13
#1
3 ts
37 =7
76 #7
polnttazze-largg-nviiite---
52 16
50
0 2o
1 1
22 =12
46 16
autrasze-mmddiunnnnvvigga----
71
68
31 =2
:15
39
75
attaaaze-largeenavigttt---
16
22:12a
28 +0
u
0.
+9
0 +0
0=00
27 +a
24 #2
7
21 +3
18 =6
29 +8
3.
3 21
1:
211
2=1
3 12
attssocerrarree--aaviiattttt
47 14
34
21
8.
11
31
cube-saingle-pllay-100
52
90
#7
40
89
cube-double-pplay-v0
35
33
3
7222
44
14
scene-play-v0
46
58 21
23 16
46 16
44 =9
72 16
puzzle--x3-pllay-v0
5
14
3
5+11
0
5
puzzle-444-plaayvv0
14
6:
1:
10 #3
1=a
23
Average
34 #1
35 #2
9 #-1
15 42
19 =2
41 #2
Table 2: Results pixel-based tasks. Dual goal representations achieve the best performance five of seven pixel-based tasks. All goal representation learning methods struggle puzzle tasks, likely due the use "late fusion" (see Section 6.2.2).
Environment
Orig
VIB
VIP
TRA
BYOL-7
Dual
vieuall--tttuzzzz-rooduumnrrrrre visuull-nnttaaze--rrree-aaavv
66 ss
18 :9
30 17
48 1:
32 is
78 :4
26
5
9
13 13
9
40 =4
visuull-uubbrrriigl---pa-y----
53 =4
18 219
39
31
35
58 2*
vinuall-cbber-dobbleeppp11---
0
15 12
visual-sceene-play--0
25
41
26
1i1mall-puzz21e-333-pppaa-0
22 12
0 :0
0 10
0.
0 4:
0 2.
visual-puzzle- 1isuallppuz2211444pppa---00
65
0
0 1:
0.
0 1:
0 :10
representation learning methods and the original representations on most tasks. Notably, our method performs particularly well cube tasks, in some cases improving performance by 3x compared to the "original" baseline (Table 6). Moreover, unlike previous algorithms such as VIB, VIP, and TRA, method shows the least sensitivity the choice of downstream offline GCRL algorithms.
6.2.2 RESULTS ON PIXEL-BASED TASKS
Next, we present the comparison results on pixel-based tasks in Table 2. We use GCIVL as the base offline GCRL algorithm, given its strongest average performance in the state-based case. As in Section 6.2.1, the results suggest that our dual goal representation outperforms the other five alternative representations (or at least performs equally well) five out of seven tasks.
Negative results. However, we found that no representation learning methods (including ours), achieve non-zero performance on visual puuzle tasks (Table 2). Given their reasonable perfor- mance on the equivalent state-based puzzle tasks (Table 1), this failure is likely related to handling visual observations.
We hypothesize that the failure stems from differences between early and late fusion of visual state and goal observations. Typically, vanilla goal-conditioned pol- icy (a 8, 9. first concatenates and g along the depth dimension (forming 64 x 64 (3+-3) -ized image) and passes to a convolutional neural network (this of- ten called "early fusion"). However, when using representation-conditiinnee policy < ppg)),
Table 3: Early late fusion.
Environment
Early
Late
visual-puzzle -aazzuuuzeeexx 3x3
22 12
0 io
visual-puzzle
65
0
we cannot directly use early fusion as needs to be processed separately. The problem is that, early fusion generally leads to better performance in visual robotics tasks (Walsman al., 2019; Huang et al., 2024). To verify this, we compare the performance of vanilla GCIVL with early and late fu- sion. Table 3 shows the results on puzzle-(3x3, 4x4), suggesting that the of late fusion is in-
8

---

Published as a conference paper at ICLR 2026
deed likely the cause of the poor performance of representation learning-basee approaches these tasks. Given this evidence, we believe this failure may addressed by modifying the representa- tion function to be aware of the current state well, which leave for future work.
6.3 Q&As
In this section, we present additional discussions and ablations through the following Q&As.
Q: Are dual goal representations indeed robust to noise?
A: Our theory (Theorem 3.2) sug- gests that dual goal representations can, in principle, be invariant to ex- ogenous noise. To empirically ify this property in practice, we eval- uate the original and dual goal rep-
antzaze-large cube-double
scene-play play
Buccesa Bare
18
resentations in three "noisy" environ- ments (with GCIVL as the down- stream GCRL algorithm), where we add Gaussian noise to the evaluation goals. This assesses the robustness of the two representations to out-of-distribution goals at test time. Figure presents the comparison results. The results tell us that our dual representation deed exhibits better robustness than the original one, suggested by the theory.
Scepa
3tepa
Stepa
Dr Original
Figure 4: Dual representations are robust noise.
Q: Can we use other aggregation (parameterization) functions for dual representations?
Table 4: Dual representations with different parameterizations.
Enviromment
Dual (Symmetric)
Dual (Inner Product)
pountazzemmediunnnnvviiteete-
73
76
pointaze--lrg---avgaat----
46
attmaazz-madiin--nariaatt----
17
75 21
attaaze-largg--arigaatevvv
23 +1
35 +7
attmazzegiiatt-naiiaatee---
20
1 10
23 12
30
buaaaosibazzeellaggeerrrrggvvv
3 +2
antroceerarree--nariggge---
ks
*a
cube-siingee-play-v0
66 14
90 14
cube-doobllepplay-r0
48 +s
61 #4
scene-play-v0
82
71
puzzle-3x3-play-vv
6
5 50
puzzle-4x4-play-ve
+1
23 43
Average
38
42
A: In Section 4, we approximate a functional using the inner product aggregation function, ((v(s), = (ss) "(().. However, this is not the only possible option. For example, other parameterizationn. such as the symmetric norm (metric) parameterization, f(((s), (pg)) = [|{{(s) - p(g) with v sp. In particular, if we employ this metric parameterization, resentations become similar to previous temporal metric embedding methods (Sermanet et al., 2018; Ma et al., 2023; Park et al., 2024a). To understand how different aggregation functions affect perfor- mance, we evaluate dual goal representations with the two parameterizations above (with GCIVL as the downstream GCRL algorithm), and present the results Table 4. The results show that our in- ner product representation generally leads to better performance than the metric one when used as goal representation. We believe this is likely due to the universality of the inner product function (Section 4), whereas the metric parameterization is provably not universal and thus potentially less expressive in representing goals (though this parameterization can still be useful for other purposes; see Ma et al. (2023); Park et al. (2024a)).
dual rep-

---

Published as a conference paper at ICLR 2026
Q: Can't directly extract policy from the temporal distance function (p
A: In our offline GCRL recipe (Algorithm I), train two goal-conditioned value func- tions: (s), ((g)) for approximating tem- poral distances to learn dual representation, and VGOLL ((()) for performing down- stream GCRL. One might wonder whether we can directly extract goal-conditions policy from the former function f. without having a separate downstream GCRL loop. This indeed possible ("Dual (Direct)" in Table 5), but it generally leads to worse per- formance (except for antmaze-large), The reason is that f has structured inner product parameterization and relatively less expressive (although theoretically universal) to extract conditiond policy than the monolithic VGCRI, (s, ((g)) function, which has structural con- straints. In other words, the inner product parameterization is expressive enough to learn a good goal representation but may not be enough extract policy (e.g.. g. in antmaze, it is sufficient fo- cus on the x-y position of the agent to learn good goal representation, but we need to know the full joint angles train a control policy), and thus we often benefit from separate downstream GCRL algorithm with a different, more expressive value function.
Table 5: Directly extracting policy from the tempo- ral distance function often degrades performance, In this experiment,
GCIVL for both f and VGCAL to enable apples-to-apples comparison.
Environment
Dual (Direct)
Dual
antmaze-large
58 1s
28 1;
humanoidmaze-meeee meddum
17 +3
28 +2
antsoccer-arena
16 +4
21 +2
cube-double
42
49 13
scene-play
50
71
puzzle-4x4
13 #:
#3
7 CLOSING REMARKS
Originally, the main idea in this work was inspired by common slogan in mathematics: "an object is uniquely determined by its relations with every other object." Building on top of this high-level idea, we introduced the concept of dual goal representation, studied its theoretical properties (sufficiency and noise invariance), proposed practical goal representation learning recipe, and demonstrated its empirical performance. We hope that the ideas introduced in this work inspire future research on state and goal representation learning reinforcement learning and sequential decision making.
LCKNOWLEDGMENT
This work was partly supported by the Korea Foundation for Advanced Studies (KFAS), ONR N00014-22-1-2773, and AFOSR FA9550-22-1-0273. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.
REPRODUCIBILITT STATEMENT
We provide the code and instructions in the supplementary material and describe the full implemen- tation details in Section D.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations (ICLR), 2017.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/ aass/600.... 160.066450, 2016.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2017.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. Henighan, Rewon Child, Aditya Ramesh, Daniel M.
10

---

Published as a conference paper at ICLR 2026
Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Neural Informa- fion Processing Systems (NeurIPS), 2020.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion, Neural Computation, 5:613 224,3-624, 1993.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference Machine Learning (ICML), 2019.
Ayoub Echchahed and Pablo Samuel Castro. A survey of state representation learning for deep reinforcement learning. Transactions on Machine Learning Research (TMLR), 2025.
Yonathan Efroni, Dipendra Misra, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Provably filtering exogenous distractors using multistep inverse dynamics. In International Con- ference Learning Representations (ICLR), 2022.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-ri with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridg- ing planning and reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2019.
Benjamin Eysenbach, Tianjun Zhang. Ruslan Salakhutdinov. and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2022.
Gerald B Folland. Real analysis: modern techniques and their applications. John Wiley & Sons, 1999.
Dibya Ghosh, Chethan Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference Machine Learning (ICML), 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). ArXiv, 2016.
Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, and Pieter Abbeel. Early fusion helps vision language action models generalize better. In CoRL. Workshop X- X-Embodiient Robot Learning. 2024.
Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference Artificial Intelligence (ICAAI), 1993.
Junsu Kim, Seohong Park, and Sergey Levine. Unsupervised-to-onniee reinforcement learning. ArXiv, abs/2408. 147855 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-
learning. International Conference on Learning Representations (ICLR), 2022.
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Rajiv Didolkar, Dipendra Misra, Dylan J Foster, Lekan P Molu, Rajan Chari, Akshay Krishnamurthy, and John Langford. Guaranteed discovery of control-ende endogenous latent states with multi-step inverse models. Transactions on Machine Learning Research (TMLR), 2022.
11

---

Published as a conference paper at ICLR 2026
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning: State-of-the-ar,, pp. 45-73. Springer, 2012.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Leurning (ICML), 2020.
Daniel Lawson, Adriana Hugessen, Charlotte Cloutier, Glen Berseth, and Khimya Khetarpal. Self-predictive representations for combinatorial generalization in behavioral cloning. ArXiv, abs/2506.10137, 2025.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020.
Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards a unified theory state abstraction for mdps. In International Symposium Artificial Intelligence and Mathematics, 2006.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning (CoRL), 2019.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via f-advantage regression. Neural Information Processing Systems (NeurIPS), 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicii pre-training. In International Conference on Learning Representations (ICLR), 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
Vivek Myers, Chongyi Zheng. Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide a metric structure for decision- making. International Conference on Machine Learning (ICML), 2024.
Vivek Myers, Bill Chunyuan Zheng, Anca Dragan, Kuan Fang, and Sergey Levine. Temporal rep- resentation alignment: Successor features enable emergent compositionality in robot instruction following. In Neural Information Processing Systems (NeurIPS), 2025.
Scohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned with latent states as actions. In Neural Information Processing Systems (NeurIPS), 2023.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICML), 2024a.
Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. In International Conference on Learning Representations (ICLR), 2024b.
Scohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. International Conference on Learning Representations (ICLR), 2025a.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. In Neural Information Processing Systems (NeurIPS), 2025b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML), 2025c.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, Abs/2204.06122 2022.
12

---

Published as a conference paper at ICLR 2026
Emily Richl. Category theory in context. Courier Dover Publications, 2017.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametrie topological memory for navigation. In International Conference on Learning Representations (ICLR), 2018.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- seff-rreicctiv representations. In International Conference on Learning Representations (ICLR), 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference Robotics and Automation (ICRA), 2018.
Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. Recon: Rapid exploration for open-world navigation with latent goal models. In Conference on Robot Learning (CoRL), 2021.
Harshit S. Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. In International Conference on Learning Rep- resentations (ICLR), 2024.
Lorenzo Steccanella and Anders Jonsson. State representation learning for goal-condd rein- forcement learning. In European Conference Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML. PKDD), 2022.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transac- tions on Neural Networks, 16:285-2888 286, 2005.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In Allerton Conference on Communication, Control, and Computing, 1999.
Ahmed Touati and Yann Ollivier. Learning representation optimize all rewards. In Neural Information Processing Systems (NeurIPS), 2021.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In International Conference on Learning Representations (ICLR), 2023.
Martin Wainwright. High-dimensional statistics: A non-asymplotic viewpoint. Cambridge Uni- versity Press, 2019.
Aaron Walsman, Yonatan Bisk, Sandia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, and Dieter Fox. Early fusion for goal directed robotic vision. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
Mianchu Wang, Rui Yang, Xi Chen, and Meng Fang. Goplan: Goal-conditioned offline reinforce- ment learning by planning with learned models. Transactions on Machine Learning Research (TMLR), 2024.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching rein- forcement learning via quasimetric learning. In International Conference Machine Learning (ICML), 2023.
Yifan Wu, G. Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. In International Conference on Learning Representations (ICLR), 2019.
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In International Conference on Learning Representations (ICLR), 2024.
13

---

Published as a conference paper at ICLR 2026
A PROOFS
(S..p. be a CMP and & Theorem A.1 (Sufficiency of Dual Goal Representations), Let M = = be its dual goal representation function. Then, there exists deterministic policy n² S 8V A that takes dual goal representation input, such that its induced policy क S x S - A defined as т (8. = (8, @ (g)) satisfies V* (x, = (*, g) for all 8, S.
Proof. We define n² 8 + Aas as 7' (s,
arg max E, s' p( (s,0) (1,))) (65') aEA
for s and f E SV. Then, for any 8, S, we have
7(8 = x (8,
arg max Eg pp (s,a) (7 ( a
arg max E pp (s,)) (s,a (V* 9))
aeA
arg max Q" (s, a.
aeA
Invoking the standard result in RL theory (Sutton & Barto, 2005), we conclude that (*( (*(,gg).
Theorem A.2 (Noise Invariance of Dual Goal Representations). Let M 2S. ,p²) be BCMP and sp' be its dual goal representation function. Let 91 92 E S be two goal observations from the same latent state; i.e. p¹ (g1) = 1f (92). Then, they have the same dual goal representation; ie., x (g))
Proof. We need to show 4
<" (g1)(s) -p (s)) for all s E S. This follows from (g))((s) log, (5,
log, (max V (ss
-ogg,
max E (1
log,
max EE Top (1/
(1) = ='(()))
=log,
inax E (r
(1)) = of (
7 7'7 (22) ) )
log,
max EE Tro
log, (max V (s(gg))))
log, V*((s
(s))
where the max is taken over the set of all non-goal-conditioned policies S A(A).
B ALGORIHTM
In this section, we describe the representation learning methods and downstream offline GCRL algorithms considered in experiments.
14

---

Published as a conference paper at ICLR 2026
B.1 PRIOR REPRESENTATION LEARNING ALGORITHMS
VIB (Shah et al., 2021; Park et al,, 2023). VIB trains goal representation based on informa- tion bottleneck (Tishby et al., 1999; Alemi et al., 2017). In experiments, a VIB representation is trained directly via the value loss in the downstream GCRL algorithm (except for GCFBC, where we train it via the actor loss). Specifically, we model downstream value function as VGCRL (s, p(g)), where (gg) denotes a stochastic latent vector sampled from ((((g), (g)) with y being a mean network and Σ being diagonal covariance network. We train this stochastic latent- parameterized value network using the downstream GCRL loss with an additional KI. loss term, (Drr(NV(((g) E(g) (O, [)), to regularize the posterior distribution with a coefficient B. In our experiments, we find that low 3 values generally work the best, and select the best 3 from {0.001, 0.003} for each environment and downstream GCRL algorithm.
TRA (Myers et al., 2025). TRA trains goal representation via symmetric InfoNCE variant (My- ers et al., 2025) the CRL loss (Eysenbach et al., 2022). We then use this representation to param- eterize downstream GCRL policy and downstream value functions. In the original paper, the TRA representation also gets gradients from the downstream GCRL loss. We perform a hyperparameter sweep on this choice, and enable gradient flow for GCFBC and disable for GCIVL and CRL.
BYOL-y (Lawson et al., 2025). BYOL- trains a representation based on temporal self preficreeee representation loss (Schwarzer al., 2021). We the bidirectional variant in the paper, and use the representation to parameterize goals in downstream policy and value networks. We do use &o to represent states to enable apples-to-apples comparisons with other goal representation learning algorithms, As in TRA, we also perform a hyperparameter sweep on gradient flows from the downstream GCRL loss the representation, and enable it for GCFBC and disable for GCIVL and CRL.
VIP (Ma al., 2023). VIP trains a representation via a metric embedding loss (Sikchi al., 2024). Specifically, it parameterizes the value function V(8.g) -1|sp(s) ( || and trains yp with dual RI. loss function (Ma al., 2022; 2023; Sikchi al., 2024). As with other methods, we use yp to represent (only) goals in downstream policy and value networks.
B.2 DOWNSTREAM OFFLINE GCRL ALGORITTMM
In our experiments, we consider three downstream offline goal-conditioned algorithms: GCIVL, and CRL, and GCFBC. For GCIVL and CRL, we use the original implementations of OGBench (Park et al., 2025a), and for GCFBC, we use the implementation by Park et al. (2025b). We refer to these works for the full descriptions these algorithms.
C ADDITIONAL RESULTS
We provide results on state-based tasks with different downstream GCRL algorithms (CRL and GCFBC) in Tables 6 and 7.
15

---

Published as a conference paper at ICLR 2026
Table Results on state-based tasks CRL as the downstream algorithm.
Environment
Drig
VIB
VIP
TRA
BYOL-Y
Dual
30 17
0.
55 17
32 12
49 26
33 At
polattaaze--lageernvviigttvo
46 1:2
1o
50
35 110
38 111
39 1:12
95
10 #1
94
77
87
93
attaazeelaargennayiaattev-
87
64
71 #1
87 #2
uttazze-glann-navigatt---000
0.
4
3
=1
14
57 #:
21
43 11
#1
=1
57 1:
huaanoidhaaze-larr---vvvviggeee
20 4:5
1
13
31
22 =7
18 14
urteocccerarrons--avvaaatttt
22 20
1 1:
3 31
7 42
9 9-
19
cube-siingle-play-v00
7
42
17
60
cube-doubbe-pplay-v0
13
21
5 5.12
24 18
scene-play-v0
#2
t:
#3
22
=2
44 #6
puzzle-333-ppay--00
4 #1
3 #1
3 +0
0
6
puz2le-4x4pplay-11
0 0.
0.
0 0.
0.10
0.
2.
Average
#2
t:
30 #1
=1
39 #:
Table Results on state-based tasks GCFBC as the downstream algorithm.
Environment
Drig
VIB
VIP
TRA
BYOL-9
Dual
pointaaze-mmdduu--navii---e
63 ia
48 :4
53 is
77 #4
77 #2
73 ia
pointmazze-larzeenvviiae---
70 14
69 22
65 25
76 14
75 24
66 17
44
10 +2
18 #2
56 +5
51
53
uttaxze-largeenaviigttee-0
20
7
29
29
uitaaze-gganttnnarigatt--00
0.0
0.10
10
0.10
0
8::
t1
=1
41
=2
8 +2
hmnanoichaaze-larre-nneerggggrr
1
+
1
1 =0
1 to
urtsoccor--arona-novviat..
1611
311
3
11 11
12
17
cube-single-plaay-v00
12
10
14
12
10
20
cube-double-plaay-r0
scene-play-v0
1944
641
18 14
45
13
27 43
puzzle-3x3-playy-00
3+1
2
2
2
2
2 40
puzzle-4x4-play-v0
211
0.
0
1
0
0 10
Average
20 40
12 to
14 #1
+1
22
23 +1
16

---

Published as a conference paper at ICLR 2026
Figure 5: OGBench environments. OGBench provides diverse state- and pixel-based goal-conditioned tasks across robotic navigation and manipulation. This figure is taken from Park al. (2025c).
D IMPLEMENTATION DETAILS
Our method and baselines are implemented on top of the default implementations given in the OG- Bench codebase (Park al., 2025a). We provide the code and instructions at https: httpps://githun com com/deepindermnan///
ndermann/ duald -ual- representations.
D.1 DISCRETE PUZZLE EXPERIMENTS
Tasks and datasets. In Section 6. we employ a discrete "Lights Out" puzzle environment. this environment, a state consists of ntr x ny binary button states (i.e., S = (0, 1)*"">,) and an action corresponds to pressing a button (i.e., A {0, 1,..,,,,. ng.., - 1}). Whenever the agent presses button, the colors of that button and the adjacent ones get toggled. We employ two different puz- zle sizes, (mz nyy) = (4, 5) and (4, 6). which we call puzzle-4x5 and puzzle-4x6, respectively. At test time, evaluate the agent with the five pre-defined state-goal configurations used in OG- Bench (Park et al., 2025a), where the maximum evaluation episode length is set to ng?y. For datasets, employ 40000 length-25 trajectories generated by uniform random policy.
Training. To compute dual representations, we first randomly sample 64 states, run breadth-firg search to compute temporal distances, and the list of temporal distances from the 64 states as a dual representation. For downstream goal-conditioned DQN, we use the -1- sparse reward function (see Section D.2). We train agents for 1M steps, and evaluate the arg max DQN policies using 15 episodes for each of the five evaluation tasks.
Hyperparameters. We provide the list of hyperparameters in Table 8. In the table, (Perr> P Proom Peraj. Prana) denotes the hindsight relabeling ratio tuple defined in Park et al. (2025a).
D.2 OGBENCH EXPERIMENTS
Visual encoders. In pixel-based tasks, there are several design choices regarding visual encoders (e.g., whether to share state or goal encoders across different networks, how to flow gradients, etc.). We test various variants regarding these choices and find the following configuration For the original representation baseline, we use separate "early fusion" (see Section 6.2.2) encoders for each of the policy and value network. For other goal representation methods, we share a single state visual encoder and a single goal visual encoder for all networks (with "late fusion")), and they get gradients from all downstream GCRL losses as well as the representation learning loss. However, do not propagate downstream GCRL gradients through the representation function yo (except for VIB, which designed to be trained with the downstream value loss). We refer to our code for the full details about gradient flows.
work best.
Metrics. Following OGBench (Park et al., 2025a), we report the average performance over the last three evaluation epochs in tables (i.e,, over 800K, 900K, and 1M gradient steps for nate-based tasks and over 300K, 400K, and 500K steps for pixel- based tasks). We use (state- based) or 15 (pixel- based) episodes for each of the five evaluation goals.
17

---

Published as a conference paper at ICLR 2026
Hyperparameters. We provide the list of hyperparameters in Table 9. Following OGBench (Park et al., 2025a), we apply layer normalization (Ba et al., 2016) to all networks except for the policies in GCIVL and CRL.
Table 8: Hyperparameters for discrete puzzle experiments.
Hyperparameter
Value
Gradient steps
(state (state-based)
Optimizer
Adam (Kingma Ba, 2015)
Leurning rate
0.0001
Batch size
1024
MLP size
[1224, 1024, 1024, 1024]
Nonlinearity
GELU (Hendrycks & Gimpel, 2016)
Layer normalization
True
Turget network update
0.005
Discount factor
0.96
DQN (Pew Peco Pieaj Prana) ratio
(0.2, 0. 0.5, 0.3)
Table 9: Hyperparameters for OGBench experiments.
Hyperparameter
Value
Gradient steps
10t (state-based), 10" (pixel-based)
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate
Batch size
1024 (state-based), 256 (pixel-based)
ML.P size
512, 512]
Nonlincarity
GELU (Hendrycks & Gimpel, 2016) 0.005
Target network update rate
Discount factor 7
0.99 (default), 0.995 (humanoi dmaze, antmaze-giant)
Goal sepresentation dimensionality
256 (default), 64 (pointaaze)
Dual representation's GCIQL expectile x
0.7 (default), 0.9 (navigate)
VIB /
0.001 (default), 0.003 (pointaaze, antmaze-large)
Representation (yeu Po
goom Piaai Prand) ratio (Dual)
0.2,0.5.0.0 (0.2,0.5,0,00 0.333 (0,1,0,0)
Representation [Peur Seuur-Pasom Paeam Pirjj Frand) ratio (TRA, BYOL-))
Downstream GCIVL expectile
0.9
Downstream policy extruction hyperparameters
Identical to OGBench (Park et al., 2025a)
Downstream policy (p Prwas (erns) Plena) ratio
(0,0,1.0)
Downstream value (peue Pgeum Pizai Pggaa) ratio (default)
Pgeum
(0.2,0.5.0.0.3)
Downstream value (peur (Peur: Pgeom: Piraj- Prona) ratio (CRL)
(0, 0,
Goal noise for Figure
0.3 (sutmazee), 0.2 (cube, scene)
Visual encoder
inpala snall (Espeholt el., 2018; Park al., 2025a)
Image augmentation probability
0.5
18


Reviews:
[{"review_id": "yhlKP2pZ29", "reviewer": "Reviewer_4M3V", "summary": "The paper introduces dual goal representations for goal-conditioned reinforcement learning (GCRL). The core idea is to represent a goal state not by its own features, but by the set of temporal distances from all other states to itself. The paper theoretically proves that this representation is sufficient for learning an optimal policy and, importantly, is invariant to exogenous noise. While the theory establishes these desirable properties to make this concept practical, the proposed algorithm sidesteps the computational challenge by learning a compact, low-dimensional vector that approximates this representation. A comprehensive set of experiments demonstrates the benefit of this approach, showing that it consistently improves the learning performance of several existing GCRL algorithms across a wide range of tasks.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel and well-motivated approach to goal representation based on temporal distances. This \"dual\" perspective is a strong conceptual contribution with clear potential for applications even beyond the GCRL setting.\n\n- The paper is well-written and clearly structured. The authors effectively build intuition by cleanly separating the ideal theoretical concept from its practical implementation, making the core ideas easy to follow and understand.\n\n- The experimental evaluation is both extensive and convincing. By testing on a diverse suite of environments and integrating their representation with three different downstream GCRL algorithms (GCIVL, CRL, GCFBC), the authors provide strong evidence for the method's general applicability and robustness.", "weaknesses": "- A significant gap exists between the theoretical framework, which analyzes an ideal functional $\\varphi^v$, and the practical implementation, which learns a finite-dimensional approximation. Is not obvious that the formal guarantees of sufficiency and noise invariance do transfer to the compressed low dimensional learned representation. The justification for the method's effectiveness therefore rests heavily on its strong empirical validation rather than a direct theoretical proof for the practical algorithm.\n\n- The proposed method employs a two-stage architecture: first learning a representation $\\varphi(g)$ by training an approximate value function ($\\psi(s)^T \\varphi(g)$), and then using $\\varphi(g)$ to train a separate downstream policy and, in many cases, a new value function (V_GCRL). This design raises significant questions about efficiency and potential redundancy.\nWhile the paper's ablation study (Table 5) shows that direct policy extraction from the initial value function performs worse, the underlying reason for this is not fully explored. It seems that a deeper analysis is needed to justify why the inner product parameterization is expressive enough to yield a powerful representation, but too restrictive for direct policy extraction. Clarifying this would explain why the apparent redundancy of learning a second value function and discarding the initial state encoder $\\psi(s)$ is a necessary design choice rather than a limitation.\n\n- A minor fix is that the paper is missing the citation to the paper \"State Representation learning for Goal Conditioned Reinforcement Learning\" Steccanella et al. that seems relevant and learn a temporal distance representation used for reward shaping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tDXboSulRj", "reviewer": "Reviewer_4LLs", "summary": "The paper proposes a way to encode goals for goal-conditioned reinforcement learning called dual goal representations. The method characterizes a goal by the collection of shortest-time distances from every state to that goal. The authors show that this representation is sufficient to recover an optimal goal-reaching policy and is invariant to observation noise that does not affect latent system dynamics. Practically, they approximate this functional by learning a goal-conditioned value model, taking its ``goal head'' as the goal embedding, and then training any downstream offline GCRL algorithm that consumes this embedding. Experiments on OGBench report average improvements over several prior representation learners across 20 state and pixel tasks, plus ablations comparing inner-product vs metric parameterizations and a study on noise robustness.", "questions": "* How sensitive is the approach to the accuracy of the learned distance surrogate? Please report correlations between predicted distances and true shortest-path steps on tasks where ground truth can be computed, and study how downstream performance varies as you degrade the value model. This could validate the core mechanism that “better distance, better policy.”\n\n* Can you evaluate noise invariance with structured distractors rather than additive goal noise, for example by adding irrelevant moving objects or textured backgrounds in pixel tasks while keeping latent dynamics fixed? This would better mirror the Ex-BCMP setup used in the theorem.\n\n* To address the early vs late fusion confound, can you try a state-aware goal embedding or a lightweight cross-attention block that allows early interaction while keeping a representation bottleneck, then re-run the visual puzzles and other pixel tasks for a fair comparison? Even a small prototype would strengthen the claim.\n\n* Please expand the ablation on representation dimensionality and the choice of inner-product vs metric forms. Are there tasks where the metric version wins once you scale capacity or adjust negative sampling for metric learning baselines, and how do results change with larger N?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The dual view, defining a goal by its temporal distances to all states, is elegant and connects clearly to the control objective. The sufficiency and noise-invariance theorems make the idea precise and motivate learning a representation that depends only on dynamics rather than raw observations.\n\n* The recipe, ``learn a goal-conditioned value model, use its goal head as the representation, then plug into standard offline GCRL,'' is straightforward and compatible with multiple downstream algorithms. This lowers integration cost for practitioners.\n\n* On OGBench, the proposed representations are competitive or best on a majority of state-based and several pixel tasks, and the paper investigates parameterization choices, the limits of directly extracting a policy from the distance model, and robustness to test-time goal noise. These analyses help interpret when the approach works.", "weaknesses": "* The key invariance result is proved under an Ex-BCMP model with disjoint observation supports per latent state, a strong assumption seldom satisfied by realistic image observations. The paper then validates ``robustness to noise'' by adding Gaussian noise to evaluation goals, which does not convincingly reflect exogenous distractors in observations or dynamics. A more faithful test would use structured nuisance variation tied to the emission model or distractor objects.\n\n* The method compares favorably on several pixel tasks, yet performance on visual puzzles collapses for all representation learners. The paper attributes this to late fusion that is required when goals are embedded, while the ``original'' baseline can use early fusion. This creates a comparison gap that complicates claims about representation quality, since the fusion strategy is entangled with the representation. The suggestion that state-aware representations could resolve this is plausible, but the paper stops short of implementing it.\n\n* The empirical approach largely repurposes a goal-conditioned value model and takes its goal head as a representation, combined with inner-product scoring. While the dual perspective is neat, the implementation feels incremental relative to prior metric and contrastive goal embeddings. The inner-product ``universality'' claim is cited, but the paper does not quantify approximation error or show cases where the metric form fundamentally fails beyond average score tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3txQosePn8", "reviewer": "Reviewer_oPkP", "summary": "The paper proposes to use estimated distances between states as the underlying latent representation in goal-conditioned reinforcement learning (GCRL). This way a goal can be implicitly represented by the distances to all other states, akin to a heuristic estimate. The approach is empirically tested in a set of benchmark domains using deep reinforcement learning on top of the learned goal representation.", "questions": "Do you actually use a different embedding for goal states as opposed to other states? What is the motivation for using two different embedding functions?\n\nIf planning using the learned distance estimates performs poorly, does this not indicate that the learned distance estimates are not very accurate?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of the work is that the proposed algorithm performs well in practice compared to other algorithms for GCRL. Concepts are clearly presented and experiments seem rather extensive.", "weaknesses": "As mentioned by the authors, several previous approaches estimate distances or metrics for GCRL by embedding states in a latent state space and applying a metric such as the L2-norm in the latent space. The authors propose using the scalar product instead, which seems highly related to the L2-norm since the squared L2-norm equals\n\\[\n\\psi(s)^\\top\\psi(s) + \\varphi(g)^\\top\\varphi(g) - 2 \\psi(s)^\\top\\varphi(g).\n\\]\nHence the main term for determining the distance (though negated) is precisely the scalar product in the third term. Another difference is that the goal embedding \\varphi is potentially different from the embedding \\psi used for other states. Though this has potential positive effects such as representing asymmetric distances, applying a metric to vectors from different embedded spaces does not have the same intuitive interpretation. I believe the motivation for these design choices could be explained better in the text.\n\nIn general the claim to novelty is a bit weak. Using the embedding for goal representations seems very similar to using the embedding to estimate distance metrics or perform planning. The inner product seems to slightly outperform the L2-norm in Table 4, but this difference is not large enough to account for the improvement in performance compared to other algorithms.\n\nThere exists another work in the literature that also learns explicit distance estimates between pairs of states from offline data:\n\nSteccanella and Jonsson, ECML 2022\nState Representation Learning for Goal-Conditioned Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qTwqvpgv3S", "reviewer": "Reviewer_U6gt", "summary": "The paper proposes a new approach to encode a goal state in goal-conditioned reinforcement learning (GCRL). Instead of using the goal state s directly, the work proposes to use a dual representation where the goal is represent as its distance to all other states. It is claimed that this representation has favorable properties which are theoretically proved and verified by the experimental results on simulated environments.", "questions": "As the authors see from my comments, I struggled to understand the practical implementation of the proposed method. I hope that the authors clarify me the concepts or explain why my questions/concerns are incorrect? I am RL practitioner and somewhat experienced coder, but I could not implement your method solely based on the manuscript.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The fundamental idea of the work is simple and interesting. Moreover, the experimental results seem to verify that the simple idea works (at least) in simulations.", "weaknesses": "**Major:**\n\n - The idea of dual representation in Figure 1 is simple, but also yields to certain questions such as dimension of the dual representation in the case of a large number of states or in the continuous cases where states are infinite - this should be clarified to serve the readers\n\n - The concept of \"temporal distance\" should be clarified. It generally suggests \"number of steps from s to g\", but it is not defined\n\n - Things get confusing when states and goal are transformed to the embeddings psi(s) and phi(g) that transform the state to N-dimensional vectors. Close to Eqs. (1) and (2) psi and phi are defined to be the output and the goal heads, but heads of what? Then the distance function is approximated by *implicit q learning* without any explanation why this is done and how it is done? Q-learning provides actor and critic networks, but are they used and how?I sought information from the appendix D.2 without finding any more explanation?\n\n - The theory is nicely explained, but the gap from the theory to actual implementation seems large?\n\n**Moderate:**\n\n - Even the simple example in Figs. 2 and 3 remains unclear as the meaning of state is not explained (i.e. what means the 64 random states - any random configuration?)\n\n - If the proposed algorithm needs pre-training with IQL, then how many interaction steps are needed for IQL? If that is substantial, then is this method really improving over other methods with the same number of steps and average over N runs with random initializations (for example, in Figure 4 are those IQL steps added to Dual results)?\n\n**Minor**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)