Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning
Authors: Aravind Venugopal
Jiayu Chen
Xudong Wu
Chongyi Zheng
Benjamin Eysenbach
Jeff Schneider
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Goal-Conditioned Reinforcement Learning
Reward Shaping
Primary area: reinforcement learning
TL;DR: We propose an novel and effective reward-shaping method for credit assignment based on generative modeling of the occupancy measure and optimal transport, demonstrating state-of-the-art performance in offline GCRL.

Abstract: The temporal lag between actions and their long-term consequences makes credit assignment a challenge when learning goal-directed behaviors from data. Generative world models capture the distribution of future states an agent may visit, indicating that they have captured temporal information. How can that temporal information be extracted to perform credit assignment? In this paper, we formalize how the temporal information stored in world models encodes the underlying geometry of the world. Leveraging optimal transport, we extract this geometry from a learned model of the occupancy measure into a reward function that captures goal-reaching information. Our resulting method, $\textrm{\textbf{Occupancy Reward Shaping (ORS)}}$, largely mitigates the problem of credit assignment in sparse reward settings. ORS provably does not alter the optimal policy, yet empirically improves performance by $\mathbf{2.2\times}$ across 13 diverse long-horizon locomotion and manipulation tasks. Moreover, we demonstrate the effectiveness of ORS in the real world for controlling nuclear fusion on 3 Tokamak control tasks.

Published as a conference paper at ICLR 2026
OCCUPANCY REWARD SHAPING: IMPROVING CREDIT ASSIGNMENT IN OFFLINE GOAL-CONDITIONES REINFORCEMENT LEARNING
Aravind Venugopal"
Jiayu Chen
Xudong Wu
The University of Hong Kong, INFIFORCE Intelligent Technology
Carnegie Mellon University
The University of Hong Kong
Chongyi Zheng Princeton University
Benjamin Eysenbach
Jeff Schneider Carnegie Mellon University
Princeton University
ABSTRACT
The temporal lag between actions and their long-term consequences makes credit assignment challenge when learning goal-directed behaviors from data. Gen- erative world models capture the distribution of future states agent may visit, indicating that they have captured temporal information. How can that temporal information be extracted to perform credit assignment? In this paper, we formalize how the temporal information stored in world models encodes the underlying ge- ometry of the world. Leveraging optimal transport, extract this geometry from a learned model of the occupancy measure into a reward function that captures goal-reaching information. Our resulting method, Occupancy Reward Shaping (ORS), largely mitigates the problem of credit assignment in sparse reward set- tings. ORS provably does not alter the optimal policy, yet empirically improves performance by 2.2x across 13 diverse long morighhoizon locomotion and manipula- tion tasks. Moreover, we demonstrate the effectiveness of ORS the real world for controlling nuclear fusion on 3 Tokamak control tasks.
Code: https: / com/ ar wwwraavindvnnn///cmmmmm htt.a//eeee//oocppp.....tttttaaa
INTRODUCTION
Reinforcement learning (RL) is fundamentally tied to credit assignment because effectively reasoming about the long-term consequences of actions is crucial to learn diverse goal- reaching behaviors from data. Although goal-conditioned reinforcement learning (GCRL) offers simple, domain- agnostic and scalable framework for RL from large of- fline datasets (Liu et al., 2022a; Yang et al., 2023), GCRL agents trained with sparse rewards struggle with credit assign- ment (Eysenbach et al., 2022: Ding et al., 2019; Wang et al., 2023a; Park et al., 2024a). Therefore, the design of well- defined and meaningful reward functions, called reward shap- ing (Ng et al., 1999) is a promising solution to address the credit assignment challenge (Yu et al., 2025).
Success rate
60
48
28
0-
GCIOL (Base Algorithm)
HIOL (Hieran (Hierarchice arierarchical RL)
n-step returns
Go-Fresh
ORS (ours)
Figure 1: Average relative success rate (task normalizz)))) on 7 challenging OGBench tasks (Park et al., 2024a). Higher is better; 100.% matches the best algorithm per task. Our method, ORS, performs best, improving over the best baseline by 31%.
While manual reward design for each goal infeasible, ex- isting work relies learning local temporal distance estima- tors from offline data (Hartikainen et al., 2019) and using them cammmnon-parametriic graphs (Savinov et al., 2018: build semi/non- parametric
Mezghani et al., 2023) and indirectly infer temporal informa- tion for credit assignment. These approaches run the risk of scaling poorly with task complexity as composing local infor- mation infer global temporal dependencies often leads to compounding errors. So, can we learn
Corresponding author; eaanugo2@anddeee

---

Published as a conference paper at ICLR 2026
Offline Data
QRS Rewards
elore dlginer coloe eaery
Gool deriibation
ORS
Agea
Future distributionally distont from goal
X
Higher rewards
Future states distributionally closer to goal
Figure 2: ORS learns occupancy measure (distribution future states) and extracts goal-reaching informa- tion into reward function such that states with future states closer to goal have higher rewards.
reward functions that accurately and directly capture long-term temporal dependencies and goal- reaching information present in offtine data?
Generative world models (Janner et al., 2020b; Schramm & Boularias, 2024; Farebrother et al., 2025) have shown remarkable ability to capture the multi-modal distribution of future states an agent may visit, i.e., the occupancy measure. This suggests that they must inherently possess the ability reason about long-term temporal effects of actions by encoding the geometry of the envi- ronment. Nonetheless, it remains unclear how to extract such temporal information from a learned world model and leverage for credit assignment.
In this paper, we formalize how world models encode world geometry. Specifically, using tools for optimal transport, show that long-term temporal dependencies can be explicitly recovered from the velocity field of a learned occupancy model trained with flow matching (Liu et al., 2022b; Lipman al., 2022). Building on this insight, we propose Occupancy Reward Shaping (ORS), reward-shaping method that extracts global goal-reaching information from the dataset occupancy measure into a scalar reward while provably preserving the optimal policy. We demonstrate that ORS significantly enhances performance, improving 2.2x over its sparse-reward base algorithm and achieving 24 -248% improvements over state-of-the-art methods on 13 challenging locomo- tion and manipulation tasks from OGBench (Park al., 2024a). Furthermore, ORS consistently outperforms baselines on tasks that involve controlling actuators in nuclear fusion reactor called Tokamak (Char et al., 2023), showing its utility in complex, real-world systems.
2 RELATED WORK
Offiine GCRL. Offline GCRL has been long-standing of research in RL. (Kaelbling, 1993; Schaul et al., 2015), stemming from a necessity to build innovative algorithms taking advantage of its rich metric structures, probabilistic goal-reaching interpretations and sub-goal compositionality. As result, diverse types of algorithms exist in offline GCRL based on behavioral cloning (Yang al., 2022; Hejna et al., 2023), actor-critic methods (Haarnoja et al., 2018; Kostrikov al., 2021), hind-sight goal relabeling (Andrychowicz et al., 2017) as well more specialized approaches based on metric learning (Wang et al., 2023a; Reichlin et al., 2024; Park et al., 2024c), contrastive rein- forcement learning (Eysenbach et al., 2022), dual RL (Ma et al., 2022; Sikchi et al., 2023b) and hierarchical RL Sahane-Sane et al., 2021; Park et al., 2023; Zhou & Kao, 2025).
Hicrarchical RL methods (Chane-Sane et al., 2021; Park et al., 2023) address the challenges posed by long- -orizon,,,,, sparse-reward tasks with a two-level policy structure where the high level policy predicts sub-goals that low-level policy learns reach. Methods such as Ahn al. (2025) address this challenge by learning an n-step critic (De Asis et al., 2018) to effectively reduce the temporal horizon. Our approach addresses this challenge using reward shaping for effective credit assignment, making ORS complementary to the methods discussed above.
Reward Shaping. The idea of using reward shaping to facilitate learning dates back to early RL search and applications (Saksida et al., 1997; Randløv Alstrom, 1998). In their canonical work, Ng al. (1999) proposes potential-base reward shaping (PBRS) that preserves the optimal policy un-
2

---

Published as a conference paper at ICLR 2026
der shaped rewards. Potential functions have been learned using expert demonstrations (Brys al., 2015; Kang et al., 2018), 8), transitions (Harutyunyan et al., 2015), uncertainty (Li et al., 2025) or with discriminators to be used exploration bonuses (Durugkar et al., 2021). Another line of research does not follow PBRS and considers reward shaping as a bonus for exploration (Pathak et al., 2017; Mguni et al., 2021; Wang et al., 2023b; Agarwal al., 2023; Ma et al., 2024; Zheng et al., 2024) and as curriculum learning (Andrychowicz et al., 2017; Eysenbach et al., 2022; Zheng et al., 2023).
Hartikainen et al. (2019) estimates local temporal distance to provide a dense reward for credit assignment in online GCRL. We address the challenge of learning shaped rewards for credit assign- ment in Offline GCRL. The closest work to ours, Mezghani al. (2023), uses both a local reward computed with local temporal distance classifier and "global" reward computed using shortest path search on a graph constructed using the local distance classifier. While graph-based methods scale poorly with task complexity, we learn single reward function capable of directly encoding global, long-horizon goal-reaching information for credit assignment by leveraging the occupancy measure. Our experiments show the effectiveness of ORS over prior state-of-the-arr work.
BACKGROUND
3.1 NOTATION
Coal-Conditioned Reinforcement Learning: We consider an infinite-horizon controlled Markov Process (a MDP (Puterman, 2014) without rewards) defined by (S, p. ?) with state space S and action space A. S x A x S + R denotes the transition function while μ denotes the initial state distribution. The discount factor is denoted b E (0, 1). policy (als)) Sx + IK is a probabil- ity distribution mapping states to actions. Each policy induces a conditional state-action occupancy
distribution (s+ 8, a) over future states s d= (s+ 8, s,a) = (1 7) Σ ? Ar-1 ((4++a
st 8, a, IT), which we refer to the occupancy measure for simplicity.
The objective of GCRL is to reach any goal, g E S. from another state 8 E S the minimum number of steps. A goal-conditioned policy [[a)s, 9): S S x A + I maps states and goals to actions. Formally, the aim is to learn the optimal shortest-path policy IT" (a(als that maxi- mizes (--p()))) 3
where t is the timestep, p(r g) (( so, g)p(s1 I so, 00) ... p(s; 5;-1, 1--1)) and 6g is a Dirac delta goal g. Such policy reaches g in the minimum number of steps. In this paper, we focus on offline GCRL. In particular, agents cannot interact with the environment for learning but have access to offline dataset of trajectories D, where each trajectory is r = (Su, ag, $1, ai....,:. 8T). We denote the dataset behavioral policy by (a s) and the dataset occupancy measure corresponding to TD by d (s+ | s.a).
Flow matching aims to learn to sample from a distribution Pdatas given fi- nite number of samples z²(1) 3...,
Flow Matching:
, ((N) E Rdd drawn from Pdata- Flow matching transforms base distribution Po, which is typieally simple Isotropic Gaussian (0, Ia) at time = 0 to the target data distribution Pdata (2) at t = 1. This transformation is defined by a velocity field (t x): [o, 1] x Rdd + Rd, parameterized by neural network and having a corresponding flow (t(t, x): [0, 1] Rd + Rd which is unique solution to the Ordinary Differential Equation (ODE) di (t. x) vo (t, (t) The velocity field is trained to minimize:
min Fang Po0, mpppl)) (-ttttt po(t x.) (01 = 0
(1)
where (1, 1)) denotes uniform sampling between 0 and 1 and 2; = -1 t) 2o + t is the linear interpolation between Ty and x1. On convergence, 0g (t) learns the velocity field which transforms samples from Po to samples from Pdata by numerically integrating the ODE. In this paper, following Lipman et al. (2024); Park et al. (2025), we use linear interpolation between base and target distributions and uniform time sampling. As in Park et al. (2025), we find that the Euler method is sufficient for solving the ODE.
3.2 MOTIVATION
In this section, we motivate our method by examining why credit assignment becomes a challenge when using sparse rewards. We do by analyzing the learning dynamics of the goal-conditioned
3

---

Published as a conference paper at ICLR 2026
value function Vss.g) the GCRL policy T(als,g) must maximize. Given goal g, initial state sus the optimal policy π (a" g), and the optimal trajectory T {So, 81, 8T = g) induced by n², the optimal value function is monotonically non-decreasing along r": vs, 8j E T", (s). 2 (*(8;, g) 3 j 2 i. However, non-monotonicity often arises in practice due to sampling or approximation errors (Park et al., 2023; Ahn al., 2025).
High non-monotonicity in ((s,g) .) can cause the policy to get stuck in sub-optimal regions of the state space, especially when is distant from 8. We hypothesize that in long-horizon tasks, under sparse rewards, credit assignment becomes a challenge because V(s g) exhibits high non- monotonicity, leading poor policies.
We now evaluate this hypothesis empirically using expert trajectories. each with a different length and a unique goal, on the sparse-reward antmaze-giant. navigate task from OGBench (Park et al., 2024a) used in Ahn et al. (2025). Trajectories are visualized in Appendix B.4. We analytically compute V(s,g) as the discounted sum of rewards. By injecting mul- tiplicative noise with variance of simulate er- rors in value estimation: V(s.g) g) r(s,a", g) + y * (V(''g) + V('', ~ NN of), with = 0.99, We evaluate the quality of (s', .g) us- ing ov. the value non-monotonicity error, computed as the proportion of states along expert trajectory where (8++1.9) V(s,,)) g).
0.45
0.35
0.30
0.25
0.20
10
10
10
Ov
Figure Average dy (y-axis) expert jectories on antmaze-giant-naviatt as func- tion of a. (x-axis). dy increases with d., denot- ing increasingly more noisy and non-monotonic Vssgg) under sparse rewards.
Figure 3 plots average by over varying of while Fig. 4 plots (s, over 5 trajectories for de = 0.0005. The results indicate that under sparse rewards, öy high even under small o, and increases with and task horizon. This motivates the central question we aim to address:
How do we design reward functions that encode long-horizon temporal structure and thereby miti- gule value non-monotonicity with effective credit assignment, leading to performative policies?
4 OCCUPANCY REWARD SHAPING FOR OFFLINE GCRL
noise d, 0.0005
First, we detail how we learn a generative model of occupancy measure (Sec. 4.1). In Sec.4.2, we then show exactly how the occupancy measure encodes state geometry (Prop.1). This allows us to extract goal-reaching temporal information from the learned occupancy model into a reward-shaping term (Prop.2). We show that these "occupancy- shaped rewards preserve the optimal goal-reaching policy (Sec.4.3) and finally, summarize method in Sec.4.4.
1
-20
2
V(s, g)
440
-60
-80
-100
0
200
400
600
800
1000
timestep
4.1 LEARNING THE OCCUPANCY MEASURE
Figure 4: V(s.g) g) (y-axis) with sparse rewards vs time (x-axis) over 5 expert trajectories (each line trajectory) from antmaze-giant-nare navigate. V g) becomes increasingly noisy as we move away from goal towards the start of trajectory (moving left x-axis) even under small
The state-action conditioned occupancy measure over future states of the dataset policy πD has the following recursive form (Sutton, 1995; Janner et al., 2020a; Schramm & Boularias, 2024):
d s,) (1 p(s' s)) + d
(st ss'aa",
V(s, a' s',a,'ss''E ED
This recursive form, reminiscent of temporal difference learning, allows learning ( 8.a) parameterized by 0 as mixture over two components: 1. the transition function and 2. target
4

---

Published as a conference paper at ICLR 2026
(s" s', s',a') over subsequent future states under Tp (parameterized by 8-, time-delayed version of 8). This approach naturally stitches together future states under intersecting trajectories in D. To accurately represent the mulli-modality inherent in d and to enable efficient computation of the ORS reward function (explained in Sec. 4.2), we learn dh using a flow matching model with the following loss (Farebrother al., 2025):
Lnow (0) (1 7) Lreat (0) + Limure (s,,a,s',') e
(3)
Lnexr
E s' d'=m~rD
[Iua(t, -(x (01-+0)ll
mo (4)
((0,)
Ltature (0) t
(110((,,, sg(vo (1,8,, a' , mm)ll
(n..((().1)
where L is the standard flow matching loss over the transition function and f regresses the velocity field (t 8, a, mr) towards a bootstrapped estimate of the velocity field of d (s+1s',
ss', a²) represented by ug_ (t, (t,s',a', a', xz). sg represents the stop-gradient operation.
4.2 EXTRACTING TEMPORAL INFORMATION FROM THE OCCUPANCY MEASURE
We now establish precise connection between the occupancy measure and state space geometry. Definition (Shortest-path distance and level sets). Let step" (s, g) denote the minimum number of steps required reach goal g from any state (s, DD We define the level sets:
S {s step' (s,g) k}, k= 0,0,,,
Clearly, Su = Lg). Under deterministic dynamics, for any S₁ with k 2 1, there exists an action a" at state such that the next-state distribution satisfies p(- 8, s.a") S Sx-1-
This characterization of distance from goal allows us to quantify how the occupancy measure en- codes geometry in terms of the squared Wasserstein-2 distance between d and 8g:
Proposition 1 (informal). Under suitable assumptions on goal reachability, dynamics, and dataset coverage, for all (s, a, E DD the average squared Wasserstein-2 distance
W2 (5g. (s {s)) = (Ba-ED(-s)) (6.. ds* (sa)))
is monotonically increasing with respect to the shortest-path layer index k. In particular, for a fixed goal g and any 1, if Sk-1 and 82 Sk, then
W (6g, d (st 1s)) S (2 d= (st 82))
Moreover, for any state-goat pair (8,g), the squared Wasserstein-2 distance minimized by the optimal action:
W2 (6g d (st (s,a') < (2 1 s,a)), Va A. a ~ (-
The assumptions and proof are provided in Appendix A.3 and Appendix A.4. This formulation captures not only how far the center of mass of d (s+ + s, a)) is from g. but also how spread out dT (s+ 8, a)) is with respect to g: if state and state B are both 5 transitions away from g but A has more future states that are farther away from the goal than B. W2()) will be lower for state B.
Intuitively, this means that by defining reward w (W,s, a, = (6 ds s, a)), we can extract the state-space geometry as goal-reaching temporal information into a scalar value that is higher for states with s+ closer to g under D. Unlike prior works (Mezghani et al., 2023; Har- tikainen al., 2019), this reward-shaping term will directly capture global long-horizon information about goal-reacha What remains is to compute W (6,, o | s.)).. While exact compu- tation involves an intractable integral, we show how to estimate y , (s, a, g) using the velocity field corresponding to d3 (s+ s,)):
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Occupancy Reward Shaping.
Input: Occupancy model dg reward function W v // Training the occupancy model:
learning rate
for each iteration = to do
Sample (8, 0, *+) D;
Compute Spretrain (0) (Eq. 8) and update <- 0 - v/ o paratrain (0);
end
for each iteration
to N do Sample (s, a, ,')'')===))))) ~ D:
Compute fion (0) (Eq. and update 0 + (Voo/oootooo
end
// Training the reward function:
for each iteration 1 to N do
10
Sample (8, a, ~ D;
11
Compute Crew (1) (Eq. 5) and update &i tb v Crew (v);
end
The squared Wasserstein-2 distance between 0g and d 8,a.) is upper bounded, up a multiplicative constant O, by the mean squared error of the associated velocity field corresponding to d s, a) and the target velocity corresponding to óg:
Proposition 2.
(62 d (x* 18,a)) CE
zi
|u(1,,, (x1 -xo)ll>
(4)
(0..a) (-Unif((O.])
where I; = tx1 + (1 2)
where v:4, 8, 0, 1T.) is the velocity field corresponding to d== at time The proof is in Appendix A.5. As mentioned in Sec. 4.1, this further justifies choice of using flow matching model to learn d Using the learned dg from Sec. 4. we can now learn r (s, a, g) using neural network vb. Notably, this does not involve running an ODE solver for multiple time-steps:
1 (5)
Lrew (1) = Ee (s,
x1=9 |lva(t, 8, (x1 20 N0, Ia) Un-Unif(
4.3 ORS PRESERVES THE OPTIMAL POLICY
It remains unclear whether using Fw from Sec.4.2, one can learn the same optimal policy as with sparse rewards, is estimated using the dataset occupancy measure d= Theorem 1. Under suitable assumptions on goal reachability, dynamics, and dataset coverage, the greedy goal-conditioned policy: greedy (a s,g) arg max aEA
where (s, coincides with the optimal shortest-path policy n² (a s.
0. denotes the optimal action-value function induced by the reward W (s, a, a,g),
Please refer Appendix A.6 for the proof. ORS thus enables efficient use of the rich goal-reaching information present in the dataset occupancy measure without having to estimate the new occupancy measure at each intermediate policy improvement step.
4.4 METHOD SUMMARY
ORS has 3 stages: 1. Train the flow matching occupancy model dh using Eq. 3; 2. Train the reward function W using Eq. 5; and 3. Train a goal-conditioned policy using any offline GCRL algorithm that uses TD-leamnnn critic. Stages and that involve learning d and r are summarized in Alg. 1. Details architecture and hyperparameters are provided in Appendix B.6.

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
In this section, we present an extensive empirical analysis of ORS across a variety of challenging long-horizon offline GCRL tasks. We then conduct detailed analyses and ablations to dissect key design choices underlying our algorithm.
Tasks: For our empirical analysis, we primarily use OGBench (Park et al., 2024a), benchmark specifically built for evaluating offline GCRL algorithms. We choose OGBench over older bench- marks such Fu et al. (2020); Tarasov al. (2023) for its task diversity, challenging oang-hoiizon sparse-reward tasks that are unsaturated as of 2025, and comprehensive multi goal evaluation.
OGBench tasks are broadly categorized into 4 types: maze navigation, cube manipulation, puzzle manipulation, and scene tasks. We select tasks with varying levels of task complexity and planning horizon. from OGBench to be representative of these 4 categories. Within each category, include tasks of varying complexity (e.g., cube-double and cube-triple) and varying dataset quality (c.g., cube-triple-play and cube-triple-noisy). We provide a detailed explanation of tasks in Appendix B.
To demonstrate effectiveness in real-world settings, we evaluate ORS on actuator control tasks in a nuclear fusion reactor called Tokamak. We use sensor and actuator data collected from the DIII-D tokamak located in San Diego, CA, USA. ORS controls 4 actuators that influence evolution of plasma in the Tokamak, system characterized by highly non-linear, stochastic dynamics and complex physics. We evaluate over 3 separate tasks, each tracking certain scalar or profile state variable: fNv (normalized ratio between plasma pressure and magnetic pressure), Electron density and Ion rotation. These quantities critically influence plasma stability and overall power output required for commercially viable nuclear fusion. Please refer to Appendix B.2 for further details.
Baselines: We compare ORS against a set of representative baselines. These include Goal Con- ditioned Behavioral Cloning (GCBC) (Ghosh et al., 2019); Goal-Conditioned Implicit Q Learning (GCIQL) (Kostrikov et al., 2021) and CondiCondi Implicit Value Learning (GCIVL) (Park et al., 2023) which approximate value functions using expectile regression (Newey & Powell, 1987) and recover policies using behavior-constrained deterministic policy gradient (DDPG + BC) and advantage-weighted regression (AWR), respectively (Park al., 2024b). Quasimetric RI. (QRL) (Wang et al., 2023a) learns a specialized quasimetrie goal-conditioned value function with a dual objective. Contrastive RL (CRL) (Eysenbach al., 2022) fits a Monte Carlo goal-conditioned value function using contrastive learning and extracts greedy policy from it.
We also compare ORS to several methods designed for long-horizon sparse- reward settings. Go- Fresh (Mezghani et al., 2023) learns a shaped reward as sum of local reward from tempo- ral distance classifier between states and a global reward computed by shortest-pan -att search on a graph constructed with the local distance classifier. SMORE (Sikchi et al., 2023a) is an occupancy- matching method that uses dual RI. to learn unnormalized densities that reflect the distance to goal.
Hierarchical Implicit Q Learning (HIQL) (Park et al., 2023) trains low-level policy to reach goals sampled from a high-level policy, both trained via a single GCIVL-style value function. Subgoal Advantage- -eeigtted Policy Bootstrapping (SAW) (Zhou & Kao, 2025) improves over HIQI by training a non- oin-hieralcliall goal-conditioned policy by bootstrapping on subgoal-conditioned poli- cies with advantage- -wwatttg-weeighted importance sampling. n-step GCIQL and GCIQL-OTA use different techniques to learn a GCIQL critic using n-step TD learning (De Asis et al., 2018; Ahn et al., 2025). These baselines ensure standalone comparison of method against two key strategies to horizon reduction in RI. Both ORS and Go-Fresh use GCIQL with a Gaussian policy, owing to its popular- ity and simplicity and to ensure fair comparison. We build on the codebase of OGBench and make use of the official implementation for Go-Fresh. All algorithms use hindsight relabeling (Kaelbling, 1993; Andrychowicz al., 20.7). Appendix B.3-B.8 provides additional information on baselines, hyperparameters used and wall-clock time.
5.1 RESULTS
Performance OGBench is measured by average (binary) success rates on 5 test-time goals of each task. We train algorithms to convergence and average the results over 8 seeds. We list the number of training iterations per task in Appendix B.5. Performance on Tokamak tasks is measured as the
7

---

Published as a conference paper at ICLR 2026
Table 1: ORS baselines: Overall average (binary) success rate (%) across the testtime goals seeds per task per algorithm. report 95% bootstrapped C.I. size after the sign.
Dataset
GCBC
GC-IVL
QRL
CRL
GC-IQL
Go-Fresh
ORS (ours)
mntmaze-largennviigee
25 +
18 +
64 + 18
90 +
34 +
88
88
antmaze-giant saaaaae-iiantnaviggaa
0 0
0
9 9
39 t
0
30
56 9
bebbedouble-pla
1
36 3
1
10
40
17 6
45
cube-triple-pla
0
1
010
3
713
18
37
puzzle-4x4-plat
0
13
040
010
26
74 6
70 5
puzzle-4x5-play
0
7
0
120
14
20
20 0
puzzle-4x6-play
0
1
0
4+
12
17
20 20:2 2
scene-play
5
42
S
19
51 4
56 : 10
80 4
antmaze-large-xxplrre
0
8+6
0+0
0+0
1
10
7
cube-triple-noisy
1+1
9+1
1+0
3+1
5
puzzle-4x4-noisy
0
20
0+0
0+0
29 7
50
puzzle-4x6-noisy
0
17 17
0
6
18 +
19
19
scene-noisy
26 +
9+1
1
26
34
40
Mean
2.5
15.9
6.9
13.7
20.0
35.8
44.2
cumulative reward per episode, where reward is the L2 distance between the goal variables to track and the actual achieved quantities. On Tokamak tasks, evaluate 10 test-time goals over seeds. We discuss the results below:
How effective is ORS? ORS achieves the best performance on most tasks, with especially large gains on more complex domains. Table summarizes results over 12 offline locomotion and ma- nipulation datasets, where * --play/*-navigate denote noisy expert datasets and *-noisy/#-explore denote highly sub-optimal datasets (Park et al., 2024a). ORS achieves 2.2x improvement in performance on average over its base algorithm GCIQL that uses sparse rewards. While see that GCIQL/GCIVL struggle on locomotion and QRL/CRL struggle on manipulation, ORS consis- tently outperforms these baselines on both domains. ORS outperforms Go-Fresh on the majority of tasks, demonstrating the effectiveness of occupancy-basee reward shaping shaping. especially in complex tasks such as antmaze-giant-nan navintt and cube-triple-pee characterized by long horizons upto 2000 steps long (Park et al., 2024a). We also see that ORS remains effective even under highly sub-optimal data the #-noisy/" eexplre datasets. Notably, only ORS and Go-Fresh produce non-zero performance Fresh outperforms ORS on this task, we hypothesize that this could be because the additional local rewards used by Go-Fresh are effective in such task.
graph-based based reward that are
antmaze-large-explor Although Go-
How does ORS compare to other strategies for long-horizon, sparse-reward offline GCRL? On average, ORS outperforms next-best method GO-Fresh by 24% and achieves 2.2x and 1.9x high performance than hierarchical (HIQL) and n-step return Opprocchs(GGIIQL-OTA respec- tively. From Table 2, we see that while hierarchical or volicy-bootstraapedd-t
methods like
Table 2: ORS vs long-horizon sparse-reward offline GCRL strategies on OGBench tasks: Overall average (binary) success rate (%) across the 5 test time goals over 8 seeds task algorithm. We report the 95% bootstrapped C.I. size after the sign.
Dataset
HIQL
SAW
SMORE
n-step GCIQL
GCIQL-OTA
Go-Fresh
ORS (ours)
antmaze-large untmawz-lange-navigatt
91
86
22 2
88 1
72 +
4
1+1
30 10
56 5
cube-double-pii
6+
40 +7
2+2
4+3
3 3+2
17 6
45 +
cube-triple-play
3
6
0
1
2 2
18 1:5
37 8
puzzlc-4x4-play
7
17 12
0+0
46 46=5
85
74 4
70 +
puzzle-4x5-pla
8+
8+44
0+0
5+2
19+1
20
puzzle-466-paay
3
8
0
14
scenc-play
38
63 6
8+2
2
42
56
antmaze-large-explore
0+00
15+88
0+0
0+0
0++0
38 10
+
cube-triplo-noisy
2+1
0+0
1+1
2
2
5
4
puzzle-4x4-noisy
3+3
3+3
0+0
0+0
0=0
50 +5
56 +
puzzle-4x6-noisy
2+1
8+66
0+0
12-6
15+3
19 +
19
socnc-noisy
25
33 1
342
2+2
3
34 1
40 4
Mean
20.0
25.8
2.8
12.7
23.2
35.8
44,2
8

---

Published as a conference paper at ICLR 2026
Table 3: ORS long- horizon sparse-reward offline GCRL strategies on 3 challenging Tokamak control tasks: Overall average cumulative return across testtime goals over seeds per task per algorithm. report the 95% bootstrapped C.I. size after the sign.
Task
HIQL
SAW
SMORE
n-step GCIQL
GCIQL-OTA
GO-Fresh
ORS
Tokamak By
45.26 + 3.83
-51,39 8,16
56.06 + 12.71
-88.7 + 44.3
-67.73 9.73
-48.20 + 11.72
44.76 +: 8.42
Tokamak density
-9.88 = 7.5
-11.0 + 2.3
-51. 10.8
-25.6 = 4.1
-27.1 + 2,5
-38.6 12.6
-26.9 + 6.6
Tokamak rotation
-24.4 1.3
-66.4 + 5.1
-2855 + 6.7
-57.4 + 46.6
-30.1 + 6.9
28.9 + 7.4
.22.4 + 0.6
HIQL and SAW are effective on antmazee locomotion but struggle on manipulation tasks. SMORE largely ineffective with a mean performance of just 2.8% while n-step return methods perform poorly on noisy datasets. In contrast, the strong performance of ORS with simple non-hierarchical policy underscores the effectiveness of occupancy- based reward shaping.
Is ORS effective in real-world tasks? We now examine the ability of ORS track goals by con- trolling actuators in real-world Tokamak control tasks. Table 3 compares ORS with basclines specif- ically designed for long-horizon, sparse-reward offline GCRL. While HIQL and SAW perform rela- tively well for rotation control, they perform poorly on other tasks. While SMORE performs poorly overall, both n-step GCIQL and GCIQL-OTA perform well on density control but show poor perfor- mance on other tasks. GO-Fresh performs poorly overall on all tasks; we believe this is because graph-based reward shaping is ineffective tasks with stochastic dynamics. ORS consistently achieves best performance across the Tokamak control, highlighting its effectiveness in complex real-world tasks.
0.0005
Rewards:: Antiaaze Giant
0.4
-100
H X-position
-200
-300
Y-posision
400
-500
Reword
6600
-700-
10
10"
10
1000
timestep
Figure 5: Left: ORS leads to lower average value non non-moootyy dy at lower noise levels (a..) over expert trajectories compared to using sparse rewards or just using (..g) rw (s, g): Center: ORS induces less noisy estimates of (s, g) over expert trajectories even for long horizons; and Right: ORS rewards over 5000 state-action pairs (denoted as dots) for single fixed goal (denoted as x) smoothly decay in magnitude with temporal distance from goal. All plots in this figure are computed over aatmaze-giant-navye
How does ORS influence value learning? We analyze how ORS influences the learning dynamics of the induced value function on antmaze-giant--nvviee
by improving credit assignment. Follow- ing the setup in Sec. 3.2, we analytically compute V(s. g) induced by ORS rewards under varying noise levels o.. Figure 5 (left) reports how the average ōy varies with a. for sparse rewards, ORS, and variant that directly uses ((s.g) g) w (s, g)) Relative to sparse rewards, ORS exhibits dy that is an order of magnitude smaller at lower noise levels. We plot the approximate value func- tion V (s.gg g) induced by ORS over the same 5 trajectories in Sec. 3.2 in Fig. 5 (center), showing estimates that are much less noisy than sparse-reward (s..g) g) over long horizons.
Finally, Figure (right) plots ORS rewards for fixed goal over 5000 uniformly sampled state- action pairs, each state displayed in the figure by its x-y position in the maze. ORS reward magni- tudes decay smoothly with increasing temporal distance from goal, with dark dots corresponding to states (eg. ends of trajectories) very distant from goal.
How do different design choices affect ORS? We now examine some key design choices behind ORS. First, we compare ORS with a baseline called L2 that computes the uses L.2 distance to goal for reward shaping. Table 4 shows that L2 rewards are ineffective and can hurt performance as shown by GCIQL with L2 rewards performing worse than sparse-reward GCIQL.

---

Published as a conference paper at ICLR 2026
Next, we ablate over design choices of the ORS reward function and Q- function by comparing ORS against a version that uses state-only rewards (s( g) named ORS-s and variant that uses the reward function itself as the Q-function, i.e., Q(s, a, Tw (s, a, named ORS-r. We keep the base algorithm the same.
Dataset
ORS
1,2
GCTQL
antmazz-giant niviga
56
3
cube-triple-paay
3
3 1
7
Table 4: ORS vs L2 rewards
In experiments on the manipulation task scene-play in Fig. 6, ORS performs best, showing the impor- tance of learning Q-function that is cumulative sum over rw (s, 0, 9), especially when the task volves stitching over trajectories to learn the optimal Q* We hypothesize that the poor performance of ORS-r is due to noisy estimates of the Q function, as evidenced by our analysis Fig. 5.
soane-play
100
Overall Success Rate
80
ORS
ORST
ORS-s
40
20
We also ablate the performance of ORS over vary- ing values of the expectile parameter x of GCIQL which is crucially linked to the density of re- wards (Kostrikov et al., 2021). The results in Table show higher performance at lower in locomotion tasks. We hypothesize that this is due to the rich learning signal from dense rewards of ORS. In manipulation tasks, task complexity increases, the x at which get best performance increases.
8.0
0.2
014
0.6
0.8
133
Learning
1e6
Figure 6: ORS ORS-s vs ORS-r over testtime goals 8 seeds on scene-play (error bars show 95% bootstrapped CI)
CONCLUSION
Dataset
k=0.6
k=0.75
==0.9
antmaze autmaze-giant-in giant-navigate puzzle-4x4-play
56 1
1 1
In this paper, we show how generative world models implicitly capture world geometry and introduce Occupancy Re- ward Shaping, a novel reward-shaping method for offline GCRL that utilizes generative occupancy model to extract global goal-reaching information into a goal-conditioned re- ward function. Unlike prior work that relies graph-building. ORS scales effectively with task complexity and enables learning performative policies over diverse datasets. ORS is simple to im- plement on top of existing GCRL algorithms and achieves state-of-the-art results over challenging lasks, including real-world Tokamak control.
70 +
21
scene-play
70
44
71
cube-triple-paay
4
+
12
37
+
Table 5: Performance of ORS over varying values of x.
Limitations. Reward shaping methods, by virtue of being sample-based, less effective in long- horizon tasks with highly combinatorial state spaces and very few ways to achieve task success (e.g. large multi-step combination lock puzzle). In such regimes, particularly in the offline setting, even ORS rewards may provide limited signal since the occupancy measure has a high Wasserstein distance from goal for almost all states. This challenge could potentially be addressed by learning an occupancy measure over a filtered set of "useful" future states and by augmenting the ORS reward with local rewards that capture short-range state dependencies.
ACKNOWLEDGEMENTS
We would like to thank Grace Liu and Mahsa Bastankhah for reviewing the paper. This material based upon work supported in part by the U.S. Department Energy, Office of Fusion Energy Sciences, under Award DE-SC0024544 and on work supported by the National Science Foundation under Award No. 2441665. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
LLM usage:
We used LLMs aid in LaTeX formatting while writing the paper.
Reproducibility Statement: We make our code available on Github. To ensure reliable compar- ison, each RI. algorithm is evaluated over 8 random seeds and 95% bootstrapped CIs reported
10

---

Published as a conference paper at ICLR 2026
eπror bars in tables and plots. Additional information baselines is provided in Appendix B.2- B.6. We state this information in Sec. 5.1 of the main paper. All proofs are in Appendix and assumptions are stated in Appendix A.3. We state this in Sec. 4.2 of the main paper.
REFERENCES
Joseph Abbate, Rory Conlin, and Egemen Kolemen. Data-driven profile prediction for diiidd Nu- clear Fusion, 61(4):046027, 2021.
Siddhant Agarwal, Ishan Durugkar, Peter Stone, and Amy Zhang. f-policy gradients: general framework for goal-conditioned using f-divergences. Advances in Neural Information Process- ing Systems, 36: 12100-121, 2023.
Hongjoon Ahn, Heewoong Choi, Jisu Han, and Taesup Moon. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning. arXiv preprint arXiv:2505.12737, 2025.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems. 30, 2017.
Tim Brys, Anna Harutyunyan. Halit Bener Suay, Sonia Chernova, Matthew Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, pp. 3352-3358, 2015.
Elliot Chane-Sane Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pp. 1430-1440. PMLR, 2021.
lan Char, Joseph Abbate, László Bardóczi, Mark Boyer, Youngseog Chung, Rory Conlin, Keith Er- ickson, Viraj Mehta, Nathan Richner, Egemen Kolemen, et al. Offline model-based reinforcement learning for tokamak control. Learning for Dynamics and Control Conference, pp. 1357-1372. PMLR, 2023.
Ian Char, Youngseog Chung, Joseph Abbate, Egemen Kolemen, and Jeff Schneider. Full shot pre- dictions for the diii-d tokamak via deep recurrent networks. arXiv preprint arXiv:2404. /2416, 2024.
Kristopher De Asis, Harrande G Holland, and Richard Sutton. Multi-step reinforcement learning: unifying algorithm. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances neural information processing systems, 32, 2019.
Ishan Durugkar, Mauricio Tec, Scott Nickum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 14:8622-8634
2021.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learn- ing as goal-conditioned reinforcement learning. Advances in Neural Information Processing Sys- tems, 35:35603-35666 35620, 2022.
Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni. Rémi Munos, Alessandro Lazaric, and Ahmed Touati. Temporal difference flows. arXiv preprint arXhv:2503.098/7, 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning reach goals without reinforcement learning. 2019.

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. Pmlr, 2018.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprint arXiv:1907.060 1907.09222.... 2019.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery, 2020. URL https://arx:: org/ hogg/bb//a 7907..0 08225.
Anna Harutyunyan, Sam Devlin, Peter Vranex, and Ann Nowé. Expressing arbitrary reward func- tions as potential-based advice. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Joey Hejna, Jensen Gao, and Dorsa Sadigh. Distance weighted supervised learning for offline inter- action data. In International Conference on Machine Learning, pp. 12882-12906, PMLR, 2023.
Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: IEGi::600.000 08415, 15, 2016.
Michael Janner, Igor Mordatch, and Sergey Levine. Gamma-models: Generative temporal difference learning for infinite-horizon prediction. Advances in neural information processing systems, 33; 1724-1735, 2020a.
Michael Janner, Igor Mordatch, and Sergey Levine. Generative temporal difference learning for infinite-horizon prediction. arXiv preprint arXiv:2010. (444:90900.144666, 2020b.
Zhengyao Jiang, Tianjun Zhang. Michael Janner, Yueying Li, Tim Rocktäschel. Edward Grefen- stette, and Yuandong Tian. Efficient planning in a compact latent action space. arXiv preprint arXiv:2208. 10291, 2022.
Leslie Pack Kaelbling. Learning achieve goals. In ICCAI, volume 2. pp. 1094-8, 1993.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations, In International conference on machine learning, pp. 2469-2478. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXi::2110.06166 2021.
Mingxuan Li, Junzhe Zhang, and Elias Bareinboim. Automatic reward shaping from confounded offline data, arXiv preprint arXiv:2505. 1478, 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hame Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Yaron Lipman. Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.06264, 2024.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.00 108299, 2022a.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.088 03003, 2022b.
Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma. Flow-based policy for online reinforcement learning. arXiv preprint arXiv:2506.1288 /28/1, 2025.
Haozhe Ma, Zhengding Luo, Thanh Vinh Vo, Kuankuan Sima, and Tze- Yun Leong. Highly efficient self-adaptive reward shaping for reinforcement learning. arXiv preprint arXiv:2408.000 03009, 2024.
12

---

Published as a conference paper at ICLR 2026
Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline coal-conditioned reinforcement learning via f-advantage regression. Advances in neural information processing systems, 15:310-323, 2022.
Marlos C Machado, Andre Barreto, Doina Precup, and Michael Bowling, Temporal abstraction in reinforcement learning with the successor representation. Journal of machine learning research, 24(80):1-69, 2023.
Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric, and Karteek Alahari. Learning goal-conditioned policies offline with self-supervised reward shaping. In Conference on robot learning. pp. 1401-1410. PMLR, 2023.
David Mguni, Jianhong Wang, Taher Jafferjee, Nicolas Perez-Nieves, Wenbin Song, Yaodong Yang, Feifei Tong, Hui Chen, Jannghhnng Zhu, Yali Du, al. Learning to shape rewards using a game of switching controls. arXiv preprint arXiv:2103.09159 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (187555))559-533, 2015.
Whitney K Newey and James L. Powell. Asymmetric least squares estimation and testing. Econo- metrica: Journal of the Econometric Society, pp. $19-847, 1987.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application to reward shaping. leml, volume 99, pp. 278-287. Citeseer, 1999.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine Hiql: Offline goal- conditioned rl with latent states actions. Advances in Neural Information Processing Systems, 36:34866-3488 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092. 2024a.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline ri? Advances in Neural Information Processing Systems, 37:79029-79056. 2024b.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. arXiv preprini 1555 2024c.
Scohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arXiv:2502.02538, 2025.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Jette Randløv and Preben Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. In ICML, volume 98, pp. 463-471, 1998.
Alfredo Reichlin, Miguel Vasco, Hang Yin, and Danica Kragic. Learning goal-conditioned policies from sub-optimal offline data via metric learning. arXiv preprint 1rXiv:2402. 2024.
Lisa M Saksida, Scott M Raymond, and David Touretzky, Shaping robot behavior using principles from instrumental conditioning. Robotics and Autonomous Systems, 22(3-4):231-249, 1997.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametri topological memory for navigation. arXiv preprint w:Xiv:1803.0053 2018
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
13

---

Published as a conference paper at ICLR 2026
Liam Schramm and Abdeslam Boularias. arXiv:2407.12163, 2024.
Bellman diffusion models.
arXiv preprint
Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Nickum. Score models for offline goal-conditioned reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023a.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.0855 2023b.
Richard S Sutton. Td models: Modeling the world at a mixture of time scales. In Machine learning proceedings 1995, pp. 531-539. Elsevier, 1995.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36:1 11111112909100 11620, 2023.
Michael L. Walker, Peter De Vries, Federico Felici, and Eugenio Schuster. Introduction to tokamak plasma control. In 2020 American Control Conference (ACC), pp. 2901-2918 8. IEEE, 2020.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforce- ment learning via quasimetric learning. In International Conference on Machine Learning, pp-
36411-36430. PMLR, 2023a.
Yiming Wang, Ming Yang. Renzhi Dong, Binbin Sun, Furui Liu, et al. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. Advances in Neural Information Processing Systems, 01138786-38791 2023b.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint 2rXiv:2202.04477 2022.
Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned r!? In International Conference Machine Learning, pp. 39543-39571. PMLR, 2023.
Rui Yu, Shenghua Wan, Yucen Wang, Chen-Xiao Gao, Le Gan, Zongzhang Zhang, and De- Chuan Zhan. Reward models in deep reinforcement learning: survey. arXiv preprint arXiv:2506. 15421, 2025.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pp. 2730-2775. PMLR, 2022.
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. arXiv preprint arXi::2310.0114,, 2023.
Chongyi Zheng, Jens Tuyls, Joanne Peng, and Benjamin Eysenbach. Can a misl fly? analysis and ingredients for mutual information skill learning. arXiv preprint arXiv:2412.08021, 2024.
John Zhou and Jonathan Kao. Flattening hierarchies with policy bootstrapping. arXiv preprint 14975 2025.
14

---

Published as a conference paper at ICLR 2026
Appendices
A PROOFS
We introduce notation in Sec. A.1. We then provide common setup used for all proofs in Sec. A.2. We introduce the assumptions used in Sec. A.3. Sec. A.4, A.5 and A.6 provide the proofs for Propositionl, Proposition2 and Theoreml, respectively.
A.1 NOTATION
Shortest-path distance: We step" (s,g) g) to denote the minimum number of steps to reach from We define level sets:
S4 (s S step" (s. k}, 0, 2.
Clearly, Sp = {g}. Under deterministic dynamics, there exists an action a at state s such that p(- 88, a²) Sk-1 whenever & E Sk. k 1.
A.2 COMMON SETUP PROOFS
The squared Wasserstein-2 distance between two probability measures μ and v is defined as:
/ ||= yll2dX v)
(2( inf xeA(a.))
where Ma, u) is the of all joint measures with marginals μ and
We consider the special case where one measure a Dirac delta, l 6g. located at single point g- In this case, any valid transport plan must move all mass from the point g. This removes the need for optimization, as the transport plan is uniquely determined to be the product measure A(z, y) = 5g (z) (z))
Substituting this unique plan into the definition yields:
(5,, / lly (z)(()
f Ily - g1l2v()ddy (by the sifting property of 6z(x))
The result is the expected squared Euclidean distance from the point g to the distribution
By replacing the general measure with the specific transition probability (s s, a) and renaming the integration variable, we obtain the final form:
(b d (st |s,a)) / ||st g (ds |s,a) M
With this setup, we now define
(ss g) 2 M ( == f (5(5 ()) (ds ss,a).
Then
( = - a,g).
Recall the expanded form of d=:
d a, = (1 - p(s' 8, 8, d ss','), V(s,a,8,a'))
Now, substitute into the definition of A as an integral, and under the assumption of deterministic dynamics:
/ (sstt, (lsttg) r (ds+ 1 s,
M (s,
/ ((st .9) [11 7) (st 8') + ydtt (st |s',a'] dst
15

---

Published as a conference paper at ICLR 2026
Using the linearity of the integral, decompose the above expression into two terms:
(M = - -0))(66 + +y (s) 1'), (1 7) + > (s'
(s*
It is notable that (s, a.g) resembles squared coal-conditioned Wasserstein-2 distance ana- logue of the successor representation (Machado et al., 2023).
We also define the state-only version "M (s, g) = (a M as,
A.3 ASSUMPTIONS
This section lists the assumptions we use to develop our proofs:
Assumption A.1 (Deterministic dynamics). We assume that the environment dynamics are deter- ministic: For any state S and action A. the state transition probubility pls' 8,0) is point mass distribution, concentrated on the unique successor state {(8, That is,
pl s, (((,a))())
Remark A.1. This is common assumption made for the sake of analytical tractability in founda- tional reinforcement learning theory. It allows for a clear analysis of the core properties of value functions and policies without the complexities introduced by stochasticity. This assumption serves as a standard basis for theoretical analysis in related work, for instance, in Hartikainen et al. (2020).
Assumption A.2 (One-step reachability).
vk 2 1, Vs E Sk. aa such that s' s'|s, |ss a" Sk-11
Remark A.2. This assumption is inherent to the definition of well-posed shortest-pan patt problem. simply formalizes that the goal is reachable and a shortest path exists from all relevant states. Assumption A.3. The potential function ((8, 9. ) strongly monotonic with respect to the shortest- path distance step" (s, g). For any two states Sas Sb. step" (sa: stepp ($b. g), then
((Sa: 9) S ((ss.g).
Furthermore, for any 2 L, 81 Sk-1 and 82 E Sg. there exists a constant Ao > such that
(s2. g) 4 (51,9) A
Remark A.3. This assumption ensures that the signal provided by the Wasserstein distance is strong and informative, providing non-trivial cost increase when moving away from the goal.
Assumption A.4 (Layer monotonicity of TD). We assume the offline policy TD is consistent with the shortest-path layer structure. For any k 1. if E S and s2 E Sg. their successors under TD. $1 = f(1 (((()) and & = (82, TT (82)) must maintain their relative layer ordering. That is,
step" ($1, 9) S step" (5/2
Remark A.4. Assumption A.4 is a condition on the quality of the offline data, which a standard requirement for obtaining theoretical guarantees in the offline RL setting (Zhan al., 2022).
16

---

Published as a conference paper at ICLR 2026
A.4 MONOTONICITY OF GOAL-CONDITIONED REWARD FUNCTION (PROPOSITION 1)
Proposition. Under suitable assumptions on goal reachability, dynamics, and dataset coverage, for all (,0, E D, the average squared Wasserstein-2 distance
(6 (s* |s)) -B--r((1ss)
WW3 (6g. dst s,a))
is monotonically increasing with respect to the shortest-path layer index k. In particular, for fixed goal g and any 1, if S and 82 Sk, then
W2 (6g (s+ (1) I ( d (st 822)).
Moreover, for any state-goal pair (8,g), the squared Wasserstein-2 distance is minimized by the optimal action:
(6 (s,a") d |s,)),, Ya E A. a' ~ ~ (.
We prove the proposition in two parts: We first prove that the average W? distance is monotoni- cally decreasing in shortest-path distance to goal and then show that W (6). d s,a")) < W d (s+ s; a)) for any (s, a. g).. Recall that we define the shortest- path distance in terms of the layer index and show that W3 (6g. (0g. (s+ [,,)) Ma (,,, g) in Sec. A.2. We can now re- re-write the first part of the proposition as:
(Layer Monotonicity of M* Under Assumptions A.I, A.3 and A.4, the function MD is mono- tonically increasing with respect to the shortest-path layer index k. For goal g and any k L, if 81 Sk-1 and $2 E Sk, we have:
RT (81, S 1M*((M22
It is important to note that the indices & here are such that h = 0 refers to the goal state, i.e. index defines a state closer to goal than index k.
Proof. Let define the Bellman operator T which maps function M new function Tho M: (7 () = (1 - ())))))( g) + (Ts,ss))))
We prove this statement by analyzing the properties of the Bellman evaluation operator 7D asso- ciated with the policy TD-
The function R is the unique fixed point of this operator, satisfying M 7 (M The ex- istence and uniqueness of this fixed point, and the convergence of value iteration (M,+1 = TDD M,) to it from any starting function Mo. guaranteed by the Banach Fixed-Point Theorem. This is because the operator 7D is a contraction mapping due to the discount factor < 1.
To show this, we take any two functions Ma; M2, and measure the distance between their outputs under the supremum norm MM Molloo max, (Ma(s) - Mo(s)]:
TD M. TD Mo max (1 7) .9) 7Ma(6'. (11-) (s's +MMo(5' y (s MM(s',9) where s' f(s, (D())
max Ma(2,g) Mo(2.9) =ES
= M, - Moll
Since < 1, the operator is contraction. Therefore, the sequence converges to the unique fixed point FD
Now, we prove that the operator TD preserves the property of monotonicity. Let M be an arbitrary function that is monotonically increasing with respect to the layer index k. We must show that the resulting function Th M is also monotonic.
Let $ E S and $2 E St- Let their successors be s = (TTD ()),2.9).
and s/2 = f(22, To (82))
We want to show that
17

---

Published as a conference paper at ICLR 2026
Consider the difference:
(T*PMM((2.)) (T () (1 - 9) - - g)1 + 7 - M($1 g)].
By Assumption A.4, we have step" ($) S step" ($/2, 9). Now we analyze the two terms in the difference:
. By Assumption A.3, the condition step" (5'1.9) < step" g) directly implies that ( I ( g)] The first term ( g) (5 g)] is non-negative.
By our premise for this step, the function M monotonic. Since (51. S d' (s2, g). follows that (s1, 9) I M($2.9). Thus, the second term (M(s) M(1 g)l is also non-negative.
Since both terms are non-negative, their sum is non-negative, and thus (T (P S (T),, g). This proves that the operator 7 preserves monotonicity.
The value iteration process starts with initial value function Mo(s) 0 for all s. The zero func- tion trivially monotonic. Since the operator T preserves monotonicity, the entire sequence of value functions (M, MM, .M.... generated by M,+1 T M, consists of monotonic functions. The final value function = is the limit of this sequence, and the limit a sequence of monotonic functions is also monotonic.
Therefore, M is monotonically increasing with respect to the layer index k:
It follows that average W3 distance (a-=p((1s) W3 (6g. d s.a)) is monotonically decreasing in thortest-path distance towards goal g.
We now prove the second part. Similar to part 1, we can re-write in terms of RD as follows:
2 (Optimal actions under M TD (s, a, (.g)). Under Assumptions A-1.A.4 and the layer monotonicity of M MT', "D for any state s E S₁ with & 2 1, any optimal, poortest-pat-indueee and any non-optimal action abad, the function RDD is strictly smaller for the shortest-path action. That is,
action a" ~ T" (.|s,, g),
M (8, a" .g) M Obads 9).
Proof. = f a²) and Sbod = f(s, (bod). By definition, s* E S and Sbod E Sj for some j 2 We analyze the difference AM Map (s, abad. 9) M0 (s, a' ,g):
AM (1 7) ( Shads f) 7 7 (Shads 9))
[ 7) (0008=,,0 ,.9) + 7M (s" [9)
(1 () (1 9) (53", ,9] + 1MM" (Sbad, 9) M "D (s", ,))
Term For Term since s* E Sk-1 and Sood Sj with j 2 k. by Assumption A.3, we have ((Sbad: g) 2
Term
4(s", .g) + Ap. Thus, Term 1 is strictly positive and bounded below by (1 A) > 0.
For Term 2, since s* Sk-1 and Sbad E Sy with 2 k. by Lemma 1, we have AD (s* .g) S MR (Stad, g). Thus, Term 2 is non-negative.
The sum of strictly positive term and non nog-neeative term is strictly positive. Therefore, AM > 0.
It follows that , (s, a=, .g) > p (s, Abad. g) or equivalently. ,: (s, "a"")))))))))) 2 a where can be any action in D state s, which concludes the proof.
A.5 ESTIMATING SQUARED WASSERSTEIN-2 DISTANCE (PROPOSITION 2)
Proposition. The squared Wasserstein-2 2 distance between 8, and (s* s.)) is upper bounded, up to a multiplicative constant C O, by the mean squared error of the associated velocity field corresponding to d d=0(s" (s | 8, a) and the larget velocity corresponding to &g:
W (8,, d=0 1 (s,a)) & CE
21-8,
|u s, a, xx) (01 -
331-xo0l3" xo>ll3
(6)
2o-N10.11)
(-Uni(((o.])
18

---

Published as a conference paper at ICLR 2026
where n; t (1 t)
We note that 21 - gives us the velocity at (Zo, 21), where of squared Wasserstein-2 distance Sec. A.2. In our case, j corresponds óg and v corresponds to dg (s+ a. As μa ōg. point mass centered at g. the transport plan is uniquely determined to the product measure A(x, s*) = ()(2) (s* a,a) (Sec. A.2). For given (s, a, g), the incquality follows from Lemma 1 ov et al. (2025) by Gronwall's inequality:
9. We start from the definition
inf dEA(Sg: 1 / || - st (A = W3 (8g, d (st ss.a)).
[ (1,8,, (xx - mo)lla (7)
S E
(0, (-Unif((o.1]
A.6 OPTIMALITY OF ORS POLICY (PROOF OF THEOREM 1)
Theorem. Under suitable assumptions on goal reachability, dynamics, and dataset coverage, the greedy goal-conditioned policy:
greody (a 8,9) arg max (s, aEA
where Q" 0. denotes the optimal action-value function induced by the reward w (s, a, g,g). coincides with the optimal shortest-path policy n² 8 s.g).
(s, a, g) is the unique fixed point The optimal action-valus value function, (s, g).. for the reward r r of the Bellman optimality operator:
Q" (s, a, = w (s.a,g) + max ("
The corresponding optimal state-value function is (*(,g) = max. (", greeedy is the policy that greedily maximizes Q*,s,a
n greedy (als, g) = arg max Q' a, arg ( (8, a. - >V (V
Lemma A.1 (Layer-wise Monotonicity of the Optimal Value Function V*). Under Assump- tions A-I-A.4 and layer monotonicity of MDD (Sec. A.4), the value function of the optimal shortest- path policy, (*(,g), g). is strictly monotonically non-decreasing with the shortest-path layer index (i.e., the value strictly higher for states closer the goal). For any l if s1 Sk-1 and 82 E Sx, we have:
(*( - (82. 2 21 1-ー-~~1)4 O.
k-
Consequently, it holds that V (s2. < (*(ss.
Proof. Let n" denote the optimal shortest-path policy. For the sake of clarity, we write that for any state s and goal g, let a" (s) (a"|s, [s, g). We first establish the difference in the one-step cost for taking a shortest-path action from two adjacent layers.
Let Sa E S and s E Sy for any j > Let their successors under n² be S/2 f(an. a" (sa)) Sj-2 and s/ f(8,, a" (st)) E S S Consider the difference in their one-step costs:
M R (st. a" (s)) g) - =0 (sa, a" (sa).
(1 7) t (5 9) + M (5/ 9) (1 71)-))(('.00)) + 7M (5'.. (5 (5a.
(1 (1->)//0 [ g) 4 (5', g)و + 7 TM (sh. - n p (5'.
By Assumption 3 (Strong Monotonicity of Φ). since S', E Sj-2 and s/1 E Sj-1,
1, we have (4(5))
((s'..)) Ag. By Lemma (Layer Monotonicity of M since s E Sy-2 and s/ Sy-1, have [M ( g) - - ( g)l 2 O. Therefore, we have lower bound on the single-step cost difference:
TH (st, a" (st). g) MD (sa, a" (8a). 0) 2 (1 7)
19

---

Published as a conference paper at ICLR 2026
Now, let's consider the two shortest-path trajectories starting from 81 Sk-1 and 82 E Sk.- Let TÍ = ($1.0: $1,1,.... $1,k-1 = 9) be the trajectory starting from $1,0 $1- Let 72 = ($2.0, $2.1..... $2, g) be the trajectory starting from $2,0 $2. By definition of the shortest-path policy, we have 81, E 1 and 82, E Sk-t for & E [0, k - 1].
The value functions the sum of discounted negative costs:
k-2
(*(81,9) =
M ($1, ((11))))
(*2,g Σ
(
1=0
Let's analyze the difference (s(,g) (52,gg))
(*($i, (5(g,
7 M RD 52, a" (52.):
M nD a"
k-2
Σ
M (sL
For each term in the summation (for t E [0, 2)), have 81, E Sk-1-t and 82,t Sk-t- Applying our single-step cost difference result, we get:
M (s2,1, a" (82.)).) M (sL,, a',,,, 2 (1 - 7)
The last term for the 7 trajectory is = ($2 a" ($2,8-1), 10-1 1),9), 9). Since 82 E S1, its successor is g E Sg. We know =D 2 O. So we can lower bound the entire difference:
k-
VW ()) VW (82.9) 2 x- 0+
Σ (11
7)
k-2
= - A.
1 k-1
= (1 - 7) Ao
k- -1
Since and 7 E (0, 1), the term (1 (1-a^-1) ) is strictly positive, As As > O, the entire lower bound is strictly positive. This completes the proof.
Theorem A.1 (Equivalence of the Q-learning Optimum and the Shortest Path). Under Assump- tions I-A.4 and noting that w (s, .g) > w (s, Abad, g) (Sec A.4), the greedy policy with respect to Q is the optimal shortest-path policy n²"
Proof. We must show that for any state (s,g), g). the shortest-path action a" = N (-|s, g) is the unique maximizer of the action-value function Q" a, and is therefore, the same action chosen by goreedy (. s. g). That is, for any non-shortest-path action abad * a":
Q" (s, "es,a"g)) .9) Q" (s, abad g).
Let's analyze the difference AQ Q" (8, a" Q" abads g). Let the successor states be s" f(s, a²) and Sbad = f(s, Abad).
AQ
W (s,a", + + ,9) 9)
Wr (s, abads 9) yV 7V* (sbad: 9))
M (s, a" 9) VV "V" (5,
MMDD abads 9)) 7 V (Shads .))
M (s, Abad. g) = (s, a" .g)
+ + .)) V V*(Sbed 9)
Term
Term 2
20

---

Published as a conference paper at ICLR 2026
We will now analyze the two terms of this expression.
The first term, _M* (s, Abad. 9) - (s,a", by part 2 of the proof in Sec. A.4, this term is guaranteed to be positive.
We now analyze the second term, (s* V (Shads g)]. By definition, the successor state s* lies in layer closer to the goal than Sbad (i.e., d3 (s" ,9) d" (Sbad: 9)). It therefore follows directly from Lemma A. that
V* 9) V*('',
Thus, the term (** V (shad, g) strictly positive.
The difference in Q-values, AQ, is the sum of a strictly positive term and a strictly positive term (Term 2). The sum therefore strictly positive.
2Q>0 0 i. ("(saa", .9) Q*
(s, abads 9)
This shows that for any state 8, the shortest-path action a² ~ n raaaaa (8 g) is the unique greedy action with respect to Q" Therefore, the greedy policy with respect Q" must be the optimal shortest-path policy π².
B
ADDITIONAL INFORMATION
B.1 OGBENCH TASKS
Antmaze datasets: Antmaze involves controlling a quadruped Ant robot with 8 degrees of freedom (DoF) to reach given goal location in the maze. The agent must learn both high-level navigation and low-level locomotion skills purely from offline data. We perform experiments on 3 types of challenging mazes: antmaze-large-navigatt, antmaze-giant- navigate and antmaze-large-explore
antmaze-giant is twice the size of the large and same size as antmaze-ultra in (Jiang al., 2022) designed substantially challenge long-horizon planning capabilities. Both are collected with noisy expert
antmaze-large-navigat and antmaze-giant-naviin SAC policies that repeatedly move towards randomly sampled goals (Park al., 2024a). antmaze-large-explrr features extremely low quality yet high coverage data consisting of random exploratory trajectories.
Cube datasets: The cube environments involve manipulation of cube blocks using a robot arm. We perform experiments on 2 cube-play datasets which are collected using a scripted policy that repeatedly picks a cube and places it in other random locations or on another cube: cube-double-pi
(2 cubes) and cube-triple-play (3 cubes). During evaluation, the agent is required to perform cube moving, stacking, swapping or permutations according to the provided goal configuration. This requires learning generalizable pick-and-place operations with multiple objects and stitching across unstructured data to achieve this. The task horizon also increases with the number of cubes. We also perform experiments on the cube-triple-noisy dataset which involves highly sub-optimal data with noisy transitions.
These environments specifically test out the combinatorial generalization abilities of the agent (upto 2 puzzle configurations and much larger state space due to robotic manipulation on the puzzle) under very long horizons. They require solving the Lights Out puzzle (Park et al., 2024a), which consists of a 2D grid with buttons in cells, where pressing buttons toggles the color the that button and buttons around it. The agent has to achieve appropriate goal configuration of colors by pressing buttons by controlling robot arm. We evaluate algorithms on play datasets puzzle-4x4, puzzle-4x5 and puzzle- 4x6; and noisy datasets of puzzle-4x4 and puzzle-4x6. These are particularly challenging tasks (Park et al., 2024a).
Puzzle datasets:
The scene tasks challenge sequential, long-horizon planning. They in- volve manipulating diverse objects: cube, window, drawer and two locks. scene-play features a dataset collected using a scripted policy that randomly interacts with the objects. At test-time. the agent has arrange objects into given goal configuration. These tasks can involve up to 8 sub-tasks e.g. unlock a drawer, then open it, then put a cube in it and close it again, etc. The agent must therefore be able to stitch together multiple skills to
Scene datasets:
21

---

Published as a conference paper at ICLR 2026
achieve success. scene-noisy involves the same tasks but the dataset is of substantially lower quality.
antmaze
cube
acene
puzzle
Figure 7: OGBench tasks.
B.2 TOKAMAK TASKS
tokamak (Figure 8) is scientific device that uses megnneo-ma fields to confine extremely hot ionized gas called plasma in a toroidal chamber, with the aim of achieving controlled nuclear fusion is considered the most promising experimental design for achieving commercial nuclear fusion at scale, opening up avenues for nearly unlimited clean energy.
We use dataset of raw sensor and actuator data collected from the DIII-D tokamak located San Diego, CA, USA. Each trajectory corresponds to plasma discharge, with control frequency of 25 ms. control task corresponds to tracking certain state variables of the plasma through the duration of discharge. While notice, we evaluated each RL algorithm using rollouts from a thoroughly tested and accurately learned model of plasma dynamics (Char et al., 2023; 2024) evaluated based on how closely it tracks a given goal state of the plasma. The reward at each time- step during evaluation corresponds the L2 distance between goal variables to track and the actual quantities achieved through control.
not able to get experiment time the reactor short
proxy. Each RL algorithm was
Position Control Coils
Ohmic Coll
Toroidal Field Coil
Ohmic
Current
Plasma Current
Vacuum Vessel
Plasma
Poloidal
Toroidal Magnetic Field
Toroidal Field Coll Current
Magnetic
Field
Figure Tokamak fusion reactor from (Walker al., 2020).
22

---

Published as a conference paper at ICLR 2026
We evaluated each algorithm over 10 goals. The offline dataset size roughly 50k samples after data cleaning and processing. The state and action spaces are outlined in the following table:
Category
Variables
Scalar States: By. Internal Inductance, Line Averaged Density,
Loop Voltage, Stored Energy
State Space
Profile States: Electron Density, Electron Temperature, Pressure,
Safety Factor, Ion Temperature, Ion Rotation
Action Space
Power Injected, Torque Injected, Total Deuterium Gas Injection, Total ECH Power
Please refer to Abhate et al. (2021); Char et al. (2023; 2024) for detailed explanations of the specific variables and actuators.
B.3 ADDITIONAL INFORMATION ON BASELINES
For baselines provided in the OGBench (Park et al., 2024a) repository, we use the best hyperparameters provided in the paper. These include: GCBC, GCIVL, GCIQL, QRT, CRL and HIQL.
GCIQL: GCIQL is a goal-conditioned variant of Implicit Q Learning (Kostrikov et al., 2021) that fits the optimal Q* using expectile regression (Newey & Powell, 1987). GCIQL learns Q and V as follows:
focrqu (V) = ((s.a))PP(sa) pi (ala) [ (Q a, -
LecIQi ('(.,,,"''pPP (a.a,)) ( (r (s, + >V -(s.aa)) a(s,
Here, Q is the target Q function (Mnih et al., 2015), is the expectile and w (s, a, g) is the ORS reward. For policy extraction, we use deterministic policy gradient with behavioral regularization (DDPG BC) (Fujimoto & Gu, 2021):
JDDPG+BC(T)) = E "P (s,0), a a (Q(s, T" (8, + a log ((a [,9)]
where (* g) = Bans(als [a]. We use double-Q learning and normalize the Q values
before policy extraction.
Go-Fresh (Mezghani et al., 2023): Go-Fresh involves 3 stages: Training the R-Net, neural network that learns local temporal distances between states, using R-Net and R-Net embeddings to create a semi-paramntric graph of the offline dataset and then training policy. While training a policy, every 1000 iterations of training, Go-Fresh creates a new dataset where each tuple corresponds to randomly sampled (s, g) pair from the dataset.
We noticed in initial experiments that even when Go-Fresh is trained using the same GCRL algorithm as ORS (GCIQL), overall success-rate was close to 0 across tasks. To ensure a fair and unbiased comparison, we modified the Go-Fresh codebase have the same goal sampling strategy from OGBench (detailed in Appendix D of (Park et al., 2024a) and in the last item of this sub- sub-section) used for all other baselines. This substantially improved performance and is required to produce non-negligible performance, as reported in Tables and 2. Furthermore, noticing that Go-Fresh is tested on relatively simpler tasks compared to ours in their paper (Mezghani et al., 2023), modify the following hyperparameters to account for the increased task difficulty, in addition to using best values for other byperparameters:
We used a memory capacity of 5000 on all tasks which we found to be sufficient (the graph building stage consistently produced graph with less than 5000 nodes). We increased the R-Net size to an MLP of 64 units (we did not see an improvement in classification accuracy 10 on all tasks. We also see that
for large sizes) and used a local distance threshold weighing the local reward by factor of 5 before summing it with the global reward gave better performance.
Goal sampling for all algorithms: The same goal-sampling method used for all base- lines in this paper as well as ORS and all variations. Goals were sampled in three different ways according to probabilities from categorical distribution defined follows:
23

---

Published as a conference paper at ICLR 2026
-Peu (gls): Dirac delta distribution centered at the current state
PPeraj (g|s): goal sampled from future states of from the same trajectory with uni- form sampling.
PPand (g|s): goal is uniformly sampled from D.
We provide information on the specific values used in Sec. B.6 and B.7 and Table 6.
n-step returns: We implement two ways doing n-step returns in the critic with sparse rewards (where r(s.g) 1(s + g) and t is the MDP timestep):
1. n-step GCIQL: (+,,, at, g) = - r(8tti; 9) + 7 ( क
2. GCIQL-OTA (based the core idea in (Ahn et al., 2025)):
Q(ac Q.. g) (1 # g) + ( x (tt
GCIQL as the base algorithm for n-step return algorithms. We provide the specific hyperparameters used for each task in Table 6. Please refer to Sec. B.7 for information all other common hyperparameters.
As evident,
Table 6: Specific hyperparameters for best performance with n-step GCIQL and GCIQL-OTA
Dataset
(BC-coeflkcient)
< (Expectile)
Actor (p Prana)
D
D
puzzle-4x--ply cube triple play
0.9
0.995
U)
3.0
0.9
0.995
(0,
untmaze-large-aavii
unimure giant navigale
0.3
0.9
50
0.995
(0, 1,
puzzle- nxi-n noisy
0.995
1,
cube-triple-noisy
0.03
0.9
50
0.995
(0. 1,0)
antmaze-lagge-xplod
0.01
0.9
0.995
(0,0, 1)
24

---

Published as a conference paper at ICLR 2026
B.4 TRAJECTORY VISUALIZATIONS. ANALYSED TRAJECTORIES
2
3
4
Figure 9: Expert trajectories of varying lengths saimaze-giant-naiieee used for analysis in Sec. 3.2 and
Sec. 5.1. The green dot represents the starting position and the represents the goal.
B.5 TRAINING ITERATIONS
Table 7: Training iterations per task. All algorithms trained for the number of iterations.
Training Iterations
cube double play
puzrle-4x4-play
IM
scene play
1M
puzzle-4x5- 4x5-paay
puzzle-xx--play
IM
cube-tripleplay
antmaae large navigule
3M
antmaze giant-navignen navigate
puzzle-44 fodee-4x--na 4xd-noisy
IM
scene-min noisy
puzzle-4xt muzzle-4x6-noisy
IM
cube triple noisy
antmaze- antmare-lurgo-eppee large-exlo---
B.6 OCCUPANCY AND REWARD MODEL. HYPERPARAMETERS
Table 8: Occupancy and reward-model hyperparameters.
Hy Hyperparameter
Value
Gradicnt steps
2M
Optimizer
Adam (Kingma & Ba, 2014)
Leamingg rate
0.0003
Batch size
256
Architecture
MLP
MI size
(512, 512, 512, 512,
Nonlincarity
GELU (Hendrycks & Gimpel, 2016)
Layer Rormalization
True
Targel net updale rale (occupancy model)
0.005
Discount factor yy (occupancy model)
Flow steps
22 (antmaze giant navigaaa 55 (antimaze luntmaaoe-lnge--uvigul nuviaaa.,, 45 (other tasks)
Reward (peur> Peract P Prana) ratio for goal sampling
ratio
(0.2, 0.5, 0.3)
We added a warm-start stage without bootstrapping to the occupancy model by training to predict future states s distributed according geometric distribution over trajectory timesteps t, Geom(1 - 7) starting the current state s:
Ppretrain (0) = EE =my = to Geeom 1))) (1,,(t, - (x1 =o)1 a00-N(0, (0. Ia) Uniif(())))
(8)
for the first 1M epochs and then training it using the bootstrapping loss in Sec. 4.1 for next next IM epochs, however did not see any consistent improvements from doing this.
25

---

Published as a conference paper at ICLR 2026
B.7 POLICY TRAINING HYPERPARAMETERS
We first list hyperparameters common all algorithms in Table 9:
Table 9: Common hyperparameters for policy training.
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba, 2014)
Learning rate
0.0003
Batch size
1024
Architecture
MLP
MI.P size (Actor and Critic)
{512, 512,
Noulinearity
GELU (Heendrycks & Gimpel, 2016)
Layer normalization
Target critic update rite
0.005
Critic (peir Piraje P Prang) ratio for goal sampling (0.2, 0.3.033)..33
We then provide specific hyperparameters used for ORS and Go-Fresh (which use GCIQL as the base algorithm train policies) in Table 10:
Table 10: Specifie hyperparameters for policy training: ORS and Go-Fresh
Dataset
(BC-ondlicieet)
(Expectilo)
Actor (y. Pho.j Plana)
rewand scaling for ORS (lincar scaling (r/x))
purdoo-xi-glay
0.6
0.6
0.999
(0, 1.0)
0.75
puzdl 1x5 play
0.6
0.5
0.99
(0, 0.5, 0.5.
0.75
nuxiee-xx--poay
0.6
0.9
0.99
0.5,
0.25
cube-Souble-ploy
cube-trile-ppayy
0.5)
scese scese-play
0.3
0,75
0.99
(1, 0.5, 0.5)
0.75
animare- bige-nniiggtee
0.15
0.6
0.995
(0, 1.11..0 B)
2.0
untmoo-gaant-navigee
0.1
0.6
0.995
(0. 1,
20
puzzle-4s4-nn Aal--44--niss anisy
0.05
puzzle noisy
cube trppee moisy
0.03
0.75
0.99
(0, 0,505.00 0.5)
sceee muisy animaze large explore
0.1
0.75
0.99
(0. ()
0.55
0.155
0.9
0.99
00,0.5. 0.5)
B.8 WALL-CLOCK TIME
The average wall-clock time per iteration is as follows:
Method
Component
Time per iteration (ms)
Occupancy model
ORS
Reward function
Policy training
6.5
R-NET training
9.5
GO-FRESH
Policy training
GO-FRESH also has a graph computation time that ranges between 1.5-10 mins depending on the task and dataset. For default GCIQL, policy training takes 7.2 ms per iteration. All times ported on a single NVIDIA RTX A6000 GPU. We typically see that reward function and occupancy model losses converge in around 500k iterations. R-Net training typically for 500k iterations.
26

---

Published as a conference paper at ICLR 2026
While ORS does lead to slightly higher computational overhead, also leads to 2.3 increase in performance over GCIQL, its base algorithm, and exhibits stronger performance than GO-FRESH variety of tasks. GO-FRESH requires training distance classifier, training a graph memory and building a graph, each of which involves hyperparameter tuning for the size of the R-Net, number of edges in the graph, number of negative pairs for training the R-Net, distance threshold for the R-Net and careful weighing of local and global rewards during policy training. In contrast, ORS is simple to train, does require storing a large graph in memory and is easy to tune. We agree that increasing the efficiency of ORS is an important direction for future work.
27


Reviews:
[{"review_id": "zQ771wfycj", "reviewer": "Reviewer_noM7", "summary": "The paper proposes to address the struggles of offline goal-conditioned RL on long horizon tasks via reward shaping. In particular, it propposes a novel reward shaping method, occupancy reward shaping, trained using flow matching to perform effective credit assignment as a reward function. Experiments show that it improves over prior Offline GCRL methods on long-horizon lcomotion and manipulation tasks in simulation.", "questions": "- What is the quality and quantity of data that is required to train such a reward model? Will there be circumstances where the reward model is not accurate? If so how is the performance affected\n- In an offline RL setting, since there is no online interaction, why can't the reward be simply obtained using distance of current state to goal?\n- Can you discuss the comparison of your approach against works like GoFar (Ma et al), where shaped reward is not used, and goal reaching behavior is direclty learned by minimizing divergence between policy and expert's  goal conditioned state occupancy?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- clear motivation and extensive discussion on background\n- provides proofs for theoretical guiarantee", "weaknesses": "- It would be more convincing if results can also be demonstrated on real-world robotics tasks, where both data quantity and quality are lower\n- There should be more discussion on other ways of computing dense reward information\n- Results seem only to be marginally better, most of the gains over GO-FRESH are on 2 tasks\n- how did the authors select the tasks in the benchmark? why are tasks like ant soccer and humanoid maze not selected?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wiv6fz5h01", "reviewer": "Reviewer_bbdp", "summary": "This paper addresses the problem of insufficient signals in offline goal-conditioned reinforcement learning (Offline Goal-Conditioned RL, GCRL) under sparse rewards and long-horizon tasks. It proposes Occupancy Reward Shaping (ORS), a learning-based reward shaping method grounded in the occupancy measure, which can capture temporal dependencies in long-horizon tasks. By integrating flow matching for fitting, ORS distills goal achievement information from the occupancy measure into a generalizable reward function.", "questions": "1.The κ parameter in the ablation is highly sensitive. Could adaptive scheduling of κ improve stability?\n2.Does the optimality guarantee in Theorem 1 depend on data coverage quality? In datasets covering only part of the optimal path, can ORS still maintain optimality?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is clearly written, logically structured and well organized. Technical details are thoroughly presented.\n2.The theoretical analysis is solid, providing proofs of convergence and an analysis of reward monotonicity. Starting from empirical evidence that sparse rewards lead to non-monotonic value functions, the paper proposes a reward shaping idea based on the occupancy measure, forming a complete logical chain.\n3.Experiments cover both locomotion and manipulation tasks. The experimental design is rigorous, and the results consistently demonstrate strong performance.\n4.The approach is compatible with existing offline goal-conditioned RL algorithms, and the analysis of the value function non-monotonicity is insightful. Experiments validate that ORS effectively alleviates this issue.", "weaknesses": "1.While multiple tasks from OGBench are used, all of them are simulated environments. There is no mention of testing on real-world data or environments with different physical properties. It remains unclear whether the algorithm remains stable under real-world physics or diverse visual conditions.\n2.The theoretical assumptions are relatively strong. Discussions or brief experiments demonstrating robustness under stochastic dynamics are needed, or clarification on whether these assumptions still hold approximately in practice.\n3.Ablation studies are insufficient. Although the text mentions “conduct detailed analyses and ablations,” the contribution of different ORS components to the final performance is not shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTKfFB5pUd", "reviewer": "Reviewer_xcHp", "summary": "Offline GCRL methods trained with sparse rewards struggles in especially long-horizon tasks, since learned value functions tend to be noisy. Task specific hand-crafted reward functions can address this problem, but the task-based reward design is challenging and not practical. It is stated that previous work mostly focuses on online GCRL and recent offline GCRL approaches employ graph based distance classifiers, which fail to scale to large datasets. In this paper, considering the importance of scalability and generalization, an occupancy measure based approach is proposed for offline GCRL. The main idea is that the occupancy measure over future states is learned via flow matching, and then a reward function is defined to guide the policy via occupancy measure matching. Overall the idea is interesting and experimental results show such an occupancy measure based algorithm improves the performance of offline GCRL in long-horizon settings.", "questions": "- How computationally complex is the proposed method? Can you please provide computational efficiency comparisons with baselines and graph-based solutions?\n- Can you please elaborate on the benefits of the proposed approach over recent literature [a-b]?\n- Can you please compare the proposed method with [a-c], in addition to the current baselines?\n- Can you please elaborate on whether this approach would also be useful in medium or short-horizon tasks?\n- Can you please clarify how $\\hat{V}(s,g)$) was trained for the analysis in Section 3.2? What algorithm was used?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Interesting Approach**: Learning the occupancy measure via flow matching and then using this similarity as the reward function is a novel and interesting approach.\n- **Improved Non-monotonicity**: It is nicely presented that the proposed approach has lower non-monotonicity when compared to sparse rewards. \n- **Analysis of the Proposed Approach**: The experimental analyses are well-presented, clearly showing the effectiveness of ORS over sparse rewards.", "weaknesses": "- **Computational Overhead**: The computational overhead of the proposed approach is not discussed. ORS requires learning an occupancy measure before training the policy, making it appear computationally more complex than baselines. Therefore, the computational overhead should be discussed and compared with both graph-based and non-graph-based offline GCRL baselines.\n- **Novelty**: Occupancy measure matching has already been employed by recent works for GCRL. The contribution over these methods is not discussed, which makes the novelty questionable. The benefits of ORS over recent literature (a-b) must be elaborated.\n- **Baselines**: In the experiments, recent relevant works (a-c) are omitted from the comparisons.\n- **Experimental Setting**: The experiments only cover long-horizon tasks. The applicability of the proposed method to short- and medium-horizon tasks should be discussed.\n- **Experimental Results**: The environments used between Table 1 and Table 2 are not the same. It is not clear why some environments are included in Table 1 but omitted in Table 2. For a clear and fair evaluation of ORS, all environments in Table 1 must also be included in Table 2.\n- **Vague Explanation**: The main assumption of the paper (in long-horizon tasks, the value function exhibits a high level of non-monotonicity) is evaluated under section 3.2, however it is not clear how $\\hat{V}(s,g)$ is trained. It is not explained how the authors obtained $\\hat{V}(s,g)$ in 3.2, or which algorithm was used and how $\\hat{V}(s,g)$ was trained.\n\n[a]: Sikchi, Harshit, et al. \"Score models for offline goal-conditioned reinforcement learning.\" The Twelfth International Conference on Learning Representations. 2023.\n\n[b]: Ma, Jason Yecheng, et al. \"Offline goal-conditioned reinforcement learning via $ f $-advantage regression.\" Advances in neural information processing systems 35 (2022): 310-323.\n\n[c]: Zhou, John Luoyu, and Jonathan C. Kao. \"Flattening Hierarchies with Policy Bootstrapping.\" Workshop on Reinforcement Learning Beyond Rewards@ Reinforcement Learning Conference 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)