From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning
Authors: Gaurav Chaudhary
Laxmidhar Behera
Venue: Accepted by TMLR ()

Abstract: Offline Reinforcement Learning (RL) aims to learn effective policies from a static dataset
without requiring further agent environment interactions. However, its practical adoption is often hindered by the need for explicit reward annotations, which can be costly to engineer or difficult to obtain retrospectively. To address this, we propose ReLOAD (Reinforcement Learning with Offline Reward Annotation via Distillation), a novel reward annotation framework for offline RL. Unlike existing methods that depend on complex alignment procedures, our approach adapts Random Network Distillation (RND) to generate intrinsic rewards from expert demonstrations using a simple yet effective embedding discrepancy measure. First, we train a predictor network to mimic a fixed target network’s embeddings based on expert state transitions. Later, the prediction error between these networks serves as a reward
signal for each transition in the static dataset. This mechanism provides a structured reward signal without requiring
handcrafted reward annotations. We provide a formal theoretical construct that provides insights into how RND prediction errors effectively serve as intrinsic rewards by distinguishing expert-like transitions. Experiments on the D4RL benchmark demonstrate that ReLOAD
enables robust offline policy learning and achieves performance competitive with traditional
reward-annotated methods.

Published in Transactions on Machine Learning Research (10/2025)
From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning
Gaurav Chaudhary
gaunauchkiit..cc.i
Deportment of Electrical Engineering Indian Institute of Technology Konpur
Laxmidhar Behera
lbehera Chitk. in
Department Electrical Engineering Indian Institute of Technology Kanpur
Reviewed on OpenReview: // openreviee. net/ net/forum? id= F6X94J12J6
Abstract
Offline Reinforcement Learning (RL) aims to learn effective policies from a static dataset without requiring further agent environment interactions. Howover, its practical adop- tion often hindered by the need for explicit reward annotations, which can be costly to engineer or difficult to obtaín retrospectively. To address this, we propose ReLOAD (Reinforcement Learning with Offline Reward Annotation via Distillation), novel ward annotation framework for offline RL. Unlike existing methods that depend complex alignment procedures, our approach adapts Random Network Distillation (RND) to gen- erate intrinsic rewards from expert demonstrations using a simple yet effective embodding discrepancy measure. First, we train a predictor network to mimic a fixed target network's 's embeddings based on expert state transitions. Later, the prediction error between these networks serves as a reward signal for each transition in the static datasct. This mechanism provides a structured reward signal without requiring handerafted reward annotations. We provide formal theoretical construct that provides insights into how RND) prediction effectively serve as intrinsic rewards by distinguishing expert-like transitions. Experiments on the D4RL benchmark demonstrate that ReLOAD enables robust offline policy learning and achieves performance competitive with traditional reward-annolated methods.
1 Introduction
Deep Reinforcement Learning (DRL) has achieved remarkable success across wide range of applications, from game-playing (Silver et al. 2017 Vinyals et al. 2019) and intelligent perception systems (Chaudhary et al. 2023) to financial modeling (Charpentier et al. 2021 Despite these breakthroughs, DRL methods typically require millions of state-action transitions to train effective policies. This sample inefficiency makes DRL impractical for many real-world scenarios and motivates the search for alternative learning paradigms.
Offline reinforcement learning (RL) is a powerful approach to mitigate these challenges by leveraging static datasets to train policies without real-time interactions. This is particularly advantageous in areas such robotics [Tag et al. 2024), autonomous driving (Kiran et al. 2021), and healthcare Yuu et al. 2021 where collecting data through trial and error may be prohibitively expensive, unsafe, or simply unfeasible. However, a common requirement across many oflline RL methods is the need for explicit reward annotations. In many practical settings, these reward signals are either scarce or costly to obtain. While techniques like Aligned Imitation Learning via Optimal Transport (AILOT) (Bobrin al.) 2024) have attempted overcome this limitation by aligning trajectories in latent intent spaces, such methods tend incur significant computational overhead and can struggle when faced with the multi-modality of intents.

---

Published in Transactions on Machine Learning Research (10/2025)
To address these concerns, we present ReLOAD (Reinforcement Learning with Offine Reward Annotation via Distillation), a novel framework that leverages Random Network Distillation (RND) (Burda et al. 2018) to generate intrinsic rewards using the discrepancy between the embeddings of the target and predictor networks. Unlike prior methods that roquire adversarial training, trajectory alignment, or latent space modeling, ReLOAD only requires standard forward passes through neural nerworks, making it simple to implement and highly scalable. Furthermore, our use of RND departs from prior novelty-based applications in offline RL (Nikulin et al. 2023) by re-framing it as a reward inference mechanism grounded in expert imi- tation. Rather than discouraging novelty, ReLOAD distills prediction error from expert-only state-transition pairs (s, s') to promote expert-aligned behavior, in contrast to prior work, which operates state-action pairs (s, a) to penalize out-of-distribution actions. This conceptual shift formalized through our theoretical analysis maalsis--nables enables scalable and effective reward inference under minimal supervision.
ReLOAD circumvents the reliance on manually engineered reward signals by training a predictor network to mimic the embeddings producod by fixed, randomly initialized target network using expert transitions. After training the predictor to match the embedding of the larget network over expert transitions, the pre- diction error between the predictor and target networks for each state transition in the offline dataset serves as an intrinsic reward. In this formulation, small errors indicate behavior similar to expert transitions, while larger errors highlight novel or suboptimal transitions. This simple yet effective mechanism facilitates robust and automated reward generation without incurring the heavy computational costs of complex alignment procedures.
Our method contributes the field in several key ways:
A Novel Reward Distillation Strategy: We introduce a technique that bypasses the need for handcrafted rewards by inferring intrinsic rewards from RND-based embedding discrepancies, leveraging accossible expert demonstrations.
Thcoretical Foundation for RND Rewards: We rigorously establish formal theorem demon- strating that RND prediction errors are a reliable intrinsic reward signal, effectively distinguishing expert-like transitions in offline RI.
Implicit Transition Quality Assessment: ReLOAD is able to assess the quality of transitions without requiring explicit action-specific labels, thereby differentiating expert behavior from subop- timal strategies.
Competitive Benchmark Performance: Extensive evaluations the D4RL benchmark Fu
el al. 2020 show that ReLOAD performs competitively with traditional reward-based ofline RL
approaches.
Compared to existing reward inference techniques, such Optimal Transport Reward (OTR) labeling (Luo et al. 2023), AILOT (Bobrin et al. 2024), and Calibrated Latent Guidance (CLUE) (Liu et al.) 2023 ReLOAD distinguishes itself through its conceptual simplicity and computational efficiency. Prior approaches (OTR and ATL.T) depend on expensive trajectory alignment. or latent space optimization methods that may falter with multimodal expert data or depend on expert action labels (CLUE). However, our framework provides intrinsie reward annotation using straightforward embedding discrepancies between state transi- tions. This makes ReLOAD particularly well-suited for large-scale and diverse offline datasets where both computational resources and adaptability are critical.
Drawing on insights from ofllin imitation learning (Kostrikov et al. 2021], optimal transport-based reward inference [Luo et al. 2023). and novelty-driven exploration (Yang et al. 2024), ReLOAD offers a practical and scalable solution to the reward scarcity problem inherent in offline RI. In the following sections, we detail our methodology, present comprehensive experimental evaluations, and discuss the broader implications of our findings for advancing the field of offline reinforcement learning.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Expert Demonstrations
Uulabeled Dataset Da
Predictor Training min (f,(, (0(8,*)
RND Reward
Offiine RL Policy Learning
De
Figure 1: Overview of our method, ReLOAD. Transitions from expert demonstrations are used to train a predictor network to align with a fixed target network. When the unlabeled transitions are processed through target and predictor networks, the embedding discrepancy on an unlabeled dataset serves as an intrinsic reward guide offiine policy learning.
2
Related Work
2.1
Offline RL with Reward Annotations
Traditional offline RL assumess datasets include reward annotations, enabling algorithms such as Conservative
Q-Learning (Kumar et al. 2020), Implicit Q-Learning (Kostrikov et al. 2022b), and Critic Regularized
Regression (Wang et al. 2020) to optimize policies using supervised learning objectives (Levine et al. 2020;
Fujimoto & Gu 2021 J. However, crafting robust and generalizable reward functions often demands extensive domain expertise, posing a significant botnee..c.. To address this, ORI. (Zolna al. 2020) employs positive- unlabeled (PU) learning (Elkan & Noto 2008) to infer rewards from reward-free -free datascts. While ReLOAD shares the goal of reward-[ree oflline RL, it diverges by distilling rewards through embedding discrepancies from expert transitions, avoiding PU learning's reliance on binary classification.
2.2 Imitation Learning (IL)
Imitation Learning (IL) offers an alternative for policy learning without explicit rewards. Behavior Cloning (BC) (Pomerleau 1988) treats policy learning supervised regression but is prone to compounding errors from covariate shift. Inverse RL (IRL) methods, such those by (Englert et al. 2013 and (Kostrikov al. 2022a infer reward functions to explain expert demonstrations, while adversarial IT,, like GAIL, (Ho & Ermon 2016), uses a minimax game between discriminator and policy. These approaches, however, can be unstable and sensitive to hyperparameters (Orsini et al. 2021). More recently, optimal transport- based IL methods (iiao et al. 2019 Dadashi et al. 2022: Cohen et al. 2021 minimize Wasserstein distances (Villani & Villani 2009) between expert and agent distributions, sidestepping adversarial instability. ReLOAD differs
by using Random Network Distillation (RND) to infer rewards directly from expert embeddings, by passsing both supervised cloning and complex distribution matching.
2.3 Reward Inference in Offline RL
Reward inference is a growing focus in offline RL, especially where intrinsic motivation, well-explored in online RL (Brown et al. 2019 Ibarz et al. 2018 Yu et al. 2020). is adapted to static datasets. Unlike online methods that often leverage annotated data or environment interactions, ReLOAD targets fully reward-fres and action-label-free offline reward inference settings. Recent approaches include Optimal Transport Reward (OTR) (Luo et al. 2023) labeling, which assigns rewards via Wasserstein- couplings between agent and expert trajectories (Villani & Villani 2009). On the other hand, Calibrated Latent Guidance (CLUE) [Lu et al. 2023) uses a conditional variational auto-encoder (VAE) (Kingma 2013) map expert and agent transitions into a shared latent space, computing rewards from Euclidean distances a collapsod expert embedding. Further, Aligned Imitation Learning via Optimal Transport. (ALIOT) (Bobrin et al. 2024) aligns transitions in structured intent representation space, reducing cost function sensitivity of OTR.
However, these methods come with notable drawbacks. OTR requires carcful selection of cost function (e.g., cosine or Euclidean), which often demannd enviromment-speeifc luning. AILOT, while addressing some of OTR's limitations by operating in a latent intent space, still relies on solving OT problems typically via the Sinkhorn algorithm-we which scales poorly with dataset size and dimensionality. Morcover, AILOT also assumes access to sufficient unlabeled trajectories to extract intents. Furthermore, in the case of multimodal- ity of intentions, its intent alignment may fail to capture diverse behaviors without additional clustering.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Similarly, CLUE's VAE-based approach struggles with such data unless supplemented with clustering, adding further complexity. CLUE also requires access to the action label to infer an intrinsic reward.
In contrast, ReLOAD employs RND to train a predictor network against a fixed traget network using transitions from De (samples expert distribution DE), then uses embedding discrepancies to quantify transition novelty across Da (samples of a broad distribution Do). This avoids the computational overhead of optimal transport, latent space modeling, and dependency on action labels, offering a lightweight and scalable solution.
3 Preliminaries
3.1
Offline RL
We consider standard Reinforcement Learning (RL) problem defined by Markov Decision Process (MDP) M = -S, A. Po,,,o, 7), where: 8 C I is the state space, A C R. is the action space, S x A - P(S) denotes the transition dynamics, r S x A + I is the extrinsic reward function, Po is the initial state distribution, y E (0, 1] is the discount factor. The goal in RL is to Iearn a policy To(als) that maximizes the expected discounted return: Eywx, [Er= 7 r(st, ac]]. Where a trajectory T = (su, ao, $1, a1,...) generated by iteratively sampling so ~ Po, at To( [8), and 8441 p(-(ss, at).
permitted. Instead, the agent learns from fixed datasets without further exploration. We assume access to reward-free offline transitions D. {(s,, (s, <()) collected from various policies (e.g., behavior policy or mixture of policies), representing a broad distribution of transitions, which we denote as samples from Dv. Additionally, we are provided with a limited set of expert transitions, D. = {(st, +()) collected from the same MDP M, assumed to reflect high-quality behavior, and treated as samples from an expert distribution Dg. By omitting actions from De, our method focuses on state transition patterns, cnsuring applicability across different dynamics. The objective to learn oflline policy that emulates expert performance by leveraging D, (expert guidance) alongside Da (broader context), without relying extrinsic rewards.
In the offline RL setting, interaction with the environment
3.2 Random Network Distillation (RND)
Random Network Distillation (RND) (Burda et al. 2018) is technique originally developed for online RL to encourage exploration by quantifying novelty and later adapted for offline RL to guide imitation or anti-
exploration [Nikulin et al. 2023). It compares the outputs two networks: a fixed target network and a trainable predictor network.
In our formulation, RND operates on transitions æ (s,55) S S, reflecting state changes in the MDP.
Formally:
Target Network: fo S x S RK, initialized with random weights v and kept fixed.
Predictor Network: go S S R trained approximate the target network's output.
The predictor is optimized over the expert distribution DE with the loss:
Lirea (0) = Ez (x)
(1)
where gradient flow through fo is halted. Once trained on the expert distribution, the prediction error [[ - ( is small for transitions x = (x, 8') similar to those in DE (expert-like behavior) and large for transitions that deviate, such as those from Dy. In offline imitation learning, this error serves as an intrinsic reward signal, guiding the policy toward expert behavior by penalizing deviations from Dg. This approach provides a principled way to label transitions when extrinsic rewards are unavailable, aligning policy learning with expert demonstrations.
4

---

Published in Transactions on Machine Learning Research (10/2025)
4 Formal Justification of the Novelty Reward
Our method repurposes Random Network Distillation (RND) for imitation learning, in contrast to its con- ventional use in exploration or novelty suppression. Prior work has applied RND in offine RL settings to penalize state-action transitions with high prediction error, thereby discouraging out-of- distribution behavior and encouraging conservative policies (Nikulin et al., 2023 In ReLOAD, however, RND reinterpreted reward distillation mechanism: the predictor trained exclusively on expert state transitions, and the resulting prediction error serves a proxy for state transition quality, lower for expert-like behavior, and higher otherwise. This reframes the inductive bias from "uncertainty avoidance" to "expert alignment"..
Moreover, ReLOAD provides formal justification for this reward structure: as we show in Theorem | the expected prediction error strictly lower under the expert distribution than under the broader offline dataset. This perspective has not been explored in prior RND-based offline RL methods and enables ReLOAD to recover expert-like behavior under minimal supervision, using a simple and generalizable framework. Below, we formalize this argument and present the theoretical basis of our reward mechanism.
Theorem 1. Let Dg and Du be probobility distributions over state transitions X (where (s, repre- sents a Imassiioo), and the support of DE is a proper subset of Du, i.e, supp Dg) & supp(Dv).
Define f + R as a fixed, randomly initialized target network, and g X + R as predictor network trained on samples from Dg to minimize the objective:
Lepraa()) Be-DellS(e
De -x()))
(2)
Let the expected prediction error on DE be:
HE = (l) -
(3)
Assume that:
1. The distribution Dy and DE agree on supp(DE)) ie,
Bz-Delllf(s 9(x) supp(DE)) ME.
(4)
2. There crists a measurable subset S C supp( Dv) supp(DE) with PraNDo (x E S) > 0, such that the expecled prediction error on S is strictly greater than pæ:
E Ex [l) ((x) x E Pg.
(5)
Then, the expected prediction error over Dy exceeds that over Dg:
Be g(x) pp-
(6)
Proof: To prove the claim, We decompose the expectation over Dy into two regions:
((fx)) supp((De)
[f(()) (d
(7)
[1((x) y(x) (1 1i (x).
Define = Pra-Da (x E supp(De)) as the probability that a sample from De lies in the support of DE- Since supp(Dz) & supp(Dr), and assuming Dy assigns positive probability supp(Dr) supp(Dr), we have 0 < 1.
Using conditional expectations, rewrite the total expectation
Ex (1 - g(x) = p Ez- [Il((x) - -=))) E supp(
(8)
(1 p) EgnDa (ll g(x)]13 a supp
5

---

Published in Transactions on Machine Learning Research (10/2025)
By assumption (1), we have:
De - g(x) supp(DE)) p..
(9)
Thus, the first term satisfies:
Re Du [II[(x) - g(x) supp(DE) = pa-
(10)
For the second region supp(Dp) (D) supp consider the expected error:
Ee~De [1) g & supp(Dg))-
(11)
By assumption (2), there exists a subset S supp with Pr (s S) > 0 and Eg Du S] μ4g-
Since S is subset of supp(Dv) supp((R)) and the error on the remaining parts of supp(Dt) supp(DFF) is non-negative, the conditional expectation over the entire complement satisfies:
De [Ilf()) - ((x) & supp(De)) 2 E. [llf(x) ((=) I S] PE-
(12)
Thus, the second term is:
(1 1-p)) p) EunDu [Ilf(z) - ( x supp(DE)] > (1 - p) me
(13)
with 1 1 > 0 because aupp(De) 2 supp(Dr)
Combining the two terms, we derive:
B 1If(x) - g(x)) = p- ME + (1 p) EELD, [If(()) - g(x))12 - supp(De))-
(14)
Since Egw (1)(()) f(x) supp(DE)) Pes it follows that:
De l1f(z)) - g())) > p PE (1 p) p)- PE= μE.
(15)
This strict inequality holds because the second term contributes the excess over JE with positive weight p.p.
Thus, we conclude:
Be- (I(x)) - 9(x) 12 pr.
(16)
This result implics that if we define a reward function r(x) - lIf(())) g transitions from D3 (expert- like, low error) yield higher rewards, while those unique to Du (novel states, high error) yield lower rewards, which align with the goals of Imitation learning.
Further, training the predictor solely on expert demonstrations induces a behavior prior. The optimization implicitly encourages policies that minimize divergence from expert-like behaviors because the learned reward function assigns higher values to expert-aligned transitions.
Unlike Inverse RL, which explicitly optimizes reward function, ReLOAD self-superviees reward labeling, Since the predictor never trains non-expert transitions, it avoids reward contamination, ensuring robust- ness against noisy, mixed-quality datasets.
Also, unlike AILOT, which requires mapping states to an intent space and solving optimal transport prob- ReLOAD's
lems- a computationally intensive process sensitive to the quality of intent representations Deveeentatioss-ReLAAADsss reward computation involves only forward passes through neural networks after initial training. This sim- plicity reduces computational demands and enhances scalability, making ReLOAD particularly well-suited for large, diverse offline datasets.

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm Sell-Supervised Reward Annotation via RND
Inputs: Offline dataset D {(s, s')}, Expert transitions D. {(s, s')), Target network fe, Predictor network go, Learning rate 7), Number of epochs Npred, Hyperparameters 8
2: Initialize: Fixed target network fø with parameters v Trainable predictor network with parameters
0
3: Step 1: Pretraining the Predictor with RND for epoch 1 Npred do Sample (s, 8") from De Compute RND loss:
prred (0) (Ifo(s,'')= s')
7:
Update predictor: 0 + -onn-Vol Vo (0)
8: end for
9: Step 2: Computing Reward for Offline Data
10: for (s, s') E do
11:
Compute intrinsic reward:
FRND s') -llff((s go(s, s')1
12: Optionally, apply exponential transformation:
F(s,8') - exxp(BrRND(s, s'))
13:
end for
14: Output: Labeled dataset D {(s, a, F(8,8),,8)) or (AND(s,
5
Method
Our approach adapts Random Network Distillation (RND) to derive an intrinsic reward signal from expert transitions for oflline imitation learning. It consists of three stages: (i) aligning a predictor network with a fixed target network using expert transitions, (ii) assigning intrinsic rewards to an unlabeled dataset based on embedding differences, and (iii) training an offline policy with these rewards to mimic expert behavior. This climinates the need for manually defined rewards in offiine reinforcement learning.
Expert Embedding Alignment: We start with an expert dataset D. = {(s,, s where each (st, s() is a state transition from expert demonstrations, assumed to sampled from an expert distribution Dg. We define two neural networks:
Target Network: Je - R a fixed, randomly initialized network with parameters e, mapping transitions to K-dimensional embedding space.
Predictor Network: ge S + K trainable network with parameters 0, mapping transitions to the same embodding space.
The predictor go is trained batch M to match the larget network's outpuls expert transitions by minimizing:
1 M
Lppeed (0)
Σ [o 81) - go 8011
(17)
M
This loss approximates Eg~Da (x) (x)) where x s'), ensuring ge learns emboddings aligned with expert. behavior while fe remains stable target.

---

Published in Transactions on Machine Learning Research (10/2025)
Intrinsic Reward Assignment via Embedding Discrepancy: We then apply the trained predictor to an unlabeled offline dataset Da {($j, x } assumed to be sampled from a broader distribution Dv. The change in subscript from in equation 17 to j here is intentional: the predictor f is trained on expert transitions (f) E DE, and then evaluated on unlabeled transitions (s), E Da. For notational convenience, we denole each transition as xy (sy, K9)- The intrinsic reward assigned a transition zy is then given by:
(18) This reward is higher (less negative) for transitions similar to those in De, where go closely matches fø. and lower (more negative) for dissimilar ionss where discrepancies are larger. This reflects the theorem's insight that Do Ilfe (x) go (z) > Ee>DE (l) go(x) prioritizing expert-like behavior.
(RND(2,) (1f(()))) go
Offline Policy Learning with Distilled Rewards: Using these rewards, we create a. labeled dataset:
δ { (T)N(()))))
(19)
A policy Te, parameterized by o, is trained using an offline RL algorithm (e.g., Implicit Q-Learning (Kostrikov ct al. 2021j) to maximize the expected return as estimated by the Q- Q-function learned from the offline dataset, which is formulated as:
max EB [QT(s,
(20)
where Q is estimated from δ with rewards FRND, and actions a are derived from transitions assuming a deterministic environment or prtaset-provided actions.
Reward Squashing for Stability: To improve training stability, we optionally transform rewards as: f(x,) = a (xp(BrRND(x;)),
(21)
where a scales the reward, and > 0 adjusts sensitivity. Since (R))( S 0, this maps rewards to (0, a), bounding variance and aiding convergence.
In conclusion, ReLOAD's intrinsic reward mechanism is simple and thcoretically grounded. By training the predictor network ga exclusively on expert transitions from De, ensure that the prediction error (z) g0(=)1 is minimized for expert-like behavior and maximized for suboptimal or novel transitions. This is formally supported by Theorem [ which proves that the expected prediction error over the broader dataset Da exceeds that over the expert dataset De, providing a clean and reliable reward signal. Unlike AILOT, which requires mapping states to an intent space and solving optimal transport, problems- computationall intensive process scnsitive the quality of intent representations ReLOAD's reward computation involves only forward passes through neural networks after initial training. This simplicity reduces computational demands and enhances scalability, making ReLOAD well-suited for large, diverse offline datasets.
6 Experiments
We evaluate our method across range of benchmark tasks to address the following key research questions:
Can our method enable offlinc RL algorithms to recover performance comparable to those using well-engineered ground-truth rewards?
How effectively does our novelty reward formulation leverage expert demonstrations to enhance policy learning?
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Performance Comparison on D4RL Locomotion Tasks. The reported results are normalized scores (mean + standard deviation) across 10 random sceds. For these methods, the results are given for 1, the number of expert trajectories. The highest scores are highlighted in green.
Dataset
TQ-Learn
SQIL
ORIL
SMODICE
ReLOAD
aeficheetah-mediiun
21.7 = 1.5
24.3 2.7
56.8 1.2
42.4 = 0.6
48.2 0.7
lallcceetal--meciuun-rplaay
6.7 + 1.8
43.8 + 1.0
46.2 + 1.1
38.3 + 2.0
42.6 + 0.8
halfcheetab--ncddun--ppprt
2.0 0.4
6.7 1.2
48.7 2.4
81.0 2.3
92.6 2.5
hopper-medium
29.6 5.2
66.9 5.1
96.3 0.9
54.8 1.2
78.7 6.9
hopper-mediuunrrelay
23.0 + 9.4
98.6 + 0.7
56.7 + 12.9
30.4 + 1.2
95.9 + 2.2
hupper-nnedium
9.1 2.2
13.6 9.6
12.8 25.1
82.4 7.7
104.8 + 9.0
walker2d-medium
5.7 4.0
51.9 11.7
20.4
13.6
67.8 6.0
81.3 1.0
rell
17.0 = 7.6
42.3 + 5.8
71.8 + 9.6
49.7 + 4.6
78.6 + 2.5
wallke2dd-meddun--xpptt
7.7 + 2.4
18.8 + 13.1
11.6 1 14.7
94.8 1 11.1
110.5 1 0.7
D4RL Locomotion total
122.5
366.9
433.6
541.6
733.2
Dataset
IQL
OTR
CLUE
AILOT
ReLOAD
halfcheetah-medium
47.4 + 0.2
43.3 + 0.2
45.6 + 0.3
47.7 + 0.35
48.2 + 0.7
hallfchottah-mmdiiu--pppaaa
44.2 + 1.2
41.3 + 0.6
43.5 + 0.5
42.4 + 0.8
42.6 + 0.8
hallcheerah-nediin-experr
86.7 5.3
89.6 1 3.0
91.4 2.1
92.4 + 1.54
92.6 2.5
hopper-medium
66.2 5.7
78.7 + 5.5
78.3 + 5.4
82.2 + 5.6
78.7 16.9
hopper-mediiu-
94.7 + 8.6
84.8 + 2.6
94.3 + 6.0
98.7 + 0.4
95.9 + 2.2
hopper--medium-expeet
91.5 14.3
93.2 20.6
96.5 + 14.7
103.4 5.3
104.8 + 9.0
walker2d-medium
78.3 + 8.7
79.4 + 1.4
80.7 + 1.5
78.3 + 0.8
81.3 + 1.0
wallker2ki-medium--eplay
73.8 7.1
66.0 + 6.7
76.3 = 2.8
77.3 - 3.1
78.6 2.5
walker2dimeddium-xxpertt
109.6 + 1.0
109.3 + 0.8
109.3 + 2.1
110.2 + 1.2
110.5 + 0.7
D4RL Locomotion total
692.4
685.6
714.5
732.8
733.2
How does our method perform in fully offline imitation learning compared to state-of-the-art offline RL approaches?
What is the impact of varying the number of expert demonstrations on performance?
We conduct extensive experiments on standard offline RL and offline IL benchmarks to answer these ques- tions, including D4RL Locomotion, Antmaze, and Adroit.
A key challenge offline RI. handling datasets that contain mixture of expert and suboptimal behaviors. We systematically evaluate our method on datasets with varying levels of expert supervision, analyzing its ability to identify and propagate useful reward signals. To further assess robustness, we conduct an ablation study by varying the number of high- demonstrations and examining the impact on learning performance.
All experiments were conducted with over 10 random seeds, with performance measured using average episode return and normalized scores relative to ground-trun woound-truth reward baselines. We also report runtime complexity assess computational efficiency. Our results demonstrate that our self-supervised reward formulation significantly enhances offline RL performance while providing strong alternative manually engincered reward functions in offline imitation learning settings.

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Performance Comparison on sparse AntMaze Tasks. The reported results are normalized scores across 10 seeds. The reposted results are given for K number of expert trajectories. The ReLOAD approach shows the most consistent performance, while AILOT achieves a slightly higher total reward.
Dataset
IQL
OTR
CLUE
AILOT
ReLOAD
antmaze-umaze
88.7
81.6 + 7.3
92.1 + 3.9
93.5 + 4.8
95 + 5.4
autmaaze-umazeddiverse
67.5
70.4 + 8.9
68.0 + 11.2
63.4 + 7.6
67.5 + 14.5
antmaze-mediun-play
72.9
73.9 + 6.0
75.3 + 6.3
71.3 + 5.2
72.9
12.5
altmaaee-medduu-ddiverss
72.1
72.5 + 6.9
74.6 + 7.5
75.5 + 7.4
76.3 + 14.1
antmaze-large-play
43.2
49.7 + 6.9
55.8 + 7.7
57.6 + 6.6
61.6 + 12.0
antmaze-large-diveree
46.9
48.1 + 7.9
49.9 + 6.9
66.6 3.1
51.4 14.2
AntMaze total
391.3
396.2
415.7
427.9
424.7
6.1
Implementation Details
Our Random Network Distilled Reward learning module can be plugged into any offline RL algorithm. For our implementation, we use Implicit-Q Learning (IQL) (Kostrikov al. 2021), which aligns with other base- lines that also build their approach this offline RI algorithm. We implement ReLOAD in JAX (Bradbury et al. 2018) and use official IQL implementattior"] We also test our approach against the Behavior Cloned (BC) policy. The details of all the hyperparameters and network architecture are in the Appendix B
6.2 Runtime
All the experiments were conducted on a single NVIDIA A40 GPU. The runtime overhead ReLOAD is less than a minute (compared to the offline RL algorithm training time), inchuding training the predictor network and annotating offine data. Thus, the total runtime of ReL.OADTTQT, equals approximately minutes. We use 100 training iterations with a batch size 64 to train the RND predictor network. Moreover, unlike previous approaches such as OTR, where computational cost increases with dimensionality, ReLOAD remains mostly invariant to the dimensionality of the state space. For high-dimensional tasks (e.g., learning from pixcls), ReLOAD is more suitable as the reward annotation depends on the embedding generated from the predictor and target network. Compared to ReLOAD, AILOT takes approximately 10 minutes overhead to train the latent space and reward relabeling.
6.3 Baselines
We compare ReLOAD with the following basclines (refer Appendix C for description of additional basclines):
IQL (ostrikov et al. 2021) is leading offline RL algorithm that mitigates cut-of-distributio ict selection by interpreting the value function as random variable. The upper bound on uncertainty is controlled via expectile regression. For evaluation, we use the ground-truth reward from D4RL tasks.
OTR
Luo et. al. 2023) defines a reward function by leveraging the optimal transport distance between state distributions from expert demonstrations and an unlabeled offline dataset.
CLUE (Liu al. 2023) constructs variational autoencoder (VAE) (Kingina 2013) for calibrating the latent space of expert and agent state-action transitions. The intrinsic reward is computed as the discrepancy between the agent's transitions and the averaged expert representation.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Performance Comparison on Adroit Tasks. The reported results are normalized scores across sceds. The reposted results are given for K number of expert trajectories. The ReLOAD approach shows the competitive performance with the current seate-of-the-ase method AILOT.
Dataset
IQL
OTR
CLUE
AILOT
ReLOAD
door-cloned
1.6
0.01 + 0.01
0.02 + 0.01
0.05 + 0.02
1.7 2.9
door-human
4.3
5.9 + 2.7
7.7 = 3.9
7.9 + 3.2
7.7 = 1.6
hammer-cloned
2.1
0.9 + 0.3
1.4 + 1.0
1.6 + 0.1
1.8 + 1.4
hammer-human
1.4
1.8 + 1.4
1.9 + 1.2
1.8 + 1.3
1.8 + 1.1
pen-cloned
37.3
46.9 + 20.9
59.4 + 21.1
61.4 + 19.5
64.5 + 24.6
pen-human
71.5
66.8 - 21.2
82.9 - 20.2
89.4 0.1
88.3 - 8.7
relocate-cloned
-0.2
-0.24 - 0.03
-0.23 + 0.02
-0.20 + 0.03
-0.11 - 0.04
habcatt-human
0.1
0.1 + 0.1
0.2 + 0.3
0.28 + 0.1
0.23 + 0.2
Adroit total
118.1
122.2
153.3
162.2
165.9
AILOT [Bobrin et al. 2024) define a reward function that uses optimal transport between the special representations of states in the form of intents that incorporate pairwise spatial distances within the data.
6.4 Results
We evaluate ReLOAD across a range of D4RL locomotion tasks using IQL as the fixed offline RL backbone. This ensures a consistent comparison across all experiments. Notably, several baselines such as OTR, CLUE, and AILOT also use IQL, making our comparisons fair and aligned. All experiments were conducted with over 10 random seeds, with performance reported normalized scores (or returns) alongside standard deviations. The best-rewarding trajectory is selected from the offline dataset as the expert trajectory. All the reward and action annotations are discarded for forward inference, and trajectories are relabeled based on intrinsic rewards. We compare ReLOAD against competitive baselines, including LQ-Learn, SQIL, ORIL, SMODICE, OTR, CLUE, and AILOT.
On D4RL locomotion tasks, ReLOAD achieves strong performance, Table T outperforming several reward- free baselines under minimal supervision (e.g., K-1 expert trajectory), without relying on action labels (as required by CLUE) costly intent alignment via optimal transport (as in AILOT). ReLOAD also performs superior to IQL with true reward-annotated trajectories. ReL.OAD matches or exceeds prior reward annotation methods on 7 out of 9 locomotion benchmarks, highlighting its ability to infer effective reward signals directly from state transitions.
In sparse-reward settings such as AntMaze and Adroit, ReLOAD demonstrates robust generalization and competitive performance relative to aligument-basen approaches. For the AntMaze task, we add reward bias of -3 to align with the binary reward nature of true reward annotation, which is in line with other reward annotation approaches. Although AILOT outperforms ReLOAD in AntMaze- largeezze-arge-iveeseee variant, ReLOAD remains highly competitive while avoiding the computational overhead of latent space modeling. Although AILOT achieves the highest cunulative reward, ReLOAD performs best on most AntMMaee navigation tasks, outperforming all baselines as shown in Table 2
In the Adroit Manipulation task, Table 3 ReLOAD performs better than all the basclines. Further, more ambiguous settings-suce as door-cloned and relocate-cloned- methods like OTR, CLUE, and AILOT fail to recover informative reward signals, likely due to the presence of mixed-quality demonstrations and
11

---

Published in Transactions on Machine Learning Research (10/2025)
IQL
ReLOAD
halfcheetah-medium
mopper-medium
walker2d-medium
80
Normalizer Score
40
20
10%
25%
50% Data Fraction
100%
10%
25%
50%
100%
10%
25%
50%
100%
Data Fraction
Data Fraction
Figure Comparison of IQL and ReLOAD on D4RL Locomotion Tasks across varying fractions of offline data (10%, 25%, 50%, 100%). ReLOAD consistently outperforms IQL, particularly in Hopper and Walker2d environments. Bars indicate mean normalized scores with standard deviation 10 random seeds.
human-induced noise. Specifically, we believe AILOT fails due to difficulty in extracting meaningful intent from noisy offline data similar. In contrast, ReLOAD continues to perform reliably, illustrating its robustness imperfect expert data. These results suggest that ReLOAD's simple embedding- reward mechanism is well-suited for procedures.
clean and noisy offline R.L datasets without explicit supervision or complex alignment
6.5 Effect of Number
of Expert Trajectories
In this subsection, investigate the sensitivity of ReLOAD to the number of expert trajectories available in the expert dataset De, comparing its performance against other basclines. Table 13 reports results for K=1 and K=5 expert demonstrraionn across D4RL locomotion tasks. Even with a single trajectory, ReLOAD achieves strong performance, highlighting its ability to generalize with minimal supervision. Increasing the number of demonstrations leads to consistent, though moderate, improvements, most notably in the Hopper and Walker2d environments, indicating that additional expert guidance refines the reward signal without requiring architectural or algorithmic changes.
These findings suggest two important properties of ReLOAD. First, it is highly data-efficient: single expert trajectory suffices to recover competitive behavior across wide range of tasks. Second, the RND-based reward formulation gracefully scales with the quality and quantity of supervision, enabling smooth reward shaping as more demonstrations bocome available. Importantly, this achieved without complex modeling of expert intent, latent alignment procedures, or reliance on action labels distinguishing ReLOAD from methods that require extensive preprocessing or structured supervision. This makes the approach particularly suitable for applications with varying levels of expert access and constrained annotation budgets.
6.6 Performance of ReLOAD Under Varying Offline Data Sizes
We compare the data efficiency of ReLOAD and IQL by evaluating their performance offline datasets of different sizes. ReLOAD utilizes intrinsic rewards generated from expert demonstrations, whereas IQL relies on true environment rewards. Our experiments focus on three D4RL Locomotion tasks: halfcheetah- medium, hopper-medt medium, and walker2d-medinne at various proportions of the full data. Both ReLOAD and IQL. were trained on these subscts, and their performance was assessed using normalized scores averaged over 10 random seeds. As illustrated in Figure 2 and Table 12 ReL.OAD consistently outperforms IQL across all tasks and dataset sizes.
For each task, we created subsets of the offline dataset
12

---

Published in Transactions on Machine Learning Research (10/2025)
These results highlight the benefits of ReLOAD's intrinsic reward mechanism, which provides more in- formative learning signal compared to the true rewards used by IQL. These findings suggest that reward annotation generated by ReLOAD propagates to efficient learning even with a low proportion of offline data. This enhanced data efficiency is particularly valuable in real-world scenarios where collecting large-scale offline datasets can be costly or impractical. The experiments demonstrate that ReLOAD can learn effect policies with significantly less data than traditional reward-based methods, making promising approach for diverse offline reinforcement learning applications.
7 Conclusion
We introduced ReLOAD, reward-free offline reinforcement learning framework that distills intrinsic rewards from expert transitions using Random Network Distillation. This reframing of RND, from novelty avoidance to expert reward inference, highlights ReLOAD's conceptual departure from prior anti-exploration work. ReLOAD provides a simple yet effective mechanism to differentiate expert-like from suboptimal behaviors without relying on action labels or handcrafted rewards by measuring embedding discrepancies between a fixed prior and a trained predictor. Extensive evaluations across D4RL locomotion, AntMaze, and Adroit tasks show that ReLOAD matches or outperforms existing reward-free methods while being computationally lightweight and robust to limited expert supervision. These results highlight the potential of embedding- based reward inference scalable alternative for real-world offline RL applications.
References
Maksim Bobrin, Nazar Buzun, Dmitrii Krylov, and Dmitry Dylov. Align your intents: Offline imitation learning via optimal transport. arXiv preprint arXiv:2402.13037 2024.
James Bradbury, Roy Frostig, Peter Hawkins, Matthow James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable trans- formations of python+ numpy programs. 2018.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on ma- chine learning, pp. 783-792. PMLR, 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint xrxix:18100 12894, 2018.
Arthur Charpentier, Romuald Elic, and Carl Remlinger. Reinforcement lcarning in cconomics and finance. Compulational Economics, pp. 1 1-38, 2021.
Gaurav Chaudhary, Laxmidhar Behera, and Tushar Sandhan. Active perception system for enhanced visual signal recovery using deep reinforcement learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Adam Cohen, Riad Akrour, Olivier Buffet, and François Charpillet. Improving adversarial imitation learning with wasserstein distance. In Proceedings of the 38th International Conference on Machine Learning, pp. 2042-2051, 2021.
Marco Cuturi. Sinkhorn distances: Lightspeed compulation of optimal transport. Advances in neural information processing systems, 26, 2013.
Robert Dadashi, Yunhao Tang, Ofir Nachum, Wojciech M Czarnecki, Lucas Hussenot, Michal Valko, and Rémi Munos. Primal wasserstein imitation learning. In International Conference on Machine Learning, pp. 4692 4703, 2022.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 213 213-220, 2008.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Peter Englert, Marc Toussaint, and Jens Kober. Model-free inverse reinforcement learning by bayesian nonparametric expectation maximization. Neural Compulation, 27(():1229-122 12600, 2013.
Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Gu. A minimalist approach offline reinforcement learning. In Advances in Newral Information Processing Systems, volume 34, pp. 20132-20145, 2021.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028 0444028-439,, 2021.
Jonathan Ho and Stefano Ermon. Generative advessarial imitation learning. Tn Advances in Neural Infor-
mation Processing Systems, volume 29, pp. 4565 4573, 2016.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint Arx:v:1312.611 2013.
B Ravi Kiran, Tbrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent. Transporlation Systems, 23(6):4909 4926, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning.
arXiv preprint vrXiv:2110.06162 2021.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via offiine reinforcement learning. arXiv preprint arXio:220..0 02005, 2022a.
Nya Kostrikov, Ofir Nachum, and Jonathan Tompson. Offline reinforcement learning with implicit q-lcarning. arXiv preprinl Xrx::22110.000 060669 2022b.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for ofllne rein- forcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 1179 1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ar preprint wxXiv220550064/3 2020.
Jinxin Liu, Lipeng Zu, Li He, and Donglin Wang. Clue: Calibrated latent guidance for oflline reinforcement learning. In Conference on Robot Learning, pp. 906 927. PMLR, 2023.
Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, and Mare Peter Deisenroth. Optimal transport for offline imitation learning. arXiv preprint ar arXiv:2303. 13971, 2023.
Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imita- tion learning via state occupancy matching. ar preprint urXiv:2202.08433) 1(2):3, 2022.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pp.
PMLR, 2023.
Manu Orsini, Odalric- Ambrryn Maillard, Alberto Maria Mételli, Tom Blankevoort, Freek Stulp, and Igor Babuschkin. What matters for adversarial imitation learning? In International Conference on Learning Representations, 2021.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. In Advances in Neural Information Processing Systems, pp. 1988.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. Xrviv preprinl X90: 11108, 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXic preprint rxXi:1712.01816: 2017.
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes, 2024. URL https://arxx) htg/abb/2408.0555
Cédric Villani and Cédric Villani. The wasserstein distances. Oplimal transport: old and new, pp. 93 111, 2009.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii DeepMind blog, 2:20, 2019.
Zhendong Wang. Jonathan J Hunt, and Mingyuan Zhou. Diffusion policics as expressive policy class for oflline reinforcement learning. Xiviv preprinl ur Xic:22208. ft::2208006133, 2022.
Ziyu Wang, Ofir Nachum, Zeb Kurth-Nelson, Samuel Ritter, Momchil Tomov, and Matthew Botvinick. Critic regularized regression. In Advances in Neural Information Processing Systems, volume 33, pp. 7768-7778, 2020.
Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6387 6394, 2019. Zhou Xiao, Ziyu Wang, Tu Hoang, Wen Sun, and Yang Gao. Wasserstein adversarial imitation learning.
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-exploration with distributional random network distillation. aixic preprint arXiv:2401.09750 2024.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healtheare: A survey.
ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Xingrui Yu, Yueming Lyu, and Ivor Tsang, Intrinsic reward driven imitation learning via generative model. PMLR, 2020.
In International conference on machine learning, pp.
Konrad Zolna, Bilal Piot, Olivier Pietquin, and Matthieu Geist. Offiine reinforcement learning with inverse dynamics regularization. In Proceedings of the 37th International Conference on Machine Learning, pp- 11448 11457, 2020.
15

---

Published in Transactions on Machine Learning Research (10/2025)
A Implementation Details of Reward Labeling Module-RNV RND
This section discusses the details of implementing the reward labeling strategy. We maintain two networks of identical architecture: the Target and the Predictor Network. The weights of the target networks are kept fixed. First, we extract the highest reward trajectory from the offline data based on the annotation provided in the offline data. The transitions (s, s') are concatenated and passed through the target and predictor networks. The predictor network is trained to minimize the mean square error loss between the output embedding of the target and the predictor network.
For oflline data reward annotation, we freeze the weights of both larget and predictor networks. The state transition from offline data is concatenated and passed through the target and predictor networks. The dissimilarity score (MSE) between target and predictor networks embedding, followed by reward squashing, is used to annotate offline data. The true reward annotation of the oflline data is discarded if available.
This section discusses the details of implementing the reward labeling strategy. We maintain two networks of identical architecture: the Target and the Predictor Network. The weights of the target networks are kept fixed. First, we extract the highest reward trajectory from the offline data based on the annotation provided in the offline data. The transitions ss, are concatenated and passed through the target and predictor networks. The predictor network trained to minimize the mean square error loss between the output embedding of the target and the predictor network. We initialize the target network independently using standard Xavier initialization. The target network is frozen after initialization and never updated, while for the prodictor network, don't any initialization strategy. We use Layer-Norm in both predictor and larget works
For offline data reward annotation, we freeze the weights of both target and predictor networks. The state transition from offline data is concatenated and passed through the target and predictor networks. The dissimilarity score (MSE) between larget and predictor networks embedding, followed by reward squashing, is used to annotate offline data. The true reward annotation of the offline data is discarded if available.
Table 4: Hyperparameters for training RND.
Sctting
MuJoCo and Antmaze
Adroit
Hidden dim
256
256
Number of layers
2
Batch size
32
16
Number of iterations
10a
10
Learning rate
le 3
3
Number of expert trajectories
1
B Hyperparameters
The performance of ReLOAD is influenced by several key hyperparameters, which we detail below. These parameters were selected based on a combination of preliminary experiments and established practices in the offline reinforcement learning (RL) literature. The offline RL algorithm used Implicit Q-Learning (IQL). and its hyperparameters are set according the recommendations in the original IQL paper (Kostrikov et al. 2021
Table presents the hyperparameters used across all experiments unless otherwise specified. For certain tasks, minor adjustments were made to optimize performance; these parameters are highlighted in 0
We adjusted the reward squashing parameters for specific tasks, such as the AntMaze and Adroit environ- ments, to suit those domains' reward scales better. Specifically, for Locomotion task we use a 10 and I = 5 except for hopper-expert we use a 10 and / 0.5, for AntMaze, we set a 10 and 1, while for Adroit, used 10 and 5. These adjustments were made based on initial experiments to ensure
16

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Hyperparameters used the ReLOAD framework. These values were used across all experiments.
Hyperparameter
Value
Description
Offline RL Algorithm
IQL
Choice of offline RL algorithm
Policy Learning Rate
3e-4
Learning rate for the policy network
Critic Learning Rate
3e-4
Learning rate for the critic network
Value Learning Rate
3e-4
Learning rate for value network
Discount Factor (7)
0.99
Discount factor for future rewards
Training Steps
1e6
Number of training steps for IQL
Batch Size
256
Batch size for IQL training
Table 6: Task dependent hyperparameters
Task
Expectile
Temperature
Locomotion
0.7
6
10
AntMaze
0.9
10
10
Adroit
0.8
0.5
10
stable training and are consistent with practices in the offline RL community for handling varying reward magnitudes.
All the reported results are across 10 random seeds with 10 evaluation episodes for each seed, consistent with previous works. We also use reward scaling, which is similar to IQL. The reward scaling factor used is
return min retrun" For the Antmaze task, we also add a reward bias of -3 to the scaled reward.
1000
C Baselines:
IQ-Learn (Garg et al. 2021 is an imitation learning approach that implicitly embeds expert policy and rewards into an inverse Q-function, enabling direct policy learning from expert demonstrations.
Diffusion-QL (Wang et al. 2022) reformulates policy learning conditional diffusion process, improving expressiveness and providing more flexible regularization mechanism for aligning with the behavior policy that generated the dataset.
SQIL (Reddy et al. 2019 introduces soft Q-learning framework that assigns fixed reward of one to expert transitions and zero to non-expert ones, effectively shaping the policy towards expert-like like behavior.
ORIL (Zolna et al. 2020 employs a discriminator to distinguish between optimal and suboptimal data in mixed datasets. The learned discriminator is then used for reward relabeling, guiding the policy toward expert-like behaviors.
SMODICE (Ma et al. 2022) an offline imitation learning method that minimizes the divergence between the state occupancy distributions of the agent and the expert. It achieves this through dual formulation of the value function, allowing for direct state-matching IL. without requiring explicit action supervision.
D Effect of Integrating ReLOAD with OTR
The integration of ReLOAD with Optimal Transport Reward (OTR) aims to explore whether combining RND-bbased embedding discrepancies with distributional alignment via optimal transport (OT) can enhance
17

---

Published in Transactions on Machine Learning Research (10/2025)
Table 7: Effect of combining ReLOAd with OTR. The reported results are normalized scores across 10 seeds. The reposted results are given for K number of expert trajectories.
Dataset
IQL
OTR
ReLOAD
ReLOAD+OTR
antmaze-umaze
88.7
81.6 + 7.3
95 + 5.4
90 + 7.6
autmazz-umazz-divess
67.5
70.4 + 8.9
67.5 + 14.5
65.7 + 16.2
antmaze-medhumm-play
72.9
73.9 + 6.0
72.9 + 12.5
73.8 + 11.9
autmaaze-nediium-diieeee
72.1
72.5 + 6.9
76.3 + 14.1
75.0 + 18.6
antmaze-larre-pla
43.2
49.7
+ 6.9
61.6 + 12.0
58.0 - 11.9
autuaaze-arge-ddivrse
46.9
+ 7.9 48.1
51.4 + 14.2
51.7 + 14.7
AntMaze total
391.3
396.2
424.7
414.2
reward inference. While ReLOAD leverages the simplicity of mean squared error (MSE) between fixed prior and predictor network embeddings, OTR assigns rewards by computing Wasserstein distances weew trajectories. This hybrid approach replaces ReLOAD's MSE-based rewards with OT-based distances, hy- pothesizing that OT's ability to capture geometric relationships between state distributions might better distinguish expert-like transitions. Specifically, we compute the Sinkhorn divergence (Cuturi 2013) between the target-predictor embeddings of agent transitions and expert demonstrations, using this as the intrinsic reward signal:
Theoretically, OT-based rewards provide more holistic measure of similarity by considering the global structure of the embedding space, whereas MSE focuses on pointwise discrepancies. However, OT intro- duccs computational overhead due to iterative Sinkhorn iterations, scaling quadratically with dataset size. Additionally, OT's sensitivity to the choice of ground cost. (e.g.. Euclidean vs. cosine) necessitates careful tuning, which may limit its plug-and-ple adaptability compared to ReLOAD's parameter free MSE.
As shown in Table 7 ReLOAD+OTH achieves competitive but inconsistent performance on AntMaze tasks.
While it outperforms standalone OTR in most cases, it generally underperforms pure ReLOAD. For instance, lartnaze-largeplbee play, ReLOAD+OTR scores 58.0 + 11.9 versus ReLOAD's 61.6 12.0, highlighting a trade-off between distributional alignment and computational efficiency. The marginal gains in antmaze- medium-diverss ( 76.3 + 14.1 + 75.0 + 18.6) suggest that OT's benefits are task-dependent and may not justify its costs in simple environments.
The ReLOAD+OTR hybrid demonstrates that distributional alignment can marginally improve performance in specific regimes but fails to consistently outperform the original framework. This reaffirms ReLOAD's core advantage: its ability to distill effective rewards without complex optimization.
E Effect of Predictor Network Training Quality
To evaluate the importance of predictor predictor training quality in ReLOAD, we conduct an ablation study comparing three regimes: (1) an under-trained predictor, trained for only a few epochs on expert transitions; (2) the standard ReLOAD setup, where the predictor is trained to convergence; and (3) an over-trained predictor, further optimized beyond convergence. Each configuration used to compute intrinsic rewards for offline RL training using IQL.
As shown in Table S performance degrades when using an under-trainen predictor across all tasks, with more significant drop on the replay and expert datascts. This degradation occurs because insufficiently trained predictors fail to produce meaningful prediction errors, thereby introducing noise into policy learn- ing. The impact is particularly highlighted when the offline dataset contains expert trajectories. In such cases, a poorly trained predictor cannot effectively distinguish between expert and non-expert transitions, undermining the ability of the reward model to guide learning. This highlights the importance of sufficiently training the predictor network to ensure that prediction errors reflect similarity to expert behavior.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Table 8: Ablation on predictor network training quality, ReLOAD uses a predictor trained to convergence on expert transitions. Undertraining leads to degraded performance, especially in medium-roplay settings. Overtraining yields similar results to ReLOAD, confirming robustness.
Dataset
Undertrained
ReLOAD (Standard)
Overtrained
halfcheetaah-mediumm
40.4 + 1.3
48.2 - 0.7
45.7 + 0.6
haafcheetal--mediun--rppaay
34.3 1 2.6
42.6 - 0.8
42.9 + 1.1
40.8 t 1.4
92.6 = 2.5
92.3 = 2.2
walker2d-medium
68.2 0.7
81.3 0.9
81.2 1.9
walker2d-mmedhum-rplly
55.9 + 3.9
78.6 = 2.5
76.2 + 2.6
walkkor2d-modduu--xpert
67.6 + 1.4
110.5 0.7
110.9 + 0.9
Table 9: Ablation study on the effect of removing the squashing function from ReLOAD. Removing the squashing function leads to performance degradation, especially in replay datasets.
Dataset
ReLOAD (Standard)
ReLOAD (No Squash)
halfcheettal-medium
48.2 + 0.7
45.09 + 0.6
halfcheetah-medinu--rpaaay
42.6 0.8
42.35 + 0.6
halfcheetah-mmeddum-xpppttt
92.6 + 2.5
88.00 + 3.0
walker2d-medium
81.3 + 0.9
41.40 + 9.2
wallurzdd-mediuu--eplaay
78.6 + 2.5
0.70 + 1.0
walker2d-medium-eeyy
110.5 + 0.7
94.52 + 27.6
In contrast, both the standard and over-trained predictors achieve consistently high performance, suggesting that ReLOAD is robust to mild overfitting of the predictor. Once the predictor has minimized its prediction error on expert data, the reward model provides stable and informative signals for policy learning.
F Effect of the Squashing Function
As shown in Table 9 removing the squashing function leads to inconsistent and often degraded performance, particularly on the replay datasets. For instance, in wallker2f-meediun-rrplae performance drops sharply to 0.70 + 1.0, while in walker2d-medium it falls from 81.3 = 0.9 to 41.40 = 9.2. Even in expert settings like malkerr2t-mdduu-cxper,, where the score remains relatively high, the variance increases substantially.
As detailed in Section ReLOAD transforms the raw prediction error using an exponential squashing function applied to its negative. This ensures that transitions predicted well by the expert-trained model receive higher intrinsic rewards, while poorly predicted transitions are smoothly down-weighted. In contrast, using the negative error directly (i.e., without squashing) results in unbounded and highly variable reward scales, which can destabilize training-especialll in mixed-quality datasets like medium-replay.
These findings highlight that the squashing function not merely numerical convenience but crucial regularizer in ReLOAD. It stabilizes the reward signal and improves robustness by preventing large prediction errors from dominating the intrinsic reward landscape.
19

---

Published in Transactions on Machine Learning Research (10/2025)
G
Effect of a AND B
Table 10: Effect of reward squashing parameters on the performance of ReLOAD.
Environment
ReLOAD (a 5, = 0.5)
ReLOAD (a 10, 5)
mallcheetal-mmddimm
46.5 + 0.8
48.2 0.65
halfcheetah medirtt--aammuu-r-eeaeaaa
43 = 1.6
42.6 = 0.8
halichoeaal-mmchu-eepprrt
92 + 2.8
92.6 + 2.52
walker2d-medium
77.3 + 5.0
81.3 + 0.96
walleer2d-medduu--eppaay
76.8 = 6.5
78.6 = 2.5
walker22d-mdduu--opprt
108.1 3.8
110.46 0.7
pen-cloned
64.5 + 24.6
64.5 + 24.6
pen-human
88.3 + 8.7
88.3 + 8.7
H
Additional Experiment
This section compares ReLOAD with Behavior Cloning (BC), traditional imitation learning method that directly learns a policy from expert demonstrations without incorporating environmental dynamics or re- wards. Table 11 highlights ReLOAD's superior performance over BC most environments within the MuJoCo Locomotion Dataset.
Table 11: Performance Comparison of BC-10 with ReLOAD on MuJoCo Locomotion Dataset
Environment
BC
ReLOAD
halfcheetab-mmedium
42.5
48.2 + 0.65
rell
40.6
42.6 + 0.8
hulfchhetaal-mediim--- expert
92.9
92.6 + 2.52
hopper-medium
56.9
78.72 + 6.9
75.9
95.86 + 2.2
hoppee-neddium-expert
110.9
104.8 + 9.0
walker2d-medium
75.0
81.3 + 0.96
walker22--nedium--rppayy
62.5
78.6 + 2.5
walker2d-meedinm-xxporr
109.0
110.46 + 0.7
Table 12: Porformance Comparison of IQL and ReLOAD on D4RL Locomotion Tasks with varying amounts of offline data. Reported scores are normalized returns (mean + std) over 10 seeds.
DataaetDatsseeeeataspppp -atasetpt>>
Data fraction 10%
Data fraction 25%
Data fraction 50%
Data fraction 100%
IQL.
ReLOAD
IQL.
ReL.OA
IQL.
ReLOAD
IQL.
ReLOAD
helfcheetall-medium hopper-medium medium walker2d- m4mmmimm
45. 05.69
45.520.8
47 17.2402
47.420.
47A20.2
47.740.4
47,410.2
48.2-0.65
58.4+5.0
63.8+7.8
55.8+7.5
10.7
51445.4
65.7+44.3
66.2+5.7
78.72+6.9
02.41:4.2
78.245.9
62. 22.247.4
78.245.8
63.927.4
79.841.
78.315.7
$1.3=0.96
20

---

Published in Transactions on Machine Learning Research (10/2025)
Table 13: Performance Comparison D4RL Locomotion Tasks. The reported results normalized
scores (mean + standard deviation) across over 10 random seeds. The results correspond to K = 5 expert trajectories. The highest scores are highlighted in green.
Dataset
IQL
OTR (K=5)
CLUE (K=5)
AILOT (K=5)
ReLOAD (K=5)
halfcheetall-meee medium
47.4 0.2
43.3 0.2
45.2 0.2
46.6 0.2
47.2 + 0.9
halfcheetah seefcheetaa-meeiun medium- replay
44.2 1.2
41.9 0.3
43.2 + 0.4
41.2 0.5
43.5 + 0.9
halfcheetah--me mulfchhetab-meeiim-esppp mepert-expert
86.7 5.3
89.9 + 1.9
91.4 + 1.4
92.4 1.0
92.8 + 1.7
hopper-medium medium
66.2 + 5.7
79.5 + 5.3
79.1 + 3.5
82.5 + 3.7
78.72 16.9
hopper- mreper--edium-iee med-im--eiiulyy replay
94.7 = 8.6
85.4 + 1.7
93.3 + 4.5
97.4 + 0.1
95.8 = 2.3
hopper-mmedium-eppert
91.5 + 14.3
90.4 + 21.5
104.0 + 5.4
107.3 + 5.6
110.4 = 5.0
walker2ddmmedium
78.3 + 8.7
79.8 + 1.4
79.6 + 0.7
80.9 1.4
81.0 + 2.5
walker22d-mediin--rcppay
73.8 + 7.1
71.0 + 5.0
75.1 +: 1.3
76.9 + 1.6
77.2 + 2.5
wellerr2d-mediim-expprtt
109.6 1.0
109.4 + 0.4
109.9 + 0.4
110.3 + 0.4
110.4 0.7
D4RL Locomotion total
692.4
690.6
721.3
735.5
737.2
21


Reviews:
[{"review_id": "zcG1XYVjpx", "reviewer": "Reviewer_Ro2J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The reward function is simple, motivated, and easily computational. \n2. The experiments are conducted over multiple datasets. \n3. The proposed method achieves strong performance.\n4. Providing ablation studies on the expert and offline dataset sizes. \n\n**Weakness**\n1. The main problem for me is that the reward function, which is the main contribution of this work, was proposed by RND and already applied in offline RL (Nikulin et al., 2023). This greatly weakens this work in terms of theoretical and technical contributions. \n2. The authors don't discuss the differences to the related works, RND and the work (Nikulin et al., 2023). Related discussions on the differences can better position this work and thus are highly encouraged.\n2. I don't understand how the prediction error works. The prediction error is the difference between outputs of the predictor network and the target network taking the same inputs. If these two networks have the same network architecture and initialized parameters, the prediction error will be always zero in the beginning of training. As a result, the prediction error cannot be used for transition assessment.\n3. It is claimed that the reward squashing technique is effective in improving training stability. The corresponding empirical results are missed for supporting the claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the problem of reward annotation in reward-free offline reinforcement learning.  \n\n**1. Reward function**  \nThe reward function is introduced by computing the prediction error between a predictor network and a target network on expert demonstrations. \n\n**2. Justification of RND rewards**  \nIntroducing a rigorous theory for the effectiveness of the RND reward. (I didn't check the proofs in details)\n\n**3. Indirect transition similarity measurement**   \nA high reward indicates the lower similarity between expert transitions and transitions from the offline dataset. \n\n**4. Reward Squashing**  \nProviding a reward squashing technique bounding reward variance for improving training stability. \n\n**5. Extensive experimental evaluations on multiple benchmarks**\n- Performing multiple experiments on different offline RL benchmarks, including D4RL Locomotion, Antmaze, and Adroit.\n- Discussing the reduced computation time compared to AILOT.\n- ReLoad achieves better performance than baselines on D4RL and Adroit tasks, while being competitive on Antmaze tasks.\n- ReLoad performs better across 2 different expert dataset sizes and 4 different offline dataset sizes than baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarifying the connections and differences to the related works, RND and the work (Nikulin et al., 2023). \n1. Providing implementation details of the target and the predictor networks, including network architecture and parameter initialization. \n2. Providing ablations on the reward squashing technique, e.g., comparing ReLoaD to ReLoaD without reward squashing.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7NYKLQZAIi", "reviewer": "Reviewer_mEAa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper is well-written and easy to read.\n\n- RELOAD is straightforward and effective.\n\n- The experiments are extensive and well-conducted.\n\n### Weaknesses:\n- I would like to ask about the distinction between the approach of generating intrinsic rewards using RND followed by offline RL learning and previous methods where RND intrinsic rewards were directly integrated into offline RL training, such as in [1]. What are the fundamental differences between these two approaches? What is the unique advantage of RELOAD, and is there any supporting evidence for this claim?\n\n- While RELOAD is presented as a plug-and-play method, I believe there are some issues with the experimental settings. It would be more convincing to compare RELOAD against existing baselines (e.g., IQL, CQL, DT) first, and then incorporate RELOAD into these baselines to demonstrate its advantages, rather than using IQL + RELOAD as a comparison against other original RL baselines. This comparison seems unfair.\n\n- A key issue with RND is that if the predictor network fails to converge, the generated intrinsic reward may reflect this non-convergence rather than an indication of an intrinsically motivating reward. Therefore, I am particularly interested in seeing a comparison of the effects of rewards generated by predictor networks under different conditions (unconverged, converged, and overfitted) when used as the dataset for offline RL training. How do these different scenarios affect the trained policies?\n---\n*Reference:*\n\n[1] Anti-Exploration by Random Network Distillation. ICML 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "RELOAD is a simple but effective reward annotation framework for offline RL. It utilizes the principles of Random Network Distillation (RND) to generate intrinsic rewards, aiding datasets that originally lack reward signals. The results show that RELOAD achieves an improvement over previous offline RL methods on the D4RL benchmark.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate a detailed, evidence-backed response to the weaknesses raised.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8pwMSgREl8", "reviewer": "Reviewer_GS7r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The method is simple and effective according to the experiment result.\n- This paper provide discussion and justification with the experiment results, which is thorough for explaining the effectiveness of the proposed method.\n\nWeakness:\n- It is a bit unclear what is the size of the training data of the predictor network? In the experiment section, I think it only said from 1-5 expert demonstrations, but how many transitions are included? I am a bit curious whether a small number of samples can really train the model effectively. And the embedding space might not be meaningful enough.\n- I have to admit I am not an expert of RL, and that is why I somehow feel that the technical novel is somehow thin. The proposed method is reusing previously proposed concepts (RND) directly. I am not saying for rejecting this paper, but I sort of want to know more on how impactful this method will be by checking authors response and other reviewers comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel reward annotation framework for offline RL. The method is based on Random Network Distillation which is simple and straightforward. The experiment results on multiple RL tasks demonstrate the successfulness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am confused by some small notation issues, and would love authors to check it and maybe it is my misunderstanding.\n- what is D_a? is it different from D_U?\n- In Sec 5 around Eq. 17 and Eq. 18, I am wondering why use f(s_i, s_i') in Eq. 17 but use x_j=(s_j, s_j') in Eq. 18? I think these uses of notations can be synced.\n- In Sec. 6, it is unclear what is the \"state-of-the-art approaches\" here means? Do you mean online or offline RL methods?", "broader_impact_concerns": "I don't see obvious impact concerns from this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)