SR-Reward: Taking The Path More Traveled
Authors: Seyed Mahdi B. Azad
Zahra Padar
Gabriel Kalweit
Joschka Boedecker
Venue: Accepted by TMLR ()

Abstract: In this paper, we propose a novel method for learning reward functions directly from offline demonstrations. 
Unlike traditional inverse reinforcement learning (IRL), our approach decouples the reward function from the learner's policy, eliminating the adversarial interaction typically required between the two. 
This results in a more stable and efficient training process. 
Our reward module, \textit{SR-Reward}, leverages successor representation (SR) to encode a state based on expected future states' visitation under the demonstration policy and transition dynamics. 
By utilizing the Bellman equation, SR-Reward can be learned concurrently with most reinforcement learning (RL) algorithms without altering the existing training pipeline.
We also introduce a negative sampling strategy to mitigate overestimation errors by reducing rewards for out-of-distribution data, thereby enhancing robustness. 
This strategy introduces an inherent conservative bias into RL algorithms that employ the learned reward, encouraging them to stay close to the demonstrations where the consequences of the actions are better understood. 
We evaluate our method on D4RL as well as Maniskill Robot Manipulation environments, achieving competitive results compared to offline RL algorithms with access to true rewards and imitation learning (IL) techniques like behavioral cloning.

Published in Transactions on Machine Learning Research (06/2025)
SR-Reward: Taking The Path More Traveled
Scyed Mahdi B. Azad
busstri@ecs.udd-frebbuu de
Deportment of Computer Science University of Freiburg
Zahra Padar
Department of Computer Science University of Freiburg
Gabriel Kalweit
kalweity Ccs www.://wswsnnrfirurrr..... de
Department of Computer Science University of Freiburg
Joschka Bocdecker
Department of Computer Science University of Freiburg
Reviewed on OpenReview: https: // openreview. net/ formm? id= Dasks/issu
Abstract
In this paper, we propose novel method for learning reward functions directly from offline demonstrations. Unlike traditional inverse reinforcement learning (IRL), approach de- couples the reward function from the learner's policy, eliminating the adversarial interaction typically required between the two. This results in more stable and efficient training cess. Our reward module, SR-Reward, leverages successor representation (SR) to encode a state based on expected future states' visitation under the demonstration policy and transi- tion dynamics. By utilizing the Bellman equation, SR-Reward can be learned concurrently with most reinforcement learning (RL) algorithms without altering the existing training pipeline. We also introduce negative sampling strategy mitigate overestimation errors by reducing rewards for outto--distributtin data, thereby enhancing robustness. This strat- egy introduces an inherent conservative bias into RL algorithms that employ the learned reward, encouraging them to stay close to the demonstrations where the consequences of the actions are better understood. We evaluate our method on D4RL as well as Maniskill Robot Manipulation environments, achieving competitive results compared to offline RL gorithms with access to true rewards and imitation learning (IL) techniques like behavioral cloning. Code available at: htttp::/giiteee com wtt.:/ore//e/eeeeeeeeeeoooo Reward
1 Introduction
Imitation learning (II.) from expert demonstrations is widely used approach for tackling sequential decision- making tasks. Methods in this domain generally fall into categories. The first focuses on directly learning policy that mimics expert behavior, such Behavioral Cloning (BC) (Pomerleau 1991]. The second, inverse reinforcement learning (IRL) (Ng & Russell 2000), infers reward function that explains expert's behavior and simultancously derives a policy from it. While both approaches have shown promisc, they come with notable limitations. BC struggles with distribution shift, failing when encountering states not covered in the demonstrations (Ross et al., 2010). IRL, while more flexible, often inherits the instabilities of adversarial training (Goodfellow al.] 2014) and requires environment interaction refine the learned reward function an impractical requirement in domains where exploration is costly unsafe, such robotics and healthcare.

---

Published in Transactions on Machine Learning Research (06/2025)
Demos
(s.a.r.s))
Offine RL Agent
Demes
[5.8,8]
Roward
Traning Phose
(s,a,s")
Offine RL
Agent
Training Phoso
Figure 1: Standard offline RL training requires rewards to be present in the demonstrations (Left). SR- Reward enables the use of offline RL algorithms in settings where the demonstrations do not include rewards (Right). Adding the next action for training the SR-Reward requires only small modification to the replay bulfer.
Offline reinforcement learning (RL) provides an alternative by learning policies from fixed datasets without interacting with the environment (Lange al., 2012). However, offline RL typically requires well-defined reward signals, which are often unavailable difficult to engineer in real-world applications. Without reward function, the agent has clear objective and fails to learn meaningful policy.
To address this challenge, we propose SR-Reward, a reward module that enables the use of offline RL agents in the absence of explicit reward functions. SR-Reward is based on the observation that frequently occurring features in expert demonstrations often capture essential aspects of expertise. By leveraging successor reprosentations
1993) to estimate feature frequencies, SR-Reward assigns higher rewards frequently observed state-action pairs. This provides a structured learning signal that helps agents align with expert strategies while retaining the flexibility to adapt to novel situations.
Successor representation provides a way to represent a state based on the frequency of its future visitation. In this work, we leverage SR as the primary mechanism to track how often state-action features appear in the demonstrations. Leveraging the SR structure allows SR-Reward to be learned via the Bellman equation which propagates information about future states and actions through temporal difference (TD) learning, Consequently, it can be integrated into existing training pipelines alongside other RL methods based on TD learning with minimal modifications. Figure T illustrates the role of SR-Reward within the training pipeline, highlighting how it integrates into the overall learning process. Unlike adversarial schemes popular with inverse reinforcement learning (IRL) methods (Ng & Russell 2000 Abbeel & Ng. 2004 Ho & Ermon] 2016), our reward function is decoupled from the policy that is being learned. Decoupling the reward from the policy climinates the instabílities associated with adversarial training and enables the use of a wide range of RL algorithms that were previously unusable due to the inaccessibility of the reward function (Fujimoto
et al. 2019 Garg et al. 2023 Sikchi al. 2023: Xu al. 2023).
Hand-engineering reward function may work for simple scenarios, but it becomes both challenging and error-prone in complex real-world tasks, such robot manipulation (Singh et al. 2009 Wu et al. 2022). In
contrast, demonstrating the desired behavior is generally more straightforward (Wu et al. 2023 Arunacha- lam et al. 2022 Rakita et al. 2017). It is in such scenarios, that SR-Reward's ability to learn a dense reward function from demonstrations can be valuable. Another avenue for using the demonstrations is to copy the expert's 's actions. Simple imitation learning methods like behavioral cloning (BC) directly mimic expert be- havior without modcling how actions lead to future states. Unfortunately, this short-sighted objective is prone to distribution shift when encountering unseen states. Re-Rew enables the use of TD learning methods, which mitigate this issue by gaining long-term view of the task via bootstrapping, making them more resilient to out-of-distribution scenarios.
We use function approximation to implement the SR mechanism in continuous state and action settings. Function approximation can lead to a significant overestimation of values for out-of-distributio data (Thrmn & Schwartz 1999). This is particularly problematic in our sotting as the expert demonstrations by definition cover only a narrow subset of the overall space. We introduce a negative sampling strategy designed counteract the overestimation error in our reward function for out-of-distribution states and actions. This is
2

---

Published in Transactions on Machine Learning Research (06/2025)
accomplished by augmenting the Bellman loss for SR-Reward that reward estimates for out-of-distribution states and actions decrease based on their distance from expert demonstrations. Incorporating negative sampling not only enhances the robustness of the reward function but also introduces a natural conservatism into the value functions and policies that rely on it. This approach encourages agents to remain near the remonstrated behaviors, where the outcomes of their actions are better underslood.
In this work, we focus on the offline inverse reinforcement learning setting, where the agent neither has access to the reward function can query the expert for any feedback. Furthermore, the transition dynamics of the environment unknown and the agent is provided with limited data in the form of expert demonstrations, In summary our key contributions are:
1. SR-Reward: reward function based Successor Representation (SR), that can be learned solely oflline using temporal difference (TD) learning simultaneously with other RL algorithms. We further describe our architecture as well as loss functions needed for training SR-Reward.
A negative sampling strategy to mitigate overestimation errors by reducing rewards for out-of- distribution data in the vicinity of the expert demonstrations, thereby increasing robustness by introducing conservative bias into RL algorithms that employ the learned reward.
2
Background
We first introduce the notation and provide more detailed review of concepts from successor representation and imitation learning.
2.1 Notation
We consider settings where the environment is represented by Markov Decision Process (MDP) and is defined a tuple M T,,,, 8 and A represent the continuous state and continuous action spaces respectively. T(s'ls, a) represents the state transition dynamics, r(s, a) represents the reward function, E 10 is the discount factor and Po represents the starting state distribution. In the offline inverse reinforcement lcarning setting, we only have access to a limited set of expert demonstrations of the form D = ((0,, ag, 81, (i, (.T)""")))') =o this paper, we are locusing on a limited setting where neither the transition dynamics T(s'ls, a) nor the reward function r(s, a) are known. The goal is learn reward function re a) from expert demonstrations such that its corresponding policy no(als) performs similarly to that of the expert.
2.2 Successor Representations
Successor Representation (SR) was originally introduced method to generalize the value function across different rewards (Dayan 1993 SR is defined as the cumulative discounted probability of visiting future states when following specific policy, effectively representing the current state (and action) in terms of potential future states (and actions).
For any given pair of states s' and actions a, a', the SR is expressed as:
M(s, a')
II(s s',at - a') s0 5. = a
where the expectation is taken over the policy <{ass) and the environment's transition dynamics [(s'(s, Similar to the Q-function, SR can be estimated using the recursive Bellman equation:
((s,, des s',a') 1(s, =' d') +E (,1,,,
This recursive formulation is particularly useful when learning SR alongside other temporal difference (TD) methods. Our SR-based reward function leverages this recursive approach, allowing the reward network
3

---

Published in Transactions on Machine Learning Research (06/2025)
Encoder
((s)
((s)
MLP
SR
Predictor
o'(s')
Figure 2: The architecture of the SR networks. The output the Encoder is concatenated with the action to produce Φ(s, a) (d()) in Equation [ The result passes through fully connected network (MLP) to create the SR(s, a) vector. The Predictor network is used for an auxiliary task to help train the Encoder. It predicts (s'),, an estimate of the true encoded next state ((8'), from ( (Ø()))
to be trained in parallel with the actor and critic networks, with minimal changes the existing training pipeline.
However, directly estimating SR using these formulations becomes computationally intractable as the number of states and actions increases, or when transitioning from discrete to continuous domains. To address this, previous research (Kulkarni et al. 2016 Machado et al. 2020 Zhang et al. 2017 has extended SR to continuous state and action spaces using Successor Features Representation (SF). SF expressed in terms of state and action features vector a(s
M (st, a.) o(st, ar) + [M (8
(1)
Here (St, (x) is the expected future occupancy of the state-action features for (t) where each element of this vector can be seen tracking the SR for single feature of the state-action space. The choice of feature extractor & is design decision that depends on the environment. Most existing work focuses on extracting
features only from the state, not the actions. In this scenario, d(s, a) can be represented as ( (*(e) ), which is a concatenation of state features and actions. In this work, we adopt this approach and use a feature extractor network to derive features from the state only.
2.3 Imitation Learning via Distribution Matching
Methods like behavioral cloning (BC), which directly learn a policy T(als) mapping states to actions, are straightforward and elfective when ample data is available. However, they are prone to distribution shift because they only match the observed action distribution (Ross et al. 2010). During inference, as the distribution of encountered states deviates from those seen during training, the accuracy of action predictions diminishes. This leads to accumulating errors that the policy cannot correct.
Nachum et al. 2019 Ghasemipour et al. 2019 Ke et al.) 22200] Kostrikov et al., 2020), are more robust to distribution shifts since they aim to match both the state and action distributions encountered during training. This helps keep the policy close the states observed in demonstrations.
Distribution matching methods, and related approaches Hoo & Ermon 2016 Fu et al. 2018
Formally, the occupancy measure of state-action pair under policy ग can be defined
TT(s =, = a )
(s, E
Σ
1=0
where is the indicator function, which equals one the condition met and zero otherwise. This is closely related to the state-action distribution d' (s) (1 >) (s, a). Considering that there is a one-to-one correspondence between the stato-action distribution and the policy (Puterman 1994). distribution matching methods aim to indirectly learn policy by minimizing the divergence between Eappet and d= A common
4

---

Published in Transactions on Machine Learning Research (06/2025)
Environments used for experiments. From left to right: 2D Toy Maze, MuJoCo environments: [Ant, Walker2D, HalfCheetah, Hopper], Adroit Hand environments: [Door, Pen, Hammer, Relocate]. We use the states provided by the environments in our experiments.
Figure 3:
choice KL-Divergence, and minimizing Dx (d (d"]eeepper) can be viewed as maximizing the following RL objective where the reward is given by the log ratio of the state-action distributions between the expert policy and the learned policy (Kostrikov et al. 2020b.
o
(Expert (s,a)
E
r log
ds
Since the state-action distribution is often unavailable, efforts typically focused on estimating the ratio of the two distributions (Ho & Ermon 2016 Nachum et al. 2019
In this paper, we propose a method that estimates SR as a proxy for the expert's state-action distribution from demonstrations and uses it as a reward for downstream RL algorithms.
2.3.1 Relationship between SR and State-Action Visitation
SR implicitly captures the state-action visitation. Many density-based II. methods, such as GAIL. (Ho & Ermon 2016), use state-action distribution or occupancy measure for their distribution matching techniques. In Appendix we derive the following close relationship between the occupancy mcasure and the successor representation
p(s' a') Papppp.ss (M a, a',''')))
The occupancy measure p(s', a') be the expectation of successor representations M(s, s', with respect to the probability of all state-action pairs (s, a) that preceded (s', a'). We learn successor features representation (SF) from the expert demonstrations and use it as a proxy for the expert's occupancy measure.
3
SR-Reward
3.1
Architecture
We the architecture shown in Figure to estimate the SR vector in continuous state and action settings. Our architecture is built upon the works of Machado et al. (2020). Kulkarni et. al. 2016), and Borsa et al. (019 with a few notable changes. First, our SR network extends the previous works to include the action when estimating the SR. This is important as our SRR-asedd reward function r(s, a) is a function of both the state and the action and needs to distinguish the reward values different actions. Second, it is common to use an auxiliary task when learning the encoder from scratch. Kulkarni et al. (2016) use the reconstruction of the state the auxiliary task, while Machado al. (2020) opt for prediction task in which the next state is predicted from the encoded state and the action. Inspired by the results Ni et al. (2024), use auxiliary task. Given the encoding of the current state o(s)
the prediction of the next encoded state and its corresponding action in the dataset we predict the encoded next state Φ(s'). We use the 12 loss for this auxiliary lask. Finally, our encoder consists of fully connected layers with ReLU activation layer as the final layer. We normalize the feature vector to ensure that all features are in the same range, such that [[)())] as suggested by Machado al. (2020). If the environment dynamics are not fully Markovian one can use a history of states as and replace the fully connected layers of the encoder with LSTM layers proposed by Borsa al. 2019

---

Published in Transactions on Machine Learning Research (06/2025)
3.2 From SR Vector to Scalar Reward
Machado et al. (2020) shows that the norm of SR implicitly counts the state visitation. Motivated by this result, we use the 7--norm of the SR vector as our reward function. Intuitively, each element of the SR vector, estimated using Equation | is the expected discounted sum of feature i of the stale according to the policy that created the demonstration dataset. Hence aggregating all the elements of the SR vector in our offline setting be seen as visitation count of the state-action pairs when following the demonstration policy. If the demonstrations are created by an expert, ||SR(s, a)llz represents how often the expert has visited (x, while performing a task. Taken as the reward for offline RI,, we set out to find a policy that maximizes the statc-action visitation of the expert. We empirically show that we can learn competitive policies using this reward function.
3.3 Negative Sampling
Neural networks tend to iverestimate the value of out-of-ddittributiin
200
data points (Thrun & Schwartz 1999. Fujimoto et al. 2018 2019
150-
Ball et al.] 2023). The overestimation error is especially concern- ing in our setup because an overestimated value of the reward for unseen states and actions will encourage the value networks and subsequently the policy to diverge from the expert demonstrations. Motivated by the idea of conservative value function via negalive sampling Luo et al. (2020), we develop our negative sampling strat- egy to combat the overestimation error of our SR network. Similar Luo et al. 2020) we create our negalive samples & and by adding a small Gaussian noise to states and actions from our expert trajec- tories. However, instead of subtracting the (2-norm of the difference vector ||s - 8112 from the reward estimate of the negative samples, we decay the values using Gaussian, exp - with a con- trolling the strength of the decay. Furthermore, we apply negative sampling not to the space of value functions but to the space of rewards. This possible in our setting where a reward function is estimated and used for learning a policy. Having control over the reward function in this setting provides the opportunity to build conservatism directly into the value functions and subscquently the policy by modifying the reward instead of forcing the value function or the policy to act conservatively
Return
Without Neqative Sampling With Negative Sampling
50-
0.0
0.2
0.6 Noise Fraction
0.4
0.8
1.0
Figure 4: Mean return of corrupted export trajectorics for Relocate envi- ronment. Negative sampling signifi- cantly reduces the reward for states and actions further away from the ex- pert demonstrations.
Fujimoto et al. 20188 2019); Kumar et al. (2020).
Our negative sampling strategy is effective primarily within the local vicinity of expert demonstrations, achieved by introducing perturbations to expert trajectories. Therefore, does not prevent reward over- estimation for state-action pairs that significantly diverge from the expert's behavior. As a result, the SR-Reward is more suitable for offline settings, where exploration is limited. In online settings, where the agent can encounter unfamiliar state-action pairs with overestimated rewards, this could lead to suboptimal policy learning.
Figure 5 shows the effect of our negative sampling strategy on a toy environment. We train an SR network using ten demonstrations with and without negative sampling and evaluate the rewards over the grid space for each one of the cardinal directions. The mean value plot Figure F showss how negative sampling during training prevents overestimation error for the rewards of the stato-action pairs not seen in the demonstrations. Plots for the four main directions show higher reward estimates for movement in the corresponding direction. For example, the Left plot shows the reward for moving left at every grid point and expected this reward is higher for the upper portion of the trajectories where the expert has moved left.
To highlight the impact of negative sampling in mitigating overestimation errors, Figure compares the mean trajectory returns of models trained with and without negative sampling. Expert trajectories are corrupted by varying levels of Gaussian noise, and their episodic returns are estimated using RR-Reward. The model trained with negative sampling exhibits significantly lower returns for corrupted trajectories, reflecting its enhanced sensitivity to out-of-distribution data. This effect is evident. in the initial drop in returns when
6

---

Published in Transactions on Machine Learning Research (06/2025)
Right
1.0
0.8
0.6
0.2
(a) Without Negative Sampling
0.8
0.6
0.4
02
(b) With Negative Sampling
Figure 5: The plots show the effect using negalive sampling for 2D Toy Maze environment (Figure whith continuous states and actions. The black lines represent the trajectories of the expert starting near the bottom-right and moving counter-clockwise towards the goal near the center. The mean of the SR-Reward over four dircetions [Up, Right, Down, Left] and the SR-Reward associated with each direction is plotted. Using negative sampling significantly reduces the extrapolation error for out-of-distribution state-action pairs.
Gaussian noise with = 0.1 is applied, followed by a much steeper decline as the noise level increases. In contrast, the model trained without negative sampling produces similar reward estimates for both expert and corrupted trajectories, with returns showing only a modest decline under substantial noise levels. Different environments may demand varying degrees of sensitivity to noise and out-of-distribution data. Our negative sampling strategy offers fiexibility, allowing customization for each environment by adjusting the perturbation noise applied to expert trajectories and fine-tuning for the decay rate in Equation 2
3.4 Training
We employ several loss functions to train our SR network. As mentioned in Section 2.2 we can estimate the SR using the Bellman equation in continuous state-action setting. The reward for the Bellman target in Equation is replaced with o(s,a) a) (o()) which is the concatenation of the encoded state and the action. We use the 12-loss to minimize the Bellman error:
Beilman (s.a,, (s, (o(s,a) + a())))
Auxiliary lasks such as predicting the reward, next state, or next latent state have been shown to facilitate the learning of useful latent representations for downstream tasks Fujimoto et al. (2024 Ni et al. (2024). To support encoder training, we incorporate an auxiliary prediction task in which the model learns to predict the next encoded state ((s') from the current encoded state Ø(s) and action a. We compute the 22-loss
Preediction = (s
[(o(s') Predictor((((s).)))
Ensuring that the reward is bounded is crucial for the stability of reinforcement learning algorithms. Prior works have addressed this in various ways, such clipping observed rewards adding penalty the magnitude of the estimated reward MMih et al. (2015); Garg et al. (2021). In our approach, we introduce an additional loss term that penalizes reward magnitudes exceeding 1, effectively encouraging a soft upper bound on the reward. This regularization was found to improve training stability. As described in Section 3.2 we define the reward as the norm of the SR vector.
Magnittdde B. [max Rewardd (8,a) 1,0)

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm SR-Rewaad + RL
Given: D: a,s,a,s',co "'anss''')'''" 1= Y: 3,
Initialize: Encoder: Enc, Fully Connected Block: MLP, Predictor: Pred
for cach training step do
Sample (8, ",,as',er s',a') ~
5: Ø(s) - Enc(s)
6: o(s') + Enc(s')
7: SR + (o()))(s),a)
Rharget ((+) (M.
C Bellman MSE(SR, SRearget)
10:
(pred(5') + Pred( (s))
11:
C Prediction
MSF. ( ((8))
12:
- ||SR|2 1
13:
C Magnitude f- (max(( - 1, 0))
14:
+ s NO. 9)
15:
r a + NO, (3)
16:
((3) + Enc($)
((x)
) (()) ll2
17:
Ocdecay + exp(
a² á
18:
SR + MLP(6(5)
19:
7t |SR|2
20:
Neg.Sampte + MSE(F, Oedecay x r)
21:
Totat t- Rallman + Prodicttion + L Magnitude + Neg.
8 (2), (4), (g), t (7)
22:
23:
RL(s,
24:
end for
Finally, we add a negative sampling loss to improve the robustness of the reward function for out-of- distribution state-action pairs. Similar to Luo et al. (020)) create negative samples and ā by perturbing states and actions from the demonstrations with noise. While might expect negative samples to overlap with the distribution of demonstrations and affect the estimation of the SR, prior work Luo et. al. (2020) shows that demonstrations typically cover only a small subset of the state-action space. As a result, negative samples are highly likely to be orthogonal to the demonstrations, an effect that becomes more pronounced as the dimensionality the environment increases. We use isotropic Gaussian noise N( 3) to create the negative samples. The hyperparaneter B controls the standard deviation of the Gaussian noise. Intuitively, we want perturbed state-action pairs (5, û) have lower reward values proportional the distance from their counterpart (s, a) from the dataset. Since SR estimates the visitation count based Φ(s, a) ( (d())) measure the distance between the negative samples and their original counterparts in the space of features and actions ( (o(+)) ). We calculate the decay factor using an exponential kernel as
a decoy = exp (-l1(s,,)) 0(5, a)22
(2)
o²
a can also be adjusted as a hyperparameter. Higher values of will produce a softer decay for the reward of negative samples. The (2-loss is used corroct the estimation of SR for negative samples:
Neg Sample B(s,a)<D [Reward (3, a) addecay x Reward a))
We train SR network using the summation of all losses total loss:
L L.ellmm Prrediction + Magnitudee &Neg.Sampl
Algorithm D shows the pscudocode for training the SR-Reward and the offline RL in the same loop. The sampled transitions used for training the SR networks have the form (s, a, 8' 8,a') which is different from the

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Normalized mean return and standard deviation over five seeds from different algorithms trained on D4RL datasets (Fu et al. 2020). Offline RI algorithms using SR-Reward perform similarly the ones using the true reward from the environment. Maniskill2 datasets Gu al. (2023) (PickCube, StackCube, TurnFaucet) do not contain rewards and only compared to BC.
f-DVL
sparseQL
Env
BC
True Reward
SR-Reward (Ours)
True Reward
SR-Reward (Ours)
Ant.
86.33 + 3.91
84.35+44.44
8.97
86.64+55.14
82.25+7.09
Hopper
108.73 4.39
110.74+1.14
108.25=3.16
110.32+2.69
109.59#0.52
Halfcheetah
104.87 + 1.55
104.70++0.66
103.80+1199
106.23 1.38
105.96 + 0.82
Walker2d
71.391 14.95
84.41:11..49
78.48=5.52
84.02+:3.75
74.48 = 7.85
Door
76.93 + 21.81
96.97 4.70
98.40 + 3.33
78.28 + 23.98
104.10 + 1.57
Hammer
114.30 = 4.59
90.53 = 10.28
111.52 = 9.74
71.32 + 18.50
117.17 = 3.71
Pen
104.95 = 5.70
109.67 = 10.06
97.79 = 5.15
108.75 = 3.91
105.00 + 3.57
Relocate
93.20 = 4.15
92.36 + 4.35
88.22 7.30
80.69 = 8.15
90.91 = 6.99
PickCube
91.79 = 4.23
88.81 5.72
96.70 = 2.21
StackCube
51.41 + 3.97
70.71 12.55
71.03 + 18.41
TurnFaucct
20.26 + 1.27
35.44 6.83
51.64 = 9.41
ones typically used for RL due the addition of the next action d'. This form of transition, however, can be easily produced with access to set of demonstrations D. We warm-start the training loop by pre-training the SR networks for 10,000 steps before using its 3R-Reward to train the RL agent.
4
Experimental Evaluation
4.1 Setup
To evaluate the proposed reward module, we integrate it with two distinct offline RL algorithms: f-DVL (Sikchi et al. 2023) and SparseQL. (Xu et al. 2023). Both algorithms, which build foundational concepts from IQL [Kosrrkovv al., 2022) and XQL Garg et al., 2023), have demonstrated enhanced stability and strong performance in oflline reinforcement learning settings. In our experiments, we replace the rewards in the offline dataset with those generated by SR-Reward, allowing the reward function to be learned in conjunction with the RL algorithms.
For empirical validation, we utilize the widely-used MuJoCo-based (Tooorov et al. 2012) environments for locomotion tasks, and the Adroit hand (Rajeswaran et al. 2018) environments for assessing performance on more realistic, hand-engineered rewards. Figure g illustrates these environments. We also include three challenging environments from Maniskil12 (Gu et al. 2202),, shown in Figure 0 These environments pose greater difficulty than MuJoCo and Adroit due to variable start and goal positions, extended time horizons, and the complexity of manipulating objects with only two fingers. As real-world applications, these datasets do not include rewards, prohibiting the use of oflline RL algorithms. We use SR-Reward in combination with SparseQL as the RL agent, for Maniskill2 environments as well in ablation studics for data size, data quality, and negative sampling (Appendix A
Figure 6: PickCube (Left), StackCube (Middle) and TurnFaucet (Right) from Maniskill2 (Gu et al. 2023
B and
We use state information from the enviromment and datasets, and hence do not use an image encoder for our experiments. Each agent is trained for one million gradient steps (two million gradient steps for ManiSkill environments), with five different random seeds used per task. For each training run (across five random seeds), we save the checkpoint that achieves the highest mean return over 25 evaluation rollouts, measured every 10,000 steps during training. After selecting the best checkpoint for each seed, then evaluate it
9

---

Published in Transactions on Machine Learning Research (06/2025)
50 fresh rollouts and report the mean and standard deviation across the five seeds (one mean return per seed). Table 1 shows the mean return and standard deviation over five seeds for different environments.
Hyperparameters remain largely consistent across environments, with key parameters listed in Table (Ap- pendix Dj. Further discussion on the hyperparameters of the negative sampling strategy and how to choose them is presented in Appendix We use the offline datasets from D4RL, (Fu al.) 2020)) and follow their normalization procedures, employing the provided scores for random and expert demonstrators (Appendix
E
Our experiments aim to answer the following key questions:
Can SR-Reward effectively replace the true reward signal for offline RL?
4.2
2. How does the performance of offline RT. algorithms using SR-Reward compare to that of BC 44.3
3. Is negative sampling strategy necessary component of SR-Reward?
4.4
4.2 Question Can SR-Reward replace the true reward signal?
To address this question, we use the true reward signal from the environment as a baseline and compare the performance of offline RL algorithms (f-DVL and sparseQL) using the true reward against the reward signal generated by SR-Rewand Reward.
The MuJoCo and Adroit Hand environments differ significantly in the complexity of their reward functions. In MuJoCo environments, the objective is relatively simple: the agent must move to the right as quickly as possible. A straightforward reward function based on the agent's velocity suffices for such tasks. contrast, Adroit Hand environments present more intricate challenges, such as rotating a pen with one hand or hammering a nail. These tasks require dense reward functions, which are constructed using multiple carefully designed sub-rewards and thresholding.
Table summarizes the performance of the offline algorithms using SR-Reward and the true reward. For the MuJoCo environments, the performance achieved with SR-Reward closcly matches that of the truc reward, remonstratinng that the dense reward generated by SR-Reward is as informative as the euvironment-proiiee reward. Notably, in the Adroit Hand environments, RR-Reward yields similar or even higher performance than the true reward. This result highlights the inherent challenges of manually engineering reward functions for complex tasks and underscores the advantages of using SR-Reward, particularly for scenarios where crafting reward function is not straight.orward
Furthermore, Figure g and Figure g present ablation studies on dataset size and quality, demonstrating that SR-Reward can produce an informative reward signal and sustain competitive performance even under limited or degraded data conditions. While overall performance decreases as the dataset size is reduced or the data quality deteriorates, this trend affects all algorithms similarly. Notably, RL agents (SparseQL) trained with SR-Reward maintain performance levels comparable to those trained with the true environment reward or to behavioral cloning agents, even when learning from suboptimal data. (See Appendix A and Appendix B for further discussion.)
4.3 Question2: How does RR-Reward + offline RL perform compared to BC?
Behavioral Cloning (BC) is a straightforward and effective method in scenarios where no reward function exists, but ample demonstrations are available. To highlight the advantages of using oflline RL equipped with SR-Reward, we compare its performance against BC across MuJoCo, Adroit Hand, and more complex ManiSkill2 robot manipulation tasks. In the latter, no true reward signal is available in the datascts, and the offline RL algorithms rely solely on SR-Reward for reward generation.
As shown in Table D combining SR-Reward with offline RL performs on par with BC for MuJoCo and Adroit Hand environments. However, as task complexity increases, the performance gap widens. This trend is particularly evident in the ManiSkil2 environments (Figure 7 (Left)). where BC closely matches the performance of ST--Reward-equipped sparseQT. for the relatively simple PickCube task but falls behind in
10

---

Published in Transactions on Machine Learning Research (06/2025)
Pickcube
Stackcube
Turffuucet
Rikcuue
Stackcube
Turffuucet
150
350
190
150-
150-
350-
500
300
220-
130-
Normalized
Normalized
23
S4-Raword
58 Renard
No..5.
N.S. (VNS)) N.S. (our))
No..s.
(VINS) N.. (aan
KeN.S.
(MN6) (RUT))
Figure Performance on Maniskil12 environments. Turning the faucet requires a contimuous connection with the edge of the handlebar throughout the movement, and stacking the cubes requires more precision than simply relocating them. SR-Reward + RL (sparseQL) outperforms BC as the difficulty of the task
increases (Left). The benefits of using Negative sampling (N.S.) become more prominent for more difficult
tasks (Right). Our negative sampling based exponential kernel outperforms that of Luo et al. (0020) (VINS), especially on the StackCube task.
the more demanding StackCube and TurnFaucet scenarios, which require higher precision for successful completion. Similar results hold when combining f-DVL with SR-Reward, although BC shows slightly better performance on the simpler PickCube lask but falls behind on StackCube and TurnFaucet.
We have limited our experiments to two offline RL algorithms (f-DVL and SparseQL), however, our findings underscore the versatility of SR-Reward as a modular component that can, in principle, be compatible with many offiine RL algorithms by replacing the reward function. This flexibility makes it well-suited for applications in robotics and other domains where obtaining demonstrations is more feasible than designing dense reward functions, and where RL algorithms have the potential to surpass the limitations of BC.
4.4 Question3: negative sampling strategy necessary component of SR- ER-Reward?
A narrow subset of the overall state-action space covered by the demonstrations can lead function approx- imation networks to predict erroneously high values when queried for out-of-distribution data. Intuitively, the negative sampling strategy presented in this paper can be beneficial in increasing the robustness of the networks used in SR-Reward by lowering the resulting reward values for state-action pairs outside demon- strations. To evaluate its effectiveness, we train SR-Reward both with and without the negative sampling strategy and compare the results in the PickCube, StackCube, and TurnFaucet environments (Figure
As shown in Figure (Right), the use of the negative sampling strategy leads to similar or improved perfor- mance across all tasks. Additionally, we benchmark this approach against the lincar decay strategy proposed by Luo et al. 22020), applied to SR-Reward (rather than the value function). Our results demonstrate that the exponential decay kernel used in our negative sampling strategy yields better performance. While the negative sampling strategy consistently enhances results across tasks, the magnitude of its contribution varies depending on the task.
5 Related Work
Learning to perform task from offline data has been extensively studied under the IL and IRL unbrella
(Abbeel & Ng. 2004 Ho & Ermon 2016 Fu et al. 2018 Garg et al.) 2021 Kostrikov et al. 2020 Kalweit et al.
2020 Pomerleau 1991). One common approach is methods based behavioral cloning (Ponerleau] 1991] which reduce imitation learning to supervised lcarning problem, i.c., learning mapping from environment states lo expert actions. They aim lo increase the probability of expert actions for the states seen in the demonstrations. Although this approach can work in simple environments with large amounts of data, it is inherently myopic and fails to reason about the consoquences of its selected actions. Consequently, such greedy approaches suffer from compounding errors due to covariant shift (Ross et al. 2010) when the agent deviates from the demonstrated states.
11

---

Published in Transactions on Machine Learning Research (06/2025)
In contrast, IRL methods incorporate information about the environment dynamics into the decision- making process by imitating the expert actions as well as the visited states (Abbeel & Ng. 2004p. Many IRI. methods, such as GAIL (Ho & Ermon 2016) and its extensions, simultaneously estimate the reward function that best explains the expert behavior and its associated policy. This optimization is done using an adversarial schome with the discriminator trying to distinguish between the expert trajectories and ones generated by the learned policy. Simultancously, the discriminator's error is used as the reward signal for training the policy. The adversarial nature of the training strategy makes such algorithms prone to training instabilities (GoodiiiesssGoooelllllll et al. 2014 Kostrikov et al.] 2019). Additionally, they require further interactions with the environment during training to create a dataset of non-expert trajectories for training the discriminator. Furthermore, there are no theoretical guarantees that show adversarial training to lead to a better performance than a two-step process, which first infers a reward function from demonstrations, followed by learning a policy using the previously inferred reward (Liu et al. 2021). , In this work, we part from adversarial training and so decouple the learning process of reward function and policy while training both simultancously.
Already there have been efforts in bypassing the adversarial optimization. Kalweit. et al. (2020) derived analytical solution for the reward based on the assumption that expert policy follows Boltzmann distribution. Their formulation applics to continuous states but is limited to discrete action spaces. While sharing a similar objective to ValueDICE (Kostrikov et al., 2020j, Garg et al. (2021) removes the need for adversarial training by formulating the reward function in terms of the value functions and maximizing them. Their objective function implicitly reduces a distance measure, such as x²-divergence, between the occupancy measure of the expert and the one of the policy being trained. This approach does not yield an explicit reward model, but a reward value can be extracted from the learned value function and policy. Without optimizing for the optimal reward function Roddy et al. (2020) uses a simple binary indicator the reward which distinguishes between expert demonstrations and online interactions. Our reward function can be seen as the continuous version of SQIL (Reddy et al. 2020) because the SR value of states and actions that are visited by the expert, will be naturally higher than the rest.
The growing need to fine-tune large language models has led to a surge of research focused on learning reward functions from human feedback and using them for model optimization (Christiano et al.) 2023 Rafailov
ct al. 2023 Ethayarajh et al. 2024). These approaches typically assume a Bradley-Terry model [Bradley
& Terry 1952) to represent the reward function and rely datasets containing both positive and negative leedback. In contrast, our work focuses on learning a reward function solely from expert demonstrations, without requiring any form of negative feedback. There is also a series of works that modify the existing enviromment reward to gain improvements during training. Vieillard et al. (2020) suggests adding log(t(a(ss) of the policy π that is being learned to the reward in temporal difference (TD) learning. The authors argue that the logarithm of the policy is a strong learning signal as is available even in a sparse reward setting and since its value is close to zero for optimal actions under optimal policy this does not conflict with the optimal control objective.
Recent works have aimed to use successor representation in reinforcement lcarning Barreto et al. 2017 Zhang et al., 2017 Filos et al., 2021 Brantley et al.) 2021 Jain et al.) 2024). While Barreto et al. (0117) and Zhhnn et al. (2017) use SR to generalize the value function to different rewards for transfer learning,
Brantley et al. (021)) focus on generalizing the representation SR over policies for small partially observable
enviromments with known dynamics. In multi-agent settings Filos et al. (2021) learn the shared features of
the environment using the estimated SR of all other agents irrespective of their goals. JJinn et al.. (2024) uses SR in the context of learning from demonstrations by matching the SR of the learner's policy to that of the expert. We share the same motivation for our work but we use SR as a reward function and do not require online interactions during training. Other works such as (Moskovitz et al.] 2022 Machado et al. 2020) modify the reward using the SR of the policy that is being learned. Machado et al. (2020) showed that the norm of the SR vector can act as the proxy for the state visitation count. They modify the reward from the environment by adding the inverse of this state visitation count during training. Moskovitz et al. (2022) suggest modification to SR only consider the first. visitation of state, hence learning the expected discounted time to reach succossor states. Similar to Machado et al. (2020), they also make use of the inverse of their modified SR norm and show improved performance, especially in scenarios where the reward in a given state will be depleted after the first visit.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Our work is similar to the one of Machado et al. (2020) in the sense that vector as proxy to state visitation count. However, we are working in offline IRL setting where there is no other reward available and the dataset is fixed. We show that in the absence of a reward signal from the environment, one can use the norm of the SR vector directly as the reward. Additionally, we extend the SR vector to continuous actions and employ a negative sampling procedure to lower the value of our SR-based reward for state-action pairs the vicinity of the demonstrations that were not present in the demonstrations dataset, hence combating the extrapolation error and creating a more robust reward function for offline RL algorithms.
also viewing the norm of SR
6 Conclusion
We introduced SR-Reward, reward function based on successor representation, which is learned from offline expert demonstrations. This reward function assigns high rewards to state-action pairs frequently visited by expert demonstrators. SR-Reward is independent of both policy and value functions but can be trained concurrently with them, enabling easy integration with various RL algorithuns without requiring significant modifications to the training pipeline.
Additionally, we implemented negative sampling strategy to encourage a pessimistic estimation of rewards for out-of-distribution state-action pairs, thereby making the reward function more resistant to overesti- mation errors. Our empirical results demonstrate that SR-Reward can effectively serve as a proxy for the true reward in scenarios where no reward function is available or where the complexity the task makes it difficult to hand-engineer sufficienly informative reward functions.
7 Limitations and Future Work
We focused our experiments on offline scttings because the negative sampling strategy can only protect the SR-Reward from overestimation errors near the expert dermonstrations, where meaningful negative samples are generated by perturbing expert trajectories. Since expert trajectories cover only small portion of the state space, high extrapolation errors can be expected in regions far from these demonstrations. Conse- quently, in online RL, when the agent explores areas distant from the expert trajectories, it may be misled by inflated rewards, leading to the learning of suboptimal policies.
SR-Reward assumes the availability of dataset of optimal trajectories. Although our empirical results indicate a degree of robustness when combining optimal and sub-optimal datasets (Figure g the presence of sub-optimal demonstrations can negatively impact SR-Reward since the training process treats optimal and sub-optimal demonstrations equally. Enhancing the ability to control the influence of demonstrations based on their quality could lead to figher-quality rewards and more data-efficient learning, offering a promising direction for future research.
Given that the successor representation is closely linked to occupancy measures and state-action distributions, the SR-Reward function proposed here can be employed to approximate the stale-action distributions both expert and non-expert actors. This paves the way for developing new algorithms imitation learning (IL) and inverse reinforcement learning (IRL), enabling the direct matching of distributions using an approximate model of state-action distributions. We consider this an exciling direction for further exploration and future research.
Finally, a thcorotical investigation rogarding the convergence propertics using SR as a roward or the efficacy using L2 Norm for converting the SR vector to a scalar reward can provide more support for our claim, however, in this paper, we have focused on supporting our claims using empirical results.
8 Acknowledgment
This work funded by Carl Zeiss Foundation throught the ReScaLe project.
13

---

Published in Transactions on Machine Learning Research (06/2025)
References
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning, In JCML '4 Proceedings of the twenty-first international conference on Machine learning. ACM, 2004.
Sridhar Pandian Arunachalam, Irmak Guzey, Soumith Chintala, and Lerrel Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. arXiv preprint arXiv:221000463, 2022.
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (ods.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Reseurch, pp. 1577 1594. PMLR, 23 29 Jul 2023.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor leatures for transfer in reinforcement learning. Advonces in neural information processing systems, 30, 2017.
Diana Borsa, André Barreto, John Quan, Daniel J. Mankowitz, Hado van Hasselt, Rémi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 -45, 1952. ISSN 00063444, 14643510. URL http://ww/
astor.org/staae org/staable/333029
Kianté Brantley, Soroush Mchri, and Gooff J Gordon. Successor feature sets: Generalizing successor repre- sentations across policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11784, 2021.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences, February 2023. URL http::/arxiv/ wwtp://///bbb/..oa0.53303
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Newral Comput., 5(4):613 624, 1993.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment prospect theoretic optinization. In Proceedings of the 4 11st International Conference Machine Leurning, ICML'24. JMLR.org, 2024.
Angelos Filos, Clare Lyle, Yarin Gal, Sergey Levine, Natasha Jaques, and Gregory Farquhar. Psiphi-learning: Reinforcement learning with demonstrations using successor features and inverse temporal difference learn- ing. In International Conference on Machine Learning, pp. 3305-3317. PMLR, 2021.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Voncouver, BC, Canada, Conference Track Proceedings. OpenReview.ned 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jemnifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Slockholmsmässann, Slockholm, Sweden, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582 1591. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (ods.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2052 2062. PMTR, 2019.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiamiin Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Newral Information Processing Systems 34: Annual Conference Newral Information Processing Systems 2021, NeurIPS 2021, 2021, virtual, pp. 4028 4088, 2021.
Divyansh Garg, Joey Hejna, Matthicu Geist, and Stefano Ermon. Extrome q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, 2023. OpenReview.net, 2023.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods, 2019.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Parley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1406. 2661, 2014.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskil12: A unified benchmark for generalizable manipulation skills. In International Conference Learning Representations, 2023.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, NIPS'16, pp. Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Red
Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Trina Rish, Glen Berseth, and Sanjiban Choudhury. Revisiting successor features for inverse reinforcement learning. In ICML 2024 Workshop Models of Human Peedback for AI Aligmment, 2024. URL https:// http::///ppppeeeeeeeeee/////raaa
Gabriel Kalweit Maria Hügle, Moritz Werling, and Joschka Boedecker. Deep inverse q-learning with con- straints. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Infor- mation Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence mininization, 2020.
Πya Kostrikow, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriiinnator-ccto--ciiii Addressing sample inefficiency and reward bias in adversarial imitation learn- ing. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nng net, 2019.
Nya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imilation learning via off-policy distribution matching, In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevicw.nee net, 2020. URL
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-lcarning. In International Conference on Learning Represenlations, 2022. URL htttp:://ppareriiee..effffoo net/ffrummiid
68n2s9ZJWF3
Tejas D. Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J. Gershman. Deep successor reinforce- ment learning. CoRR, bbs/1606.02396 2016.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative qlearning for offline rein- forcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781 71329538.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning, pp. 45-73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-27644--3, doi:
2.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, Pp. 809-817, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450383073.
Yuping Luo, Huazhe Xu, and Tengyu Ma. Learning self-correctable policies and value functions from demon- strations with negative sampling. In 8th International Conference on Learning Represenlations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ng net, 2020.
Marlos C Machado, Mare G Bellemare, and Michael Bowling, Count-based exploration with the successor representation. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 5125- 5133, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattic, Amir Sadik, Toannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540))529 533, February 2015. ISSN 00280836. URL httpp://dt www.doi.....o..oooo..... org/10. 1ttg:/10..008/maauree422333
Ted Moskovitz, Spencer R Wilson, and Maneesh Sahani. A first-occupancy representation for reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreevie:
Bet//forum?id-DJAAee2yyll
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections, 2019.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement lcarning. In Pat Langley (cd.), Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000, pp. 663-670. Morgan Kaufmann, 2000.
Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self-predictive ri, 2024.
Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Newral Comput.. 3(1):88 97, 1991. doi: 10.116//NE00.190 1991.3.99933....
Martin L. Puterman. Markow Decision Processes: Discrate Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn, Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https://arxiv.. org/ abs/ 18290
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018
Daniel Rakita, Bilge Mutlu, and Michael Gleicher. A motion retargeting method for effective mimicry- based telcoperation of robot arms. In 2017 12th ACM/TEEE International Conference on Human-Robot Interaction (IRI, pp. 361 370, 2017.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: imitation learning via reinforcement learning with sparse rewards. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopio, April 26-30, 2020. Open net, 2020.
Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/101 t/s//111.0000 10666, 2010.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Nickum. Dual rl: Unification and new methods for reinforcement and imitation learning, 2023.
Satinder Singh, R. Lewis, and A. Barto. Where do rewards come from? 01 2009.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. 1999. URL https: www.:sssaaaatcciiooeeeeeoo httpss//ppp...eeantttttcto.o.... :tt.oooa.....rrrooprprii011.51..
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFE/RSSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. TEEE, 2012. doi: 10.1109/IRO0 A...........66... 2012.6386109.
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems,
2020.
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning. Conference on Robot Learning, 2022.
Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Picter Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators, 2023.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ Interna- tional Conference Intelligent Robots and Systems (IROS), pp. 2371-2378. IEEE, 2017.
17

---

Published in Transactions on Machine Learning Research (06/2025)
A
Data- Size Ablation
Walker2d
Ant
Haftcheetah
Hopper
150
150
150-
150
Normalzze
0-
10
100
550
10
100
502
13
50
300
550
10
50
105
500
Data Size
Data Size
Data Size
Data Size
BC
Thus Reaard
SR Rexard
Hammer
200
203
150
150
150
Normaized
100
1.0-
103
50
50
50
Thus Rowaard
3h-Roward
Figure 8: Effect of data size performance. RL agents (SparseQL) using SR-Reward show competitive performance compared to BC and the RI. agents that use the true reward.
To investigate the impact of data size on SR-Reward, we trained each algorithm using varying numbers of expert trajectories from the D4RL dataset. We used sparseQL as our offiine RL algorithm. Specifically, we evaluated performance using [10, 50, 100, 500] demonstrations for MuJoCo and Adroit hand environments. As shown in Figure g agents trained with true reward do not significantly outperform those trained with SR-Reward across different data sizes in all MuJoCo environments.
As the number of demonstrations decreases, performance declines for all agents, regardless of the reward function used. This trend suggests that informative rewards can still be learned even with limited data. Therefore, the performance drop observed with fewer demonstrations likely reflects the data inefficiency of the offline RL algorithms rather than a significant decline the quality of the learned reward.
18

---

Published in Transactions on Machine Learning Research (06/2025)
B Rata-Quality Ablation
Walker2d
Halfcheetah
Hopper
:50
150
150
100
203
Nomalzen aomallddd
medium
medium PERRR Quality
metkmm
meduu PIRAT Dota Quality
excet
medium
medin, MOORT Quality
medum
medimn PRENT Data Quality
Ronerd
Figure 9: Effect of data quality performance. RL agents (SparseQL) using SR-Reward show similar performance to the basclines. SR-Reward is robust to the mixing of sub-optimal demonstrations (medium- expert) as there is no significant. drop in performance compared to the agents that were trained on the true reward.
Depending on the environment, creating a set of high-quality expert demonstrations can quickly become a cumbersome task. Therefore, is important to know the effect of sub-optimal demonstrations when used for training the SR-Reward. We conduct our experiments on MuJoCo environments using three datasets with different quality demonstrations from D4RL with the mmediumeexpert dataset being a combination of both expert and medium demonstrations. Figure g shows that agents using SR-Reward have similar performance to the oncs trained using true environment reward. The mixing of expert and medium datasets does not show significant negative impact on agents trained with SR-Reward as compared to other agents. In fact including the sub-optimal trajectories results in higher returns, especially for the more difficult Walker2D environment which can benefit from larger datascts. Having low scnsitivity to sub-optimal demonstrations is a desirable altribute of SR-Reward since collecting expert demonstrations can be tedious and error-prone, which increases the possibility of including sub-optimal demonstrations.
19

---

Published in Transactions on Machine Learning Research (06/2025)
C Occupancy Measure and Successor Representations
We will restrict ourselves to the occupancy mcasure of the state only (instead of state and action). The extension to state and action is trivial via a second summation over the actions.
The expectation with respect to starting state distribution po, the policy that is followed n, and the transition dynamics of the environment T.
We can write the definitions of occupancy measures p(s) and successor representations Ms, in terms of probabilitics p.
M(8 8') = E
? (T( = 86) = s]
t=0
2
"E[[Iss s')ls0
Σ
7'((s: s'|s0
1=0
and similarly for the occupancy measure p(s):
p(s) = E
*I(sr = =]]
0=0
20
'B[TT &t = $)]
7'p(si s)
t=U
Below we show that p(s') (.s)==Ep(a) (s))
p(5') = Σ pp(s = =s)
Σ 7 Σ p(s)p(s; =s'sso s)
Ep(s)
p(s)
r'p(s: 5'1s0 =s s)
1=
-p(s)
20

---

Published in Transactions on Machine Learning Research (06/2025)
D
Hyperparameters
Table Most important Hyperparameters used the experiments.
Hyperparameter
Value
Noise 3 (MuJoCo)
1.0
Noise (MuJoCo)
3.0
Noise 3 (Adroit))
0.1
Noise a (Adroit)
0.3
Noise 3 (Maniskill))
0.03
Noise (Maniskill2)
0.3
LR (Critic, Value)
0.0003
LR (Actor, SR- (e-RReward)
0.0001
Encoder MLP
[256, 128
SRNet MLP
[128]
Predictor MLP
[128, 32
Critie MLP
[256, 256
Actor MLP
118, 128
ValueNet MLP
[128, 128
Batch Size
128
Training Steps
1000000
21

---

Published in Transactions on Machine Learning Research (06/2025)
E D4RL Return Normalization
We follow the same normalization procedure described in D4RL with min and max scores for cach task taken from the D4RL datasets as below:
Table 3: Min and Max scores for each D4RL environment
Environment
Min
Max
Walker2d
1.629
4592.3
Ant
-325.6
3879.7
HalfCheetah
-280.178
12135.0
Hopper
-20.272
3231.3
Door
-56.512
2880.569
Hammer
2274.556
12794. 134
Pen
96.262
3076.833
Relocate
-6.425
4233.877
For Maniskill2 environments (PickCube, StackCube, TurnFaucet), the minimum score is considered when the task not solved, and the maximum score is:
A scoremaz = 1.0 + (1.0 MAXSTEPS
where MAXSTEPS is set to 500 and & the steps of the simulation hence giving more rewards to the successful tasks that are completed fewer steps. The expert can complete the tasks approximately 150 steps hence the maximum score for these environments set to 1.7.
The returns normalized for all plots using the Min and Max scores of each environment as follows:
Return Scoremin
ReturTnooraatser Score Scoremin
22

---

Published in Transactions on Machine Learning Research (06/2025)
F Policy Comparison in the Action Space
We visualize the actions proposed by two policies on the Adroit Door task to assess their behavioral similarity, Specifically, we compare two models trained using the f-DVL algorithm: one trained with the true reward provided by the environment, and the other trained with our learned SR-Reward. Since both models achieve comparable mean returns (sec Table 1 we examine whether they also produce similar actions, which would indicate that SR-Reward leads to a policy behaviorally close that of the true reward.
Figure 10 illustrates the actions suggested by each policy when evaluated on the same observations drawn from 50 trajectories in the offine datasct. Each subplot corresponds to dimension of the action space. The alignment between the two sets of actions across all dimensions suggests that the policies behave simi- larly- supporting the idea that policies trained using SR-Reward behave similarly to the ones trained using the true reward.
Action Dimension
Action Dimension
Action Dimension
Action Dimension 3
Arien Sa3e
0.5
1iv_rrue
0.0
0.0
-0.5
-0.3
-02
0.5
20
40
20
40
40
Action Dimension
Action Dimension
Action Dimensian
Action Dimension
-022
0.9
0.4
-02
25
40
20
40
Action Dimension
Action Dimension
Action Dimersion 10
Action Dimension 11
10 186
0.6
00-
0.4
0.55
-05
-1.0
-1
25
40
20
40
40
40
Action Dimension 12
Action Dimension
Action Dimersion
Action Dimension 15
-0.1
-1.3
-10
20
40
20
40
40
0
40
Action Dimansion
Action Dimension
Dimension
-04
-1.0
-05
40
Action Dimansion
Action
Dimension
Ac5inn Vilue
0..4
0.550
8.925
20
Figure 10: Action trajectories for f-DVL. model trained true reward SR-Reward the Ardoit Door environment.
23

---

Published in Transactions on Machine Learning Research (06/2025)
G Negative Sampling Hyperparameters
The negative sampling method introduced in Section 3.3 is governed by two hyperparameters: B and a. The parameter / controls the amount of noise added to expert stale-action pairs to generate negative samples, while a determines the width of the Gaussian penalty kernel used to measure the similarity between expert and negative samples in the latent space. To study the sensitivity of our method to these hyperparameters, we trained a series of SparsoQL + SR-Reward models on the StackCube environment using a grid of different 3 and values. Figure 11 presents the normalized returns across these settings. The red circle marks the hyperparameter configuration used in our main experiments (Section 4
Overall, high values of ß tend to degrade performance, likely because the resulting negative samples become out-of-distribution with respect to the demonstration data, which can negatively affect training. A similar trend is observed for large values a: when the penalty kernel is too wide, both expert and perturbed samples receive similar rewards, making harder to distinguish optimal from suboptimal behaviors, ultimately harming performance. As a practical guideline, we set deviations across all observation and action dimensions in the demonstration data. Figure 12 shows the distribution of values across these dimensions, with the red line marking the median standard deviation. This choice typically yields noise level that reflects the natural variability present in the expert data. For 0, we choose a value between 3 to 7 times the selected 3. This ensures that the reward function remains sensitive enough to distinguish perturbed samples from expert oncs puos--pnaliiing them appropriately-whil not being so narrow as lo overly suppress reward values for slightly perturbed but still useful samples.
approximately the median of the standard
46.04
3.53
1.0
0.5
22.06
29.15
0.3
72.50
76.32
50.19
43.27
33.66
0.1
0.01
0.03
0.05
0.1
0.5
Figure 11: Normalized return for SaaaseQLSSRRRwaarr for different hyperparameters 3 and of the nega- tive sampling strategy. The red circle indicated the return from the hyperparameters used for our exporiments in Table
24

---

Published in Transactions on Machine Learning Research (06/2025)
Volues dimers data stackcube
Values sach cimansion
Vallues of each dimancian halfcheetah
2.5
2.0-
1.3
| 11140
11
1
Volues
Volues
-10
-1.
-15
-50
Stdev of each dimension data stackcube
Stdev of each dimension of data door
Stdey of each dimension of data halfcheetah
Median
Meilan
Median
Standand Dellation
Standard Dovlation
Standard Deviiation
0-6
0.4
0.-
10
20
20
40
92
60
0
20
10
15
20
Dimensions of observation and actions
mensions of observations and actions
Dimensions of observatiors and actions
Figure 12: (TOP): Values every dimension of observation (blue) and actions (orange) for 50 demonstrations of StackCube, Door, and HalfCheetah environments. (BOTTOM): Standard deviation of each observation and action dimension for 50 demonstrations for the same environments. The red line indicates the median of the standard deviations along all dimensions of observations and actions.
25

---

Published in Transactions on Machine Learning Research (06/2025)
H Online Reinforcement Learning
SR-Reward can be trained independently of the reinforcement learning (RI) algorithm using only the offline dataset. In this section, we address the question of whether SR-Reward, once trained offline, can be effec- tively used as a reward function for online reinforcement learning. Figure 13 compares the performance of TD3 trained using the environment's true reward versus using SR-Reward trained on an offline dataset of demonstrations. The TD3 agent was trained on the HalfChoctah environment for million gradient steps using online interactions.
The results highlight a key limitation of SR-Reward in online settings. While incorporating negative sampling improves its robustness to some extent, there is still a notable drop in performance when SR- 3R-Reward is used in place of the environment's 's true reward. This outcome is expected: since SR-Reward is trained solely on expert demonstrations, it does not generalize well to parts of the state-action space that lie far from the distribution of the offline dataset. As the online agent explores new regions during training, SR-Reward may assign incorrect values to unfamiliar state-action pairs, potentially misleading the agent and resulting in suboptimal behavior.
HalfCheetah | Offinneto-Onnline Reward
Normalized Return
Online (Thue Reward)
Online (SR-Rteward - Neg Sampling)
Oniie (SR-Ateward)
Figure 13: Normalized reward of TD3 algorithm trained using true reward and SR- Reward.
26


Reviews:
[{"review_id": "F2CSdOivuq", "reviewer": "Reviewer_RrXb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1.  **Reward from Successor Feature Norm:** The paper proposes a novel reward function derived from Successor Features (SF), $M_{\\theta}(s, a) \\in \\mathbb{R}^k$, learned from offline expert data $\\mathcal{D}\\_E$. Specifically, $r_{\\theta}(s, a) = ||M_{\\theta}(s, a)||\\_2$, where $M_{\\theta}(s, a)$ approximates the expected discounted sum of future feature vectors $\\phi(s', a')$ under the expert policy $\\pi_E$. This differs from standard SF usage which typically employs learned linear weights $w$ to recover task-specific value functions $\\Big( Q(s, a) \\approx M(s, a)^\\top w \\Big)$. Using the $L_2$ norm provides a task-agnostic measure of expected future feature occupancy density under $\\pi_E$.\n2.  **Offline TD-based SR Learning:** The SF network $M_{\\theta}$ is trained via minimizing the Temporal Difference (TD) error based on the Bellman equation for SFs, using transitions $(s, a, s', a')$ sampled from $\\mathcal{D}\\_E$:\n    $\\mathcal{L}\\_{\\text{Bellman}} = \\mathbb{E}\\_{(s,a,s',a') \\sim \\mathcal{D}\\_E} \\Big[ || M_{\\theta}(s, a) - (\\phi(s, a) + \\gamma M_{\\bar{\\theta}}(s', a')) ||^2 \\Big]$\n    This TD-based learning integrates naturally with offline RL algorithms that also rely on TD updates (e.g., IQL, f-DVL, SparseQL), enabling concurrent training without fundamentally altering the RL algorithm's optimization process, beyond substituting the reward signal.\n3.  **Stable Non-Adversarial Learning:** The reward $r_{\\theta}(s, a)$ is learned independently of the agent's policy $\\pi_{\\text{RL}}$, avoiding the complexities and potential instabilities of adversarial frameworks (e.g., GAN-like optimization in GAIL involving $\\min_{\\pi} \\max_{D}$ objectives). The optimization involves standard supervised/TD loss minimization for $M_{\\theta}$ and standard RL objectives for $\\pi_{\\text{RL}}$ using $r_{\\theta}$.\n4.  **Feature-Space Negative Sampling for OOD Regularization:** The proposed negative sampling strategy combats reward overestimation near the data manifold. By generating perturbed samples $(\\tilde{s}, \\tilde{a}) \\sim \\mathcal{N}((s,a), \\beta^2 I)$ and minimizing $\\mathcal{L}\\_{\\text{Neg.Sample}} = \\mathbb{E} \\Big[ ( r_{\\theta}(\\tilde{s}, \\tilde{a}) - \\alpha_{\\text{decay}} \\times r_{\\theta}(s, a) )^2 \\Big]$ with $\\alpha_{\\text{decay}} = \\exp\\left(-\\frac{||\\phi(s, a) - \\phi(\\tilde{s}, \\tilde{a})||_2^2}{\\sigma^2}\\right)$, it explicitly encourages lower reward estimates for samples diverging from the demonstrations *in the learned feature space $\\phi$*. This provides targeted regularization against function approximation errors for OOD inputs close to the support of $\\mathcal{D}_E$.\n5.  **Empirical Results on Diverse Tasks:** The method demonstrates efficacy by integrating with state-of-the-art offline RL agents (f-DVL, SparseQL) and achieving high performance on D4RL benchmarks (matching or exceeding true reward performance, especially on Adroit) and challenging ManiSkill2 tasks (significantly outperforming BC), validating its utility in practical reward-free offline scenarios.\n6.  **Well-Defined Architecture and Training:** The SR network architecture (Fig 2: Encoder, MLP, auxiliary Predictor) and the composite loss function ($\\mathcal{L}_{\\text{Total}}$ including Bellman, prediction, magnitude penalty, and negative sampling terms) are clearly specified (Sec 3.1, 3.4, Alg 1), providing a concrete implementation blueprint.\n\n**Potential Areas of Improvement:**\n\n1.  **Limited Scope of OOD Constraint:** The negative sampling mechanism primarily imposes conservatism locally around the expert trajectories. For state-action pairs $(s,a)$ far outside the support of the expert distribution $\\rho^{\\pi_E}(s, a)$, the reward estimate $r_{\\theta}(s, a)$ relies mainly on the extrapolation behavior of the $M_{\\theta}$ network driven by $\\mathcal{L}_{\\text{Bellman}}$, and lacks explicit regularization. This might be insufficient for online settings or policies that explore significantly beyond $\\mathcal{D}_E$, potentially leading to large extrapolation errors compared to methods with global OOD penalties (e.g., CQL).\n2.  **Sensitivity to Expert Occupancy Measure:** SR-Reward fundamentally reflects the discounted state-action occupancy $\\rho^{\\pi_E}(s, a)$ of the expert. If the expert demonstrations contain significant portions of suboptimal behavior, these regions will still yield high $||M_{\\theta}(s, a)||_2$ and thus high reward, potentially misleading the RL agent. The method lacks an explicit mechanism to filter or down-weight such suboptimal data segments.\n3.  **Theoretical Justification for $L_2$ Norm Reward:** While the $L_2$ norm of $M_{\\theta}(s,a)$ correlates with expert visitation frequency, its optimality as a surrogate reward lacks rigorous theoretical grounding. It's unclear to me if maximizing $\\mathbb{E}\\_{\\pi_{\\text{RL}}}[\\sum \\gamma^t ||M_{\\theta}(s_t, a_t)||\\_2]$ directly optimizes for the underlying unknown true task reward, unlike, e.g., MaxEnt IRL frameworks which often connect to matching occupancy measures via $r(s, a) \\propto \\log \\pi_E(a|s)$ or similar forms. Establishing a more formal relationship between the proposed objective and standard MDP objectives would be useful. \n4.  **Dependence on Feature Quality $\\phi(s)$:** The effectiveness hinges on the quality of the learned state representation $\\phi(s)$ from the Encoder. The Bellman updates for $M_{\\theta}$ and the distance calculations in $\\alpha_{\\text{decay}}$ rely critically on these features accurately capturing state similarities relevant to the dynamics and expert behavior. The choice of the auxiliary prediction task is heuristic, and alternative representation learning objectives could potentially yield better performance.\n5.  **Comparison to Related Offline Reward Inference:** The paper could benefit from a more detailed discussion and potential empirical comparison with other non-adversarial offline reward inference methods. Examples include techniques based on inferring rewards from learned Q-functions (e.g., variants of IQ-Learn adapted for offline use) or contrastive methods that learn rewards by discriminating expert data from non-expert data (which might be synthesized or come from a different source in the offline setting).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce SR-Reward, a novel method for learning a reward function directly from offline expert demonstrations, designed explicitly for settings where the true reward is unknown and environment interaction is not possible. The core objective is to improve learning efficiency in Inverse RL. In particular, key ideas presented in the paper include:\n\n1.  **Reward from Successor Representations:** The core idea is to leverage the Successor Representation (SR), specifically Successor Features (SF) in the continuous domain, to derive a reward signal. The paper proposes using the $L_2$ norm of the learned state-action SR vector, $M(s, a)$, as the reward function:    $r_{\\theta}(s, a) = || M_{\\theta}(s, a) ||\\_2 $\n    The intuition is that state-action pairs frequently leading to future states visited by the expert (as captured by the SR learned from expert data $D_E$) should receive higher rewards. $M_{\\theta}(s, a)$ represents the expected discounted sum of future state-action features $\\phi(s, a)$ under the expert's policy.\n\n2.  **Offline Temporal Difference Learning:** SR-Reward is learned using a Temporal Difference (TD) approach based on the Bellman equation for SR/SF:\n    $M_{\\theta}(s_t, a_t) \\approx \\phi(s_t, a_t) + \\gamma M_{\\bar{\\theta}}(s_{t+1}, a_{t+1})$\n    This allows the SR network (and thus the reward function) to be trained solely from the offline demonstration dataset $\\mathcal{D}\\_E = \\{(s, a, s', a')\\}$, concurrently with standard offline RL algorithms, without requiring modifications to the RL agent's core learning process beyond using the generated reward $r_{\\theta}(s, a)$. The features $\\phi(s, a)$ are typically derived by concatenating learned state features $\\phi(s)$ with the action $a$.\n\n3.  **Decoupled Reward and Policy Learning:** Unlike many Inverse Reinforcement Learning (IRL) methods, particularly adversarial ones (like GAIL), SR-Reward learning is decoupled from the learning of the agent's policy $\\pi$. This avoids the instability often associated with adversarial training frameworks.\n\n4.  **Negative Sampling for Robustness:** To address the potential for function approximation errors leading to reward overestimation for out-of-distribution (OOD) state-action pairs (which are common in offline settings), the paper introduces a negative sampling strategy. Negative samples $(\\tilde{s}, \\tilde{a})$ are generated by perturbing demonstration samples $(s, a)$ with noise. A loss term is added to penalize the reward $r_{\\theta}(\\tilde{s}, \\tilde{a})$ for these OOD samples, encouraging it to be lower than the reward $r_{\\theta}(s, a)$ of the original sample, scaled by a decay factor $\\alpha_{\\text{decay}}$ based on their distance in the feature-action space:\n    $\\mathcal{L}\\_{\\text{Neg.Sample}} = \\mathbb{E}\\_{(s,a) \\sim \\mathcal{D}\\_E} [ ( r\\_{\\theta}(\\tilde{s}, \\tilde{a}) - \\alpha\\_{\\text{decay}} \\times r\\_{\\theta}(s, a) )^2 ]$\n    with $\\alpha_{\\text{decay}} = \\exp\\left(-\\frac{||\\phi(s, a) - \\phi(\\tilde{s}, \\tilde{a})||_2^2}{\\sigma^2}\\right)$. This introduces a bias to preserve the learned reward function near the demonstrated trajectories.\n\n5.  **Empirical Validation:** The effectiveness of SR-Reward is demonstrated through experiments on benchmark offline RL datasets (D4RL MuJoCo, Adroit) and challenging manipulation tasks without predefined rewards (ManiSkill2). The results show that offline RL agents using SR-Reward achieve performance comparable to agents trained with the true environment reward (when available) and significantly outperform Behavioral Cloning (BC) on complex tasks, highlighting the benefit of combining the learned reward with RL's planning capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Please Note: All proposed adjustments are in the spirit of strengthening the work, but none are deemed strictly critical for acceptance given the current level of contribution and validation.)*\n\n1.  **Strengthen Discussion on OOD Limitations:**\n    *   *Suggestion:* Explicitly state in Section 7 (Limitations and Future Work) that the learned reward $r_{\\theta}(s, a)$ may suffer from unreliable extrapolation far from the support of the expert data distribution $\\mathcal{D}_E$, as the negative sampling primarily enforces local conservatism. Useful addition would be other potential choices to elucidate why the given choice of metric is sufficient. Also clarify potential implications for online RL or highly exploratory offline policies.\n    *   *Motivation:* Improves transparency about the method's scope and manages reader expectations regarding its applicability beyond the offline setting tested.\n\n2.  **Acknowledge Demonstration Quality Assumption:**\n    *   *Suggestion:* Add discussion acknowledging that SR-Reward, by rewarding proximity to the demonstrators visitation $||M_{\\theta}(s,a)||_2$, implicitly assumes $\\mathcal{D}_E$ predominantly contains desirable or near-optimal behavior with sufficient coverage. Noting that performance could degrade if demonstrations are mixed with exploratory/potentially suboptimal trajectories.\n    *   *Motivation:* Clearly states a key underlying assumption influencing the quality of the learned reward signal.\n\n3.  **Further Discussion on Theoretical Basis of $L_2$ Norm Reward:**\n    *   *Suggestion:* Either adding more formal connections of a brief discussion (perhaps in Limitations or Conclusion) acknowledging that the choice of the $L_2$ norm as the reward function, while empirically successful, currently lacks a formal theoretical justification linking its maximization directly to the recovery of the optimal policy for the unknown ground-truth task reward, unlike prior works in the IRL paradigms (e.g., MaxEnt).\n    *   *Motivation:* Solidifying core design decisions  and highlights an area for future investigation.\n\n4.  **Include Hyperparameter Sensitivity Analysis (Appendix):**\n    *   *Suggestion:* Provide a brief sensitivity analysis in the Appendix for the key hyperparameters introduced, particularly the negative sampling parameters $\\beta$ (noise std dev) and $\\sigma$ (decay kernel width), showing performance variation on one or two representative environments as these parameters change around their reported optimal values.\n    *   *Motivation:* Increases confidence in the robustness of the empirical results and provides practical guidance regarding HP tuning stability.\n\n5.  **Expand Related Work Comparison to Non-Adversarial Offline IRL:**\n    *   *Suggestion:* Enhance Section 5 (Related Work)  and potentially relevant baseline, by briefly discussing and differentiating SR-Reward from other relevant *non-adversarial* techniques for offline reward inference or IRL (e.g., methods based on preferences, contrastive learning, or direct value function inversion like offline adaptations of IQ-Learn).\n    *   *Motivation:* Better contextualizes SR-Reward within the broader landscape of modern offline reward learning approaches beyond BC and adversarial methods.", "broader_impact_concerns": "No significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UeQzJxGPI9", "reviewer": "Reviewer_aD43", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe paper addresses an interesting problem of learning rewards using successor representations in offline RL. The proposed technique is indeed intuitive and novel. \n\nWeaknesses: \n\nSince this paper is a resubmission, I went through the entire paper to check if the problems highlighted by the previous set of reviews have been adequately addressed. Unfortunately, I think this is not the case. Particularly, the previous action editor has highlighted three important requirements: \n\n1. Testing the method on tasks where the advantages can be appreciated.\n\n 2. Theoretical justification for why this method works. \n\n3. Considering the hybrid offline-online setting. \n\nThis resubmission should have highlighted the changes in response to these comments. However, these details are missing. I would like the authors to discuss the changes in their paper in response to these comments. \n\nIn addition, I have the following points to highlight in the paper. \n\n1. There are several design choices in Section 3 which do not have adequate experimental support to showcase its effectiveness. I expect to see more ablation studies to better understand the advantages of the design choices. For example, it is mentioned that an extra reward was added to penalize rewards greater than 1, but no experimental justification is included in the paper. Another example is the use of auxiliary prediction tasks, which is lacking details and justification in the paper. \n\n2. I am not convinced that the experiments are conclusively capturing the intended outcomes. The authors aim to show that SR-Rewards can replace the true reward signal. To demonstrate this they compare the performances of two offline RL algorithms using the true reward and SR-Rewards. However, this comparison should have used the final policies learned by the offline RL algorithms instead of reward values obtained as done in the paper. If the converged optimal policies are similar using both reward functions, it can be claimed that the SR-Rewards can replace the true rewards as they do not change the optimal policies. The paper instead shows that the reward values are similar, and I am not sure what can be inferred from this observation. \n\n3. If I understood correctly, the paper mentions that the best checkpoint during training is then used for evaluation in Section 4.1. This seems very strange. Why was the best checkpoint used instead of reporting the statistics for all trials? \n\n4. More discussions and experiments are needed to demonstrate the need for a negative sampling strategy. When is this helpful and why? The current experimentations and discussions are very limited and I am not sure if this is indeed helpful in general.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides an alternative approach (different from the standard inverse reinforcement learning (IRL)) for learning the reward function in offline RL settings. The key insight for the proposed SR-Reward method is to use successor representations as a proxy for feature (of state-action pairs) frequencies pertaining to the demonstration policy. To address the problem of overestimation in out-of-distribution experiences, the paper proposes reducing the rewards obtained to induce a conservative bias as done previously. The effectiveness of SR-Rewards are demonstrated by using them for standard offline RL training (with comparisons to training using ground truth rewards) in classic offline testbeds. The paper also provides comparisons of learning with SR-Rewards to imitation learning (behaviour cloning) in classic offline testbeds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please highlight the changes in response to the previous reviews. Also please address the other weaknesses listed in my review.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AsFxvZdinD", "reviewer": "Reviewer_fZcR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. The method is fairly straightforward, but appears to improve offline RL performance compared to BC for some offline RL algorithms.\n2. SR-Reward is evaluated in the context of two different offline RL algorithms.\n3. A variety of tasks and task difficulties is assessed.\n4. The impact of negative sampling and the corresponding design choices are ablated.\n**Weaknesses**:\n1. Table 1. is a bit difficult to parse and pull out the key pieces to the story. This makes it difficult to assess exactly how much SR-Reward improves performance. It might better to report the difference in returns between BC vs. SR-Reward and True Reward vs. SR-Reward. The analysis talks about the performance gap between the SR-Reward and the other two methods, so showing the gap will make that easier to follow.\n2. There is not inverse reinforcement learning background nor is an explanation given for why one is not provided.\n3. There is no mention of the dataset size experiments in the main body of the paper. Given that in contrasting BC versus SR-Reward it is called out that BC performs well when there is a lot of data, it is important to talk about the impact of training dataset size between the two methods. Is the amount of data trained on of the size that BC is expected to perform well? Despite the claims that it is easier to provide a demonstration than to write a reward function, providing many, expert-level demonstrations can be a non-trivial task. So then how much does SR-Reward help to address this. At a minimum, reference Figure 8.\n4. It is not discussed how sensitive SR-Reward is to its hyper-parameters nor is the sensitivity of f-DVL and sparceQL to their hyper-parameters when training on the True versus SR-Reward.\n5. It is not discussed how state-space complexity and diversity impact the quality of the learned SR-Reward. The tasks evaluated on are rather simple in terms of the diversity of the states, e.g. there the background is fixed across episodes.\n6. Figure 1 should detail the main contributions of the paper: (1) the successor representation and (2) the negative sampling. The current figure only showing that the data is augmented is unnecessary.\n7. It appears that the SR-Reward method does not always beat BC when the True Reward does, especially for f-DVL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a new method, SR Reward, to infer a reward function from demonstration data containing states and actions. The reward function is learned via a Successor (Features) Representation (SR) and is computed as the l2-norm of the successor feature representation. When learning the SR network, an auxiliary objective is used to make sure the state-action representations is predictive of the next states. Negative sampling is used to prevent the reward model from assigning too much value to unseen state action pairs. This takes the form of an auxiliary loss that reaches the reward model to decay the value assigned to a state-action pair as a function of how dissimilar it is to the state-action pairs in the expert demonstrations. The approach is distinct from inverse reinforcement learning because the policy and reward learning is decoupled. \n\nThe results demonstrate that using offline RL with the SR Reward performs better than behavior cloning. The benefits of the negative sampling auxiliary loss is verified qualitatively through visualizations of the reward with and without negative sampling in a 2D toy maze environment and by ablating the negative sampling loss on three robotic manipulation tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please fix the formatting issues on pages 7 and 9 where the text stops partway across the page and then there are large blank spaces.\n2. To accompany the claims that, \"[t]hese findings underscore the versatility of SR-Reward as a modular component that ban be integrated with any offline RL algorithm,\" make sure to discuss both the offline RL algorithms in the sections above. Right now only sparseQL is discussed. This is important to discuss as there are different performance gaps between BC and f-DVL versus BC and sparseQL. Additionally, the claim that SR-Reward can be integrated with \"any offline RL algorithm\" should be softened as only two are evaluated.\n3. It should be stated how the states were represented for training the SR-Reward and for training the policies. Where image renderings used? Or were joint positions used?\n4. Introduce the experiments referenced in Figure 9 prior to the conclusion.\n5. Reference the data set size ablations in the main body of the paper and provide some take aways. This is a key analysis.\n6. Add a citation supporting the claim, \"In contrast, demonstrating the desired behavior is generally more straightforward.\"", "broader_impact_concerns": "I do not see any to warrant concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers tackles the IRL problem, and proposes a new way to learn a reward function from offline trajectories using Sucessor Representation.

The reviews were overall positive, highlighting the simplicity and efficiency of the method, and all recommend acceptance.The authors answered to the reviews and updated their paper accordingly. This paper is a re-submission: the main reason it was originally rejected was too much noisy evaluations, which has been addressed in this revision. 

I recommend acceptance.
Decision: Accept (poster)