Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
Authors: Arnav Kumar Jain
Harley Wiltzer
Jesse Farebrother
Irina Rish
Glen Berseth
Sanjiban Choudhury
Venue: ICLR 2025 Poster (2025)
Keywords: Inverse Reinforcement Learning
Imitation Learning
Successor Features
Primary area: reinforcement learning

Abstract: In inverse reinforcement learning (IRL), an agent seeks to replicate expert demonstrations through interactions with the environment.
Traditionally, IRL is treated as an adversarial game, where an adversary searches over reward models, and a learner optimizes the reward through repeated RL procedures.
This game-solving approach is both computationally expensive and difficult to stabilize.
In this work, we propose a novel approach to IRL by _direct policy search_: 
by exploiting a linear factorization of the return as the inner product of successor features and a reward vector, we design an IRL algorithm by policy gradient descent on the gap between the learner and expert features.
Our non-adversarial method does not require learning an explicit reward function and can be solved seamlessly with existing RL algorithms.
Remarkably, our approach works in state-only settings without expert action labels, a setting which behavior cloning (BC) cannot solve.
Empirical results demonstrate that our method learns from as few as a single expert demonstration and achieves improved performance on various control tasks.

Published as a conference paper at ICLR 2025
NON-ADVERSARIAL INVERSE REINFORCEMENT LEARNING VIA SUCCESSOR FEATURE MATCHING
Arnav Kumar Jain Harley Wiltzer Jesse Farebrother 3,
Irina Rish 1.2 Glen Berseth Sanjiban Choudhury'
Mila Québec Al Institute Université de Montréal McGill University *Cornell University
ABSTRACT
In inverse reinforcement learning (IRL), agent seeks replicate expert demon- strations through interactions with the environment. Traditionally, IRL is treated adversarial game, where adversary searches over reward models, and learner optimizes the reward through repeated RL procedures. This game-solving approach both computationally expensive and difficult stabilize. In this work. we propose novel approach to IRL. by direct policy search: by exploiting lin- ear factorization of the return as the inner product of successor features and a reward vector, we design IRL algorithm by policy gradient descent on the gap between the learner and expert features. Our non-adversarial method does not require learning explicit reward function and can be solved seamlessly with existing RL algorithms. Remarkably, our approach works in state-only settings without expert action labels, setting which behavior cloning (BC) cannot solve. Empirical results demonstrate that our method learns from as few as single ex- pert demonstration and achieves improved performance various control tasks.
INTRODUCTION
In imitation learning (Abbeel & Ng, 2004; Ziebart et al., 2008; Silver et al,, 2016; Ho & Er- mon, 2016; Swamy et al., 2021), the goal is to learn decision-making policy that reproduces behavior from demonstrations. Rather than simply mimicking the state-conditioned action distribution as in behavior cloning (Pomerleau, 1988), interactive approaches like Inverse Rein- forcement Learning (IRL; Abbeel & Ng. 2004; Ziebart et al., 2008) have the more ambitious goal of synthesizing policy whose long-term occupancy measure approximates that of the expert demonstrator by some metric. As re- sult, IRL methods have proven to be more ro- bust, particularly in regime with few expert demonstrations, and has lead to successful de- ployments in real-world domains such au- tonomous driving (e.g.. Bronstein et al., 2022; Vinitsky et al., 2022; Igl et al., 2022). However, this robustness comes at cost: IRL approaches tend to involve a costly bi-level optimization.
Mean Normalized Return
1.0
0.8
0.4
0.2
0.0
BC
IQ-Leam
OPOLO
MM
GAllO
SFM
Figure 1: Comparing Mean Normalized Return on 10 tasks from DeepMind Control suite (Tunyasu- vunakool al., 2020) of our method SFM against offline Behavior Cloning (Pomerleau, 1988), the non-adversarial IRL method IQ-Learn (Garg et al., 2021). and the state-only adversarial meth- ods OPOLO (Zhu et al., 2020), MM (Swamy et al., 2021) and GAIfO (Torabi et al., 2018), where the agents are provided a single expert demonstration. Our state-only non-adversarial method SFM achieves a higher mean normalized return. Error bars show the 95% bootstrap Cls.
Specifically, modern formulation of many IRL methods (e.g.. Garg et al., 2021; Swamy et al.,
Correspondence to Senav-kumar: jainemill la quebec.
Author order decided by chili-eating contest; result was inconclusive.
Our codebase is available https: com.ub.ccomaaaaa htttuu..co//aaaa/aiiia hamm/arnavkjjj

---

Published as a conference paper at ICLR 2025
2021) involve min-max game between an adversary that learns reward function to maximally differentiate between the agent and expert in the outer loop and a Reinforcement learning (RL)
subroutine over this adversarial reward in the inner loop. However, all such methods encounter set of well-documented challenges: (1) optimizing an adversarial game between the agent and the expert be unstable, often requiring multiple tricks stabilize training (Swamy et al., 2022), (2) the inner loop of this bi-level optimization involves repeatedly solving computationally expensive RL problem (Swamy et al., 2023), and (3) the reward function class must be specified in advance. Moreover, many approaches to imitation learning require knowledge of the actions taken by the demonstrator. This renders many forms of demonstrations unusable, such as videos, motion- captiue data, and generally any demonstrations leveraging an alternative control interface than the learned policy (e.g., a human puppeteering a robot with external forces). As such, it desirable to build IRL. algorithms where the imitation policies learn from only expert states.
These challenges lead us to the following research question: Can a non-adversarial approach to occupancy matching recover the expert's 's behavior without action labels? To address this question, we revisit the earlier approaches to feature matching (Abbeel & Ng. 2004; Ziebart et al., 2008; Syed & Schapire, 2007; Syed et al., 2008), that is, matching the accumulation of discounted state or state-action features along the expert's 's trajectory. For this task, we propose to estimate expected low-variance, cumulative sum of features using Successor Features (SF; Barreto et al., 2017) fully online algorithm that employs temporal-differenee learning. Leveraging the benefits of SFs, we demonstrate that feature matching can be achieved by direct policy search via policy gradients. In doing so, we present a new approach to IRL, called Successor Feature Matching (SFM), which provides a remarkably simple algorithm for imitation learning.
Interestingly, when the learned features are action-independent, we show that SFM can imitate an expert without knowledge of demonstrators' actions. This accommodates a variety of expert demon- stration formats, such as video or motion-capture where action labels are naturally absent. Addi- tionally, rather than manually pre-specifying class of expert reward functions (Swamy et al., 2021). I), SFM adaptively learns this class from data using unsupervised RI. techniques. Our experiments val- idate that SFM successfully learns to imitate from as little as a single expert demonstration. As a result, SFM outperforms its competitors by 16% on mean normalized returns across wide range of tasks from the DMControl suite (Tunyasuvunakon et al., 2020) as highlighted in Figure 1. To summarize, the contributions of this work are as follows:
1. Occupancy matching via reduction to reinforcement learning. In this work, we propose an algorithm for feature matching that can be achieved by direct policy search via policy gradients for inverse RI. In doing so, our method Successor Feature Matching (SFM) achieves strong imitation performance using any off-the-shelf RI. algorithms.
2. Imitation from a single state-only demonstration. Our method learns with demonstrations without expert action labels by using state-only features to estimate the SFs. To our knowledge, SFM is the only online method capable of learning from single unlabeled demonstration without requiring an expensive and tefficult-to-stablll bilevel optimization (Swamy et al., 2022).
2 RELATED WORK
Inverse Reinforcement Learning (IRL) methods typically learn via adversarial game dynamics, where prior methods assume base features are known upfront (Abbeel & Ng, 2004; Ziebart et al., 2008; Syed & Schapire, 2007: Syed al., 2008). The advent of modern deep learning architectures led to methods that do not estimate expected features, but instead learn a more expressive reward function that captures the differences between the expert and the agent (Ho & Ermon, 2016; Swamy et al., 2021; Fu et al., 2018). The class of Moment Matching (MM; Swamy et al., 2021) methods offers a general framework that unifies existing algorithms through the concept of moment matching, or equivalently Integral Probability Metrics (IPM; Sun et al., 2019). In contrast to these methods, our approach is non-adversarial and focuses on directly addressing the problem of matching expected features. Furthermore, unlike prior methods in Apprenticeship Learning (AL; Abbeel & Ng, 2004) and Maximum Entropy IRL (Ziebart et al., 2008), our work does not assume the knowledge of base features. Instead. SFM leverages representation learning technique extract relevant features from the raw observations. The method most similar to ours is IQ-Learn (Garg et al., 2021), 1), a non-adversarial approach that utilizes inverse Bellman operator to directly estimate the value
2

---

Published as a conference paper at ICLR 2025
function of the expert. Our method is also non-adversarial but offers significant advantage over IQ-Learn: it does not require knowledge of expert actions during training providing a state-only imitation learning algorithm (Torabi et al., 2019). However, many existing state-only methods also rely on adversarial approaches (Torabi et al., 2018; Zhu et al., 2020). For instance, GAIfO (Torabi et al., 2018) modifies the discriminator employed in GAIL (Ho & Ermon, 2016) to state-only inputs, while OPOLO (Zhu et al., 2020) combines similar discriminator with an inverse dynamics model predict actions for expert transitions to regularize the agent. Similarly, R2I (Gangwani & Peng, 2020) proposed learning an indirect function to enable imitation when the transition dynamics change. In contrast, SFM is aoveraarral method that learns from state-only demonstrations.
Successor Features (SF; Barreto et al., 2017) generalize the idea of the successor representa- tion (Dayan, 1993) by modeling the expected cumulative state features discounted according to the time of state visitation. Instead of employing successor features for tasks such as transfer learning (Barreto et al., 2017: Lehnert et al., 2017; Barreto et al., 2018; Abdolshah al., 2021; Wiltzer al., 2024b;a), representation learning (Le Lan et al., 2022; 2023b;a; Farebrother et al., 2023; Ghosh et al., 2023), exploration (Zhang et al., 2017; Machado et al., 2020; Jain et al., 2023), or zero-shot RL (Borsa et al., 2019; Touati & Ollivier, 2021; Touati et al., 2023; Park et al., 2024), harnesses SFs for IRL, aiming to match expected features of the expert. Within the body of work on imitation learning, SFs have been leveraged to pre-train behavior foundation models capable of rapid imitation (Pirotta et al., 2024) and within adversarial IRL. typically serves as the basis for estimating the value function that best explains the expert (Lee et al., 2019; Filos et al., 2021; Abdulhai al., 2022). In contrast, our work seeks to directly match SFs through a policy gradient update without requiring large, diverse datasets or costly bilevel optimization procedures.
approach
PRELIMINARIES
Reinforcement Learning (RL; Sutton & Barto, 2018) considers a Markov Decision Process (MDP) defined by MM = (S, A, ,, where S and A denote the state and action spaces, P : S A + ASS) denotes the transition kernel, r S + E is the reward function, 7 E [0, 1) is the discount factor, and Po E A(S) is the initial state distribution. Starting from the initial state So (-(-) an agent takes actions according to its policy π S + A(A) producing tra- jectories {Sn. A1, S1, S1.... ~ Pr. the probability measure over trajectories where So ~ Po. A, (( S,), and S ~ P(- St, A.). We write E,, denote expectations with respect states and actions sampled under Pr,. The performance of a policy can be measured as the cumulative discounted sum of rewards obtained from an initial state, given by
r((++t) St S, A, =]]
J(;r)) = Bs- Pos.
(1)
(7(4,4)
where OF : S x A + R is referred to as the action-value function. When the reward function is unambiguous, write J(T) and Q* in place of ((T;)) and QF.
Successor Features (SF; Barreto et al., 2017; 2020) allow us to linearly factorize the action-value function (T, a) s5 as.a) [T,,,,, w, with the components: (1) 2b S x A + Rd being the expected discounted sum of state features Tht a) = E. Ee x " (Se((+))) S, = 8, A. a) after applying the feature map S -> Rdd and (2) wt IRd being linear projection of the reward function r onto the components of defined as w, (Cov, (o())) -1 E., [r(S) d(S) (Touati & Ollivier, 1021). In practice, we can learn parametric model 25 ~ 26" via Temporal Difference (TD) learning (Sutton, 1988) by minimizing the following least-squares TD objective.
Csr (8: 0) E(,,A, (' ('=x(11s") [II((S) + " (S', A') +5 (5, A)]
(2)
where the tuple A, is state-action-nex
transition sampled from dataset B. The param- eters 0 denote the "target parameters" that are periodically updated from 0 by either taking direct copy or a moving average of 8 (Mnih et al., 2015).
Inverse Reinforcement Learning (IRL; Ng et al., 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) is the task of deriving behaviors using demonstrations through interacting with the environ- ment. In contrast RL where the agent improves its performance using the learned reward, Inverse
3

---

Published as a conference paper at ICLR 2025
Reinforcement Learning (IRL) involves learning without access to the reward function; good per- formance signalled by expert demonstrations. As highlighted in Swamy et al. (2021), this corre- sponds to minimizing Integral Probability Metric (IPM) (Sun et al., 2019) between the agent's state-visitation occupancy and the expert's which is framed to minimize the imitation gap given by:
((TE) - J(T) & sup fEFo
B
Σ t=0
((S)s)
- EE Σ E+'s(s))) 1=0
(3)
where Fa S IR denotes the class of reward basis functions. Under this taxonomy, the agent being the minimization player selects a policy π E II to compete with a discriminator that picks a reward moment function f E Fs maximize the imitation gap. This leads natural framing as a mix-max game min, fa J(TE) J(T).
By restricting the class of reward basis functions be within span of some base-features such that Fo (f(s) = =(s)" w: ||0|22 B), the imitation gap becomes:
J(TB) - -J(x v sup Emp |
Y (O(S)) w)
E
t=0
t=
w, (4) where (s, a) denotes the successor features of the expert policy ae for given state s and ac- tion a. Under this assumption, the agent that matches the successor features with the expert will minimize the performance gap across the class of restricted basis reward functions. A vector w* optimizing the supremum in (4) is referred to as witness- it describes a reward function that most clearly witnesses the distinction between π and the expert demonstrations. Solving the objective of matching expected features between the agent and the expert has been studied in prior meth- ods where previous approaches often resort to solving an adversarial game (Ziebart et al., 2008; Abbeel & Ng, 2004; Syed & Schapire, 2007; Syed et al., 2008). In the sequel, introduce a non- adversarial approach that updates the policy to align the SFs between the expert and the agent, and does not require optimizing explicit reward function to capture the behavioral divergence.
(ES- |
sup
F A)]
Es (1))))))))
Naturally, the aforementioned assumption requires that ø induces a class Fs that rich enough to contain the expert's 's underlying reward function. It is not generally possible ensure this without privileged information instead, we jointly learn features Φ using recent advances in representation learning for RI. (e.g., Farebrother al., 2023; Park et al., 2024). We posit that such features, which are meant to distinguish between a diverse set of behaviors, will be rich enough to include the expert's 's reward in their span. Our experimental results validate that indeed this can be achieved.
SUCCESSOR FEATURE MATCHING
In this section, deseribe Successor Feature Matching (SFM) state-only, non-adversarial algorithm for matching expected features between the agent and expert. The key concept underlying SFM is that, leveraging successor features, the witness w equation 4 can be approximated in closed form, yielding a reward function for RL policy optimization. Specifically, the difference in successor features between the agent and expert can itself act the witness in this case, w is parallel to the feature matching objective, implying that this witness maximally discriminates between the agent and expert's 's performance. Coneretely, we have that,
-
w bm
B
arg max (5 Thx) w. where 1 and T == Es A (N (S, AS)
T711 l1/
(5)
TE = ES-Po() (
(5, A)]
Remarkably, this observation allows us to bypass the adversarial reward learning component of IRL by directly estimating the imitation-gap gww--aaammmiing.... reward function r² -,. given by
xx Joーoag -xg (x) = <(x)] T " & d(x) (Ø+ - Φ5)
(6)
4

---

Published as a conference paper at ICLR 2025
Algorithm Successor Feature Matching (SFM)
Require:
Expert demonstrations - = { dgr... of a 1 1.M
Require: Base feature loss Lteat and initialized parameters O (0, f)
Require: Initialized actor Ty. SF network and target o, 4
replay buffer B
1: while Training do
2: Rollout Ty and add transitions replay buffer B
3: Update expected features of expert & with EMA using (7)
4: Sample independent minibatches D' C S A S from B
5:
Update SF network via (.S,
[1(((S) + + - b0 A)1
6:
Estimate witness vo = G d- using Proposition and minibatch D'.
7:
Update actor via V (TT s (bs) ú)) using Proposition 2 and minibatch D
8: Update base feature function via V Bree Lreat (0feat)
9: end while
This insight enables us to replace the costly bi-level optimization of IRL in favor of solving a single RL problem. The remaining challenge, however, is determining how to estimate w
A natural first step in estimating w "-o is to leverage the provided expert demonstrations consisting of M trajectories {r' = .s....., sT, m These demonstrations allow us to compute &. as,
M
T
FF = M
((s))
A naïve approach might then attempt to estimate Tn from initial States-howeed this proves to be challenging, as it precludes bootstrapped TD) estimates, and Monte Carlo estimators have pro- hibitively high variance for y near 1. Instead, we leverage a key result that allows us to estimate this quantity more effectively by bootstrapping from arbitrary transitions from the environment.
Proposition 1.
Let B denote buffer of trajectories sampled from arbitrary stationary Markovian policies in the given MDP with initial state distribution Pq. For any stochastic policy æ,
$ = (1 8') E (1))( [ (S, (S] -BBA (S' AS]]
(8)
The proof of Proposition 1 is deferred to Appendix A. Notably, key aspect of this estimator is its ability to use samples from different state-visitation distribution, effectively allowing us to use replay buffer B of the agent's experience to estimate C.
With estimates of T and 6ht in hand, might be tempted apply any RL algorithm to optimize r= However, this approach overlooks the structure provided by the learned successor features. Instead, show how these features be directly leveraged to derive novel policy gradient method that optimizes the policy to directly match the difference in features.
4.1 A POLICY GRADIENT METHOD FOR SUCCESSOR FEATURE MATCHING
Instead of directly optimizing the reward function r - now derive policy gradient (Sut- ton et al., 1999) that directly aligning the successor features of the agent and expert. To this end, leverage our learned successor features which are already required for estimating the wit- ness = - to calculate the value-function under the reward r -r eliminating the need for separate critic to estimate Q QT;- This defines the following off off-policy policy gradient objective (Degris et al., 2012) for the policy TH parametrized by
F mmggg = ES ANT, 35 (S, " 15
where 3: S + A(A) is policy different from Th. In practice, can view p being replay buffer containing experience collected throughout training. Given this objective, we now derive the policy gradient with respect to μ in the following result with the proof given in Appendix A.
5

---

Published as a conference paper at ICLR 2025
Proposition 2. For stochastic policies : S + ((A) the policy gradient under which the return most steeply increases for the reward function r an defined in equation 6 is given by.
V VaU (Tp) (w=)+) (Es- (mx [V log (A S) V9 (S)
(10)
Alternatively, for deterministic policies π S + A, the deterministic policy gradient (Silver al., functionn
2014) for the reward function v. T5- defined in equation 6 is given by.
V UFUiFi
(E
(Esupu (T) V (5,
(11)
From Proposition 2 we can see that the SFM policy gradient operates by directly optimizing the alignment between the agent's 's successor features and the expert's by changing the policy in the di- rection that best aligns Th" with sF. This approach simplifies policy optimization by leveraging the computed successor features to directly guide the alignment of the agent's 's feature occupancy with that of the expert. Furthermore, Proposition 2 along with Equation provide drop-in replacements to the actor and critic losses found in many popular methods (e.g.. Fujimoto et al., 2018; 8; 2023; Haarnoja et al., 2018) allowing for the casy integration of SFM with most actor-critic methods.
The overall SFM policy gradient method, summarized in Algorithm I, encompasses both estimating the witness - as well as the policy optimization procedure to reduce the imitation gap using this witness. However, so far, we have assumed access to base features when estimating the successor features. In the following, we describe how these features, too, can be learned from data.
4.2 BASE FEATURE FUNCTION
We described in $3 that SFs depends on a base feature function e S R2. In this work, SFM learns the base features jointly while learning the policy. Base feature methods are parameterized by pairs feat = (o. f) together with a loss Lieat, where Φ S - Fd is state feature map, f is an auxiliary object that may be used to learn d, and Lieat is a loss function defined for ø and f.
Before discussing the learning of base features, we note an important point: when we don't have ac- cess to expert actions, we can still compute & and the policy gradient by using action- ieteeeenndeeendent base features. While our method can handle problems where rewards depend on both states and actions (requiring expert action labels), in many practical applications expert actions are unavail- able. As our experiments in $5 demonstrate, SFM can effectively learn imitation policies without requiring access to expert actions substantially broadening the applicability of our approach.
Below, we briefly outline the base feature methods considered in this work.
Random Features (Random): Here, á is randomly-initialized neural network, and f is discarded. The network remains fixed during training (Crest = 0).
Autoencoder Features (AE): Here, & S + Rd compresses states to latents in Rd and f: Rd + S tries to reconstruct the state from the latent. The loss Creat is given by the AE loss CAE.
CA Es-B [llf()(S)) S112]
0feat (Ø, f).
(12)
Inverse Dynamies Model (IDM; Pathak et al., 2017): Here, f: Rd x Rd A function that tries to predict the action that lead to the transition between embeddings Φ : S - Idd of consecutive states. The loss Lteat is given by the IDM loss LIDM.
LIDM (fteat) ((S....s) [[l [[(((((S) A1l]] O (d.
(13)
Forward Dynamics Model (FDM): Here, f Rd x A + S a function that tries to predict the next state in the MDP given the embedding of the current state and the chosen action. The loss Creat is given by the FDM loss LFDM.
SFDM(Oteat EES.A, (1lf([(()(()))__ s'1 Oreat = fQ.
(14)
Hilbert Representations (HR; Park et al., 2024): The feature map o : S - Rd of HR is meant to estimate a temporal distance: the idea is that the difference between state embeddings f (s.

---

Published as a conference paper at ICLR 2025
llo(s) - ا()وه approximates the amount of timesteps required to traverse between the state E S and randomly sampled goal g E S. Here, f is discarded, and Liest is the HR loss LHR,
LHR( (Oreat ( G~B [& (- (S + G) >sg " G)} (2, G))] Oteat (1) 0)
(15)
where sg denotes the stop-gradient operator, ? is the discount factor, and & is the 7-expectile loss (Newey & Powell, 1987), as a proxy for the max operator in the Bellman backup (Kostrikov al., 2022). In practice, /g( (S", G)} is replaced by fg(S G), where is delayed target network tracking o, much like a target network in DQN (Mnih et al., 2015).
features, although we maintain that a key advantage of the framework is that it does not require adversarial training. To demonstrate the influence of such features, we consider training base features via IRL.
Finally, our framework does not preclude the use of adversarially-trainee
Adversarial Representations (Adv): The embedding ó : S + Fd is trained to maximally distin- guish the features states visited by the learned policy from the expert policy. That is, Lieat is given by LAdu which adversarially maximizes an imitation gap similar to equation 4,
LAdv (freat)
Bn [o())] -En [o(S')]
Preat (6.0).
(16)
In our experiments, we evaluated SFM with each of the base feature methods discussed above. A comparison of their performance is given in Figure 7. Our SFM method adapts familiar determin- istic policy gradient algorithmss, particularly TD3 (Fujimoto et al., 2018) and TD7 (Fujimoto et al., 2023), for policy optimization through the actor loss of equation 11. We further provide results for stochastic policy gradient methods in Appendix B. Full implementation details are provided in Appendix C, and we now demonstrate the performance of SFM in the following section.
EXPERIMENTS
Through our experiments, we aim to analyze (1) how well SFM performs relative to competing non-adversarial and state-only adversarial methods at imitation from a single expert demonstration, (2) the robustness of SFM and its competitors to their underlying policy optimizer, and (3) which features lead to strong performance for SFM. Our results are summarized in Figures 2, 4, and 7, respectively, and are discussed in the subsections below. Ultimately, our results confirm that SFM indeed outperforms its competitors, achieving state-of-the-art performance on a variety of single- demonstration tasks, even surpassing the performance of agents that have access expert actions.
5.1 EXPERIMENTAL SETUP
We evaluate our method 10 environments from the DeepMind Control (DMC; Tunyasuvunakool et al., 2020) suite. Following the investigation in Jena et al. (2020) which showed that the IRL algorithms prone to biases in the learned reward function, where all episodes are truncated after 1000 steps in the environment. For each task, we collected expert demonstrations by training TD3 (Fujimoto et al., 2018) agent for 1M environment steps. In our experiments, the agent is provided with a single expert demonstration which is kept fixed during the training phase. The agents are trained for 1M environment steps and we report the mean performance across 10 seeds with 95% confidence interval shading following the best practices in RLiable (Agarwal et al., 2021). For the RLiable plots, we use the returns obtained by random policy and the expert policy to compute the normalized returns. Our implementation of SFM is in Jax (Bradbury et al., 2018) and it takes about ~2. hours for one run on a single NVIDIA A100 GPU. We provide implementation details in Appendix C and hyperparameters. in Appendix D.
consider infinite horizon tasks
Baselines. Our baselines include state-only version of moment matching (MM; Swamy et al., 2021), which is an adversarial IRL. approach where the integral probability metric (IPM) is replaced with the Censon-Shannon divergence (which was shown to achieve better or comparable performance with GAIL (Swamy et al., 2022)). We implemented state-only MM by changing the discriminator network to depend only on the state and not on actions. Furthermore, we replace the RL optimizer in MM to TD7 (Fujimoto et al., 2023) keep parity with SFM. We compare SFM to another state- only baseline GAlfO (Torabi et al., 2018) where the discriminator learns to distinguish between the state transitions of the expert and the agent. Since, to our knowledge, official implementation of
7

---

Published as a conference paper at ICLR 2025
Median
IQM
Mean
Optimality Gap
BC IQ-Leam OPOLO
MM
GAIFO
SFM
0.25 0.50 0.75
0.25 0.50 0.75
0.25 0.50 0.75
0.25 0.50 0.75
Normalized Return
Figure 2: RLiable (Agarwal al., 2021) plots of the proposed method SFM with an offline method BC, non-adversarial method IQ-Learn that uses expert action labels and adversarial state-only methods: OPOLO, MM and GAIfO across 10 tasks from DMControl suite.
Walker
Walker
500
900
500
Average
Cheetah Run
Quecdnuped Walk
Quadruped Stand
Dusdruped Run
Quadruped jump
500
300
300
660
600
140
602
200
362
200
IQ-Learn
Expert
OPOLO
Per-task learning curves of IRL methods with the TD7 (Fujimoto et al., 2023) policy optimizer on mangle-demonstration imitation in DMC. Notably, IQ-Leam and BC require access to expert actions, while (state-only) MM, GAIfO, and SFM learn from expert states alone. Results are averaged across 10 seeds, and are shown with 95% confidence intervals.
Figure 3:
GAIIO available, we implemented our version of GAIIO with similar architecture the MM framework. Here, we replace their use of TRPO (Schulman et al., 2015) with either TD3 or TD7 to maintain parity. Additionally, for adversarial approaches, we impose gradient penalty (Gulrajani et al., 2017) on the discriminator, learning rate decay, and Optimistic Adam (Daskalakis et al., 2018) to help stabilize training. We also compare with OPOLO (Zhu et al., 2020) and use the official im- plementation for our experiments. Apart from state-only adversarial approaches, behavior cloning (BC; Pomerleau, 1988) which is a supervised learning based imitation learning method trained to match actions taken by the expert. Lastly, we compare SFM with IQ-Learn (Garg et al., 2021) a non-adversarial IRL. algorithm which learns the Q-function using inverse Bellman operator (Piot et al., 2016). Notably, BC and IQ-Learn require the expert action labels.
compare with
5.2 RESULTS
Quantitative Results Figure 2 presents the RLiable plots (Agarwal et al., 2021) aggregated over DMC tasks. We observe that SFM learns to solve the task with single demonstration and sig- nificantly outperforms the von-adversarial BC (Pomerleau, 1988) and IQ-Learn (Garg et al., 2021) baselines. Notably, SFM achieves this without using the action labels from the demonstrations. We believe behavior cloning (BC) fails in this regime of few expert demonstrations as the agent is un- predictable upon encountering states not in the expert dataset (Ross & Bagnell, 2010). We further observe that SFM outperforms our implementation of state-only adversaria baselines. Furthermore, SFM has a significantly lower optimality gap, indicating that the baselines are more likely to per- form much worse than the expert. Among the state-only adversarial approaches, GAIfO leverages more powerful discriminator based on the state transition as compared to only states used MM and thereby performs better. To further analyze the gains, report the average returns across each task
8

---

Published as a conference paper at ICLR 2025
Median
IQM
Mean
Optimality Gap
MM (TD3)-
GAIFO (TD3)
SFM (TD3)
0.30 0.45 0.60 0.75
0.4
0.6
0.8
0.45
0.60 0.75 0.15 0.30 0.45 0.60
Normalized Return
Figure 4: Performance of state-only IRL algorithms under the weaker TD3 policy optimizer.
Median
IQM
Mean
Optimality Gap
Random
Hilp
IDM
AE
Adv
FDM
0.45 0.60 0.75
0.45 0.60 0.75
0.45
0.60
0.75
0.15
0.30
0.45
Normalized Return
Effect of different base features on the performance of SFM. Here, compare with Random, Inverse Dynamics Model (IDM), Hilbert Representations (Hilp), Autoencoder (AE), Ad- versarial (Adv) and Forward Dynamics Models (FDM). FDM was found work best across DMC tasks. Note that all base feature functions were jointly learned during training.
Figure 5:
Figure 3. We observe that OPOLO does well on walker and cheetah domains; however, it strug- gles with the quadruped domain where posit its reliance learned inverse dynamics model becomes problematic due to the challenges of accurately modeling these more complex dynamics. We observe that SFM converges faster when compared leading methods, suggesting improved sample efficiency relative to its competitors. SFM does not use techniques like gradient penalties which are often required when training adversarial methods (Swamy et al., 2021; Ren al., 2024). Lastly, SFM outperforms MM and GAIfO on most tasks across the quadruped and walker domains.
Robustness with weaker policy optimizers In this work, the network architecture for SFM and the state-only baselines is inspired from TD7 (Fujimoto et al., 2023). TD7 is a recent algorithm presenting several tricks to attain improved performance relative its celebrated predecessor TD3. To evaluate how robust these methods are to the quality of the RL algorithm, we also study the performance characteristics when employing the relatively weaker TD3 optimizer. The RLiable plots in Figure 4 present the efficacy of SFM to learn with the less performant TD3 optimizer. Remarkably, the performance of SFM (TD3) is similar efficacy of our non-adversarial method to learn with other off-the-shelf RL algorithms. However, the adversarial baselines did not perform as well when deployed with TD3. To further understand the performance difference, in Figure we that SFM attains significant performance gains across tasks in the quadruped domain. contrast, the adversarial state-only baselines perform similarly on tasks in the cheetah and walker domains for both RL optimizers.
the SFM (Figure 2) demonstrating the
Ablation over base features In Figure 7, we study the performance SFM with various base feature Φ. As discussed in $4.2, experiment with Random Features, Inverse Dynamics Mod- els (IDM; Pathak et al., 2017) 7), Hilbert Representations (Hilp; Park et al., 2024), Forward Dynamics Model (FDM), Autoencoder (AE), and Adversarial (Adv) features. Through our experiments, we observe that FDM achieves superior results when compared with other base feature functions (Fig- ure 5). In Figure 7 and Table 3, we see that, IDM features performed similarly FDM on walker and cheetah domains, but did not perform well on quadruped tasks. We believe it is challenging to learn IDM features on quadruped domain which has been observed in prior works (Park al., 2024; Touati et al., 2023). Similar trends were observed for Hilp features and we suspect that learning the notion of temporal distance during online learning is challenging as the data distribution changes while training. Random features performed well on quadruped domain, but did not perform well on cheetah and walker tasks. Autoencoder (AE) and adversarial (Adv) features did well across RLiable suspect that leveraging structure
metrics, however FDM features achieved better performance- from the dynamics leads to superior performance. Moreover, learning adversarial features required tricks like gradient penalty and learning rate decay. We believe SFM leverage any representa-

---

Published as a conference paper at ICLR 2025
tion learning technique to obtain base features and potential avenue for future work is to leverage pretrained features for more complex tasks to speed up learning.
6 DISCUSSION
We introduced SFM-a novel non-adversarial method for IRL. that requires no expert action labels. Our method learns to match the expert's successor features, derived from adaptively learned base features, using direct policy search as opposed to solving minmax adversarial game. Through experiments on several standard imitation learning benchmarks, we have shown that state-of-the-ar imitation is achievable with non-adversarial approach, thereby providing an affirmative answer to our central research question.
Consequently, SFM is no less stable to train than its online RL subroutine. This is not the case with adversarial methods, which involve complex game dynamics during training. Much like the rich literature on GANs (Goodfellow et al., 2014; Gulrajani et al., 2017; Kodali al., 2018), adversarial IRI. methods often require several tricks to stabilize the optimization, such as gradient penalties, bespoke optimizers, and careful hyperparameter tuning.
Beyond achieving state-of-the-ar performance, SFM demonstrated unexpected feat: is excep- tionally robust to the policy optimization subroutine. Notably, when using the weaker TD3 policy optimizer, SFM performs almost as well as it does with the relatively stronger TD7 optimizer. This is in stark contrast to the baseline methods, which performed considerably worse under the weaker policy optimizer. As such, we expect that SFM can be broadly useful and easier deploy on resource-limited systems, which is often a constraint in robotics applications.
Interestingly, SFM follows recent trend in model alignment that foregoes explicit reward modeling for direct policy search. This was famously exemplified in RLHF with DPO (Rafailov et al., 2024) and its subsequent extensions (Azar et al., 2024; Munos al., 2024). It is worth noting that SFM, unlike DPO, does require modeling state features. However, the state features modeled by SFM are pask-agnostic, and we found in particular that state embeddings for latent dynamics models suffice. We emphasize that this is reflection of the more complicated dynamics inherent to general RL problems, unlike natural language problems which have trivial dynamics.
SFM is not the first non-adversarial IRL. method; we note that IQ-Learn (Garg et al., 2021) similarly reduces IRL to RL. However, we showed that SFM substantially outperforms IQ-Learn in practice, and more importantly, it does so without access expert action labels. Indeed, to our knowledge, SFM is the first non-adversarial state-only interactive IRL method. This opens the door to exciting possibilities, such as imitation learning from video and motion-capture data, which would not be possible for methods that require knowledge of the expert's 's actions. We believe that the simpler, non-adversarial nature of SFM training will be highly useful for scaling to such problems.
Limitations While SFM is simpler than IRL methods, it still doesn't 't theoretically alleviate the exploration problem that IRL methods encounter. A promising direction of future work could com- bine SFM with resets (Swamy et al., 2023) or hybrid IRL (Ren et al., 2024) to improve sample efficiency. Alternatively, SFM can leverage existing exploration algorithms with certain methods leveraging successor features being particularly amenable (e.g., Machado et al., 2020).
ACKNOWLEDGEMENTS
The authors would like to thank Lucas Lehnert, Adriana Hugessen, Gokul Swamy, Juntao Ren, Andrea Tirinzoni, and Ahmed Touati for their valuable feedback and discussions. The writing of the paper benefited from discussions with Darshan Patil, Mandana Samiei, Matthew Fortier, Zichao Li and anonymous reviewers. This work was supported by Fonds de Recherche du Québec, Na- tional Sciences and Engineering Research Council of Canada (NSERC), Calcul Québec, Canada CIFAR AI Chair program, and Canada Excellence Research Chairs (CERC) program. The au- thors are also grateful to Mila (mila.quebec) IDT and Digital Research Alliance of Canada for com- puting resources. Sanjiban Choudhury is supported in part by Google Google Faculty Research Award, OpenAl SuperAlignment Grant, ONR Young Investigator Award, NSF RI #2312956, and NSF FRR#2327973.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2004.
Majid Abdolshah, Hung Le, Thommen Karimpanal George, Sumil Gupta, Santu Rana, and Svetha
Venkatesh. A new representation of successor features for transfer across dissimilar environments. In International Conference on Machine Learning (ICMI.) 2021.
Marwa Abdulhai, Natasha Jaques, and Sergey Levine. Basis for intentions: Efficient inverse rein- forcement learning using past experience. CoRR, abs/2208.04919, 2022.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Neural Information Pro- cessing Systems (NeurIPS), 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2017.
André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Rémi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning (ICML), 2018. 2018.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences (PNAS), 1 117(48):3007--3008 2020.
Diana Borsa, André Barreto, John Quan, Daniel J. Mankowitz, Hado van Hasselt, Rémi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In International Conference on Learning Representations (ICLR), 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //github. com/ google/jax.
Eli Bronstein, Mark Palatucci, Dominik Notz, Brandyn White, Alex Kuefler, Yiren Lu, Supratik Paul, Payam Nikdel, Paul Mougin, Hongge Chen, al. Hierarchical model-based imitation learning for planning in autonomous driving. In International Conference on Intelligent Robots and Systems (IROS), 2022.
Robert Dadashi, Adrien Ali Taiga, Nicolas Le Roux, Dale Schuurmans, and Marc G Bellemare. The value function polytope in reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. In International Conference on Learning Representations (ICLR), 2018.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-624, 1993.
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor critic. In International Conference Machine Learning (ICML), 2012.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G Bellemare. Proto-value networks: Scaling representation learn- ing with auxiliary tasks. In International Conference on Learning Representations (ICLR), 2023.

---

Published as a conference paper at ICLR 2025
Angelos Filos, Clare Lyle, Yarin Gal, Sergey Levine, Natasha Jaques, and Gregory Farquhar. PsiPhi- learning: Reinforcement learning with demonstrations using successor features and inverse tem- poral difference learning. In International Conference on Machine Learning (ICML), 2021.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference on Learning Representations (ICLR), 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning (ICML), 2018.
Scott Fujimoto, David Meger, and Doina Precup. An equivalence between loss functions and non- uniform sampling in experience replay. In Neural Information Processing Systems (NeurIPS), 2020.
Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For SALE: State-action representation learning for deep reinforcement learning. Neural Information Processing Systems (NeurIPS), 2023.
Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. In International Conference on Learning Representations (ICLR), 2020.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. IQ-Iearn: Inverse soft-q learning for imitation. In Neural Information Processing Systems (NeurlPS), 2021.
Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforeement learning from passive data via latent intentions. In International Conference on Machine Learning (ICML). 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Pro- cessing Systems (NeurIPS), 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Neural Information Processing Systems (NeurIPS), 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning (ICML), 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Neural Information Processing Systems (NeurIPS), 2016.
Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucei, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In International Conference on Robotics and Automation (ICRA), 2022.
Arnav Kumar Jain, Lucas Lehnert, Irina Rish, and Glen Berseth. Maximum state entropy explo- ration using predecessor and successor representations. In Neural Information Processing Systems (NeurIPS), 2023.
Rohit Jena, Siddharth Agrawal, and Katia Sycara. Addressing reward bias in adversarial imitation learning with neutral reward functions. CoRR, abs/2009.0946) 2020.
Diederík P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Naveen Kodali, James Hays, Jacob Abernethy, and Zsolt Kira. On convergence and stability of GANs. In International Conference on Learning Representations (ICLR), 2018.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations (ICLR), 2022.
Charline Le Lan, Stephen Tu, Adam Oberman, Rishabh Agarwal, and Mare G. Bellemare. On the generalization of representations in reinforcement learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
12

---

Published as a conference paper at ICLR 2025
Charline Le Lan, Joshua Greaves, Jesse Farebrother, Mark Rowland, Fabian Pedregosa, Rishabh Agarwal, and Marc G. Bellemare. A novel stochastic gradient descent algorithm for learning principal subspaces. In International Conference Artificial Intelligence and Statistics (AIS- TATS), 2023a.
Charline Le Lan, Stephen Tu, Mark Rowland, Anna Harutyunyan, Rishabh Agarwal, Mare G. Belle- mare, and Will Dabney. Bootstrapped representations reinforcement learning. In International Conference on Machine Learning (ICML), 2023b.
Donghun Lee, Srivatsan Srinivasan, and Finale Doshi- Truly batch apprenticeship learning with deep successor features. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Lucas Lehnert, Stefanie Tellex, and Michael L Littman. Advantages and limitations of using suc- cessor features for transfer in reinforcement learning. CoRR, abs/1708.00102, 2017.
Marlos C Machado, Mare Bellemare, and Michael Bowling. Count- exploration with the successor representation. In AAAI Conference on Artificial Intelligence, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529- 533, 2015.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback. In International Conference on Machine Learning (ICML), 2024.
Whitney K Newey and James L Powell. Asymmetric least squares estimation and testing. Econo- metrica: Journal of the Econometric Society, pp. 819-847, 1987.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2000.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICML), 2024.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), 2017.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Bridging the gap between imitation learning and inverse reinforcement learning. JEEE transactions on neural networks and learning systems, 28 (8):1814-182) 2016.
Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imi- tation via behavior foundation models. In International Conference on Learning Representations (ICLR), 2024.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Neural Information Processing Systems (NeurIPS), 1988.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Neural Information Processing Systems (NeurIPS), 2024.
Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J Andrew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2024.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In International Conference Artificial Intelligence and Statistics (AISTATS), 2010.
13

---

Published as a conference paper at ICLR 2025
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. De- terministic policy gradient algorithms. In International Conference on Machine Learning (ICML), 2014.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, 1lya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):444488, 2016.
Wen Sun, Anirudh Vemula. Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International Conference on Machine Learning (ICML), 2019.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9 44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Neural Information Processing Systems (NeurIPS), 1999.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. moments and match- ing: A game-theoretic framework for closing the imitation gap. In International Conference Machine Learning (ICML), 2021.
Gokul Swamy, Nived Rajaraman, Matt Peng, Sanjiban Choudhury, Drew Bagnell, Steven Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation. In Neural Information Processing Systems (NeurIPS), 2022.
Gokul Swamy, David Wu, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Inverse reinforce- ment learning without reinforcement learning. In International Conference on Machine Learning (ICML), 2023.
Umar Syed and Robert E Schapire. A game-theoretic approach apprenticeship learning. In Neural Information Processing Systems (NeurIPS), 2007.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear pro- gramming. In International Conference on Machine Learning (ICML), 2008.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. 2018.
CoRR, abs/1807
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize rewards. In Neural Information Processing Systems (NeurIPS), 2021.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In Internutional Conference Learning Representations (ICLR), 2023.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm-control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. In Neural Information Processing Systems (NeurIPS), 2022.
14

---

Published as a conference paper at ICLR 2025
Harley Wiltzer, Jesse Farebrother, Arthur Gretton, and Mark Rowland. Foundations of multivari- ate distributional reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2024a.
Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang, André Barreto, Will Dabney, Mare G. Bellemare, and Mark Rowland. A distributional analogue to the successor represen- tation. In International Conference on Machine Learning (ICML), 2024b.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. In Inter- national Conference on Intelligent Robots and Systems (IROS), 2017.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-po imitation learning from observa- tions. In Neural Information Processing Systems (NeurlPS), 2020.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008.
15

---

Published as a conference paper at ICLR 2025
A PROOFS
Before proving Proposition we begin by proving some helpful lemmas. First, we present simple generalization of result from Garg et al. (2021).
Let i denote any discounted state-action occupancy measure for an MDP with state space S and initial state distribution Po. and let V denote vector space. Then for any f : - V, the following holds,
Lemma 1.
((,A)~ - (155)))11Sss--p((11111111111 = >) [f(8))
Proof. Firstly, any discounted state-action occupancy measure μ identified with a unique policy T² as shown by Ho & Ermon (2016). So, is characterized by
p(dsda) = -1-1)) -1- (da)
>> (ds),
where (S) Pr. (S, E S) the state-marginal distribution under policy Th at timestep Expanding the LHS of the proposed identity yields
(( (f(S) (1 - -)
(1 7)
y EBs [f(S)] - B (S.A)) -1s
1=
= (1 7)
Σ
y E'Sspp [S(S)] - (1 7)
++ r+ Br
1=
1=
20
(1 - 7)
7 /'SSsp' (S(S)) - (1 77 Σ
(+ -.tEEspp [S(S)
(1 7
E s [f(S) - (1 7
E Sn [f(S)]
1=
= (1 7) Es (S(S))
since vo Po (the initial state distribution) for any μ.
Intuitively, we will invoke Lemma with ff denoting the successor features to derive an expression for the initial state successor features via state transitions sampled from a replay buffer.
Proposition 1. Let B denote buffer of trajectories sampled from arbitrary stationary Markovian policies in the given MDP with initial state distribution Po. For any stochastic policy ा,
Φ = (1-7)-1BBg B(s,
EA (A A)] -7Ar [b A']]
(8)
Proof. Suppose B contains rollouts from policies { for some N E N. Fach of these policies Th induces a discounted state-action occupancy measure μ4a Since the space of all discounted state-
action occupancy measures is convex (Dadashi al., 2019), it follows that μ + Ek Pa is itself a discounted state-action occupancy measure.
Consider the function f S Rd given by f(s) E ( (s, A). We have
16

---

Published as a conference paper at ICLR 2025
B -l(()
-
B1 (f Yf(S))
((S,A)y [f()) - 7 11Ss0)/sss1-
(1 - 7) Bs [[S)]
where the final step invokes Lemma 1, which is applicable since is a discounted state-action occu- pancy measure. The claim then follows by substituting f (S) for (= (S, A) and multipying through by (1-7)-1.
Proposition 2. For stochastic policies : S (AA) the policy gradient under which the return most steeply increases for the reward function r n -ax defined in equation 6 is given by,
V log (A 5) 35 (S, A)]) (10)
Alternatvell, for deterministic policies π S A, the deterministic policy gradient (Silver al., 2014) for the reward function -, +tg defined in equation 6 is given by,
VaU(Fpir' 7, = (wT) (Esmpe (p) V V (S. A)])
(11)
Proof. That v (Ti r - is the direction that most steeply increases the return under the reward function xx+ -n is established in Degris al. (2012). It remains derive expression for (TU r - Since r - is linear in the base features, given an estimate va of the successor features for policy The the action-value function is given by
Q0(8,a) (8))8, TTo))=",,r -
Then, we have that
V VaU( PTUiTipT x = Esuyp V, Tx(a (S)oo(S,)
(17)
= Es V, x, () a)T "== -ap-
When Tu is stochastic, the log-derivative trick yields
V (mpit +mo) = Es- ANTT-1S)) V log np S) A) " u -
(wi T (Es-po E 9) V log (, (),
Alternatively, for deterministic policies T. (where, with notational abuse, we write (s)s)) E A), the deterministic policy gradient theorem (Silver et al., 2014) gives
VnU TAU(TppP P my
a)T -76) V (S)
S-upa
(ST (
18=m,(9)
Snipa
(w+ +Rg)7 ST
(I (S) (s)
Snippl
as claimed.
17

---

Published as a conference paper at ICLR 2025
B EXTENDED RESULTS
In this section, we provide the tables with average returns across tasks from DMControl Suite (Table
1, 2 & 3) and per-environment training runs for our study with weak policy optimizers and base feature functions ((Fig. 6 & 7).
Task
BC
IQ-Learn
OPOLO
MM
GAIfO
SFM
Cheetah Run
77.0 + 11.1
1.4 1.4
747.6 6.7
781.6
80.7
777.2 = 45.0
648.8 + 35.9
Cheetah Walk
371.1 + 163.3
5.7
6.9
919 + 26.3
896.6
128.2
863. + 236.
945
33.9
Quadruped
150.5
29.7
260.7
12.0
198.9 1: 50.6
489.
104.6
505.8
192.3
799.1
47.8
Quadruped
62.1 22.4
174.7
+
347.4
280.4
227.3
65.9
Quadruped Stand
351.6
G8.5
351.1
26.7
378.7
+ 37.3
732.2
+ 271.9
804.7 + 211.5
941.6
25.7
Quadruped Walk
119.0 d: 40.9
171.6 8: 11.2
391.8
1: 57.9
844.7
138.
656.1 4 321.2
759.9
177.5
Walker Flip
39.6
2.2
2.5
24.5
706.2
496.9
264.3
090.7
101.9
653.6
26.7
Walker Smad
168.5 2 48.9
181.1 +
135.8
846.2 + 256.9
574.2
209.3
810.4
250.3
900 4 96.9
Walker Walk
36.
29.6
20.3
52
2.6
738.9
399.T
726.3
234.8
792.8
242.2
916.5
48.4
Table 1: Returns achieved by BC, IQ-Leamm, OPOLO, state-only MM, GAIfO and SFM across tasks on the DMControl Suite. The average returns and standard deviation across 10 seeds are reported.
Vibiker Walk
Ier Run
Walker Fiip
Walker Stannd
Che stah Walk
60.0
900
900
9:0
600
R00
600
6:0
60D
300
300
300
300
Cheetah Run
Quadruped Walk
Quadruped ded Stand
Quadruped Run
Quadruped Jump
900
300
500
900
900
600-
600
600
600
600
300
300
300
M
www
0.50
9.00
6.00
8.c0
5.00
6.00
Environment Stops
106
Enwirnment Stepo
106
Environment Stops
166
Environment Stops
106
En EnvinonmentSteee 106
MM (TD3)
GAILO (TD3)
SFM (TD3)
Expert
Figure 6: Comparison of state-only IRL methods using the weaker TD3 policy optimizer. Notably,
only SFM consistently maintains strong performance with the weaker policy optimizer.
Environment
MM (TD3)
GAlIO (TD3)
SFM (TD3)
Cheetah Run
439.6 138 6
674.0
514.7 = 77.9
Cheetah Walk
859.6 = 165.3
873.9 58
829.7 226.3
Qundruped Jung
308.8 115.9
334.3 1 159.8
821.6 1 27.5
Quadruped Run
107.0 +: 22.8
94.4 23.1 23.1
705.6 + 57.3
Quadraped Stand
449.7 206.1
381.8 216.0
946.3 20.5
Quadruped Walk
201 = 175.8
317.3 246.4
829.3 = 86.9
Walker Flip
7:4.4
37.7
Walker Run
530.2 163.
600.9 105.0
606.5 = 30.2
Walker Stand
375.8 245.8
764.8 220.7
934.0 : 49.6
Walker Walk
395. 3
769.7 1 258.1
880.1 75.8
Table Comparison of state-only IRL methods using the weaker TD3 policy optimizer. This table presents returns achieved by state-only MM (TD3), GAIFO (TD3) and SFM (TD3) across tasks on the DMControl Suite. The average returns and standard deviation across 10 seeds are reported.
18

---

Published as a conference paper at ICLR 2025
Wallber Walk
Walker Run
Walker
Walker Stand
Cheetah Walk
900
909
600-
600
600
600
600
Average
300-
100
300
Cheetah Run
Quaciruped Walk
Quadruped Stand
Quadruped Run
Quadruped jump
902
903
900
0,00
1.00
8.00
0.00
Environment Stops 166
Environment Stops
166
Envfronment Stops
166
Er Envronment Stoge
168
Enwirommnnt Steps
1e6
FDM
IDM
Hilp
Random
AE
Adv
Expert
Figure 7: Effect of different base feature functions on the performance of the agent. Here, we com- pare with Random, Inverse Dynamics Model (IDM) (Pathak et al., 2017), Hilbert Representations (Hilp) (Park et al., 2024), Autoencoder (AE), Adversarial representations (Adv) and Forward Dy- namics Models (FDM). FDM was found to work best across DMC tasks. Note that all base feature functions were jointly learned during training.
Environment
Random
AE
Hilp
IDM
FDM
Ady
Cheetah Run
484.4
45.4
585.7 +: 93.6
417.8 + 118.8
622.0
69.3
648.8
35.9
374.1 + 113.7
Cheetah Walk
107.6
38.9
244.5
Quadruped Jump
744.4
79.9
59.0
47.8
144.1
Quadruped Run
356.5
92.7
493.4
57.8
311.8
94.5
118.0
86.1
671.7
65.9
737
196
Quadriped Stand
914.0
33.1
896.3 + 94.2
258.1 + 102.0
222.0
G3.3
941.6
25.7
806.4 + 140.3
Quadiuped Walk
402.8
145.6
Flip
101.8
771.9 + 197.0
912.8
36.9
856.9
64.5
439.0
Walker Run
506.6 + 181.8
632.2 + 41.7
616.7
53.8
689.5
139.0
658.6 2 26.7
620.5 + 158.2
Walker Stand
716.9 + 160.9
836.2
140.0
934.8
42.8
960.6
22.9
9U9.4 +: 96.9
964.2
31.8
Walk
189.7
259.8
Table 3: Effect of different base feature functions the performance of the agent. Here, pare with Random, Inverse Dynamics Model (IDM) (Pathak et al., 2017), Hilbert Representations (Hilp) (Park al., 2024), Autoencoder (AE), Adversarial representations (Adv), and Forward Dy- namics Models (FDM). The table reports the returns achieved by each base feature function when trained with SFM across tasks on the DMControl Suite. The average returns and standard deviation across 10 seeds are reported. FDM was found to work best across DMC tasks. Note that all base feature functions were jointly learned during training.
19

---

Published as a conference paper at ICLR 2025
B.1
SFM WITH STOCHASTIC POLICY
To extend SFM to stochastic policies, we propose having an agent comprising of a stochastic actor parameterized to predict the mean and standard deviation of multi-variate gaussian distribution. Here, for a given state 8, the action is sampled using a ~ (-1s))) = - (*), ( where my S + IR" and of : S - RT for A = R". The SF network architecture the is the same as the SFM (TD3) variant, where the network estimates the SF for state-action pair. The SF-network can be updated using 1-step TD enror using the base features of the current state similar to equation 2. The actor is updated using the update rule described in Proposition 2 where we estimate the policy gradient via reparameterization trick with Gaussian policies- particularly, we consider the class of Gaussian policies with diagonal covariance as in Haarnoja et al. (2018). Finally, to prevent the pearly-deterministic one, also include a policy entropy bonus
policy from quickly collapsing our actor updates. We conduct experiments over the tasks from DMControl suite and present environment plots in Figure 8 and returns achieved in Table 4. We provide the implementations of SFM with stochastic policy in Appendix C,3
Walker
402-
900
500
Average Returna
300
300
200
300
300
0
Cheetah
Quadruped
Quadruped
Quadruped
Quadruped
900
900
900
602-
800
600
600
600
Average
300
500
300
500
300
0.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
8.00
0:25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1,00
6.00
0.25
0.50
0.75
1.00
Endiromment Steps
Environment Steps
145
Enddronmmnt Staps
166
Erwiroement Staps
14b
Environment Steps
106
SFM (TD7)
SFM (Stochastic)
SFM (TD3)
Expert
Comparison of variants of SFM with TD7, TD3 and an entropy regularized stochastic policy. We observe that SFM can be trained with stochastic polices. However, the variants with deterministic policy optimizers work better some tasks than the stochastic policy.
Figure 8:
Environment
SFM (TD7)
SFM (TD3)
SFM (Stochastic)
Cheetah Run
648.8
$14.7 77.9
500.9 136.3
Cheetah Walk
945.1 + 33.9
829.7 + 226.3
918.8 #: 21.7
Quadruped Jump
7
821.6 27.5
764.0 + 84,8
Quadruped Run
671.7
705.6 57.3
614.9 113.4
Quadruped Stand
24.
946.3 20.5
829.5 224.1
Quadruped Walk
759.9 177.5
829.3 1 86.9
K21.9 a: 56.3
Walker Flip
64.5
865.5 37.7
830.4 45.4
Walker Run
66.76
606.5 1: 30.2
630.6 1 17.5
Walker Stand
909.4
934.0 49.6
925.7 + 38.9
Walker Walk
936.5
880.1 + 75.8
916.3 + 43.9
Table 4: Comparison of SFM with a stochastic policy and variants based on deterministic policy
optimizers (TD3 & TD7). The table reports the returns achieved by each base feature function when trained with SFM across tasks on the DMControl Suite. The average returns and standard deviation across 10 seeds are reported.
20

---

Published as a conference paper at ICLR 2025
C IMPLEMENTATION DETAILS
Since SFM does not involve learning an explicit reward function and cannot leverage an off-the- shelf RL algorithm learn a Q-funtion, we propose novel architecture for method. SFM is composed of 3 different components- actor Tu. successor features (SF) network ho. base feature function & and f. Taking inspiration from state-of-the-ar RL algorithms, we maintains target net- works. Since, SF network acts similarly to critic in actor-critic like algorithms, SFM comprises of networks to estimate the SF (Fujimoto al., 2018). Here, instead taking minimum over estimates of SF from these two networks, our method performed better with average over the two estimates of SF. To implement the networks of SFM, we incorporated several components from the TD7 (Fujimoto et al., 2023), TD3 (Fujimoto et al., 201 8), and SAC (Haarnoja et al., 2018) 8) algo- rithm which are described in this section. Moreover, SFM does not require techniques like gradient penalty (Gulrajani et al., 2017), the OAdam optimizer (Daskalakis et al., 2018) and learning rate scheduler.
C.I TD7- NETWORK ARCHITECTURE
The architecture used in this work is inspired from the TD7 (Fujimoto al., 2023) algorithms for continuous control tasks (Pseudo 1). We will describe the networks and sub-components used below:
Two functions estimate the SF (sbo,- 10,z)
Two target functions to estimate the SF (tho.. bb,,)
policy network n
A target policy network Tp
An encoder with sub-components fv, 9v
A target encoder with sub-components fo.
A fixed target encoder with sub-components fp. 96
A checkpoint policy and the checkpoint encoder fe
A base feature function Ø
Encoder: The encoder comprises of two sub-networts to output state and state-action embedding,
such that 2 = [s(()) and sa = (2 ((2",)), a). The encoder is updated using the following loss:
Encoder (ncoder(Juss = (go(Ju()), - -100())
(18)
( 12
(19)
where s' denotes the sampled transitions and Ix is the stop-gradient operation. Also, we represent fo(s), gea (0 "i0z,,a),. , ~² fz(s), and zea ga (3" a).
SF network: Motivated by standard RL algorithms (Fujimoto et al., 2018; 2023), SFM uses pair of networks estimate the SF. The network estimate SF are updated with the following loss:
(sb()),)) |target - bb, (=*a ,", (...))
(20)
target o(s) 2 clip(lbe, (x) + va (x)), a min: mmax),
(21)
x = [ s' ze' ,a) a' = ( .s')
(22)
e, where clip(N((.
(23)
Here, instead of taking the minimum over the SF networks for bootstrapping at the next state (Fuji- moto et al., 2018), the mean over the estimates of SF is used. The action at next state a' is samples similarly to TD3 (Fujimoto et al., 2018) and the same values of (z*, 2*)) are used for each SF net- work. Moreover, the algorithm does clipping similar to TD7 (Fujimoto et al., 2023) the predicted SF at the next state which updated using larget (equation 32) at each time step, given by:
Vmin +- min(1mmin target)
(24)
h min target)
(25)
21

---

Published as a conference paper at ICLR 2025
Policy: SFM uses single policy network which takes [a", s] as input and is updated using the following loss function described in $4.
Upon every tar sarget.updaate_freeuee the current network parameters and remains fixed:
training steps, the target networks updated by cloning
(01, (01, 02)
(26)
# f
(27)
(2) + (i1 122)
(28)
(2) D2)
(29)
(30)
Moreover, the agent maintains checkpointed network similar TD7 (Refer Appendix of TD7 (Fujimoto et al., 2023) paper). However, TD7 utilizes the returns obtained in the environment
for checkpointing. Since average returns absent the IRL tasks, is not clear how checkpoint policies. Towards this end, we propose using the negative of Mean Squared Error (MSE) between the SF of trajectories generated by agent and the SF of demonstrations as a proxy of checkpointing. To highlight some differences with the TD7 (Fujimoto et al., 2023) algorithm, SFM does not utilize a LAP (Fujimoto et al., 2020) and Huber loss to update SF network, and we leave investigating them for future research.
22

---

Published as a conference paper at ICLR 2025
Pseudo SFM (TD7) Network Details
Variables:
phi_ thi_dim 128
zs_dim = 256
Value SF Network:
D SFM uses two SF networks each with similar architechture and forward pass.
10 = Linear (state_dde _ddimmmmm + action dim, 256)
Linear (2s (2s_dim 2 + 256, 256)
12 Linear (256, 256)
13 Linear (256, phi_dim)
SF Network by Forward Pass:
input concatenate (state, action))
AvgL1Norm (10 (1nuupt)))
concatenate [zsa, zs, x])
ELU (11 ELU (x))
ELU (12 (x))
sf = 13 (x)
Policy π Network:
10 Linear (state_dimg 256)
11 Linear (zs_dim + 256, 256)
12 Linear (256, 256)
13 Linear (256, action dim)
Policy Forward Pass:
input = state
AvgLINorm (10 (input))
concatenate ( [zs, x])
ReLU (11 (x))
ReLU (x)
action tanh (x))
State Encoder Network:
11 Linear (state dim, 256)
12 Linear (256, 256)
13 Linear (256, za_dim)
State Encoder / Forward Pass:
input = state
ELU (11 (input)) )
ELU (x))
zs AvgT.1Norm (13 )
State-Action Encoder g Network:
11 = Linear (action_dim + dim, 256)
12 Linear (256, 256)
13 Linear (256, zs_dim)
State-Action Encoder Forward Pass:
input concatenate ([action, zs])
ELU (11 (input))
ELU (12 (x))
zsa 13 (x)
23

---

Published as a conference paper at ICLR 2025
C.2 TD3-BASE NETWORK ARCHITECTURE
The architecture used in this work is inspired from the TD3 (Fujimoto et al., 2018) algorithms for continuous control tasks (Pseudo 2). We will describe the networks and sub-components used below:
Two functions to estimate the SF (sboL. 1622)
Two target functions estimate the SF (1bg, v82))
A policy network "M
A base feature function Ø
SF network: Motivated by standard RL algorithms (Fujimoto et al., 2018; 2023), SFM uses pair
of networks estimate the SF. The network estimate SF updated with the following loss:
LsF ||target (s, as.
(31)
target ((5) 2 * (va, (x) (x), (z))
(32)
I [s'
(33)
d' = ===(((s)) + where e ~~ clip(N(0,,
(34)
Here, instead of taking the minimum over the SF networks for bootstrapping at the next state (Fuji-
moto et al., 2018), the mean over the estimates of SF is used. The action at next state a' is samples
similarly to TD3 (Fujimoto et al., 2018). Lastly, target networks to estimate SF updated via polyak averaging (with polyak factor of .995, given by
0, - atl + (1 - a),,, for = 1,2.
(35)
Policy: SFM uses a single deterministic policy network which takes state to predict action a.
Pseudo 2. SFM (TD3) Network Details
Variables:
phi_dim 128
Value SF Network:
P SFM uses two SF networks each with similar architechture and forward pass.
10 Linear (state_de _im + action_dim, 256)
11 Linear (256, 256)
12 =
Linear (256, phi__im)
SF Network th Forward Pass:
input concatenate (state, action))
ReLU (10 (inuput))
ReLU (11 (x))
12
Policy Network:
10 Linear (state_diae 256)
11 Linear (256, 256)
12 =
Linear (256, action_ding
Policy Forward Pass:
input state
ReLU (10 (input)) )
ReLU (11 (x))
action = tanh (12 )
C.3 SFM WITH STOCHASTIC POLICY
In light of Proposition 1, for a stochastic policy. Proposition can again be used to update the policy parameters. Indeed, following the proof of Proposition 2, have that
24

---

Published as a conference paper at ICLR 2025
V (T 7m
IE
V
Ty(a| (0)o a)da
BmB
IE
(5, a) " + Va s)da
(36)
( s / a) a) s)da
The integral above be estimated without bias by Monte Carlo, using the log-derivative (RE- INFORCE) trick, or in the case of certain policy classes, the reparameterization trick (Kingma & Welling, 2014; Haarnoja et al., 2018); the latter of which tends to result in less variance in practice (Haarnoja et al., 2018), 1), so we use it our experiments. Towards this end, let y denote a measurable space with @ probability measure over V and suppose there exists a. y A such that
Tx (- 8) Law Law(ap(s; e)), e 0.
Under such parameterizations, have
= Eevve V = (. a)la=a
Example 1. Gaussian policies- that is, policies of the form + N(m(s). E(s)) for m S + Rd and Σ -> Raxd _can can be reparameterized the aforementioned manner. Taking y Rdd
and @ = (0, Ia), construct the map By S x x + by
(p( E) = m)()) En(s)e.
Clearly, it holds that Law :)) (m,, (8)), accommodating any Gaussian policy. Altogether, our gradient estimate is computed as follows,
Vud(TTi Ty -x))
W N
n V.
④ = (1 7)
a - 2 Te
N₂
ر iid B,
{ 10)
((82))82))) iid B, { N/O,, In), ( I.)
(37)
We note that, to compute an unbiased gradient from samples, minibatches of (s, s') pairs must be sampled independently for the estimator úi of w = and likewise action samples must be drawn independently in the computation of t..
Finally, to prevent the policy from quickly collapsing to a nearly-determiniss one, we also include policy entropy bonus in our actor updates. The architecture used in this work is inspired from the SAC (Haarnoja al., 2018) algorithms for continuous control tasks (Pseudo 3). We will describe the networks and sub-components used below:
Two functions to estimate the SF (zbo,, vb,,)
Two target functions estimate the SF (sbū,- sbaz)
A policy network T.
25

---

Published as a conference paper at ICLR 2025
A base feature function @
SF network:
Motivated by standard RL algorithms (Fujimoto et al., 2018; 2023), SFM uses a pair of networks to estimate the SF. The network estimate SF are updated with the following loss:
CsF ( = ||target - Da, (8, a)l
target o(s) + 2 (bb,, (x) 1o, (z)),
[s', a' (s()
Here, instead of taking the minimum over the SF networks for bootstrapping at the next state (Fuji- moto et al., 2018), the mean over the estimates of SF is used. The action at next state a' is samples similarly to TD3 (Fujimoto et al., 2018). Lastly, target networks to estimate SF updated via polyak averaging (with polyak factor of .995, given by
0 - ad, + (1-a));, for 1,2.
(38)
SFM uses a single Gaussian policy network which takes state to sample an action a as described earlier.
Policy:
Pseudo 3. SFM (Stochastic) Network Details
Variables:
phi_dim = 128
Value SF Network:
& SFM uses two SF networks each with similar architechture and forward pass.
10 Linear (state dim + action_diig 256)
11 Linear (256, 256)
12 Linear (256, phi_dim)
SF Network by Forward Pass:
input = concatenate [state, action))
ReLU (10 (inuput)) )
ReLU (11 (x))
x 12 (x)
Policy Network:
10 Linear (state_dd) dim, 256)
11 Linear (256, 256)
1m Linear (256, action_dimg
1s Linear (256, action_dim)
Policy π Forward Pass:
input state
ReLU (10 (input)) )
x ReLU (11 (x) )
mean = tanh (1m (x)
std softplus (1s (x)
C.4 BASE FEATURES
Since base feature function é, have two networks- 1) To provide the embedding for the state, and 2) To predict the next state from the current state and action. Pseudo provides the description of the network architectures and the corresponding forward passes.
26

---

Published as a conference paper at ICLR 2025
Pseudo Base Feature Network Details
Variables:
phi_ dim 128
Base Feature Network & to encode state:
10 Linear (state_ddie) 512)
12 Linear (512, 512)
13 =
Linear (512, phi_
Base Feature é Forward Pass:
input = state
Layernorm (11 (x))
tanh (x)
ReLU (x)
phi_ S = LZNorm (x)
FDM Network:
Linear _dm + action_dim, 512)
11 = Linear (512, 512)
12 Linear (512, action_dim)
FDM Network Forward Pass:
input concatenate [phi_s, action))
x ReLU (1)
ReLU (11 (x)
action = tanh (12(x))
C.5 STATE-ONLY ADVERSARIAL BASELINES
For the state-only MM method, we used the same architecture as TD7 (Fujimoto et al., 2023) or TD3 (Fujimoto et al., 2018) for the RL subroutine. We kept the same architecture the discrim- inator provided in the official implementation. However, only states as inputs. Additionally, we used gradient penalty and learning rate decay to update the discriminator, and OAdam optimizer (Daskalakis et al., 2018) for all networks. For GAIfO (Torabi et al., 2018), we used the same architecture as state-only MM. However, the discriminator takes the state-transition denoted the state and next-state pair as input.
modified the discriminator to take
27

---

Published as a conference paper at ICLR 2025
D HYPERPARAMETERS
In Table 5, we provide the details the hyperparameters used for learning. Many of our hyper- paramters are similar to the TD7 (Fujimoto et al., 2023) algorithm. Important hyperparameters include the discount factor 7 for the SF network and tuned with values [0.98, 0.99, 0.995] and report the ones that worked best in the table. Rest, our method was robust to hyperparameters like learning rate and batch-size used during training.
Name
Value
Batch Size
1024
Discount factor 7 for SF
.99
Actor Learning Rate
5e-4
SF network Learning Rate
5e-4
Base feature function learning Rate
5e-4
Network update interval
250
Target noise
.2
Target Noise Clip
.5
Action noise
Environments steps
le6
Table Hyper parameters used to train SFM.
28


Reviews:
[{"review_id": "BLsT4riSIU", "reviewer": "Reviewer_7726", "summary": "This work considers inverse RL in the state-only setting. Leveraging the linear structure of returns, as inner product of successor features and reward weight, the authors could model inverse RL without adversarially optimizing the reward weight, while is often done in prior works. This allows one to directly optimize policy through minimizing the gap between the successor features of expert policy and of the learned policy.", "questions": "- what are the hyperparameter search space for SFM and baselines respectively?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well organized and easy to follow\n\n- The motivation is sound and the resulting algorithm is straightforward (without requiring min-max optimization)", "weaknesses": "- It is quite surprising that IQL failed most of the tasks in Figure 3\n\n- The choice of baselines could be expanded. While SFM considers the state-only setting, GAIfO is the only baseline that originally proposed for this setting. It might be reasonable to include a few baselines that are designed for state-only setting, for example [1, 2] (with public implementation if my memory serves me right) and some more up-to-date baselines are appreciated. \n\n----\n\nNote: I have not actively followed the recent literature on successor features, so I am uncertain about its novelty when applied to the IRL setting. As a result, I have a low confidence score.\n\n[1] Gangwani, Tanmay, and Jian Peng. \"State-only imitation with transition dynamics mismatch.\" arXiv preprint arXiv:2002.11879 (2020).\n[2] Zhu, Zhuangdi, et al. \"Off-policy imitation learning from observations.\" Advances in neural information processing systems 33 (2020): 12402-12413.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZnH17Wts91", "reviewer": "Reviewer_w2Zs", "summary": "This paper introduces Successor Feature Matching (SFM), a novel non-adversarial approach to Inverse Reinforcement Learning (IRL). The key innovation is reformulating IRL as a direct policy optimization problem that matches successor features between the agent and expert demonstrations. The method has three notable contributions:\n\n1. A non-adversarial approach that avoids the computational expense and instability of traditional adversarial IRL methods\n2. The ability to learn from state-only demonstrations without requiring expert action labels\n3. Strong performance from as little as a single expert demonstration\n\nThe method leverages successor features to estimate expected cumulative features and uses policy gradient descent to minimize the gap between learner and expert features. The authors demonstrate superior performance compared to both adversarial and non-adversarial baselines across multiple control tasks.", "questions": "1. How would the method perform on tasks with sparse rewards or requiring significant exploration?\n2. Could the approach be extended to work with stochastic policies?\n3. How does the method compare to recent offline IRL approaches?\n4. What are the computational requirements compared to adversarial methods?\n5. How sensitive is the performance to the choice of successor feature architecture?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The non-adversarial approach using successor features is novel and elegantly simple compared to existing methods\n2. The approach is well-justified with clear theoretical analysis and proofs\n3. The method works with state-only demonstrations and single examples, making it widely applicable\n4. SFM integrates seamlessly with existing actor-critic frameworks\n5. Demonstrates superior performance across multiple tasks and metrics\n6. Shows consistent performance even with weaker policy optimizers", "weaknesses": "1. The paper doesn't fully address how SFM handles the exploration problem inherent in IRL\n2. The current implementation is tied to deterministic policy gradients, potentially limiting its applicability\n3. The method currently only works with state-only base feature functions\n4. While comprehensive, the evaluation is limited to DMC suite tasks; testing on more diverse environments would strengthen the claims", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HbI5wi9daG", "reviewer": "Reviewer_YndG", "summary": "The paper introduces Successor Feature Matching (SFM), a novel method to Inverse Reinforcement Learning (IRL) which is non-adversarial and does not require expert action labels. SFM directly optimizes a policy to match the expert's successor features (SFs) using policy gradient ascent. The paper claims that SFM offers several advantages over existing IRL methods, including simplified training, state-only learning and robustness to optimizer choice.\nThe paper supports these claims through empirical evaluations on the DeepMind Control suite, demonstrating that SFM outperforms state-of-the-art adversarial and non-adversarial baselines on a variety of single-demonstration tasks. Additionally, the paper explores the impact of different base feature functions on SFM's performance, finding that Forward Dynamics Models (FDM) lands the best results.", "questions": "- How does SFM handle scenarios where expert demonstrations do not cover the entire state space, particularly in sparse-reward or high-dimensional tasks?\n- Can the authors elaborate on potential extensions of SFM to stochastic policy optimizers, and any early experiments indicating feasibility?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- **Novel and well-motivated approach:** SFM provides a new angle in IRL by leveraging SFs for direct policy optimization, offering an alternative to adversarial methods.\n- **Strong empirical results:** Experiments demonstrate the superiority of SFM over sota methods on normalized return and optimality gap across single-demonstration tasks, demonstrating its effectiveness.\n- **State-only learning:** The ability to learn from state-only demonstrations is a great contribution.\n- **Robustness to RL optimizer choice:** SFM's performance with both strong (TD7) and weaker (TD3) optimizers shows versatility, making it potentially useful for resource-limited applications.\n- The experiments are conducted in clear logic and the analysis and observations are novel and interesting.", "weaknesses": "- **Dependence on deterministic policy gradients:** The current formulation of SFM is tied to deterministic policy gradient algorithms. Exploring extensions to stochastic policies would broaden its scope. However, it's clearly mentioned by the authors in the discussion part.\n- **Exploration Challenges:** as stated by the authors, the presented method does not fully resolve IRL exploration issues. I wonder how the performance may vary significantly across domains requiring extensive exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ufRb2dJRD3", "reviewer": "Reviewer_ZRQG", "summary": "This paper proposes Successor Feature Matching (SFM) in the Inverse Reinforcement Learning (IRL) problem.  The authors focused generalized reward-like properties of  successor features (Barreto et al. 2017), e.g. recovering reward, discounted accumulation. These allowed SFM for alternative formulation of policy optimization using L2 loss in Eq. (4). The experimental results suggest that the proposed SFM yields good scores in single-demonstration benchmarks and much more impressive scores among state-only IRL algorithms.", "questions": "1. How applying l2-loss  in Eq.(4) can be understood for the policy $\\pi_\\mu$? \n2. I think SFM (random) performed well; what is the reasoning behind this and presenting Eqs? (9-11)? Could the base feature only trained with arbitrary unsupervised learning with only states without the next states, actions, or goals?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed SFM, in contrast to adversarial methods, is more rooted in the historical context of early IRL studies.\n2. In my opinion, IRL (or even RL) methods do not have to restrict themselves to formulating scalar reward signals. Adaptation SF (Barreto et al., 2017) expands the notion of IRL into feature learning, which can relax some rigidity in RL-IRL formulation. Since the features have successfully formulated vectorized signals in the transfer learning tasks in RL, this could be a significant contribution that relaxes studies of IRL. \n3. The proposed method achieved good performance among modern IRL methods.\n4. Propositions 1-3 are straightforward to understand.", "weaknesses": "1. I do not understand why \"non-adversarial\" property is framed as the main contribution throughout the paper. Adversarial learning and SFM are not mutually exclusive. Eq. (3) shows that the base feature can be trained with adversarial learning by setting $\\mathcal{L}_\\mathrm{feat} = \\mathrm{Eq. (3)}$. If the author intended to deepen our understanding of the role of adversarial learning in IRL, they should have provided (1) another theoretical justification of non-adversarial learning and (2) ablation studies, including Eq. (3) in Fig. 6, and (3) results of combining various base feature losses that verifies adversarial losses might deteriorate IRL performance.  \n2. Compared to Figs. 4 and 5, single-demonstration task performance in Fig 3 is only sometimes good, suggesting that SF is more focused on representing states. For some scenarios, action representation performance might be necessary.   \n3. To demonstrate scalability, including more complex (or demonstrative) benchmarks in Fig. 3 would be beneficial to IRL researchers to grasp the model's performance. \n4. There are no supporting tables in the appendix that measure the performance of the learning curves in Figs. 3, 5, 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)