IOSTOM: Offline Imitation Learning from Observations via State Transition Occupancy Matching
Authors: Quang Anh Pham
Janaka Chathuranga Brahmanage
Tien Anh Mai
Akshat Kumar
Venue: NeurIPS 2025 poster ()
Keywords: Learning from Observations
Imitation Learning
Primary area: reinforcement_learning

Abstract: Offline Learning from Observations (LfO) focuses on enabling agents to imitate expert behavior using datasets that contain only expert state trajectories and separate transition data with suboptimal actions. This setting is both practical and critical in real-world scenarios where direct environment interaction or access to expert action labels is costly, risky, or infeasible. Most existing LfO methods attempt to solve this problem through state  or state-action occupancy matching. They typically rely on pretraining a discriminator to differentiate between expert and non-expert states, which could introduce errors and instability—especially when the discriminator is poorly trained. While recent discriminator-free methods have emerged, they generally require substantially more data, limiting their practicality in low-data regimes. In this paper, we propose IOSTOM ($\textit{Imitation from Observation via State Transition Occupancy Matching}$), a novel offline LfO algorithm designed to overcome these limitations. Our approach formulates a learning objective based on the joint state visitation distribution. A key distinction of IOSTOM is that it first excludes actions entirely from the training objective. Instead, we learn an $\textit{implicit policy}$ that models transition probabilities between states, resulting in a more compact and stable optimization problem. To recover the expert policy, we introduce an efficient action inference mechanism that $\textit{avoids training an inverse dynamics model}$. Extensive empirical evaluations across diverse offline LfO benchmarks show that IOSTOM substantially outperforms state-of-the-art methods, demonstrating both improved performance and data efficiency.

IOSTOM: Offline Imitation Learning from Observations Via State Transition Occupancy Matching
Quang Anh Pham, Janaka Chathuranga Brahmanage, Tien Mai, Akshat Kumar Singapore Management University
{qa.pham. pw.pppa....m. 2025, janakat 2022) smu. edu.edu.... fatmai, akshatkumar) @smu, edu. sg
Abstract
Offline Learning from Observation (LfO) focuses on enabling agents to imitate expert behavior using datasets that contain only expert state trajectories and separate transition data with suboptimal actions. This setting is both practical and critical in real-world scenarios where direct environment interaction or access to expert action labels is costly, risky, or infeasible. Most existing LIO methods attempt to solve this problem through state or state-action occupancy matching. They typically rely pretraining discriminator to differentiate between expert and non-expert states, which could introduce errors and instability -eppciialy when the discriminator is poorly trained. While recent discriminator-free methods have emerged, they generally require substantially more data, limiting their practicality in low-data regimes. In this paper. we propose IOSTOM (Imitation from Observation via State Transition Occupancy Matching), a novel offline LfO algorithm designed to overcome these limitations. Our approach formulates learning objective based on the joint state visitation distribution.
key distinction of IOSTOM is that it first excludes actions entirely from the training objective. Instead, learn an implicit policy that models transition probabilities between states, resulting in more compact and stable optimization problem. To recover the expert policy, we introduce an efficient action inference mechanism that avoids training an inverse dynamics model. Extensive empirical evaluations across diverse offline LfO benchmarks show that IOSTOM substantially outperforms state-of-the-att methods, demonstrating both improved performance and data efficiency.
Introduction
Imitation learning is a framework in machine learning where agents learn to perform tasks by mimicking expert demonstrations rather than learning through trial-and-error or explicit reward This approach is particularly useful in environments where designing reward functions is difficult or costly. Its practical relevance spans wide range domains, including robotics, healthcare, and autonomous driving, where expert behavior is available but reinforeement learning either too risky, cime-consuming, or expensive to deploy [35] 46 24]. By leveraging expert demonstrations, imitation learning enables faster deployment of intelligent systems and facilitates safer exploration in complex, real-world environments.
signals 33 37 15
A variant of this framework, known as Imitation from Observations or Learning from Observations (LfO), focuses on learning policies using only state trajectories without access the expert's 'ss actions. This setting presents unique challenges, such as inferring intent and disambiguating optimal behavior from partial information, but also broadens applicability to scenarios where action data is unavailable or hard to record. For example, in video-based learning from human demonstrations in household
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tasks (e.g., cleaning or cooking), it is often infeasible to capture the precise motor commands or control actions, making observation- only learning a practical and valuable approach.
Recent developments in imitation learning from observations have increasingly focused on scenarios where limited set of expert state-only trajectories is complemented by sub-optimal state-action demonstrations. While this setup has practical appeal, many existing methods rely on distribution- matching frameworks that operate over complex imput tuples such as (s, a, s') or (8, s', s"),, where These formulations appear to be sample-inefficient due to the structural complexity of the inputs. Furthermore, some approaches require estimating a
8, s' represents a state and a an action [8 38
discriminator to support training (27 48). which can be unreliable in low-data or biigh-iimensionll settings [39]. Other methods rely on learning an inverse dynamics model to recover unobserved expert actions. which introduces approximation errors that may degrade the quality of the learned policy 43 50). To the best of our knowledge, no existing method in the I,0 setting addresses all of
these limitations simultaneously.
We aim address the aforementioned limitations in this work. Our central idea to ignore sub- optimal actions and instead focus on learning the transition probabilities between consecutive states, leading simple and compact learning objective that only involves joint state pairs (s,s'). We then develop an efficient method to recover the expert policy without requiring an inverse dynamics model. Specifically, our contributions are as follows:
(i) By first disregarding sub-optimal actions in the demonstration data, we propose learn state-to-state transition probabilities, which can be interpreted as an implicit policy that encapsulates the actual state-action policy. We then formulate the learning problem as matching joint state visitation distributions and leverage convexity and Lagrangian duality to derive a tractable joint-state Q-learning procedure. This training formulation, in addition to being discriminator-free is significantly simpler and more compact than prior approaches that rely on action annotations, as it only involves consecutive state pairs (5,8").
(ii) We further introduce two novel strategies for efficiently extracting policy from the learned Q-function. First, we propose a Q-weighted behavior cloning (BC) approach, which is theoretically equivalent the standard Evvattag--wiihtee BC but offers more compact and stable formulation. Second, we propose single-stage process for recovering the expert policy without estimating an inverse dynamics model, thereby avoiding approximation errors that could degrade policy quality.
(iii) We validate our LfO framework using state-of-the-art benchmarks, demonstrating that our algorithm, IOSTOM, significantly outperforms existing methods. The implementation of IOSTOM is publicly available at /httts::/ggthhii com/ cctm//ganggah1199/000000
2
Related Work
Learning from Observations Different from Learning from Demonstrations (LfD) 37 35 using expert state-action dataset, Learning from Observations (LfO) 45] addresses the challenge of imitation learning when expert actions are unavailable, relying instead state-only expert trajectories. LfO research can be broadly distinguished into online and offline paradigms. In online LfO setting, the agent can actively interact with the environment [44 49). Recent advancement in online LfO focuses on improving adversarial imitation learning (AIL) approaches 14, [] The core idea of AlL relies on generative adversarial networks (GANs) [ where generator policy learns imitate an expert, while a discriminator differentiates between agent-generated and expert data. In addition to online LfO, its offline setting has also received significant interest due to practical constraints of many real-world scenarios, where continuous interaction is costly or risky. It assumes access to state-only expert demonstrations and an action-labeled background dataset from other interactions [50]. A common approach trains an inverse dynamics model (IDM) on background data to infer expert actions, then applies Behavior Cloning (BC) 43 However, BC needs extensive, high-quality expert data and can suffer from compounding eπrors, exacerbated by IDM inaccuracies [34] Another line adapts the Distribution Correction Estimation (DICE) framework (28). These methods (e.g.. PW-DICE [48), [8] ) use a discriminator to estimate density ratios as pseudorewards for downstream RL. While avoiding an explicit IDM, their success depends on discriminator quality and RL robustness. Recently, DILO (38) bypass both IDM and discriminator learning by solving the dual
SMODICE [27]. LobsDICE

---

of an occupancy matching objective, directly optimizing a utility function. This function, measuring long-teni long-term divergence from expert visitation, is used to extract the imitation policy.
Imitation Learning via Distribution Matching: Distribution Matching objective is powerful tool in Reinforcement Learning (RL) that has demonstrated its effectiveness in exploration [25], goal-conditioned RL [26, M. and especially Imitation Learning (IL). Many popular IL methods such as BC, GAIL [I3. and DAgger [36] can be formulated as statistical divergence minimization
problems [TO]. This minimization can be performed over the state, state-action, trajectory space, resulting in different II. approaches [30] The well-known DICE-family algorithms [21]]]] [8] 27]
also optimizes state state-action visitation distribution matching problems between the learner and expert via their dual formulations [29]. They often require learning discriminator to estimate the log-ratio for distribution correction. Recently, [3 introduce ReCOIL, a discriminator-ffee method that also optimizes the duality of the state-action occupancy matching problem. This work is closely related our IOSTOM, both learn a score function that assigns high values expert samples and low values to non-expert samples. However, IOSTOM focuses on solving the state-transition occupancy matching problem instead of the standard state-action one address the LfO problem. Our setting is generally considered more challenging than the L.fD setting targeted by ReCOII. [7 mainly due to the absence of expert actions in LfO.
3 Background
Markov Decision Process. We consider a Markov Decision Process (MDP) defined by the tuple M = (S, A, T, R. po: 7). where S denotes the set of states, A set of actions, Po represents the distribution of initial states, R : S x A R defines the reward function for each state-action pair, and T S x A- + S' is the transition function, i.e., ((s'|s, a) is the probability of reaching state E S when action a E AA is taken at state 8 E S. The parameter y E [0, 1) is the discount factor. In reinforcement learning (RL), the goal is to find a policy that maximizes the expected long term accumulated reward, i.c., max, (E(s.a)=a=- [R(s,a)] ] where d' (s, a) is the occupancy measure (or state-action visitation distribution) of policy क. The definitions of d' (s, a) and other common visitation distributions are include in Table
State-Action Distribution
Transition Distribution d' (8, 8
State Distribution
Joint Distribution
Notation
d" (s)
d' (s.a) a)
d' 88,
Support
S x m) d' (s(s)(aa)s) o' (s.4))((5(s. Σn ds,
AxAXS
SXS
Definition
(1 Di P(s
Table 1: Overview on different stationary distributions adapted from 50
Offline Imitation Learning from Observations. Different from standard Imitation Learning, Learning from observations (LfO) relaxes the requirement of action in expert dataset. In offline LfO setting, we assume access to an expert observation-only dataset DE {8, 8'} and a suboptimal interaction dataset D, {s, a, s'). We also denote the respective visitation distributions of the expert and suboptimal datasets DE and D, as d and d'. Several methods have been proposed to handle this challenging scenario. For instance, SMODICE (27. state-of-the-art approach for learning from observations (LfO), minimizes an upper bound of KL-divergence Dxx (s) ds()s via the objective:
d' (s) min Ear()) log (s)
+ DD, (I) (s,
where Dy denotes an f-divergence between two distributions. LobsDICE [1] proposes a similar formulation:
min DxL (d<(ss d (s,s') + a DKL (s. (
DILO 3 introduces a discriminator-free approach via solving another objective:
min
+ (1- d' (s, 3 d (s, s',d)) + (1-3) B) d' (s)))
where d' (s,.s',aa)) ds, ('sss)))((a))))
3

---

Most methods (except DILO) rely on a learned discriminator to predict distribution correction ratios based on s or (s, s'), which can be unreliable in low-data or nigh-dimensional settings [39 Although DILO is discriminator-fres it requires costly triplet samples (8, s"") and non-standard visitation structure, which limits its sample efficiency. contrast, our proposed method IOSTOM is discriminator-free and only involves joint state pairs (s, s') during learning. This leads to more compact representation and improved sample efficiency over prior approaches. Furthermore, IOSTOM is the only approach directly minimizing D Dy|d" (s, ss) dd (s, which is the main objective of LfO
IOSTOM Imitation Learning via State Transition Occupancy Matching
We present a novel framework for imitation learning from observations, which is structured into two sequential stages. The first stage focuses on recovering the state-transition probabilities, denoted as g(s | 8), which represents the probability of transitioning state s. the second stage, we recover a policy based on the learned state transition model g(8' (). The key insight behind our method is to simplify the LfO problem by initially ignoring the action information in the dataset. By doing so, we treat the state transition model g(s' s) as a form of "implicit policy" that governs the behavior of the demonstrator. This abstraction allows us to bypass the need for explicit action labels during the early phase of learning.
the next state s' given the current
4.1 Joint State Q-learning Formulation
Our approach centers on recovering the transition probability between states, denoted as g(s' s), which can be viewed as an implicit policy. This transition model can be computed based on the underlying policy and environment dynamics as: g(s' s) = -Ea (a s)T(s' s. To facilitate training. we define the joint visitation distribution over state pairs (s, s') as:
d' (3, s') = d" (s)g(((s)) - (s s,
where (s, a) is the statenaction visitation distribution under policy æ, recursively computed via the single-step transpose Bellman equation 22
dd (s, a) (1 ( +
Σ d" s', a')T(a s),
with do(s) denoting the initial state distribution. To support learning from observations, our goal to remove the dependence on actions in the visitation distribution. We introduce the following proposition to this end:
Proposition The joint visitation distribution d3 (s, 8') can be expressed as:
d (s) = -1 (1-))dd(s)g(s' s) + 7g(s') Σ d' (3,))
We note that the flow equality Equation ① depends solely on the joint state visitation distribution
d (s, s') and the state transition function g(s' |s) shethe two key quantities we aim to recover.
Using the flow constraints described above and following the approach in [39] our main objective is minimize the divergence between two joint state visitation distributions: d'ix s") and d (s, s'). defined as follows:
d (8, 5') ad(s, s') + (1 add (s, and d (s, = ad (8, + a)d 8"),
where a E (o, 1) is a mixing hyperparameter. Here, (s, s') represents a mixed visitation distribution combining the learned Ktatettransitin behavior with that of the suboptimal dataset, while ai (s, 8') represents a mixed distribution combining expert behavior and suboptimal behavior. Combining this with the flow constraints in (T). we formulate our learning problem as follows:
(Dy (s, s') [ (s,s')
max g.d20
s.t. d(s, = (1 ( (g(s'(s) Σ d(3,
(2)

---

The constrained problem described above is convex in d when the transition function is fixed. Following a similar approach to that [39] the maximization over d can be equivalently reformulated as an unconstrained optimization problem via Lagrangian duality. We formalize this result in the following proposition, with the full derivation provided in the appendix.
The constrained optimization problem in Equation unconstrained max-min problem:
Proposition 2.
2 is equivalent to the following
{a(1 77) (B wo [Q + (e [s" (x1Bg" 1Q((', (Q(6))0000))))(0)0))))))))))))) Q(,, s))]
max min Q(s,s') - a1 B(sss)aal (g [Q [Q( Q(s,
-
(3)
where ((s, are the Lagrange multipliers and f denotes the convex conjugate of a chosen convex function f.
For notational convenience, we define Q(s, = ~g( [Q(s, s')]. Using this shorthand, we write the objective function (3) as (See Appendix B.3 for complete derivation):
max min 9 Q(a,e")
(1Q.9) = a(1 7) Be [Q)s + ass If (7Q(5' .) ((s,
(1 (1) )adr [T (>Q(()) -(,=)))
where (²) - f. This formulation learns a joint-state value function Q(s, where actions are entirely ignored. While being more compact and manageable than prior formulations that rely on (s, a, s') or even (s, s', tuples, our approach benefits from ignoring suboptimal actions in the data, This design helps mitigate the imbalance in offline datasets, where expert demonstrations lack action labels, whereas suboptimal trajectories contain fully observed actions.
4.2 Extreme V-learning
Solving the maximin objective in can be done via dual optimization by alternating between optimizing (s,, 8') and g. Specifically, we minimize L(Q.g) g) over Q, and then maximize it over g with Q fixed. Following [47) the maximization over g can be approximated by computing max, y (s,.g) g) for each state 8, which requires sampling from g. In offline RI., this is challenging due to ofststtributtonn (OOD) issues when querying Q on unseen state transitions
[23 To address
this, we adopt the in-sample soft estimation from (9]. replacing the hard maximization with a KL-
regularized soft value: (s) (e)= (Q(s, 8') 3 log g a(>(s))] where f is the behavior policy
and 3 controls the KL strength, keeping close to μ to avoid OOD samples. This supported by the following proposition:
Proposition 3. m (s) can be approximated via the following Extreme-V objective:
min { J(V Q)= E (a.0"))at E. (exp(w(s, s')) + (') -11
(5)
where wss s') = (Q(s, 8 V(s))BB.
Using this estimate, the Q-learning objective becomes:
min L(Q.V) = a(1 - 7) 18 (oo [Q(s, + aE (a If" (7V(5') ss,,s1)
+ (1 a (
F (7V(s') - Q(s,
(6)
We optimize Q and V jointly: Q by minimizing L(Q, V), and V by minimizing J(V Q). Crucially, L(Q, V) is concave in Q, and J(V Q) is convex in V, forming a bi-concave/convex structure that ensures stable and convergent optimization.
Proposition 4. Under any convex function f, I(Q.V) is convex in Q, and the Extreme- loss J(V Q) is convex in V.
5

---

4.3 Policy Extraction
Typically, once the and V functions have been learned through Q-learning, a policy can be recovered using advantage-weighted behavior cloning (AW-BC) [3.. In our context, where we operate with an implicit policy g(s s). the policy can be recovered by solving the following optimization:
max
(et (o,00) (exp (F(Q(s, - (()))) log [8],
(7)
Here, T > is parameter controlling the sharpness of advantage weighting. We use D'' instead of mix in e as the policy ı is extracted solely from D'. This is sufficient under
the mixed dataset D E. the common coverage assumption that d' (s,s') > 0 whenever (s. > 0, as adopted in prior
works 19 18 27
limitation of objective 7 is that the value function (s)) in our setting is only an approximation obtained via the extreme-Y V surrogate, which can introduce noise and bias in the computation of advantages. To address this, we propose an alternative approach based solely on the Q-function. The following proposition shows that this alternative objective can, in theory, recover the same optimal BC formulation.
implicit policy as the original advantage-weighhe
Proposition 5. The following Q-weighted behavior cloning objective returns the same optimal implicit policy as the original advantage-weighter BC formulaionn:
max
E (add)dd jexp (TQ(s, s')) logg(s' |s]]-
(8)
Given the learned implicit transition model g(s' s), existing approaches often recover the expert policy π (als) by training an inverse dynamics model (IDM), denoted as I(a s, s's)). This is typically done by optimizing the following objective: maxt (s llgI[(a s, 8'] and then defining the recovered policy (a s) E. I(a 5,)')g((' s). While intuitive, this two-step approach has several limitations. First, decoupling the learning of g(s' 8) and the recovery of aa 8) via separate inverse model introduces additional sources of bias. Second, training the inverse dynamics model I(a 8, typically requires a significant amount of high-quality data. When the offline dataset d' contains a large proportion of low-quality suboptimal data, the inverse model may be inaccurate-resule
in compounding approximation errors, as also noted in learning-from- observation (LfO) literature [18]
To address limitations of IDM- recovery, we propose a single-stage policy extraction method that avoids training inverse dynamics model. Our approach leverages the identity g(s' s) Σ. T(x' 8, a) m (a s). Using this, we rewrite the Q-weighted BC objective
(8) as a direct
optimization over
max = exp(TQ(ss. (s, s')) log ET(s' ( (aa))(als))
The objective (T),, however, involves log-sum over actions, making difficult to optimize directly. We develop a tractable lower bound on this objective, which resembles a weighted behavior cloning loss over log T s).
Proposition The objective F(T) is lower-bounded by the following surrogate function F(T),, up to an additive constant: F(T) = ((e,,")> [exp (TQ(s, E. I(a s') log π (a) |s]]
While F(T) is lower bound of the original objective F(T). maximizing this surrogate function still promotes the maximization of (TT) in practice. The primary advantage the surrogate objective F(T) is that it contains the term Σ. I(a (a) s), which can be empirically approximated using offline samples, thus avoiding the need to learn the inverse dynamics. In particular, we can empirically approximate F(T)) as:
F(x) ( [exp (rQ(s, s')) log TTa s)]
where the expectation taken over offline trajectories (s, a, 8'). We note that a similar weighted behavior cloning formulation was used in (38] although without providing theoretical justification. Empirically, their results demonstrate that this single-stage approach can outperform the traditional two-step method involving inverse dynamics modeling.

---

Practical Algorithm
The common choices of f-divergence function in the literature be KL. Pearson x². In IOSTOM,
we choose the x² divergence function with its convex conjugate function ("(x) = I + æ. Our
objective (6) becomes (complete derivation can be found in the Appendix B.8
min I(Q,V) =(1 B) (e Q(s, 8') + yI and [V(+)] ad 68) Q
1
+ 4a s E. miz
(s') Q(s, $)?]
The minQ -a -a."","")= [Q(s, s')] term in the above objective which effectively encourages maxi- mizing the Q-values of expert transitions can lead to unbounded growth in Q. potentially resulting in learning instability. To address this issue, we adopt a technique from [2] that constrains the expert Q-values, and propose the following practical Q-learning objective (with derivation in Appendix B.9h:
()Q = (1-7) Eddo [Q(s, 1-a
Ed
(GV())) +))
4a
E [(o(s.e))
2
27
1-
Finally, to estimate the term ) [Q(s, s')], we sample (s, pairs uniformly from the offline dataset rather than from a policy rollout. This empirical estimation, adopted in prior works 88 38),
helps reduce overfitting and improves the robustness the learned policy by leveraging diverse range of initial transitions. We present main steps of our IOSTOM algorithm in Algorithm [
Algorithm 1 IOSTOM
6 Experiments
1: Input: Expert datasct D suboptimal dataset D
2: Initialize Q, V functions and policy networks Qs, V., ma
In this section, we compare IOSTOM with
3: Set target network parameters ø' <-
previous state-of-the-ane approaches on di- verse sets of environments and tasks from the D4RL benchmark (7]. and real world data. Particularly, we aim to answer the following main questions: (Q1) Can IOS- TOM outperform other baselines on stan- dard LfO benchmarks? (Section 6. (Q2) Is our algorithm still robust with limited
4: for i 1, .N do
Sample mini-batches from DE and D
6: # Update using (V, Q) in Equation 5
7:
w + w - JVa )
Update Q using T.QQ, V) Equation
9:
& + aLQQr V.)
10:
# Update policy via weighted BC
11:
u(s,&') ))= exp (rQrrss
expert data? (Section 6.2) (Q3) How well
12:
+- 0 4 # Update target network
[w](s, 8") log (o [)]]
IOSTOM perform when learning from ex- perts of different dynamics? (Section 6.3)
13:
14:
o' + >x + A)Ø'
(Q4) What is the performance of IOSTOM
15: end for
on real-world instances (Section 66.4 We
16: Output: Imitation policy me
also provide implementation details and additional experiments in the Appendix C
Baselines and experimental setup main baselines: SMODICE [27], PW-DICE [48], and DILO 38]. Both SMODICE and PW-DICE require learning discriminator. The main difference between them is that SMODICE aims to minize the KL-divergence distance state visitation distributions between learner and expert while PW-DICE uses Wasserstein distance [16] instead. DILO is the recent SOTA discriminator free method for LfO. We train all algorithms for million gradient steps with random seeds and monitor the normalized score 100
We choose three SOTA LfO methods in the literature as
expert score random score during training. The average normalized score of last 10 evaluations used assess the performance of different methods.
method score random
6.1 Offline IL from Observations
To answer the question (Q1), we use the same offline I.0O benchmark from DILO (39 with datasets constructed from the D4RL framework [7]. Specifically, we evaluate methods on 8 Mujoco envi-
7

---

LtD approaches
approoches
Export
Suhoptimal Dataset
Env
BC (expert data)
BC (Blll dataset)
ReCOIL
SMODICE
PW-DICE
DILO
1OSTOM
rundom+
halfcheetahh
hopper
4.52a1.4
2.25
5.64
60.20....1
108. 18
106 56 s0
85.55 +1.00
108. 09
86,11
91.53
35
20:07
109.32
93,02
111.33
88.83
expent
walker2d
0.6000
102.16
107.92 #1.20
107. 08 45.5
108.31 20.
40 21
ant
5.17 es
30.661. .as
126.24g445
20.79
126.89 21
22.17
130.75
mindom+
hallchhetahh
hopper
20.93
4.34
2.24
3.0.0.54
76. 16.42
58,30 10.00
3.19
75.04 114.2:
4.02
104.27
43.65
14.74
107. 28
88.77
13.00
111.33
88.
few-expert
walker2d
0.
0.74
83 13.23
3.93
36.11
108.35
108,40
106.92
ant
35.3
g..0
6.59 16.00
99,90
22.50
110.79
120.09
15.12
130.75
mediuma expert
halfeheetah
hopper
16.09
59.25
42.45
21.15
55.74 =2
24.18
58.74
25.06
1188
17
21.03
20
20.01
88.83
walker2d
2,43
72.76
108.5
108.54
0.91 20.71
105
20.35
$6.59
108.12
s0 13
106.92
ant
0 0.6627.22
95. 95.4743037
120.36.7:.7
104.00
108.
14 #1.00
98
124.72
130.75
medium hopper
1:
45.8755
$3.50
57.24
96. 95
100.96
111.39
halfcheetah
-1.15
42.21
75.96 45.46...54
42.58
27.85
26.05
59.40 60.80
89.47 10.02
88.83
lew-esperi walker2d
2,02
70-422246
25
27.06
74.35
106.92
ant 10045
81.63
110.38
88.20
90.34
90.77
+ 50
120.36 11 m5
L30.75
pen 13.95
34.4421100
95.04.4.44
15.71
a13.36
23.39
44.56
26.48
1:333
82.77 14
106.42
clonol4exper
doot 20.2220.0
8.011
102.75
1.57
0.07
91.29
102.77
109.94
hammer
24
5.4577.4
1.07
1.29
29:32
91.80
4:2
94.59
125.71
pen 13.83
90.76
103.72 3..00
58.62 a7.00
-2.56 $1.30
31.95 47:43
95.77
106.12
humantespent
dour
-0,03
103.71.1.2
104.70
29.84 212.17
20.00
0.11 20.40
103.94
hammer
0.18
122.6gg14
125.19
33.28
g16.43
2.02
20:77
6.93 go
41
125.71
partilleepert
kiichen
2.5
45.5g1a7
60.0as.70
36.67
45.77
12.33
55.36
23.00
$20.87
58.95 22.27
75.0
mixeddespen
kiichen
2.2a5s
42.3
$2.0a1a
48.33 16.20
7,50
24.6
29.1 17 assan
46.45 ao
75.0
Table 2: Average normalized return over last 10 evaluations of IOSTOM against baselines on the D4RL suboptimal datasets with expert trajectory. The mean and std are obtained over random seeds. LIO methods with avg. perf within the std-dev of the top performing LfO approach is in bold.
ronments: 4 locomotion (Hopper, HalfCheetah, Walker2d, Ant) and 4 manipulation (Pen. Door, Hammer, Kitchen) [42). Each task's expert dataset contains one trajectory. Suboptimal datasets for locomotion mix D4RL 'random' or 'medium' data with 200 ('expert' or 30 "feeeexpert") expert trajectories. For manipulation, D4RL non-expert datasets ('mixed' and 'partial' for Kitchen; 'human' and 'cloned' for others) are mixed with up 30 expert trajectories. This results in 24 diverse tasks for comparing IOSTOM against baselines, with manipulation tasks being more challenging due to larger state spaces. More details on environment and dataset are included in the Appendix.
Table 2 presents results for IOSTOM and baselines. We also include the results of some Learning from Demonstration (LfD) methods such as Behavior Cloning (BC) and ReCOIL (39) to serve as the reference upper bound of LfO methods because they have access to expert actions during learning. We choose these two approaches as ReCOIL is the SOTA offline LID method while BC is the most popular IL algorithm. Their results are taken directly from ReCOIL's paper which uses a similar setting. As shown in Table 2 IOSTOM leads on 23/24 tasks, only marginally underperforming DILO on 'walker2d random+expert' while still matching expert performance. Discriminator based methods (SMODICE, PW-DICE) degrade significantly with few expert examples manipulation tasks due to discriminator overfitting. While DILO's discriminator-fi free nature mitigates this, it still struggles in 'few-expert' settings (e.g., 'halfcheetah") ") and 'human+expert" tasks where training can diverge (see Appendix for further discussion). BC methods with access to expert actions also exhibit poor performance on most tasks. Notably, IOSTOM's performance is comparable to, and sometimes surpasses, ReCOIL on locomotion tasks, showcasing its effectiveness and potential to bridge the gap between LID and LfO.
high-dimensional
6.2 LfO with subsampled expert
This section focuses on benchmarking the sample efficiency of our approach (Question (Q3)). We
adapt the subsampled expert trajectory setting from LID literature [ (00) to construct a subsampled state-only expert dataset. Specifically, expert trajectories are sub-sampled by keeping transition every 20 time steps (i.e. subsampling rate is 20) starting with random offset. This process will
create incomplete expert trajectories which makes both BC and DICE method like ValueDICE (21) fail as shown in [51]. This setting may not be valid in case of DILO because requires the triplet (s, s'.s") which is equivalent to two transitions inside action-labeled expert dataset; we still adapt the 2-transition version of the subsampling procedure only for DILO. LobsDICE also considers the similar setting for Lf0 like us on locomotion tasks, but they construct DE using 50 sub-sampled 50
expert trajectories, which means using
20 = 2.5 times of total transitions of an expert trajectory.
This makes this setting still easy to deal with for both our approach and basclines. Therefore,
8

---

Suboptinal Datoset
SMODICE
JOSTOM
(full)
(sub)
(Tall)
(sub)
(full)
(sub)
(fall)
(sub)
randoms
halfcheetal
lopper
106.56 20.s
85.55
108.
78.63
20.45
108.09
86.11
37.95
97.35
86.3 35
91.53
13.25 $12.05
92.05
109,32
93.02
109.94
93.23
expert
walker2d
21:20
+0.51
av.as
00.559 =1.11
+0.1s 125.
450.00
107.98
20.20
108.01 =0.10
ani
126
c0
124.13 63.76
126,89 e1.17
112.99
+2.37
30.27 22.47
128.19 $1.02
126.23 =2.87
nandom+
hallcheetah
hooper
58.30
3.19
58,44
3.06
75:04
4.02
48.30
3.91
11:17
104,27
13.65
92.52
44.22
101.28
88.77
105.00
86.09
13.42
few-expert
walker22d
20.76
20.32
10.19
26.29 $10.57
108.35 26.15
=5.00
108.40 20.21
104.32 =4.02
ant
6.59 16.55
6.33 13:35
99,90 19.59
82.81
6.00
110.79 11:30
31.91 10.05
120.09 15.17
123,83
1.20
medium-
halfcheetalh
hopper
53.80
55.74
54.24
50,06
65.99
58.74 21
62.78
03.05
108 22
88.54 1:377
42.61 g0
54.42
93.12
110.20
109.72
92.97
expert
walker2d
2..71
11.43
10.33
118.45
$6.59 112.33
83.41 1es.57
108.12 10.13
108.59
16-17
ant
104.00 g3.02
99.52
81:08
108.14 11:90
110.68
$4:20
98 11.44
108.75
124.72 13.49
124.06
23.00
medium
hallcheetah
hopper
53.50
88 =0.05
54.26
+0.74
57.24
20.03
50.51
11:99
=3.01
59.40
96.95
+5.50
53.53
55. $5.50
#7.52
108 96
89.47 =0.82
107.31
87.45
23.57
few-expert
wallker2d
9.08
23.07
3,05
22.22
27.05
52.95
111.00
20.50
52.90
108.15 20.03
108.45
25.30
ant
88. 20
88.80
15.18
90.34
89.69
90.77
90.00 g3.00
120.36
117.29
Table 3: Comparison of normalized returns obtained by different offlie LfO methods expert dataset with I expert trajectory denoted as (full) or subsampled expert trajectories (subsampling rate is 20) denoted as (sub). The mean and std are obtained over 5 random seeds. Methods on subsampled expert dataset with avg. perf within the dev of the top performing method is in bold. Methods with greater than 5% performance decrease subsampled expert datasets are highlighted in blue.
construct D from subsampled trajectories only (i.e. 0.25x total transitions of an expert trajectory) and evaluate all LfO methods on locomotion tasks with the same suboptimal dataset in Section 6.1
Thows the comparison results on the subsampled setting. IOSTOM continues outperform all baselines on these challenging tasks. Furthermore, its performance does not change much compared to using complete expert trajectory even when the total number of expert samples reduced by
times. SMODICE is also robust on 12/16 tasks but its performance on 'few-expert" setting is still poor. Both DILO and PW-DICE face a large drop (>5 %) on the performance of most tasks in the scenario of less samples and incomplete trajectories.
6.3
LfO with mismatched expert
10STOM
SMODICE
DILO
PV-DICE
HallCheetah Shart
Ant: Disabled
100
140
120
Normalz.e Retuum
100
80
0.0M
0.2M
0.AM
0.6M
0.8M
IM
0.0M
0.2M
0.4M
0.6M
0.BM
IM
Gradient Steps
Gradient Steps
Figure 1: Comparison results for LfO with mismatched experts
To evaluate IOSTOM performance when learning from experts of different dynamics, we adopt SMODICE's s mismatched dynamics setting. We test on 'HalfCheetah- Short' (halved torso) and 'Ant- Disabled' (partially amputated front leg) (See Appendix H of SMODIE [27] for illustration), using one expert trajectory from these modified agents. The suboptimal dataset remains the 'random+expert data (Section 6.T) from the original agents. This setting creates clear mismatch between expert and interaction datasets. Figure | shows IOSTOM outperforning baselines on these challenging tasks, while DILO performs worst. The poor performance of DILO can be due the use of visitation distribution d(s, s', a') in its objective which matches the wrong a' in the mismatched expert dataset.
6.4 Lf0 for marine navigation
We next test IOSTOM in real-world domain, the maritime navigation problem. Our goal is to learn II. policies that can behave like human experts (ship pilots) for navigating vessels (mainly large tankers and cargos). These polices offer significant benefits for operational safety and efficiency.

---

ADE [m]
Near Miss Count
CPD:
82
25
20
10-
Expert
BC 1MO0ICEOIO IOSTON
Expert BC SNcorCe DND RISTOM
Goal Reach Rate
Acceleration (x10 ms
3.5
20-
3.6
10-
0.5
2.4
1
0.2
Expert
BC SWODKED FOSTOM
Expert
BC SMODKGE DIL KISTOM
Figure 2: Left: Visualization an episode from our maritime simulator, Blue vessels follows their historical trajectory; red vessel is controlled by IL policy (destination * marker), green dots denote historical trajectory of ego vessel. Right: Comparison results on various performance metrics. Details on metrics are in the Appendix.
For instance, they can be integrated into Vessel Traffic Information Systems to provide port watch operators with accurate short-term predictions of vessel movements, especially in congested ports such as Singapore strait. Learned IL policies also enable to do what-if analysis such as how would safety be affected when there is traffic surge (e.g., by simulating additional vessel arrivals and assigning them IL. policies). Prior RL-based approaches to maritime traffic management 40 41 rely on online learning, requiring costly simulator interactions and accurate simulation of vessel dynamics. In contrast, our IL method directly learns vessel behavior from large offline datasets, offering easier and more accurate modeling as shown in our results (see later metrics such as ADE, goal rate).
We collect large amount of historical navigation data (~ 2 years) of vessels operating in a hotspot region in Singpaore strait (among top 5 busiest ports) recorded in the Automatic Identification System (AIS). We use these data with ShipNaviSim. a data-driven maritime traffic simulator [32 to construct realistic environment featuring an IOSTOM-controlled ego agent (red) and log-play agents (blue) that follow their actual trajectories, as illustrated in Figure 2 The ego agent controlled by IL. policy tries to reach the goal ('x') while avoiding collisions with other log-play agents. The ego agent can also observe past states (blue and red dots) of its and close surrounding agents (its observation space). Because AIS data does contain any action information, we use an inverse kinematics model (IVM) to construct the action space and generate action for AIS data. The state-only expert dataset in this setting is easy obtain due the action-free nature of AIS data. We generate the suboptimal dataset by adding random noise action-labeled expert trajectories. Further details about enviromment and dataset generation can be found in the Appendix.
We evaluate our approach in maritime navigation setting against BC, SMODICE, and DILO. Results shown in Figure 2 using metrics relevant to this domain, introduced in [2], which reflect how well the learned agent imitates expert behavior. ADE (Average Displacement Error; lower is better) measures how far, on average, the agent's 's trajectory deviates from the expert's. ss Goal reach rate (higher is better) indicates how often the ship reaches the goal. Near-miss count captures the number of close-quarter situations, defined as scenarios where two ships come close to each other posing a collision risk; lower values indicate reduced collision risk, and average acceleration should closely match that of the expert. Mean and standard deviations Our approach outperforms across all three baselines in ADE, mear-miss count, and goal reach rate, while maintaining an accelcration profile similar to the expert. DILO is the second-best performer. SMODICE struggles due to nigh-dimensional observation space-which includes nearby ships and trajectory history; leading to poorly trained state discriminator and worse performance than BC.
over 5 seeds for each method.
7 Conclusion
We presented IOSTOM, a discriminator- Q-learning framework for offline imitation learning from observations. By learning an implicit policy in the form of state-to-state transitions and matching joint state visitation distributions, our method avoids reliance on action labels for value function learning (Q-learning) and eliminates the need for inverse dynamics models in policy extraction. Extensive experimental results and ablation studies demonstrate that IOSTOM achieves strong empirical performance and improves sample efficiency compared to prior approaches.
10

---

Acknowledgments and Disclosure of Funding
This research/project is supported by the National Research Foundation, Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2020-017),
References
[1] Siddhant Agarwal, Ishan Durugkar, Peter Stone, and Amy Zhang. f-policy gradients: A general framework for goal-conditioned using f-divergences. Advances in Neural Information Processing Systems, 36:12100-12123, 2023.
[2] Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. LS-IQ: Implicit reward regularization for inverse reinforcement learning. In The Eleventh International Confer- ence on Learning Representations, 2023.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint arXiv::607.064550 2016.
[4] Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances Neural Information Processing Systems, 25244699-24654, 2022.
[5] Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional distributions via dual embeddings. In Artificial Intelligence and Statistics, pages 1458-1467.
PMLR, 2017.
[6] Anish Abhijit Diwan, Julen Urain, Jens Kober, and Jan Peters. Noise-conditioned mergy-based annealed rewards (NEAR): A generative framework for imitation learning from observation. In The Thirteenth International Conference on Learning Representations, 2025.
[7] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072119 2020.
[8] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028 4039, 2021.
[9] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations 2023.
[10] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence mini- mization perspective on imitation learning methods. In Conference on robot learning, pages 1259-1277. PMLR, 2020.
[11] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[12] Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, al. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems, 36:7730-7742, 2023.
[13] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[14] Bo-Ruei Huang, Chun-Kai Yang, Chun-Mao Lai, Dai-Jie Wu, and Shao-Hua Sun. Diffusion imitation from observation. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024.
11

---

[15] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning survey of learning methods. ACM Computing Surveys (CSUR). 50(2):1-35, 2017.
[16] Leonid V Kantorovich. Mathematical methods of organizing and planning production. Man- agement science, 6(4):366-422, 1960.
[17] Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation learning from observation alone. Advances in Neural Information Processing Systems, 34:28598-2861 2021.
[18] Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Kee-Eung Kim. Lobs- dice: Offline learning from observation via stationary distribution correction estimation. Ad- vances in Neural Information Processing Systems, 35:8252 8264, 2022.
[19] Geon- Hoyong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference Leurning Representations, 2021.
[20] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. pnseriiinttor-acto---rii Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
[21] Ilya Kostrikov, Ofrr Nachum, and Jonathan Tompson. Imitation learning via off- off-policy distribu- tion matching. In International Conference on Learning Representations, 2020.
[22] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179- 1191, 2020.
[24] Luc Le Mero, Dewei Yi, Mehrdad Dianati, and Alexandros Mouzakitis. survey on imitation learning techniques for end-to-end autonomous vehicles. IEEE Transactions Intelligent Transportation Systems, (3(9):14128-1411 2022.
[25] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint (0KV::/906.05274, 2019.
[26] Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Oshert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in Neural Information Processing Systems, 35:310-323 2022.
[27] Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference Machine Learning, pages 14639-14663, PMLR, 2022.
[28] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Dehavior-agnoste estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
[29] Ofir Nachum and Bo Dai. Reinforcement learning via fenchel- duality, arXiv preprint arXiv:2001.01 0666, 2020.
[30] Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference Robot Learning, pages 529-551. PMLR, 2021.
[31] Xue Bin Peng, Aviral Kumar, Grace Zhang and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
12

---

[32] Quang Anh Pham, Janaka Chathuranga Brahmanage, and Akshat Kumar. Shipnavisim: Data- driven simulation for real-world maritime navigation. In International Conference on Au- tonomous Agents and Multiagent Systems, AAMAS '25, page 1641-1649, Richland, SC, 2025. International Foundation for Autonomous Agents and Multiagent Systems.
[33] Dean Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
[34] Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits imitation learning. Advances in Neural Information Processing Systems, 33:2914-2922 2020.
[35] Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3(1):297-330, 2020.
[36] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[37] Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
[38] Harshit Sikchi, Caleb Chuck, Amy Zhang, and Scott Niekum. A dual approach imitation learning from observations with offline datasets. 8th Annual Conference on Robot Learning, 2024.
[39] Harshit Sikehi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual RL: Unification and new methods for reinforcement and imitation learning. In The Twelfth International Conference on Learning Representations, 2024.
[40] Arambam James Singh, Akshat Kumar, and Hoong Chuin Lau. Hierarchical multiagent reinforcement learning for maritime traffic management. In International Conference on Autonomous Agents and Multiagent Systems, pages 1278-1286, 2020.
[41] Arambam James Singh, Due Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Multiagent decision making for maritime traffic management. In AAAI Conference on Artificial Intelligence, pages 6171-6178, 2019.
[42] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 5033-503..200 IEEE, 2012.
[43] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceed- ings of the Twenty-Sevene International Joint Conference on Artificial Intelligence, VICAI-18, pages 4950-4957. International Joint Conferences on Artificial Intelligence Organization, 2018.
[44] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint
2018.
[45] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. In Proceedings of the Twenty-Eighth International Joint Conference on Artifi- cial Intelligence, IJCAI-19, pages 6325-6331. International Joint Conferences on Artificial Intelligence Organization, 2019.
[46] Lu Wang, Wenchao Yu, Xiaofeng He, Wei Cheng, Martin Renqiang Ren, Wei Wang, Bo Zong, Haifeng Chen, and Hongyuan Zha. Adversarial cooperative imitation learning for dynamic treatment regimes. In Proceedings The Web Conference 2020, pages 1785-1795, 2020.
[47] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RI. with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023.
13

---

[48] Kai Yan, Alex Schwing, and Yu-Xiong Wang. Offline imitation from observation via primal wasserstem state occupancy matching. In Forty-first International Conference on Machine Learning, 2024.
[49] Chao Yang, Xiaojian Ma. Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagree- ment. Advances in neural information processing systems, 32, 2019.
[50] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. Advances in neural information processing systems, 33: 13:22-11224,,, 2020.
[51] Li Ziniu, Xu Tian, Yu Yang, and Luo Zhi- Zhi-Quan. Rethinking valuedice does it really improve performance? In ICLR Blog Track, 2022. http:///wobbooo........oo/o///// valuedice/.
14

---

APPENDIX
Contents
Limitations and Future Work
16
Missing Proofs and Derivations
16
B.1
Proof of Proposition
16
B.2
Proof of Proposition
16
B.3
Complete derivation of transtorming objective function (3)
18
Proof of Proposition
18
B.5
Proof of Proposition
19
B.6
Proof of Proposition
20
B.7
Proof of Proposition
20
B.8
Complete Derivation of L(Q, V) using Pearson X² divergence
................................
B.9
Complete Derivation of L(Q, V) for bounded Q-lcarning
21
Experimental and Implementation Details
22
Mujoco tasks
22
Maritime Navigation task
23
Architecture and Hyperparameters
24
Baseslines
25
D Additional Experiments
26
D.I
Comparison with LobsDICE
26
D.2
Sensitivity to Subsampling of Low- cow-Quality Data
27
D.3
Comparison with other variants of IOSTOM
27
D.4
Ablation
28
D.5
Ablation
29
15

---

A Limitations and Future Work
Despite its strong empirical performance, IOSTOM has some limitations. First, like most LfO methods, it assumes access to high-quality consecutive state pairs (s, 8'), which may not always be available in real-world datasets. Second, we assume that actions are fully observable in the sub-optimal dataset, which might not hold in practice. While these limitations are beyond the scope of this work, they highlight important directions for future research.
B
Missing Proofs and Derivations
B.1 Proof of Proposition
Proposition.
The joint visitation distribution d (s, s') can be expressed as:
(s = (1-=)))(11))6())df (8) rg('s) () (3)
(10)
Proof. We recall that d= (s, (1=))do(S)T(a + > d ds a')a)((aa ss)
Therefore, we have:
d (s E (ss (s,a)
Σ (1 7) s)T(s' s. + ETT(s' s,a) EF (3)))
3,))a(a
dd(())g( ())+ ss Σ d' (3(3,a)T(( 3,
(1 (s(())( (s) + ng(s') (s) {s) (s.
as desired.
B.2 Proof of Proposition?? 2
Proposition. The constrained optimization problem in Equation is equivalent to the following unconstrained max-min problem:
min {a(l 71) Q(s,s') -(1 aa1-a)E
(77)) (E,,"')) (()s + (e,d) [f* (118) [Q(s', Q(s, s')]
1 (a.') (yEonrgt(or) [Q sn)] - ;
where (ss,s)) are the Lagrange multipliers and f denotes the convex conjugate of chosen convex function f.
Proof. We recall that the primal formulation in Equation (2) is as follows:
max -Dy (s, E... (
g.d20
s.t. d(s, = (1 7)du(s))(ss())) (g(s'(s) Σ d(3
16

---

We first apply duality on the inner maximization problem of the above formulation:
max min (d200((ss))
-Dy (8, ( (s
Ed(o,o) (1-7)do(s) rg(6'(s)) Σ d(3, -(3,)))((s,55)
(II)
5,8'
max min a(1 - dddte))o00)) (Q(s,))))))(( [Q({, a, Q(s,s))
a rd
Σ -"0"" Q((,,))) - Dy Dy (d 8 Mid (8,8)) 8))
(12)
Step 11 to 12 is equivalent to changing the following order of summation:
Σ (ss (')(s'ls)))
Σ d(3,s)
Σ d(3,s) s)
Q(s s'g((ss)))
Σ d(s,s')
Es ss)
By adding and subtracting another term below,
12 becomes:
max min a(1 (d [0ss, (((ss)) g.d20 ((a,"")
aBB..'' and Σ g(s")ss) 8n) Q(5,5')
+ (1 e
? Σ s's(ss Q(s.s')
(1 Beg
Σ
- Q(s,
Dy (dimee (8, (8,6') Iid
(13)
We can swap max and minQ in (13 due to strong duality.
13
max min max a(1 (11001))) [Q(s,s']] (((ss)))))
9 ((a,0)) d'ner (2.0))20
+ E ddt
D Dy(dmrnn (8, (s, silla (s)
s'))
- (1 Bs.' (((o"))) Q(5,
(14)
= max min a+1 - 1)aas ((+,'))
(
f 7 Σ g(s" -
+ E
- (1 - 0c) Es wal
y " s²) -(s))(())')))
(15)
17

---

where f is convex conjugate of convex f- f-divergence function. following equation using the interchangeability principle [5]:
14 to 15 can be proved by the
max
B Be
Σ 9(s" [9s Q(s,
D, (d minl (s, d s'))
d' (e,0))20
(n [(( -Q(s,)) - d (
d' (8,6))2 ad E.J
max
a (8,
ft 7 g(5" s')((s)) Q(s,5'))
=B d BE.I mii
Finally, the objective
(15) the unconstrained dual problem of Equation
B.3 Complete derivation of transforming objective function to
a(1 - (a.)) [Q(s, + (s ( [5 (y [Q(s', Q(s,
(1 aa) E (5, )oud (>(gg"~tt-<>) [Q 8 (ss
(16)
a(1 (s do
[(s + (o,e"" R. [f (nQ(s' .9) Q(sss')))
(1 - a) E (s, hQ((s' (s)
(17)
ax1 7I (o.0)) [Q(s,'') + alE (8.0') (
(nQ(5', Q(s,
(1 a [ (s.a)) [f" (nQ(s' .9) Q(s, 5 (1- hQ((s {(s,s')
(18)
a(1 7) (s.x')~do [Qss,s'] + alE /d [ ()Q(s'.g) s(s,
a (
[[" (1Q(s' 9) Q 8 (7Q('', Q(s,,'))]
(19)
a(1 7) (s,8)) do
Q(ss, 8'), alE (s,)) [/" (nQ(s',g) Q(s,6)))
a.) E )a.')dd
f (7Q(s' Q(s,
(20)
16 17 by defining (8, Eg (Q(s s')]. (8) due d d (s,
add 5)
addd 5')
19 to 20 by defining (+(() (t) t.
B.4 Proof of Proposition
Proposition. maxg (V) can be approximated via the following Extreme-V objective: min V { (J(V) Q) ( (exp(u(ss 8/) + 88 - 113
where w1s 8') = (Q( V(s))BB.
Proof. Recall that:
(g(()= = Ex) Q(s, - Bog g(s') s)
uls' 3)
which is the expected reward under transition distribution g(- s), regularized by the KL divergence from reference distribution u(- s). Moreover, the problem max, (g(s) } is classic entropy- regularized expected reward maximization problem. The optimal solution has a closed form:
(Q(5.8))
max
Vi(s)} Blg Σ H(s' cxp
exp
(21)
3
We now write the function J(V Q) as:
J(V Q)) Σ ulss 8) exp [(ss - V(s)
Q(s, s') V(s)
-1]
(0.0*)
For any state s, and fixed Q, the function J(V) Q) is convex in ((s) because:
18

---

The exponential function exp ( is convex in Vss),
The linear term (Q(s, 8') B also convex (affine),
The sum and non-negative weights preserve convexity.
To find the minimum of J(V Q) with respect to V, we take the derivative with respect to V(s) and set zero:
aJ(V Q)
E uus' s) [ exp
Q((,,')) V(s)
1
=
(s)
B
3
Rewriting:
Σ (ss'1s) (((,,5))-10 - V(s)
Σ u(s's ((s'1ss. (s)
8
We have Eg u(s' 8) 1, this gives:
(((,,5)) V(s) 1.
Enls' exp exp
B
Bringing the constant outside the exponential:
exp (-1 V(s)
u(s' ex)
(-1
1,
(- V(()
= exp
Ev 11s' (8) exp Q(5,s')
Taking the logarithm of both sides and solving for (ss). we obtain the closed-form solution to miny Q(VQQ) as:
(Q(s, 8
((s,s') s')
(s) = log Em(s' sx
(22)
Combined
21 and 22
we get:
(*(s)= max (s}
max
g
as desired.
B.5 Proof of Proposition
Proposition. Under any convex fanction J, I)Q,V) is concave in Q, and the Extreme-V V loss J(V Q) is convex in V.
Proof. We rewrite the objective L(Q, V) as:
min L(Q, a(1 7) EE (s, [Q(s,'') + aE (a,))DD if (yV(s') Q(s,s'))
+ (1 - a) ( (s.s')>D (nV((s') Q(s,
We now analyze the convexity of T(Q,V) with respect to Q. Note the following:
The first term, ED, (Q(s, s')], linear in Q, and hence convex.
The functions f and f are convex (as they convex conjugates of proper convex functions).
The composition convex function with an affine function (i.e., 8') convex in Q.
19

---

Expectations of convex functions preserve convexity.
Therefore, each term in L(Q, V) is convex in Q, and the entire objective L(Q, V) is convex Q, as desired.
The convexity of J(V Q) in V follows directly from the discussion in the proof of Proposition
B.6 Proof of Proposition
Proposition. The following Q-weighted behavior cloning objective returns the same optimal implicit policy as the original ddvantage-weiihhee BC formulation:
max
(a, (exp
[s]]
g
Proof. We write the objective function as:
F(g) = Σ (s) (TQ(s, logg(s' |ss,
(a:81)
where se' (s') 88 is the state-to-state transition probability (i.e., the implicit behavior policy) for the dataset D For each fixed state s, the expression
En'(s' 8) exp (TQ(s, log (s' 8)
a weighted log- liglliiklhoodd, where the weights y² () cxp(TQ(ss, s')) are known. Maximizing this with respect to g(- s) under the constraint that g(- 8) is valid probability distribution (i.e.,
Eg g(s' {) = 1) leads to standard result from maximum likelihood estimation with importance weights. The closed-form solution is:
1e (s' exp exp (rQ(s,s'))
9" (s's' (6 E.i u (s" 8) exp (TQ(s, (s))"))
We now consider the advintage-weighted behavior cloning objective:
max (sss))) jexp (T(Q(s,, V(s))) logg(s' [s]]
In similar fashion to soft behavior cloning, this yields the following closed-form optimal "implicit policy":
0'" ('(s' (s') s) exp (T(Q(s, - VV()))
En (y () exp (T(Q(s, y) V(s)))
where Vss) appears in both the numerator and denominator and thus cancels out. This simplifies the expression and leads
(8 8) g=" indicating the equivalence between the advantage- weigtted behavior cloning and the Q-weighted behavior cloning formulations.
B.7 Proof of Proposition
Proposition. The objective F(TT) is lower-hounded by the the following surrogate functtooo F(TT), up to an additive constant: F(T)) = (s.'' (exp (TQ(s, En I(a s, s') log π TTa] s)].
Proof. We write the objective function as:
ΣT(s' s,a))(al 1s) s
F(T) = B(
20

---

Given that the logarithm function is concave, we apply Jensen's 's inequality. Define:
(88,0) = Σ T(s's 18,a),
Then we have:
T(s' s, x(a m(als) log A(s,
Σ T('' (s,))(((ls)) log
log
(23)
A(s,8)
T(s' s, log 8) + log Ass,s')
(24)
( s')
I(a s, s') log ((a s) log (s)
(25)
Substituting this back into the original objective yields the lower bound:
F(T) ( exp Txp (TQ(s,))))
Σ I(a ( s') log T(a s) + ( lexp (TQ(s, 8')) log A s')]
The second term is independent of and can be treated constant during training. Therefore, can optimize the surrogate lower bound:
F(T) = (O exp (TQ(s,))) Σ I(a s, log T(a (s)
B.8 Complete Derivation of L(Q, V) using Pearson x² divergence
We recall that the objective function L(Q, V) has the following form:
min L(Q,V) a(1 - 7) 16 (o [Q(s, + al (e.0 If -(V(s))(((sss)))) ((s,
Q
+ (1 (e.'"a
F (7V(s') Q(s,
(26)
where f* is the convex conjugate divergence function f and - r. Under Pearson
x2 divergence, its convex conjugate f*(x) = + and the associated f*(x)= - The objective
26 with Pearson x2 divergence becomes:
min L(Q,) = a(1 - 7) )(o 8Q) + a (B (VV(5') ((s,
E ( (3V((') Q(s,(s 8()
E Eis [67V(5)) Q(s,
(27)
min I(Q,V) (1 -7) ( [Q(s, (e (TV(s') Q(s,
1
a.E | (7/V(')) Q(s 1))) + - (1 a) ((V((')) - Q(s, 51)3]
+
(28)
4a
4a
@ min L(Q, V) = (1 -1- (1 ((o.') [Q(s, + Eende [vV( - E( (Q(s,8'))
1
E
s E. [ - Q(. s(s
(29)
4a
B.9 Complete Derivation of ī(Q, V) for bounded Q-learning
The operator ming- - or >D [Q(s, s')] in 29 which effectively encourages maximizing the Q-values of expert transitions can lead unbounded growth in Q. potentially resulting learning instability. To address this issue, adapt technique from 22 that bounds the expert Q-values. First,
21

---

looking at the objective 28 Let's define ss( =Q(s,') -V((s) Vs, ~ d.. The training
objective becomes:
min L(Q, =1 (1-
11--) 7) ) [Q(ss (sss')ddd" (-rg(s,
oF (2.')) [r](),
- Q(+, - (1 - ax ( (
+
4a
4a
min L(Q, V) (1 (lo [Q(s 1
E ((V()) Q((,s))
4a
E Rd (-r@(1, Be, (r
1 1- E
+ min L(Q,V) = (1--)) [Q(s,s']] +
( Q(s,)))))
4a
1
Bedd'aad (-rr (8,88 80)
Re
s s')
-
+ min L(Q,V)= = min (1 - Q(a,a')
(1-7))B 7) B (Q
[((
+
1 1-
E, E ((>V(s') Q(s,
4a
+ En Ea 2)
(30)
[Q(s,s'] min L(Q, (1 (1dd) (1])s))((000)((((ss))
1 1- E - Ea (( -
Q
4ex
+ Eg [(Q 8') - (>>(((())))) + 2])
(31)
Following 2 the minimum of the third term in 30 is reached when rQ (s, s') 2. This will lead to Q(), Ei= 222 - Vs, 8' ~ d Therefore,
replace the target VV('') 2 in (31)
with fixed target 2 to have the following modified objective with bounded Q. I-y
min i(Q. V) = (1 (ddo [Q(s,s']
a E. s'd [(]V(5)) Q(s,
+ 4a
+ 4 Brat (((6)-))11
1- -
C Experimental and Implementation Details
Our method is implemented in JAX version 0.5.3 (with CUDA 12 capabilities). We conduct experiments using a computing cluster with NVIDIA RTX 3090 GPUs. For each IOSTOM run, five distinct training seeds are processed simultaneously on shared hardware set comprising single GPU, 32 CPU cores, and 128 GB of RAM. This parallel execution on shared resources enables the completion of 1 million training steps for all five seeds in about 60-90 minutes.
C.1 Mujoco tasks
We use the same offline LfO benchmark from DILO 138 which utilizes datasets derived from the D4RL (7) framework, and tests on Mujoco environments. The state-only expert dataset in all tasks includes only one expert trajectory. In terms of locomotion tasks, suboptimal datasets, labeled 'random+expert', 'random+few-expeer", 'medium+expent" and tnedium-few-exper'', are generated by mixing expert trajectories with lower-quality trajectories from D4RL's 'random-v2 22' and 'medium- v2' datasets, respectively. The 'random+expert' and 'mediumeeppert datasets combine 200 expert trajectories with roughly 1 million transitions from the corresponding 'random-v2" or 'medium-v2" dataset. The "xteew-expert" variants are similar but incorporate only 30 expert trajectories. In manipulation environments, all suboptimal 'x+expert' datasets are formed using 30 expert trajectories mixed with the complete 'x' D4RI. dataset. We also use "-v0' variant of D4RI. datasets for all manipulation tasks. Table 4 gives overview about our LfO Mujoco tasks.
22

---

Task
State Dim
Action Dim
Horizion
Suboptimal
Data Points
Dataset
Hopper
11
1000
randon+expert
le6 random transitions 200 expert trajectories
meedium+expert
le6 medium transitions + 200 expert trujectories
random+few
expeee
lef random transitions +30 expert trajectories
medium+few expper
medium transitions + expert trajectories
Walker2d
17
1000
random+expert
le6 random transitions 200 expert trajectories
mediumm+xxper
le6 medium transitions + 200 expert trujectories
random+few expeet
lef random transitions + 30 expert trajectories
medium+few-expen
le6 medium transitions + expert trajectories
Halfehoetah
17
1000
random+expert
le6 random transitions 200 expert trajectories
medium texpert
leb medium transitions + 200 expert trujectories
random+few-coe cow-exeeet
le6 random transitions + 30 expert trajectorics
medium-few-expe few-expent
medium transitions + expert trijectories
Ant
27
1000
random+expert
random transitions
expert trajectories
medium+expeet
1c6 medium transitions + 200 expert trajectories
random+few-exe
le6 random transitions 30 expert trajectories
medium=lew-expen
medium transitions + expert Imaecctriis
Pen
45
24
cloned+expert
Se6 cloned transitions 30 expert trajectories
human+expert
5000 human transitions 30 expert trajectories
Door
39
28
200
cloned+expert
le6 cloned transitions 30 expert trajectories
humantexpert
6729 human transitions + 30 expert trijectories
Hammer
46
26
200
cloned+expert
166 cloned transitions + 30 expert trajectories
humantexpert
11310 human transitions + expert Injectories
Kitchen
59
partial-expert
136950 partial transitions + expert trajectories
mixed+expert
136950 mixed transitions expert trajectories
Table 4: Overview of D4RL tasks and their repsective suboptimal dataset we use in LfO setting
C.2 Maritime Navigation task
The Maritime Navigation task was created using historical data from a hotspot in the Singapore Strait, following the AIS-driven simulation paradigm adopted in recent maritime traffic simulator ShipNaviSim (3). We selected the area with the highest traffic density and collision where numerous ships cross paths, as shown in Figure - as our planning region. We collect large amount of historical navigation data (~ 2 years) of vessels operating in this hotspot region recorded in the Automatic Identification System (AIS) from MarineTraffiu
The AIS data of each vessel contains two types of information: static and dynamic. The static data contains some information like width, length, type, and ID of vessel. Other vessel movement information like latitude, longitude, speed, heading are included in dynamic data. To generate trajectory data, we selected tankers and cargo vessels as they represent the riskiest class due to their larger size (200-300 meters) and lower navigational agility. All trajectories were then interpolated at 10-second intervals. The final dataset comprises approximately 125,000 trajectories, totaling around 14 million environment transitions. The average trajectory length in dataset is around 100-150, We used 80% of the data for training and reserved the remaining 20% for evaluation.
and course-over-grouu
The observation space is defined from the perspective of the ego agent (the vessel being controlled). At any given time, the agent observes a historical sequence of its own trajectory and those of the 10 closest nearby ships (each over a configurable number of past steps). For the ego agent and nearby ships, and for each historical point, the available features include the x and y coordinates, speed v, and heading angle/ h. Additionally, the agent observes its goal location. Observing past states and nearby ship information helps capture multi-ship interactions and provides context for decision-maie making. For simplicity, all algorithms used the same neural network architecture to process the observation space. We did not use any complex structures; instead, we flattened the observation space and provided it as input to the neural network.
The action space is modeled as a straightforward, 3-dimensional continuous space. An action is defined as (dz, dy, da),, representing the changes in the and y coordinates and the change in heading h, respectively. The vessel's speed at the next time step is derived from the distance traveled (calculated from u V d3 + d2//67 divided by the time interval 87, which is set to 10 seconds.
https://ww.marrinn www.mmmmaaaic..ccoom
23

---

MU
Planning Region
Anchorege
Landmass
Traffic Seporation SchemerTSB)
Figure 3: The red region is used as the environment area. The gray areas indicate anchorage zones, the green areas represent landmasses, and the arrows and regions with dark blue borders represent the Maritime Traffic Separation Scheme (TSS). The high density of crossing points in the red area makes it more challenging region for navigation, providing suitable setting for testing advanced planning techniques.
This also known as a delta action space [2] and can be used for any moving object. Because action in our environment represents the difference between some state features of current and next timestep, we can have a simple Inverse Kinematies Model computing this difference infer action between two consecutive states of state-only trajectories in datasets.
Following vessel-specific metries introduced in ShipNaviSim [32 used evaluate navigation policies, comparing learned agent behavior to human expert data.
Goal-Conditioned ADE (GC-ADE) measures the average displacement between the learned policy's 's trajectory and the original historical trajectory in the 2D plane. Given Tm of length T, and Tp of length Tp, GC-ADE computes the error over the minimum of the two lengths.
min TTn.TTTTp Tp)
GC-ADE = min(Tm: Tp)
Σ E=1
(xT - x2(2 2 + (vi" y²²²²
Goal Rate is the percentage of times the ego agent successfully reaches its designated goal location. Success is defined as coming within radius of 200 meters of the goal.
Near Miss Count represents the average number of time-steps per episode during which the ego agent approaches another vessel within a distance of cable lengths (555 meters), which is considered near-miss by domain experts. The 'near-miss' metric is interpreted broadly proxy for high traffic density and increased potential for navigation risk; it does not always imply that vessels in "near-miss' situation about collide.
C.3 Architecture and Hyperparameters
Our implementation builds upon the official implementations of ReCOIL 39) and XQL 9 We keep most of their parameters and network settings as shown Table 5 We also add Layer Normalization in V-function network to improve training stability as suggested in XQL. The regularization was tuned by searching over [3, 5, 10, 15, 20J. For locomotion tasks, we set 3 = 20 for standard LIO setting, and 15 for subsampled setting. In terms of manipulation tasks,
10 works best in most cases except 'pen-cloned" setting where 3 is set to 3. The policy temperature is often set to in previous works [22 39). However, find that this value results in very bad performance for IOSTOM because we do not use advantage for updating policy. We tune 7 via via hyper- parameter sweeps over [0.01, 0.04, 0.08, 0. 0.2]. T = 0.04 is the best-performing hyperparameter in most tasks except for the 'human" Adroit and 'mixed' Franka Kitchen manipulation tasks, where 0.01 was used. For maritime navigation task, we set / = 20 and 0.04 which is similar LfO setting of locomotion tasks.

---

Type
Hyperparameter
Value
Actor
Network Size
2256, 256]
Activation Function
ReLU
Learning Rate
3e-4
Weight Decay
le-3
Training Length
1M steps
Batch Size
512
Optimizer
Adam
Dropout Rate
0.1
LR decay schedule
cosine
Critic
Network Size
[256. 256]
Activation Funetion
ReLU
Learning Rate
3e-4
Training Length
1M steps
Batch Size
512
Optimizer
Adam
Mixture Ratio a
0.5
Polyak Update Rate À
0.005
Discount Factor 7
0.99
Table 5: Hyperparameters of IOSTOM
C.4 Baseslines
To evaluate the performance of approach, conduct comparative evaluations against three established oate--ofttheaate (SOTA) techniques: SMODICE (27). PW-DICE [48), and DILO (38). The SMODICE and PW-DICE algorithms both operate by training a discriminator learned policy. Their fundamental difference lies in the divergence measure employed: SMODICE seeks to minimize the KL-divergence between the state occupancies of the learner and the expert, while PW-DICE alternatively uses the Wasserstein distance for this aligmment. DILO offers distinct, more recent SOTA paradigm for LfO, notable for its discriminator-free learning process. For all comparative methods, we utilize the publicly accessible codebases provided by their authors. To ensure fair comparisons, we use hyperparameter settings recommended their original publications the default configurations within their code. The only exception is DILO where we can not reproduce consistent results as reported the paper using their default parameters. After some tuning effort, find that using Layer Normalization (3] can help to improve DILO performance. However, the training still diverges in some tasks as shown in Figures and
guide the
DILO (reproduce)
DILO (paper)
HalfCheetah-v2 (random-fewweeperr" task)
HalfCheetah-v2 ("mediam+few-expert task)
70
60
70
Normliized Return
50
Normalized Return
60
50
40
10
0.0M
0.2M
0.4M
0.6M
0.8M
1.0M
0.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Gradient Steps
Gradient Steps
Figure 4: Training divergence DILO on locomotion tasks
25

---

DILO (reproducs)
DILO (peper)
Door-vö (humettoxpert task)
Pen-vö exumantexpertt task)
Hammer-v0 (hhmmn+expprt" task)
80
Normalzed Return
Normatzed Ritur
100
80
Nermalzed
80
80
60
80
40
20
40
20
0.2M
0:4M
0.6M
0.8M
1,0M
0.0M
0.2M
0.4M
0.5M
0.8M
0.0M
0.2M
0.4M
0.8M
0.8M
1.0M
Gradient Steps
Gradient Steps
Gradient Steps
Figure 5: Training divergence of DILO on manipulation tasks
D Additional Experiments
D.1
Comparison with LobsDICE
Suboptimal Dataset
Env
PW-DICE
LobsDICE
IOSTOM
random+expert
hopper
108.09 12.39
99.64
15.01
109.32 11.08
halfcheetah
86.11
14.39
80.76
13.17
93.02 10.40
walker2d
107.48
10.53
107.54
10.13
107.98
10.20
ant
126.89
+1.17
122.65
=0.72
128.19
#1.52
random+few-expert
hopper
75.04 14.22
74.25
111.23
107.28
133.2
halfcheetah
4.02 +1.74
11.61
45.66
88.77
1.22
walker2d
36.11 19.19
101.29
25.53 93.84
108.40
40.21
ant
99.90 42.59
17.51
120.09
15.17
medium+expert
hopper
65.99 18.05
74.43
45.28
110.20
+0051
halfcheetah
58.74 11.84
71.09
1:3.76
93.12
+0.32
walker2d ant
105.41 +0.33
103.34
=2.11 114.96
108.12
10
108.14
+1.90
-4.05
124.72
43.49
medium+few-expert
hopper
57.24 +3.03
67.00 +6.43
108.96
+1.33
halfcheetah
27.85 #6.03
44.76
+3.94
89.47
=0.82
walker2d
75.22 17.05
95.39
+5.22
108.15
4:0.3
ant
90.34
#2.56
95.33
+1.08
120.36
+1.25
cloned+expert
pen
23.39 14..6
29.83 44.59
82.77
14.8
door
0.07
#0.14
0.02 +0.00
102.77
+0.96
hammer
1.29 +0.12
0.55 10.19
94.59
29.39
human+expert
pen door
-2.56 +1.30
42.09 +5.08
95.77
+8.91
0.15 #0.02
10.98 19.71
100.77
#1.08
hammer
2.02 10.77
17.06
113.44
93.34
17.41
pextial+expert
kitchen
12.33 +5.38
40.33
+2.24
58.95
#2.27
mixed+txppert
kitchen
7.50 14.16
45.67 +2.75
46.45
+0.84
Table 6: Average normalized return over last 10 evaluations of IOSTOM against LobsDICE and PW-DICE on the D4RL suboptimal datasets with I expert trajectory. The mean and std are obtained over 5 random seeds. LfO methods with avg. perf within the std-dev of the top performing LfO approach is in bold.
To further strengthen our empirical study, we additionally include a comparison against LobsDICE 18. alongside PW-DICE and our method (IOSTOM), under the same experimental settings. Al- though previous work [48] has suggested that PW-DICE generally outperforms LobsDICE on the
26

---

D4RL "random-expeer" benchmark, we perform direct comparison here for completeness and to ensure fair evaluation across representative DICE-based baselines.
The corresponding results are reported in Table 6 We observe that IOSTOM consistently and substantially outperforms LobsDICE across all tasks and dataset regimes. is common among DICE- based methods, LobsDICE exhibits degraded performance in the "few-expert" and manipulation tasks, whereas IOSTOM remains robust. Our findings also confirm that PW-DICE surpasses LobsDICE in the "random+expert" setting, but PW-DICE becomes unstable and underperforms in several other configurations, illustrating the general sensitivity of discriminator- based approaches. In contrast, IOSTOM achieves both stronger performance and greater robustness, reinforcing the benefits of its stable, discriminator-fre free formulation.
D.2 Sensitivity to Subsampling of Low-Quality Data
Suboptimal Dataset
Env
IOSTOM(full)
IOSTOM(sub5)
IOSTOM(sub20)
rendortteppert
hopper
109.32 11.08
110.24 10.63
110.66 10.16
halfcheetah
93.02
10.40
90.81
+2.10
85.59 =5.09
walker2d
107.98
#0.20
107.91
10.16
107.53 #0.11
ant
128.19
11.52
127.26 22.48
127.30 1.33
random-few-eepert
hopper
107.28 #3.92
108.99 #0.76
109.34 #1.62
halfcheetah
88.77 #1.26
86.62 +1.98
80.49 =6.41
walker2d
108.40
#0.21
108. =0.29
107.78
=0.22
ant
120.09
#5.17
119.63 =2.40
105.98
#7.03
medium+expert
hopper
110.20 10.51
109.89 10.22
109.58
11.18
halfcheetah
93.12 10.32
91.07
11.72
76.30
113.56
walker2d
108.12
10.13
108.31
10.21
108.03
10.15
ant
124.72
13.49
127.71
12.72
128.61
10.71
mediun=ffw-expert hopper
108.96 =1.33
109.68 =0.56
108.20
=0.83
halfcheetah
89.47 10.82
90.20
11.94
77.65 111.48
walker2d
108. 15 20.43
107.88
10.83
107.39
20.92
ant
120.36 11.25
118.23
=5.04
102.14
#7.26
Table 7: Average normalized return over the last 10 evaluations of IOSTOM under different sub- sampling rates of low-quality data (sub5 and sub20) on the D4RL suboptimal datasets with expert trajectory. Performance drops exceeding 5% relative to IOSTOM(ful)) are shown in bold.
To assess the sensitivity of our method to the amount of available low-quality data, we conducted additional experiments where the random (or medium-quallity) portion of the dataset was subsampled to only 20% and 5% of its original size. These two variants are denoted as IOSTOM(sub5) and IOSTÓM(sub20), while the original version is referred to as IOSTOM(full) The corresponding results are summarized in Table Performance drops exceeding 5% relative to IOSTOM(full) are highlighted in bold.
When reducing the low-quality data to 20% (IOSTOM(sub5)), our method exhibits strong robustness: across all tasks and datasets, performance remains very close (OSTOM(full), with significant degradation observed. However, when the low-quality portion is aggressively reduced to just 5% (IOSTOM(sub20), we observe a more noticeable performance decline up 18% particularly in halfcheetah and ant. Nonetheless, IOSTOM still achieves robust performance on 10 of 16 tasks, even under this extremely limited data regime.
tasks,
D.3 Comparison with other variants of IOSTOM
To validate our algorithmic designs, we compare IOSTOM with other variants: IOSTOM-IDM (Using Inverse Dynamies Model), IOSTOM-Adis Ady (Using advantage instead Q to update policy), and IOSTOM-IQL (Using Implicit Q Learning 22 objective train V-function network). Table shows these comparison results. Overall, 1OSTOM has the best performance on 17/24 tasks and consistently produces high-quality results compared to other variants. IOSTOM-IQL is the second
27

---

Saboptimal Dataset
Eay
LOSTOM-ADV
JOSTOM- IDM
1OSTOM-IQL
IOSTOM
Expent
random+
halfcheetah
hopper
55.93 #6.07
6.43 =2.51
97.06 #4.57
80.53
109.74 #0020
92.82
10.71
109.32
93.02
1.08
111.33
88.83
expert
walker2d
104.20
=4.07
79.70 22
108.43 #0.14
107. 98 =0.20
106.92
ant
120.13
23.65
128.99
4:311
128.71
23.73
128.19
=1.02
130.75
random+
hopper
20.47 1:0.06
50,63
4:02 03
106.59
1::05
107. .8
23300 233992
111.33
hallcheetah
2.13 =0.11
68.79 26
86.44
+1.55
88.77
1:126
88.83
few-expert
walker2d
8.38 22.88
69.11
222.28
108.37
1:0005
108.40
20021
106.92
ant
41.18
12.00
123,86
1:977
125.15
4:450
120.09
=5.17
130.75
hopper
53.46 113.16
97.62
110.71
110.20
111.33
medium++ expert
halfcheetah
49,00 4:3.44
85.21
=1 86
91.45
+:149
93.12 +0.32
88.83
walker2d
85,96 +221.8
94,90 +22.34
108.45 1:017
108.
=0.13
106.92
ant
118.74
24.55
128.32
124.66
124.72
130.75
medium hopper
42.79 #:2.6
68.32 #17.80
106.02
108.96 =1.83
111.33
halfcheetah
42.31 2:0.00
76.44
78.18
22224
89.47 20082
88.83
fow-expert walker2d ant
74.16 12.14
107.89 1:0.2
108.38 1:018
108. 15 20:43
106.92
99.89
#1.92
121.80
+1778
121.64
4:2.6
120.36
#1.25
130.75
pen
41.65 1:542
63.39
1:11.46
10.54
11.51
82.77
1:484
106.42
cloped+expeet
door
13.87
#8.26
18.68
#:1244
32.25
#13.08
102.77
=0.06
103,94
hammer
11.77
+10.66
47,83
57.04
#6.34
94,59
43,30
125.71
pen
92.73 #3.73
81.72
13
95.26
=:0.16
95.77
4:8.1
106.42
human-expert
door
95.08
#1.00
78.50
=20.05
99.47
#:3.07
100.77
=1.68
103,94
hammer
88,23
25.77
82.12
438.01
68.32
1:3.06
93.34
J:7.41
125.71
exptit
kitchen
56.08
3.0.
66.30
57.75
42200
$8.95
4:3.07
75.0
mixed+expert
kitchen
49.42
$0.08
28.95
+30.02
47.92
#1.23
46.45
75.0
Table 8: Average normalized return over last 10 evaluations of TOSTOM against other variants the D4RL suboptimal datasets with expert trajectory. The and std are obtained over random seeds. LIO methods with avg. perf within the std-dev of the top performing LfO approach in bold.
best method, but its performance still significantly worse than IOSTOM on 'cloned' tasks. The results in 'ew-expert' setting of IOSTOM-IDM is very bad compared to 'expert' setting which clearly shows the weakness of training Inverse Dynamics Model with low-quality data. IOSTOM-ADY has the worst performance in most tasks.
D.4 Ablation
Suboptimal Dataset random=expert
Env
0.1
0.3
0.5 (our)
0.7
0.9
goPaara((9)
hopper
18.81
110.21 10.55
109.32 $1.08
30.31 92.99
110.29 20:0
1.00
halfehestah
93,09 50.22
92.90
20.31 20.31
93.02
20.40
+0.08
92.89
=0.18
walker2d
107.82
107.97
107.98
10.20
108. 108.16.0 16.17
108.38
0.52
0.37
128.37
126.67
128.19
e3.02
125.78
127.30
2.02
0.14
random--le-ccpeert
happer
107.41 13:90
108.59 10.04
107.2 28 33.00
104.75 30.30
107.25 g::1
3.54
1.21
halfeheerah
87.29 20.rs
87.52
-2.11
88.77
88.42 +1.00
86.61
2.43
0.00
walker2d
108
20.04
108.2 21
108. 40 10.21
108.45 20.12
108 24
0.33
0.05
125.43
#2.77
121.19
+2.00
e5.17
$1.00
123.46
#1.17
mediam-expert
hopper
109.79 10.0
110.44 10.51
110.20 10.51
110.30 10.54
110.61 1007
0.74
0.37
halfcheerah
93.16
92,82
93.12
93.00
92.68
0.52
0.04
walker2d
107.54
10.45
19.19 124.90
108.12
10.18
28 $1.44
20:37
ant
129.00
20.00
#2.36
72 $3.40
125.52 #2.30
127.86
#2.00
3.32
medium--eww-expe
hopper
107.95
110.08
108. 08896...
106.99
104.34
5:21
1.02
halfcheerah
87.98 +1.17
87.79
+1.79
89.47
88.66 +1.31
89.17
1.88
0.00
walker2d
108.
20.31
20 10.3
108, 15 10.
108.16 16 20
108.47 20:00
0.30
0.30
ant
120.44
21.07
119.36
119.12
122.32
2.62
1.60
clonestexpert
50.08 117.20
72.62
17:45
82.77
73,96
76.37 1e.50
39.19
0.00
s0.0a
20.58 103.79
102.77 20.00
+1..8
20000
1.10
hammer
101.98 14.00
110.07
17.00
94.59
105.58 11..35
106.81
1.:.1
14.06
14.06
humansexpert
95.75
96.68
24.25
95.27
98.29
97.74
2,58
2.56
13.45
100.91
13:05
111.00
12.31
101.13 11:07
hammer
95.24 25.79
101.71 24.07
93.34 37.41
101.04 26.30
103.79
10.07
10.07
partial-expert
kitchen
64.00
61 61.83
58.95
61,33
60.08
7.89
7.89
mixeedexpert
kitchen
$3.38
45.42 2405
20.54
21.35
45.33 21.00
Average
4.54
2.07
Table 9: Average normalized return over last 10 evaluations of IOSTOM with different a values on the D4RL suboptimal datasets with expert trajectory. Method with the best avg. perf is in bold.
28

---

This section presents an ablation study to evaluate the impact of the hyperparameter on IOSTOM's performance. Trepooss performance for each setting, with the best-perg a for each task highlighted in bold. We also include two additional metrics: gapworst, representing the percentage gap between the best and worst a; and gopdefault: indicating the gap between the best-performing a and our default setting of a = 0.5.
According to the table, the average gapworat value across all tasks is just 4.54%, which is relatively small. This indicates that IOSTOM's performance is not highly sensitive to the choice of a. Further- more, the average performance gap between the task-specife optimal a and our default choice of = 0.5 is even smaller at 2.07% %. The gapde value is also under 5% in all but 3 of 24 tasks. This confirms that a 0.5 is a robust and effective hyperparameter choice, consistently providing near-optimal performance.
D.5 3 Ablation
Suboptimal Dataset
Env
J=3
9=5
8=10
9=15
f9=20
Expert
random+
halfcheetah
hopper
52.40 111.29
13.72
92.64
5.56
10.90
41.39
93.10
1:90.18
20.05
110.36 10..5
93.18
10033
109.32
93,02
10.00
111.33
88.83
expert
walker2d
1.18
60.97
1,0.00
6.75
114.25
107.67
10.14
107.98
106.92
ant
118.94 27.00
126.17
22:00
128.02
1:500
128.20
23.52
128.19
11.52
130.75
random+
halfcheetah
hopper
10.68 13.56
2.24 10:01
2.20
6.10 10.76
10.00
35,94
83.37
1:55.77
=2
102.30
85.20
11.01
15.12
107,28
88,77
11.00
23.92
111.33
88.83
few ewweexpert
walker2d
1.20 +0.24
11,65 =3.33
29.87
+31.00
108.21
=0.41
108.40
=0.21 120.09
106.92
ant
45.72 155.95
97.4
#14.19
125.28
+:3.0
122,76
#3.04
130,75
medium+ expert
hopper
31.99 #24.17
61.48
#10.37
979.97..
#:042
110.02
#1.00
110.20
=0.51
111.33
halfehoctah
43.20 10.47
54,63
:3.08
92.63
92.96
1:0.2)
93.12
10.52
88 83
walker2d
71.70 #4.50
107.83
+0.76
108.31
+0027
=0.12
12 =0.18
106,92
ant
98.70
11.81
102.01
12.00
121.
1:2.9
124.12
22.93
124.72 23.49
130.75
medium
hopper
46.86 29.74
61.88
#22.27
105. 83 $4.07
106.80
=2.29
108.96 =1.35
111.33
halfchoctah
42.85 10.27
43.17
20.12
15:15
83.52
11:72
89.47
10.82
88.83
lew-eapert
walker2d
66.58
69.35
108.33
108.46 10:13
108.15
20:43
106.92
ant
92.15
#1.80
94.62
=3.44
=1.67
111.45 =3.09
120.36 =1.25
1.30.5
pen
82,77 +4.84
56.05
=7:20
11.30
=2:64
10.33 #2.07
10.13 +3.00
106.42
clonedvexpert
door
40.58
2.5.52
80,99
#46..8
102.77
+0.06
100.08 =2.09
86.06 #6.14
103.94
hammer
88.63
233.84
94,88
#16.51
94.59
=:: 30
100.59
=10.00
90.27 218.70
125.71
pen
99.27
#5.22
101.27
95,77
96, 14 +7.58
99. 99.96 15.000
106.42
human-expert
doar
99.90
#1.00
102.22
#:.42
100.77
+1.68
+22.06
=2.00
103,94
hammer
87.11
#10.28
102.10
+15.66
93.34
#7.41
97.37
#15.82
93.39
125.71
partiallexpert
kitchen
49.80
#14.81
61.10
=9.24
57.75
=2:00
63.00 #3.33
59.70
#6.26
75.0
mixed+expert
kitchen
46.75 1...5
45,80
:2.05
47.92
46.85
1:100
45.40
1.3.4
75.0
Table 10: Average normalized return over last evaluations of IOSTOM with different 3 values on the D4RL suboptimal datasets with expert trajectory. Method with the best avg. perf is in bold.
This section presents an ablation study to evaluate the impact of the hyperparameter 33 on IOSTOM's performance. Table 10 summarizzs these results. For locomotion tasks (e.g., Hopper, HalfCheetah, Walker2d, Ant), higher 3 values, typically 15 or 20, generally yield superior scores compared to lower values such as 3 or 5. Conversely, for manipulation tasks (c.g.. Pen, Door, Hammer, Kitchen), optimal performance is often achieved with 3 values of 5 or 10. However, the performance differences across various 8 settings for these tasks are less pronounced. The only exception is the 'pen' environment within the 'clonedtexpert dataset, where decreasing B leads to improved results, with 3 = 3 achieving the highest score.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:[Yes
Our main claims made in the abstract and introduction precisely reflect our paper's 's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are included in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

The proofs for all propositions are included in the Appendix. All assumptions are clearly stated both the main text and the proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The source-code of our algorithm, along with environment details and exact commands to run are included in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The D4RL datasets is publicly available. have submitted source code with detailed instructions. We can not publish the AIS data used in Section 6.4 due to confidential agreement with a third-party data provider.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
wwriiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details are described in the main paper and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
We have reported the mean and std of results obtained by running with different random sets in the all experimental sections of main paper. In the code, provide scripts for generating all training curves constructed from mean scores and shaded by standard error.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have shown all these information in our Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //te:///neeuee newrips........
Answer: [Yes]
Justification:
I confirm that paper conforms to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper provides a general offline imitation learning algorithm that only tests the simulated environments. As such, we do not foresee any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has no risk for misuse because we do not release any dataset or pretrained model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: instructions.
Our source code is submilted alongside the paper, accompanied by sufficient
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not conduct any crowdsourcing experiment and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do not conduct any experiment related human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https://neuri)) htct:::://eeripss.cc//ooeeeeeeee /ct////oefeeeeneee/22////.. for what should or should not be described.
36


Reviews:
[{"review_id": "hRoXPlAcbu", "reviewer": "Reviewer_ahUM", "summary": "This paper introduces ISOTOM, a framework for offline imitation learning from observation (IfO). ISOTOM aims to address the offline IfO problem while improving training stability by eliminating the need for a discriminator and an inverse dynamics model. Specifically, ISOTOM relies on state-transition distribution matching to remove dependence on action information. The paper first presents a joint-state Q-learning formulation to recover the implicit policy, followed by policy learning through advantage-weighted behavioral cloning. Empirical results demonstrate that ISOTOM outperforms baseline methods in MuJoCo environments and a marine navigation scenario.", "questions": "- Have the authors conducted any experiments with subsampled random data to assess the sensitivity of the method to the amount of random data?\n- Have the authors conducted any experiments in image-based environments? Such settings may better reflect the nature of imitation from observation scenarios, where actions are not available and only visual state information is provided.\n\nI am willing to increase the score if most of the concerns in the weakness and questions sections are solved.", "limitations": "Yes (in the appendix)", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n\n(1) Quality\n- The paper provides both theoretical analyses and empirical results demonstrating the effectiveness of the proposed method. However, I did not examine the proofs in the appendix in detail.\n- The experiment section includes numerical results not only for MuJoCo simulated environments but also for a real-world marine navigation dataset.\n\n(3) Significance\n- The proposed method demonstrates superior performance compared to the baselines in both MuJoCo simulation environments and a real-world marine navigation scenario.\n\n(4) Originality\n- The paper introduces an objective function that excludes actions, which appears to be a novel contribution in the context of offline imitation from observation.\n\n[Weaknesses]\n\n(2) Clarity\n- I believe the primary issue that makes me hesitant to recommend solid acceptance is the lack of clarity in the paper. While this may partly reflect my own limitations in expertise, the presentation feels unstructured and densely written, which hinders comprehension. For example:\n    - In Section 3, the definitions of $d^E(s)$ and $d^I(s)$ are missing or unclear.\n    - In Section 4.1, $d_0(s)$ is not properly defined.\n    - The transition from Equation (3) to (4) is not clearly explained. I was able to derive (4) from (3), but it required effort that could have been avoided with a brief clarification.\n    - The paper does not specify where the proofs of the main theorem can be found, even though they are in the appendix.\n    - Additionally, several parts of the text (e.g., lines 121, 124, 146, 148, 150, 225, 235, 244) are written in a way that makes them difficult to parse.\n\nThese issues in notation, definition, and mathematical transitions collectively hindered my ability to follow the technical development of the paper. As a result, it was difficult to clearly identify the main contributions and understand the reasons behind the strengths of the proposed method compared to prior work. Improving the logical flow between sections and organizing the key concepts more clearly would significantly enhance readability and overall comprehension.", "was_revised": true, "final_justification": "As I mentioned in my comment, the authors have addressed the concerns raised in the review, particularly by conducting experiments to demonstrate the robustness of the proposed method. Additionally, the clarity issue does not appear to be a significant problem for other reviewers. Therefore, I am inclined to raise my overall score to 4 (borderline accept).", "rebuttal": "We thank reviewer for your helpful comments and feedback. We provide our responses below to address your concerns.  \n\n> 1. Issues about paper's clarity\n\nWe thank the reviewer for the valuable comments regarding the clarity of our paper. We will revise the manuscript accordingly based on your suggestions.\n> In Section 3, the definitions of $d^E(s), d^I(s)$ and are missing or unclear.\n\nThey are occupancy measures of the expert and sup-optimal datasets. We will clarify them.\n\n> In Section 4.1, $d_0(s)$ is not properly defined.\n\nThis is the visitation distribution of  state $s$ at time 0. We wil clarify this. \n\n>The transition from Equation (3) to (4) is not clearly explained. I was able to derive (4) from (3), but it required effort that could have been avoided with a brief clarification.\n\nWe have removed some intermediate derivations due to space limitations. We will clarify this transition in the revised version.\n\n> The paper does not specify where the proofs of the main theorem can be found, even though they are in the appendix.\n\nThank you for the comment. We will add references to the corresponding proofs in the appendix.\n> Additionally, several parts of the text (e.g., lines 121, 124, 146, 148, 150, 225, 235, 244) are written in a way that makes them difficult to parse.\n\nThank you for pointing out these detailed issues. These parts were compressed due to space limitations, which may have caused some lack of clarity. We will revise them to make the content clearer and easier to follow.\n\n> 2. Have the authors conducted any experiments with subsampled random data to assess the sensitivity of the method to the amount of random data?\n\nWe thank the reviewer for the insightful question. In response, we applied subsampling rates of 5 and 20, meaning only 20% and 5% of the low-quality trajectories (\"random\" or \"medium\") from the original suboptimal dataset were used, respectively. The table below compares the performance of IOSTOM(full) with these subsampled variants—IOSTOM(sub5) and IOSTOM(sub20). Drops in performance exceeding 5% relative to the full dataset are highlighted in bold.\n\n\nWhen reducing the low-quality data to 20% of its original size (IOSTOM(sub5)), our method demonstrates strong robustness. Across all tasks and datasets, the performance remains very close to the results on the full dataset across all tasks with no significant degradation observed. When the low-quality data is aggressively subsampled to just 5% (IOSTOM(sub20)), we observe a more noticeable performance drop (up to about 18%), particularly in th halfcheetah and ant environments. However, IOSTOM is still robust on 10/16 tasks in this extremely limited data scenario.\n\n| Suboptimal Dataset | Env         | IOSTOM(full) | IOSTOM(sub5) | IOSTOM(sub20)    |\n|--------------------|-------------|--------------|--------------|------------------|\n| random+expert      | hopper      | 109.32±1.08  | 110.24+-0.63 | 110.66+-0.16     |\n|                    | halfcheetah | 93.02±0.40   | 90.81+-2.10  | **85.59+-5.09**  |\n|                    | walker2d    | 107.98±0.20  | 107.91+-0.16 | 107.53+-0.11     |\n|                    | ant         | 128.19±1.52  | 127.26+-2.48 | 127.30+-1.37     |\n| random+few-expert  | hopper      | 107.28±3.92  | 108.99+-0.76 | 109.34+-1.62     |\n|                    | halfcheetah | 88.77±1.26   | 86.62+-1.98  | **80.49+-6.41**  |\n|                    | walker2d    | 108.40±0.21  | 108.13+-0.29 | 107.78+-0.22     |\n|                    | ant         | 120.09±5.17  | 119.63+-2.40 | **105.98+-7.03** |\n| medium+expert      | hopper      | 110.20±0.51  | 109.89+-0.22 | 109.58+-1.18     |\n|                    | halfcheetah | 93.12±0.32   | 91.07±1.72   | **76.30+-13.56** |\n|                    | walker2d    | 108.12±0.13  | 108.31+-0.21 | 108.03+-0.15     |\n|                    | ant         | 124.72±3.49  | 127.71+-2.72 | 128.61+-0.71     |\n| medium+few-expert  | hopper      | 108.96±1.33  | 109.68+-0.56 | 108.20+-0.83     |\n|                    | halfcheetah | 89.47±0.82   | 90.20±1.94   | **77.65+-11.48** |\n|                    | walker2d    | 108.15±0.43  | 107.88+-0.83 | 107.39+-0.92     |\n|                    | ant         | 120.36±1.25  | 118.23+-5.04 | **102.14+-7.26** |\n\n> 3.  Have the authors conducted any experiments in image-based environments? Such settings may better reflect the nature of imitation from observation scenarios, where actions are not available and only visual state information is provided.\n\nWe thank the reviewer for the insightful question. In this work, we focused on the well-established D4RL state-based benchmarks to allow for direct and rigorous comparison with recent state-of-the-art LfO methods (e.g., SMODICE, PW-DICE, DILO), which are also primarily evaluated in these settings.\n\nWhile we agree that extending to image-based tasks would broaden the applicability of our method, experiments on domains like V-D4RL [1] require careful design of visual encoders and significant effort to ensure fair integration with both our framework and existing baselines. As most prior LfO methods also do not evaluate in image-based settings, a thorough and fair comparison remains challenging.\n\nWe will include a discussion in the revised paper highlighting the extension to vision-based tasks as a key direction for future work.\n\n**References:**\n\n[1]: Lu, Cong, et al. \"Challenges and opportunities in offline reinforcement learning from visual observations.\" arXiv preprint arXiv:2206.04779 (2022).\n\n\n---\n\n**We hope our revisions and our response  address the reviewers’ concerns and further clarify our contributions. If there are any additional questions or comments, we would be happy to address them.**"}, {"review_id": "JoQpFdoQyw", "reviewer": "Reviewer_gGMk", "summary": "This paper introduces a novel approach (IOSTOM) for offline LfO problem by joint state visitation distribution matching. This method specifically avoids using discriminators or inverse dynamics models, and instead learns an implicit policy via state-to-state transition probabilities without using action label. Empirical results on D4RL datasets demonstrate the efficacy of the method.", "questions": "What is the intuition of using a mixture of joint state visitation distributions $d_\\mathrm{mix}^I, d_\\mathrm{mix}^{E,I}$? How does the mixing parameter $\\alpha$ influences the training? It would be helpful to introduce the intuition before deriving the full equation.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper presents a novel LfO approach grounded in occupancy matching.\n\n- The theoretical framework and derivation of the algorithm is very solid.\n\n- The method is proved to be  practical and efficient with its discriminator-free and IDM-free design.\n\n- Demonstrates better performance on diverse and challenging benchmarks, outperforming all baselines, especially inclusing an interesting real-world case study on marine navigation.\n\n- Demonstrates robustness to low-data settings and sub-sampling.\n\n  \n\n**Weaknesses:**\n\n* Access to fully observable actions in sub-optimal dataset is hard to guarantee. It would be helpful to discuss how realistic it is.", "was_revised": true, "final_justification": "Thank you for the detailed replies and additional experiments. I have no further concerns and am keeping my score.", "rebuttal": "We thank reviewer for your helpful comments and feedback. We provide our responses below to address your concerns.  \n\n> 1. Access to fully observable actions in sub-optimal dataset is hard to guarantee. It would be helpful to discuss how realistic it is.\n\nWe thank the reviewer for the insightful comments. We believe that access to action-labeled suboptimal datasets is highly realistic in many practical settings. For example, in the robotics community, collecting demonstrations via teleoperation from non-expert users is often significantly easier and more cost-effective than acquiring expert demonstrations, which typically require domain-specific training or expert operators [1].\n\nBeyond robotics, similar trends are observed in other domains. In autonomous driving, for instance, large-scale datasets are often gathered from regular human drivers who may not always exhibit optimal behavior. These action-labeled suboptimal logs still provide valuable data for training and evaluation. In healthcare applications, clinician interactions recorded in electronic health records can be viewed as suboptimal demonstrations, yet they form the basis for many decision-support models. Similarly, in gaming and digital environments, user gameplay logs—despite being noisy or suboptimal—are widely used for imitation and reinforcement learning research.\n\nThese examples underscore that suboptimal, action-labeled data are not only easier to obtain but also more abundant and representative of real-world behavior. As such, they offer a practical and scalable foundation for learning from demonstrations in many domains.\n\nWe will incorporate these examples into the revised paper to better illustrate the practicality of accessing action-labeled suboptimal datasets.\n\n> 2. What is the intuition of using a mixture of joint state-visitation distributions $d_{mix}^I, d_{mix}^{E,I}$? It would be helpful to introduce the intuition before deriving the full equation.\n\nWe thank the reviewer for the insightful question. The use of a mixture of joint state-visitation distributions is adopted from some recent state-of-the-art papers on RL and IL (e.g., DualRL [2]), which highlights its advantage in avoiding on-policy sampling.\n\nSpecifically, prior work typically adopts matching between $d^{\\pi}$ and $d^{E}$, but this approach requires on-policy samples from $\\pi$ to solve. This is not ideal in the offline setting, as these samples can be out-of-distribution. This limitation has been discussed in previous LfO works such as LobsDICE (Section 2.2 in [3]) and SMODICE (Section 3.1 in [4]). Using a mixture of joint state-visitation distributions helps address this issue by avoiding reliance on on-policy data while maintaining effective alignment between expert and policy behavior. We will include this discussion earlier in the updated paper to better clarify the motivation behind this design choice.\n\n\n> 3. How does the mixing parameter $\\alpha$ influence the training?\n\nWe thank the reviewer for the question. In response, we conducted an ablation study on the sensitivity of $\\alpha$, testing five values across all 24 tasks. The table below reports performance for each setting, with the best-performing $\\alpha$ for each task highlighted in bold. We also include two additional metrics: $gap_{worst}$, representing the percentage gap between the best and worst $\\alpha$, and $gap_{default}$, indicating the gap between the best-performing $\\alpha$ and our default setting of $\\alpha = 0.5$.\n\nAccording to the table, the average $gap_{worst}$ value across all tasks is just 4.54%, which is relatively small. This indicates that IOSTOM's performance is not highly sensitive to the choice of $\\alpha$. Furthermore, the average performance gap between the task-specific optimal $\\alpha$ and our default choice of $\\alpha=0.5$ is even smaller at 2.07%. The $gap_{default}$ value is also under 5% in all but 3 of 24 tasks. This confirms that $\\alpha=0.5$ is a robust and effective hyperparameter choice, consistently providing near-optimal performance.\n\n| Suboptimal Dataset | Env         | $\\alpha=0.1$     | $\\alpha=0.3$     | $\\alpha=0.5$ (our)    | $\\alpha=0.7$     | $\\alpha=0.9$     | $gap_{worst}$(%) | $gap_{default}$(%) |\n|--------------------|-------------|------------------|------------------|-----------------|------------------|------------------|---------------|-----------------|\n| random+expert      | hopper      | 109.72+-0.84     | 110.21+-0.68     | 109.32+-1.08    | 109.19+-0.21     | **110.29+-0.64** | 1.00          | 0.88            |\n|                    | halfcheetah | **93.09+-0.22**  | 92.90+-0.31      | 93.02+-0.40     | 92.99+-0.08      | 92.89+-0.18      | 0.21          | 0.08            |\n|                    | walker2d    | 107.82+-0.23     | 107.97+-0.08     | 107.98+-0.20    | 108.16+-0.17     | **108.38+-0.26** | 0.52          | 0.37            |\n|                    | ant         | **128.37+-1.89** | 126.67+-2.89     | 128.19+-1.52    | 125.78+-3.00     | 127.30+-1.85     | 2.02          | 0.14            |\n| random+few-expert  | hopper      | 107.41+-1.27     | **108.59+-2.04** | 107.28+-3.92    | 104.75+-3.36     | 107.25+-4.11     | 3.54          | 1.21            |\n|                    | halfcheetah | 87.29+-0.73      | 87.52+-2.11      | **88.77+-1.26** | 88.42+-1.00      | 86.61+-1.84      | 2.43          | 0.00            |\n|                    | walker2d    | 108.09+-0.24     | 108.21+-0.06     | 108.40+-0.21    | **108.45+-0.12** | 108.24+-0.14     | 0.33          | 0.05            |\n|                    | ant         | **125.43+-2.77** | 121.19+-2.09     | 120.09+-5.17    | 122.18+-1.90     | 123.46+-1.17     | 4.26          | 4.26            |\n| medium+expert      | hopper      | 109.79+-0.95     | 110.44+-0.51     | 110.20+-0.51    | 110.30+-0.54     | **110.61+-0.07** | 0.74          | 0.37            |\n|                    | halfcheetah | **93.16+-0.19**  | 92.82+-0.32      | 93.12+-0.32     | 93.00+-0.23      | 92.68+-0.17      | 0.52          | 0.04            |\n|                    | walker2d    | 107.54+-0.45     | 107.96+-0.12     | 108.12+-0.13    | 107.28+-1.44     | **108.22+-0.27** | 0.87          | 0.09            |\n|                    | ant         | **129.00+-0.59** | 124.91+-2.56     | 124.72+-3.49    | 125.52+-2.26     | 127.86+-2.20     | 3.32          | 3.32            |\n| medium+few-expert  | hopper      | 107.95+-1.72     | **110.08+-1.12** | 108.96+-1.33    | 106.99+-4.23     | 104.34+-5.88     | 5.21          | 1.02            |\n|                    | halfcheetah | 87.98+-1.17      | 87.79+-1.77      | **89.47+-0.82** | 88.66+-1.31      | 89.17+-0.60      | 1.88          | 0.00            |\n|                    | walker2d    | 108.15+-0.31     | 108.24+-0.33     | 108.15+-0.43    | 108.16+-0.22     | **108.47+-0.22** | 0.30          | 0.30            |\n|                    | ant         | 120.44+-1.57     | 119.36+-1.84     | 120.36+-1.25    | 119.12+-0.58     | **122.32+-0.73** | 2.62          | 1.60            |\n| cloned+expert      | pen         | 50.08+-17.30     | 72.62+-7.45      | **82.77+-4.84** | 73.96+-5.58      | 76.37+-6.52      | 39.49         | 0.00            |\n|                    | door        | 102.68+-0.63     | **103.79+-0.63** | 102.77+-0.96    | 102.65+-1.37     | 103.64+-0.92     | 1.10          | 0.98            |\n|                    | hammer      | 101.98+-4.88     | **110.07+-7.29** | 94.59+-9.39     | 105.58+-12.35    | 106.81+-2.94     | 14.06         | 14.06           |\n| human+expert       | pen         | 95.75+-7.18      | 96.68+-4.83      | 95.77+-8.91     | **98.29+-5.11**  | 97.74+-5.11      | 2.58          | 2.56            |\n|                    | door        | 100.41+-3.85     | 100.94+-3.65     | 100.77+-1.68    | 99.99+-2.31      | **101.13+-1.07** | 1.13          | 0.36            |\n|                    | hammer      | 95.24+-5.79      | 101.71+-4.97     | 93.34+-7.41     | 101.04+-6.36     | **103.79+-8.80** | 10.07         | 10.07           |\n| partial+expert     | kitchen     | **64.00+-7.66**  | 61.83+-1.38      | 58.95+-2.27     | 61.33+-4.13      | 60.08+-3.21      | 7.89          | 7.89            |\n| mixed+expert       | kitchen     | 45.25+-2.38      | 45.42+-4.05      | **46.45+-0.84** | 45.08+-1.13      | 45.33+-2.01      | 2.95          | 0.00            |\n| Average            |             |                  |                  |                 |                  |                  | 4.54          | 2.07            |\n\n**References:**\n\n[1]: Lynch, Corey, et al. \"Learning latent plans from play.\" Conference on robot learning. Pmlr, 2020. \n\n[2]: Sikchi, Harshit, et al. \"Dual rl: Unification and new methods for reinforcement and imitation learning.\" ICLR 2024.\n\n[3]: Kim, Geon-Hyeong, et al. \"Lobsdice: Offline learning from observation via stationary distribution correction estimation.\" NeurIPS 2022.\n\n[4]: Ma, Yecheng, et al. \"Versatile offline imitation from observations and examples via regularized state-occupancy matching.\" ICML 2022.\n\n---\n\n**We hope our revisions and our response  address the reviewers’ concerns and further clarify our contributions. If there are any additional questions or comments, we would be happy to address them.**"}, {"review_id": "Bsc02XbKOX", "reviewer": "Reviewer_4HqZ", "summary": "The paper proposes IOSTOM, an offline imitation learning from observation method based on state transition occupancy matching that is discriminator-free and data-efficient. The proposed method operates on a state-only expert dataset and state-action suboptimal dataset. The proposed method consists of several steps. First, IOSTOM recovers a state-transition probability distribution (\"implicit policy\") by minimizing $\\chi^2$-divergence between the learned state-transition and expert state-transition (regularized by suboptimal dataset policy), which can be converted to an unconstrained bilevel Q-value optimization via Lagrange multipliers and convex conjugates. Soft value estimates are adopted to avoid Out-Of-Distribution (OOD) samples. Finally, the policy is retrieved by advantage-weighted behavior cloning from the suboptimal state-action dataset (as a lower bound surrogate for optimizing implicit policy with Inverse-Dynamic Model (IDM)). The proposed algorithm is evaluated on multiple environments which outperforms prior methods.", "questions": "I have several questions:\n\n1. Is it possible to add a teaser figure for this paper? I feel people who are less familiar with the state occupancy matching framework can understand the idea more easily with an illustration.\n\n2. Is there a more direct ablation / empirical evidence (e.g. visualization) to show in some cases discriminators fail other than final performance difference?\n\n3. Why do the authors choose $\\alpha=0.5$ for distribution mixture with suboptimal dataset, which seems quite large? Can the authors provide an ablation on different values of $\\alpha$?\n\n4. Why do the authors choose to update policy during the training loop of other components (as shown in Alg. 1) instead of finishing the other parts of training first, which could potentially cause instability? $\\theta$ is not involved in other parts of training after all.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-written and easy to follow; though the paper is math-heavy, the high-level idea behind the formulae (first optimize state-pair divergence, then use Lagrange multiplier & convex conjugate to convert to unconstrained problems, then soft values to avoid OOD issues, and finally extract policy via lower bound of implicit policy with IDM) is very clearly conveyed.\n\n2. The idea of this paper is interesting and well-supported by mathematical derivations: bypassing discriminators by optimizing \"implicit policy\", which is a novel notion.\n\n3. The evaluation results look solid with sufficient amount of ablations. I particularly appreciate the video demonstrations in the supplementary material.\n\n**Weaknesses**\n\n1. the training objective in Eq. 9 seems potentially unstable, as there is a term $(Q(s,s')-\\frac{2}{1-\\gamma})^2$. When $\\gamma$ is close to 1 with long horizon tasks, this can lead to a very large value of $Q$ which causes numerical issues during optimization.\n\n2. The number of expert dataset used in the experiment is quite large, especially for the MuJoCo datasets. In the appendix, the authors stated that they use 30 expert trajectories for \"few-expert\" scenario. In contrast, both SMODICE (https://arxiv.org/pdf/2202.02433, Sec. 6.1) and PW-DICE (https://arxiv.org/pdf/2311.01331, Appendix D.2) uses **single** expert trajectory. I would like to see how the proposed algorithm work with even less amount of expert trajectories than the current setting.\n\n**Minor Weaknesses**\n\n1. Some reference format are incorrect. For example: \"[34] introduce ReCOIL\" -> \"Sikchi et al. [34] introduce ReCOIL\"; \"Following [40]\" -> \"Following Xu et al. [40]\".\n\n2. The period in Tab. 4, Tab. 5 and Fig. 3 caption is missing.\n\n3. The main paper uses \"inverse dynamic model\", but the appendix line 648 uses \"inverse kinematics model\".", "was_revised": true, "final_justification": "The authors' response has addressed my concern satisfactorily. However, as some other reviewers feel their concern is unaddressed, I will keep my current score.", "rebuttal": "We thank the reviewer for the constructive comments and valuable feedback. Below, we provide our detailed responses to address your concerns. \n\n> 1. The training objective in Eq. 9 seems potentially unstable, as there is a term $(Q(s, s')-\\frac{2}{1-\\gamma})^2$. When $\\gamma$ is close to 1 with long horizon tasks, this can lead to a very large value of $Q$ which causes numerical issues during optimization.\n\nWe thank the reviewer for the insightful comment. In our work, the discount factor $\\gamma$ is set to 0.99 (see Table 5 in the Appendix), which is a standard choice in RL. With $\\gamma = 0.99$, the term $(Q(s, s') - \\frac{2}{1 - \\gamma})^2$ becomes $(Q(s, s') - 200)^2$. Our training remains stable with this formulation. Prior imitation learning methods such as LS-IQ [1] have also adopted this formulation to promote stable learning. The goal of  $(Q(s, s') - \\frac{2}{1 - \\gamma})^2$ is only assigning large and bounded $Q$ values to expert transitions as pointed in Section 5 and Appendix B.8 of our work. Therefore,  one could simply cap the target value (e.g. at 200) in the case of $\\gamma$ > 0.99 (which is uncommon in deep RL). This would preserve the objective's intent to assign a high and stable Q-value to expert data while avoiding any potential numerical issues.    \n\n\n> 2. The number of expert dataset used in the experiment is quite large, especially for the MuJoCo datasets. In the appendix, the authors stated that they use 30 expert trajectories for \"few-expert\" scenario. In contrast, both SMODICE (https://arxiv.org/pdf/2202.02433, Sec. 6.1) and PW-DICE (https://arxiv.org/pdf/2311.01331, Appendix D.2) uses single expert trajectory. I would like to see how the proposed algorithm work with even less amount of expert trajectories than the current setting.\n\nWe thank the reviewer for raising this point. We believe there may be a misunderstanding. In Appendix C.1, we stated: *“The state-only expert dataset in all tasks includes only one expert trajectory.”* In our paper, the term “few-expert” refers to a suboptimal dataset constructed by combining \"random\" or \"medium\" trajectories with 30 expert trajectories.\n This is similar to the experimental setting used in SMODICE and PW-DICE, where the suboptimal dataset contains 200 expert trajectories alongside lower-quality ones. The key difference across scenarios lies in how the suboptimal dataset is constructed, as detailed in **Table 4** of the Appendix.\n\nWe would greatly appreciate it if the reviewer could point us to the specific part that caused confusion, so we can revise it for greater clarity.\n\n> 3. Is it possible to add a teaser figure for this paper? I feel people who are less familiar with the state occupancy matching framework can understand the idea more easily with an illustration.  \n\nWe thank the reviewer for the valuable suggestion, which could indeed improve the readability of the paper. We will include such a teaser figure in the revised version. Due to the NeurIPS rebuttal policy, we are unable to upload images at this stage, but it will be added in the updated version of the paper.  \n\n> 4. Is there a more direct ablation / empirical evidence (e.g. visualization) to show in some cases discriminators fail other than final performance difference? \n\nWe thank the reviewer for the valuable suggestion. Visualizing the reward landscape is indeed a helpful way to illustrate how the discriminator assigns high rewards to non-expert states in tasks where discriminator-based methods perform poorly. As the NeurIPS review system does not allow image uploads during the rebuttal phase, we will include these visualizations in the revised version of the paper.  \n\n\n> 5. Why do the authors choose $\\alpha =0.5$ for distribution mixture with suboptimal dataset, which seems quite large? Can the authors provide an ablation on different values of $\\alpha$? \n\nWe thank the reviewer for the insightful question. We set $\\alpha = 0.5$ as it is a commonly adopted choice in prior works [1, 2, 3] that employ mixture distributions for $f$-divergence minimization.\n\nIn response to your question, we conducted an ablation study on the sensitivity of $\\alpha$, testing five values across all 24 tasks. The table below reports performance for each setting, with the best-performing $\\alpha$ for each task highlighted in bold. We also include two additional metrics: $gap_{worst}$, representing the percentage gap between the best and worst $\\alpha$, and $gap_{default}$, indicating the gap between the best-performing $\\alpha$ and our default setting of $\\alpha = 0.5$.\n\nAccording to the table, the average $gap_{worst}$ value across all tasks is just 4.54%, which is relatively small. This indicates that IOSTOM's performance is not highly sensitive to the choice of $\\alpha$ . Furthermore, the average performance gap between the task-specific optimal $\\alpha$ and our default choice of $\\alpha=0.5$ is even smaller at 2.07%. The $gap_{default}$ value is also under 5% in all but 3 of 24 tasks. This confirms that $\\alpha=0.5$ is a robust and effective hyperparameter choice, consistently providing near-optimal performance.\n\n| Suboptimal Dataset | Env | $\\alpha=0.1$ | $\\alpha=0.3$ | $\\alpha=0.5$ (our) | $\\alpha=0.7$ | $\\alpha=0.9$ | $gap_{worst}$(%) | $gap_{default}$(%) |\n|---|---|---|---|---|---|---|---|---|\n| random+expert | hopper | 109.72+-0.84 | 110.21+-0.68 | 109.32+-1.08 | 109.19+-0.21 | **110.29+-0.64** | 1.00 | 0.88 |\n|  | halfcheetah | **93.09+-0.22** | 92.90+-0.31 | 93.02+-0.40 | 92.99+-0.08 | 92.89+-0.18 | 0.21 | 0.08 |\n|  | walker2d | 107.82+-0.23 | 107.97+-0.08 | 107.98+-0.20 | 108.16+-0.17 | **108.38+-0.26** | 0.52 | 0.37 |\n|  | ant | **128.37+-1.89** | 126.67+-2.89 | 128.19+-1.52 | 125.78+-3.00 | 127.30+-1.85 | 2.02 | 0.14 |\n| random+few-expert | hopper | 107.41+-1.27 | **108.59+-2.04** | 107.28+-3.92 | 104.75+-3.36 | 107.25+-4.11 | 3.54 | 1.21 |\n|  | halfcheetah | 87.29+-0.73 | 87.52+-2.11 | **88.77+-1.26** | 88.42+-1.00 | 86.61+-1.84 | 2.43 | 0.00 |\n|  | walker2d | 108.09+-0.24 | 108.21+-0.06 | 108.40+-0.21 | **108.45+-0.12** | 108.24+-0.14 | 0.33 | 0.05 |\n|  | ant | **125.43+-2.77** | 121.19+-2.09 | 120.09+-5.17 | 122.18+-1.90 | 123.46+-1.17 | 4.26 | 4.26 |\n| medium+expert | hopper | 109.79+-0.95 | 110.44+-0.51 | 110.20+-0.51 | 110.30+-0.54 | **110.61+-0.07** | 0.74 | 0.37 |\n|  | halfcheetah | **93.16+-0.19** | 92.82+-0.32 | 93.12+-0.32 | 93.00+-0.23 | 92.68+-0.17 | 0.52 | 0.04 |\n|  | walker2d | 107.54+-0.45 | 107.96+-0.12 | 108.12+-0.13 | 107.28+-1.44 | **108.22+-0.27** | 0.87 | 0.09 |\n|  | ant | **129.00+-0.59** | 124.91+-2.56 | 124.72+-3.49 | 125.52+-2.26 | 127.86+-2.20 | 3.32 | 3.32 |\n| medium+few-expert | hopper | 107.95+-1.72 | **110.08+-1.12** | 108.96+-1.33 | 106.99+-4.23 | 104.34+-5.88 | 5.21 | 1.02 |\n|  | halfcheetah | 87.98+-1.17 | 87.79+-1.77 | **89.47+-0.82** | 88.66+-1.31 | 89.17+-0.60 | 1.88 | 0.00 |\n|  | walker2d | 108.15+-0.31 | 108.24+-0.33 | 108.15+-0.43 | 108.16+-0.22 | **108.47+-0.22** | 0.30 | 0.30 |\n|  | ant | 120.44+-1.57 | 119.36+-1.84 | 120.36+-1.25 | 119.12+-0.58 | **122.32+-0.73** | 2.62 | 1.60 |\n| cloned+expert | pen | 50.08+-17.30 | 72.62+-7.45 | **82.77+-4.84** | 73.96+-5.58 | 76.37+-6.52 | 39.49 | 0.00 |\n|  | door | 102.68+-0.63 | **103.79+-0.63** | 102.77+-0.96 | 102.65+-1.37 | 103.64+-0.92 | 1.10 | 0.98 |\n|  | hammer | 101.98+-4.88 | **110.07+-7.29** | 94.59+-9.39 | 105.58+-12.35 | 106.81+-2.94 | 14.06 | 14.06 |\n| human+expert | pen | 95.75+-7.18 | 96.68+-4.83 | 95.77+-8.91 | **98.29+-5.11** | 97.74+-5.11 | 2.58 | 2.56 |\n|  | door | 100.41+-3.85 | 100.94+-3.65 | 100.77+-1.68 | 99.99+-2.31 | **101.13+-1.07** | 1.13 | 0.36 |\n|  | hammer | 95.24+-5.79 | 101.71+-4.97 | 93.34+-7.41 | 101.04+-6.36 | **103.79+-8.80** | 10.07 | 10.07 |\n| partial+expert | kitchen | **64.00+-7.66** | 61.83+-1.38 | 58.95+-2.27 | 61.33+-4.13 | 60.08+-3.21 | 7.89 | 7.89 |\n| mixed+expert | kitchen | 45.25+-2.38 | 45.42+-4.05 | **46.45+-0.84** | 45.08+-1.13 | 45.33+-2.01 | 2.95 | 0.00 |\n| Average |  |  |  |  |  |  | 4.54 | 2.07 |\n\n> 6. Why do the authors choose to update policy during the training loop of other components (as shown in Alg. 1) instead of finishing the other parts of training first, which could potentially cause instability? $\\theta$ is not involved in other parts of training after all.\n\nWe thank the reviewer for the insightful question. The reason is that the policy does not influence the value function updates, so policy extraction can be performed either concurrently with or after value learning. In our implementation, we follow the XQL [4] codebase, which updates the policy jointly with the value networks. This joint training approach is also adopted by other LfO methods such as DILO and LobsDICE (see Section 4.3 of [3] and Appendix G.2 of [5]).\n\n**References:**\n\n[1]: Al-Hafez, Firas, et al. \"Ls-iq: Implicit reward regularization for inverse reinforcement learning.\" ICLR 2023.\n\n[2]: Sikchi, Harshit, et al. \"Dual rl: Unification and new methods for reinforcement and imitation learning.\" ICLR 2024.\n\n[3]: Sikchi, Harshit, et al. \"A dual approach to imitation learning from observations with offline datasets.\" CoRL 2024.\n\n[4]: Garg, Divyansh, et al. \"Extreme q-learning: Maxent rl without entropy.\"  ICLR 2023. \n\n[5]: Kim, Geon-Hyeong, et al. \"Lobsdice: Offline learning from observation via stationary distribution correction estimation.\" NeurIPS 2022.\n\n\n---\n\n**We hope our revisions and our response  address the reviewers’ concerns and further clarify our contributions. If there are any additional questions or comments, we would be happy to address them.**"}, {"review_id": "uoj1MnOaFj", "reviewer": "Reviewer_y8cD", "summary": "The paper presents imitation learning from observation via state transition occupancy matching (IOSTOM), a new offline LfO algorithm that excludes actions entirely from the training objective. The proposed algorithm learn an implicit policy that models transition probabilities between states, rather than leveraging action information of non-expert demonstrations. In the experiments, the authors demonstrate that IOSTOM outperforms other baseline algorithms on diverse sets of environments and tasks from D4RL benchmark.", "questions": "- What is the main reason for considering mixed state visitation disruption instead of starting from directly matching the objective with $d^E$? (What is the main problem if we derive it in the same way as direct matching for $d^E$?) It would be better if the reason for this part was explicitly explained in the paper.\n- How was the value of hyperparameter $\\alpha$ selected? How sensitive is IOSTOM's performance to hyperparameter $\\alpha$?\n- The optimal value of the hyperparameter alpha seems to be different for each environment. Is there no way to automatically determine $\\alpha$ (also for $\\beta$)?\n- LobsDICE is also a representative comparison algorithm that can be considered in the same setting. Why didn't you compare it? Can you add a comparison experiment?\n- In the paper, it was explained that the absence of discriminator learning is an advantage, but instead, $V$ learning was introduced. Isn't it similar? What makes it better?\n- Is it true that all the experimental learning was done with only offline datasets? Is it true that hyperparameter search and final model selection were also done without testing the environment?\n- In all experiments, were you only training using the offline dataset? Was there no access to the test environment even during hyperparameter search and final model selection?", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed method is theoretically well-grounded, with clearly derived formulations based on flow constraints, Lagrangian duality, and Q-weighted behavior cloning.\n- The method is evaluated across a wide range of environments and consistently achieves strong performance.\n\n**Weaknesses**\n\n- There is no experimental comparison with LobsDICE, one of the major baseline algorithms, considering the same problem setting.\n- Since hyperparameter selection for the real environment is not possible in offline settings, analysis of hyperparameter alpha is thought to be necessary.\n- It requires an additional hyperparameter beta, and its performance is very sensitive to $\\beta$.\n- It seems necessary to check whether fair experiments and comparisons were conducted in an offline setting. (ex. when determining hyperparameter $\\alpha$ and $\\beta$, whether the settings with the best performance in the test environment were selected and reported as the final results.)", "was_revised": true, "final_justification": "I have raised my rating accordingly (3: Borderline reject). I appreciate the author's detailed response, but concerns remain regarding the experimental aspects of the paper, making it still difficult to change the rating to accept.", "rebuttal": "We thank reviewer for your helpful comments and feedback. We provide our responses below to address your concerns.\n\n> 1. What is the main reason for considering mixed state visitation distribution instead of starting from directly matching the objective with $d^E$? (What is the main problem if we derive it in the same way as direct matching for $d^E$?)...\n\nWe thank the reviewer for the insightful question. The objective of directly matching between $d^{\\pi}$ and $d^{E}$ requires on-policy samples from $\\pi$ to solve which is not ideal in offline setting because these samples can be out-of-distribution. This issue is already discussed in previous LfO works like LobsDICE (Section 2.2 in [1]) and SMODICE (Section 3.1 in [2]). To mitigate this, these methods need to integrate the action-labeled suboptimal dataset into learning process by solving surgroate objectives as we note in **Section 3** of our paper.  Through our formulation in Eq. 2, we are also able to leverage the suboptimal dataset in training process. However, the key distinction is that our objective is mathematically constructed to have the same optimal solution as the original, direct-matching problem.\n\n\n> 2. How was the value of hyperparameter $\\alpha$ selected? How sensitive is IOSTOM's performance to hyperparameter $\\alpha$?\n\n**Please refer to Point 5 of the responses to reviewer 4HqZ**\n\n> 3. The optimal value of the hyperparameter alpha seems to be different for each environment. Is there no way to automatically determine $\\alpha$(also for $\\beta$)? \n\nWe thank the reviewer for the questions. As pointed out in our response to Reviewer 4HqZ, $\\alpha$ is not a sensitive parameter and can be fixed to 0.5 following previous works. The hyperparameter $\\beta$, which controls the regularization strength in the extreme value learning framework, is often sensitive and may need to be tuned per environment to achieve the best performance, as shown in [3]. However, we found very minimal hyperparameter tuning was required in our framework. As shown in Appendix C.3, we fix $\\beta$ to 20 for all locomotion tasks and $\\beta$ to 10 for most manipulation tasks except for \"pen-cloned\" where β is set to 3. The ablation study of $\\beta$ in Appendix D.3 also shows that in IOSTOM, $\\beta$ is only sensitive per class of environment, not per individual environment. Specifically, high values like 15 or 20 are consistently effective for locomotion tasks, while lower values like 5 or 10 achieve good results on manipulation tasks. This provides a straightforward heuristic for setting this parameter: one can either select a value based on the task type (locomotion vs. manipulation) or perform a quick search on a single representative task from a class and apply that value to all others in the same class.\n   \n> 4. LobsDICE is also a representative comparison algorithm that can be considered in the same setting. Why didn't you compare it?...\n\nWe thank the reviewer for bringing up LobsDICE. Initially, we did not include it in our baselines because PW-DICE—a more recent and related method—was already shown in [4] to outperform LobsDICE on the “random+expert” D4RL setting.\n\nTo address your question and provide a more complete comparison, we have conducted additional experiments comparing LobsDICE, PW-DICE, and our method (IOSTOM), as shown in the table below.\n\nThe results clearly demonstrate that IOSTOM significantly outperforms LobsDICE across all tasks. Like other DICE-based methods, LobsDICE performs poorly on the “few-expert” and manipulation tasks, while our method remains robust. Our findings also reaffirm that PW-DICE surpasses LobsDICE in the “random+expert” scenario. However, PW-DICE underperforms in several other settings, highlighting the general instability of discriminator-based methods. These comparisons further emphasize the advantages of our stable, discriminator-free approach. We will include the full results in the updated paper.\n\n| Suboptimal Dataset | Env | PW-DICE | LobsDICE | IOSTOM |\n|---|---|---|---|---|\n| random+expert | hopper | 108.09±2.39 | 99.64+-5.01 | **109.32±1.08** |\n|  | halfcheetah | 86.11±4.39 | 80.76+-3.17 | **93.02±0.40** |\n|  | walker2d | 107.48±0.53 | 107.54+-0.13 | **107.98±0.20** |\n|  | ant | 126.89±1.17 | 122.65+-0.72 | **128.19±1.52** |\n| random+few-expert | hopper | 75.04±14.21 | 74.25+-11.23 | **107.28±3.92** |\n|  | halfcheetah | 4.02 ±1.74 | 11.61+-5.66 | **88.77±1.26** |\n|  | walker2d | 36.11 ±9.19 | 101.29+-5.53 | **108.40±0.21** |\n|  | ant | 99.90 ±2.59 | 93.84+-7.51 | **120.09±5.17** |\n| medium+expert | hopper | 65.99 ±8.05 | 74.43+-5.28 | **110.20±0.51** |\n|  | halfcheetah | 58.74 ±1.84 | 71.09+-3.76 | **93.12±0.32** |\n|  | walker2d | 105.41 ±0.33 | 103.34+-2.11 | **108.12±0.13** |\n|  | ant | 108.14±1.90 | 114.96+-4.05 | **124.72±3.49** |\n| medium+few-expert | hopper | 57.24±3.03 | 67.00+-6.43 | **108.96±1.33** |\n|  | halfcheetah | 27.85±6.03 | 44.76+-3.94 | **89.47±0.82** |\n|  | walker2d | 75.22±7.05 | 95.39+-5.22 | **108.15±0.43** |\n|  | ant | 90.34±2.56 | 95.33+-1.08 | **120.36±1.25** |\n| cloned+expert | pen | 23.39±4.56 | 29.83 +- 4.59 | **82.77±4.84** |\n|  | door | 0.07±0.14 | 0.02 +- 0.00 | **102.77±0.96** |\n|  | hammer | 1.29±0.12 | 0.55 +- 0.19 | **94.59±9.39** |\n| human+expert | pen | -2.56±1.30 | 42.09 +- 5.08 | **95.77±8.91** |\n|  | door | 0.15±0.02 | 10.98 +- 9.71 | **100.77±1.68** |\n|  | hammer | 2.02±0.77 | 17.06 +- 13.44 | **93.34±7.41** |\n| partial+expert | kitchen | 12.33±5.38 | 40.33 +- 2.24 | **58.95±2.27** |\n| mixed+expert | kitchen | 7.50±4.16 | 45.67 +- 2.75 | **46.45±0.84** |\n\n\n> 5. In the paper, it was explained that the absence of discriminator learning is an advantage, but instead $V$, learning was introduced. Isn't it similar? What makes it better?\n\nWe thank the reviewer for the insightful question. In our setting, discriminator learning and value function learning serve fundamentally different purposes. The discriminator acts as a reward function, assigning high scores to expert states and low scores to non-expert ones. In contrast, our value function does more than binary classification—it imposes a structured ranking over suboptimal states by assigning higher values to those from which expert states are more easily reachable. This provides a meaningful recovery signal, guiding the agent back toward expert-like behavior when it deviates. Such nuanced guidance is something a simple reward-like discriminator cannot provide.\n\n> 6. In all experiments, were you only training using the offline dataset? Was there no access to the test environment even during hyperparameter search and final model selection?\n\nWe thank the reviewer for the question. In our work, the entire training process for IOSTOM is conducted strictly offline, using only the provided datasets. This stands in contrast to online IL methods like GAIL [6], which require continuous interaction with the environment to populate a replay buffer.\n\nHowever, we do require limited access to the environment for hyperparameter tuning and performance evaluation. This is a standard practice in both offline RL (e.g., Table 4 in [3]) and offline IL (e.g., Table 7 in [5]), where returns from the environment are used to select the best-performing hyperparameters for each task or task class.\n\nWhile fully offline alternatives exist, they are less applicable in our setting. For example, off-policy evaluation (OPE) methods [7] are commonly used in offline RL but require reward information, which is not available in our imitation learning setup. Another alternative—using validation loss to guide selection—is often unreliable due to its poor correlation with final policy performance [8].\n\nTherefore, we adopt the commonly accepted assumption of having limited evaluation access to the environment. In our case, since $\\alpha$ is fixed at 0.5, we only tune $\\beta$ from a small candidate set {3, 5, 7, 15, 20}, requiring at most five evaluation trials.\n\nIn terms of your concern about fair comparisions where we might change the $\\alpha$ and $\\beta$ per environment to beat other approaches, we want to note that we keep the same $\\beta=20$ across all locomotion tasks. We only use different values of $\\beta$ ($\\beta=3$ for \"pen-cloned\" task and $\\beta=10$ for the rest) when solving manipulation tasks. When dealing with different type of tasks like manipulation, previous LfO methods like SMODICE or PW-DICE also needs to modify their agorithmic choices: changing from KL divergence to $\\chi^2$ divergence in SMODICE (Appendix E.2 in [2]) and change in hyperparameter in PW-DICE (Table 1 in [4]). If we set a consistent $\\beta=10$ for all manipulation tasks, IOSTOM is still the leading approach 7/8 tasks. Therefore, we believe that IOSTOM outperforms all baselines under a fair and standard comparison protocol. \n\n**References:**\n\n[1]: Kim, Geon-Hyeong, et al. \"Lobsdice: Offline learning from observation via stationary distribution correction estimation.\" NeurIPS 2022.\n\n[2]: Ma, Yecheng, et al. \"Versatile offline imitation from observations and examples via regularized state-occupancy matching.\" ICML 2022.\n\n[3]: Garg, Divyansh, et al. \"Extreme q-learning: Maxent rl without entropy.\"  ICLR 2023. \n\n[4]: Yan, Kai, Alexander G. Schwing, and Yu-Xiong Wang. \"Offline imitation from observation via primal wasserstein state occupancy matching.\" ICML 2024.\n\n[5]: Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" The International Journal of Robotics Research (2023)\n\n[6]: Ho et al. \"Generative adversarial imitation learning.\" NeurIPS 2016.\n\n[7]: Fu, Justin, et al. \"Benchmarks for deep off-policy evaluation.\" ICLR 2021.\n\n[8]: Mandlekar, Ajay, et al. \"What matters in learning from offline human demonstrations for robot manipulation.\" CoRL 2021.\n\n---\n\n**We hope our revisions and our response  address the reviewers’ concerns and further clarify our contributions. If there are any additional questions or comments, we would be happy to address them.**"}]

Author rebuttal:

Meta-review: (a) Summary of Claims and Findings

This paper introduces IOSTOM, a novel algorithm for offline Learning from Observations (LfO). IOSTOM learns expert policies from state-only expert data and a separate suboptimal state-action dataset, avoiding the common need for discriminators or inverse dynamics models. The authors propose a new objective based on matching the joint state transition occupancy between the learned policy and the expert. This is formulated as an unconstrained Q-learning problem, and following weighted behavior cloning to recover the implicit policy. Empirical results on D4RL benchmarks and a real-world navigation task show that IOSTOM significantly outperforms state-of-the-art methods. 

(b) Strengths

The reviewers agree on several key strengths of the paper:
- Novelty: The action-free objective based on state transition occupancy is a novel and valuable contribution to the LfO field.
- Theoretical Soundness: The method is well-grounded in theory, with clear mathematical derivations that connect Q-learning and the implicit policy recovery.
- Strong Empirical Performance: The paper presents extensive and convincing results, with IOSTOM consistently outperforming baselines on diverse and challenging tasks.

(c) Weaknesses

Initial weaknesses raised by the reviewers included:
- Clarity: One reviewer (ahUM) initially find the paper's presentation dense and difficult to follow, citing missing definitions and unclear derivations.
- Experimental Gaps: Reviewers question the sensitivity to certain hyperparameters (y8cD, 4HqZ), the lack of comparison to a key baseline LobsDICE (y8cD), and the realism of the data requirements (gGMk, 4HqZ). Moreover, one reviewer (y8cD) raised concerns about the fairness of the offline evaluation.

(d) Reasons for Recommendation

I recommend accepting this paper. It presents a high-quality, technically solid contribution to the important problem of offline LfO. The proposed method, IOSTOM, is novel, theoretically well-motivated, and demonstrates superior performance. 

The authors' rebuttal has addressed nearly all concerns raised by the reviewers. The initial clarity issues are not significant and the authors have committed to revisions to improve the paper's readability. The authors also provide substantial new experiments, including a full comparison against the requested LobsDICE baseline and an extensive ablation study on the key hyperparameter α. While one reviewer remains concerning about the hyperparameter sensitivity, the overall consensus on the paper's strengths—its novelty, theoretical rigor, and state-of-the-art results—outweigh the remaining concerns.

(e) Summary of Discussion and Rebuttal

The discussion period is very productive. The authors provide detailed responses and new experimental results that addressed the reviewers' main concerns.
- Clarity: In response to Reviewer ahUM, the authors agreed to clarify all missing definitions and derivations. This satisfied the reviewer, who subsequently raised their score.
- Hyperparameter Sensitivity: In response to Reviewers y8cD and 4HqZ, the authors provided a new, comprehensive ablation study across 24 tasks for the hyperparameter α. The results convincingly showed that the method is not overly sensitive to this parameter and that the default value is a robust choice.
- Comparison to LobsDICE: As requested by Reviewer y8cD, the authors ran a full suite of experiments comparing IOSTOM to LobsDICE. The results clearly demonstrated IOSTOM's superior performance.
- Offline Evaluation: Reviewer y8cD questioned the use of environment access for hyperparameter tuning. The authors clarified that this is a standard and widely accepted practice in the field, as fully offline model selection remains an open problem, and that their tuning was minimal and fairly conducted.

Following the rebuttal, three of the four reviewers lean toward acceptance, with two raising their scores. While Reviewer y8cD's final rating remains a borderline reject due to lingering concerns about the hyperparameter sensitivity, their primary technical and experimental questions are successfully addressed. The productive discussion and the authors' thorough response have further solidified that this submission is ready for acceptance.
Decision: Accept (poster)