A Dual Approach to Imitation Learning from Observations with Offline Datasets
Authors: Harshit Sikchi
Caleb Chuck
Amy Zhang
Scott Niekum
Venue: CoRL 2024 (2024)
Keywords: Learning from Observations
Imitation Learning
TL;DR: A dual approach to LfO that is principled, computationally efficient, and empirically performant.

Abstract: Demonstrations are an effective alternative to task specification for learning agents in settings where designing a reward function is difficult. However, demonstrating expert behavior in the action space of the agent becomes unwieldy when robots have complex, unintuitive morphologies. We consider the practical setting where an agent has a dataset of prior interactions with the environment and is provided with observation-only expert demonstrations. Typical learning from observations approaches have required either learning an inverse dynamics model or a discriminator as intermediate steps of training. Errors in these intermediate one-step models compound during downstream policy learning or deployment. We overcome these limitations by directly learning a multi-step utility function that quantifies how each action impacts the agent's divergence from the expert's visitation distribution. Using the principle of duality, we derive DILO (Dual Imitation Learning from Observations), an algorithm that can leverage arbitrary suboptimal data to learn imitating policies without requiring expert actions. DILO reduces the learning from observations problem to that of simply learning an actor and a critic, bearing similar complexity to vanilla offline RL. This allows DILO to gracefully scale to high dimensional observations, and demonstrate improved performance across the board.

A Dual Approach to Imitation Learning from Observations with Offline Datasets
Harshit Sikchi
Caleb Chuck
UT Austin Selkchi@utexas..o edu
UT Austin
calebchuckkutexas..u edu
Amy Zhang UT Austin, Meta Al
Scott Niekum UMass Amherst
Abstract: Demonstrations are an effective alternative to task specification for learning agents in settings where designing a reward function difficult. However, demonstrating expert behavior the action space of the agent becomes unwieldy when robots have complex, unintuitive morphologies. We consider the practical setting where an agent has a dataset of prior interactions with the environment and is provided with observation-onis -nll expert demonstrations. Typical learning from observations approaches have required either learning inverse dynamics model or a discriminator as intermediate steps of training. Errors in these intermediate one-step models compound during downstream policy learning or deployment. We overcome these limitations by directly learning a multi-step utility function that quantifies how each action impacts the agent's 's divergence from the expert's vis- itation distribution. Using the principle of duality, we derive DILO (Dual Imitation Learning from Observations), algorithm that can leverage arbitrary suboptimal data to learn imitating policies without requiring expert actions. DILO reduces the learning from observations problem to that of simply learning an actor and critic, bearing similar complexity to vanilla offline RI. This allows DILO to gracefully scale to high dimensional observations, and demonstrate improved performance across the board. Project page (code and videos): hari-sikehi, github.io/dilo/
Keywords: Learning from Observations, Imitation Learning
Introduction
Imitation Learning [1] promises to leverage few expert demonstrations to train performant agents. This setting is also motivated by literature in behavioral and cognitive sciences [2, 3] that studies how humans learn by imitation, for example when mimicking other humans or watching tutorial videos. While learning from a small number examples is often the motivation, many imitation learning methods [4, 5, 6, 7, 8] typically assume the impractical setting where the learning agent is allowed to interact with the environment as often as needed. We posit that the main reason humans can imitate efficiently is due to their knowledge priors from previous interactions with the environment; humans are able to distill skills from prior interactions to solve a desired task. Examples of expert behavior are commonly available through the ever-increasing curated, multi-robot or cross-embont eroos-embodied, datasets and even through tutorial videos. However, leveraging these expert datasets efficiently presents two challenges: (a) The expert data often comes in the form of observation trajectories lacking action information (e.g. tutorial videos in the observation space of agent, cross-embodiment demonstrations, etc.) (b) The learning agent should be able leverage its collected dataset of environment interactions to mimic expert behavior. This collected dataset may not contain expert transitions and are referred to suboptimal datasets datascts of arbitrary quality. These challenges serve as our key motivation to bring imitation learning closer to these practical settings. We consider
Equal contribution
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Classical
Offline LfO
Dual Imitation from Observation (DILO)
Learn Discriminator/IMM
Input
Output
RU/BC Training
Learn ('(+F)
Extract (+4)))
Offine Interectiono
Expert Observations
Policy
1
(" 2
V' Rt) 10
possible expert
Offine Interactions Expert Observations
Pollcy
Figure 1: DILO Method Overview: Classical offline LIO methods require learning a Discriminator/IDM prior the RL/BC step suffering from compounding errors during training/deployment respectively. DILO directly learns multi-step utility V* (s. x') of transitioning to state in minimizing cumulative divergence with an expert avoiding errors arising due to using learned intermediate models for subsequent optimization.
the setup of offline imitation learning from observations, where the agent has access to an offline dataset of its own action-labeled transitions of arbitrary quality and is provided with potentially few task-relevant expert demonstrations in the form observation trajectories.
Learning from Observations (LfO) has been widely studied [9, 10, 11, 12] in the online setting, where the agent is allowed to interact with the environment. A common denominator across LfO methods is the use of learned one-step models to compensate for missing expert actions. However, in the offline setting learning accurate one-step models with limited data is challenging and can result compounding errors during downstream policy learning. These models have either taken the form of a discriminator to predict single-step expert rewards or Inverse Dynamics Models (IDM) to predict expert actions. Methods that learn a discriminator distinguish the states or state-next states between expert and the suboptimal policy data seek to match visitation distributions of expert and the agent [8, 11, 13, 4, 14]. The learned discriminator serves of policy optimization. In the offline setting, with limited data, the discriminator is susceptible to overfitting and any errors will compound during RL when treating the discriminator as an expert reward function [15]. A negative side-effect of using discriminator-base
reeaad for the next step
distribution matching in LfO is also its reliance on minimizing a proxy upper bound rather than the true objective [11, 13]. The other popular family of algorithms for LfO involves learning an IDM [16, 17], where the agent uses the offline data predict actions from consecutive states and uses it annotate the expert trajectories with actions. The policy is then extracted via behavior cloning on inferred expert actions. Aside from the well-known compounding error issue with behavior cloning (the errors in learned IDM only serve to exacerbate the issue), this approach discards the wealth of recovery behaviors that could be learned from offline datasets to better imitate the expert. Thus, the key question is: Can we derive an efficient, lightweight yet principled off-policy algorithm for learning from observations that (a) learns from offline datasets of arbitrary quality, (b) bypasses the learning of intermediate one-step models, and (c) does not resort to minimizing loose upper bounds?
In this work, we frame Imitation Learning from Observations as modified distribution matching objective between joint state-next state visitations of the agent and expert that enables leveraging off-policy interactions. The distribution matching objective can be written as a convex program with linear constraints. Using the principle of duality, we propose Dual Imitation Learning from Observations or DILO, which converts the distribution matching objective its dual form, exploiting the insight that the state leaks information about missing actions. DILO longer requires knowing expert actions in the agent's action space and instead requires sampling multiple consecutive states in the environment. An overview of our method can be found in Figure 1. DILO presents three key benefits over prior work: (1) DILO is completely off- off-policy and optimizes for exact distribution matching objective without resorting to minimizing upper bounds (2) DILO learns a multi-step utility function quantifying the effect of going to a particular next-state by minimizing long term divergence with the expert's 's visitation distribution, avoiding the compounding errors persistent in methods that learn intermediate one-step models. (3) DILO solves single-player objective, making the learning stable and more performant. Our experimental evaluation suite of MuJoCo [18] environments

---

with offline datasets from D4RL [19] and Robomimic [20] shows that DILO achieves improved performance consistently over the evaluation suite. We demonstrate that DILO scales to image observations seamlessly without extensive hyperparameter tuning. Finally, DILO shows improved real robot performance compared to prior methods, which are observed be more sensitive to the quality of the suboptimal dataset.
2 Related Work
Learning from Observations: Imitation Learning from Observations (LfO) considers the setting where the expert trajectories are available in the form of observations but missing action labels. This setting is more practical as performant algorithms developed for LfO can unlock learning from a plethora of video datasets and develop ways to transfer skills across embodiments. Unfortunately, learning from observations alone has been shown to be provably more difficult compared to the setting where expert actions are available [21]. As result, current methods in LfO restrict themselves to small observation spaces and involve complicated learning algorithms that first train model using offline interaction data to either predict expert actions [22, 11, 23] or learn state-only reward function [11, 13] in the form of a discriminator. This learned model is used for subsequent Behavior Cloning, as BCO [16], or for RL [13, 11]. As a result, prior methods suffer from compounding errors either during training or deployment. The issue of compounding errors in the offline setting with BC approaches or RL with a learned reward function has been investigated theoretically and empirically in prior works [24, 25, 15]. These errors can be fixed with repeated online interaction but can lead to substantially poor performance in the offline setting.
Duality in RL and IL: The duality perspective in reinforcement learning has been explored in the early works of [26, 27] and has gained recent popularity in the form of Dual RL [28, 29] and DICE [30, 31, 32, 33, 34, 35] methods. Dual approaches formulate RI. as convex program under linear constraints and leverage the Lagrangian or the Fenchel Rockefeller duality obtain an unconstrained and principled objective for RL. The appeal of the dual perspective stems from the ability of dual approaches to learn from arbitrary off-policy data without being sensitive to distribution shift or losing sample efficiency as traditional off-policy methods [36, 37]. This behavior is attributed to the fact that dual approaches compute the on-policy policy gradient using off-policy data in contrast to traditional off off-cy methods, which perform Bellman backups uniformly over state space. Duality has been previously leveraged imitation learning from observations [35, 13, 11] by first creating an upper bound to the distribution matching objective of imitation learning such that resembles (return maximization) RL objective and then solving it using dual RL algorithms.
3 Preliminaries
We consider learning agent in Markov Decision Process (MDP) [38, 39] which is defined as tuple: .M- = (S,A, p, R, dod) where S and A denote the state and action spaces respectively, p denotes the transition function with p(s'|s, a) indicating the probability of transitioning from to s' taking action a; A denotes the reward function and E (0, 1) specifies the discount factor. The reinforcement learning objective to obtain a policy π : S - ((AA that maximizes expected return: E,, IΣi- 7'r(s,, J.)]. where we use E, to denote the expectation under the distribution induced by at ~ x(-18)), 8t+1 p(-|st, 0z) and (AA) denotes a probability simplex supported over A. f-divergences define a measure of distance between two probability distributions given by
(P(()) = E p(z) where f is convex function.
Visitation distributions and Dual RL: The visitation distribution in RL is defined as the discounted probability of visiting particular state-action under policy म, i. d (s, = (1 - ())((s)) Ei= P(s s| and uniquely characterizes the policy π that achieves the visita- tion distribution as follows: 7 (als) = Σ. d=.a))) - Our proposed objective is motivated by the recently proposed Dual-V class of Dual RL [28] methods that formulates regularized RL (with conservatism
(s.a)
3

---

parameter and offline visitation distribution d0) as convex program with state-only constraints:
max Bd(s,a))r(s, a)] -DDaaaaa)_DD(s......._ || (s)
d30
s.t EEEA d(s, = =1 (1-1) + - d(s', ps)sss "'ssss"')),, Ys S.
The above objective is constrained and difficult to optimize, but the Lagrangian dual of the above objective presents an unconstrained optimization that results in a performant Dual-RL algorithm.
(Iresc)++ + - v(6) [
min - (1) [v() + a (.)) 13
(2)
where (v) = maxzeR (x+ y) - (x) s 3 O. Our proposed method builds upon and extend this formulation to an action-free LfO setting.
Imitation Learning from Observations: We consider the LfO setting where the expert provides state- only trajectories: DE {[s .f.............f... s]. Our work focuses the offline setting we have access to an offline inferaction data
where in addition to the expert that consists of potentially suboptimal raalleewwe reward-free {s, a, 5'} transitions coming from the learning agent's 's prior interaction with the environments. We denote the offline dataset by d0 consisting of (state, action, next-state} tuples and p(s, a, )') as the corresponding visitation distribution of the offline dataset. Distribution matching techniques aim match the state visitation distribution of the agent that of expert. Although we use 8 as a placeholder for states, the method directly extends to fully-obserrable MDP's where perform visitation distribution matching in the common observation space of expert and agent.
4 Dual Imitation Learning from Observations
Classical offline LfO approaches that rely on learning a discriminator and using it as a psuedoreward for downstream RL are susceptible to discriminator errors compounding over timesteps during value bootstrapping in RL [40, 21, 41, 15]. The discriminator is likely to overfit with limited data especially when expert observations are limited or high dimensional. Methods that Iearn IDM and use behavior cloning (BC) only perform policy learning on expert states and suffer compounding errors during deployment as a result of ignoring the recovery behaviors that can be extracted from offline, even suboptimal datasets [24, 25]. The key idea of the work is to propose objective that directly learns utility function quantifying how state transitions impact the agent's 's long-term divergence from the ex- pert's 's visitation distribution. We derive our method below by first framing LfO as specific visitation distribution matching problem and then leveraging duality to propose an action-free objective.
4.1
LfO as (s, 8') Joint Visitation Distribution Matching
To derive our method, we first note a key observation, also leveraged by some prior works [10], that the next-state encodes the information about missing expert actions as the next-state a stochastic function of the current state and action. We instantiate this insight in the form of distribution matching objective. We define {s, s'} joint visitation distributions denoted by d (s, s', a') = (1
(a'((')s)) Ee 7'p(#1+1
8',&t 81 s/Tr.. Intuitively, it extends the definition of state- action visitation distribution by denoting the discounted probability of reaching the {state, next-state) pair under policy and subsequently taking action a'. Under this instantiation, the LfO problem
reduces to finding a solution of:
min Dj (s, (s, (s)
as at convergence, d' d' s', s'.'s''') d' s', a') holds, which implies (s, = d= s') and also d' (s) = (s) by marginalizing distributions. Unfortunately, the above objective (a) requires computing an on-policy visitation distribution of current policy (d=) (b) provides no mechanism to incorporate offline interaction data (d"), and (c) requires knowing expert actions in the action space of the agent (a').

---

4.2 DILO: Leveraging Action-free Offline Interactions for Imitating Expert Observations
We now show how framing imitation (Eq. 3) as a constrained optimization objective w.r.t visitation distributions allows us to derive an action-free objective. First, in order to leverage offline interaction data p. we consider a surrogate convex mixture distribution matching objective with linear constraints:
max - (M(Mix p) d>0
8.1 Ea ((("',"",a")) = (1 - dod( ( 88)) + Es. d(s, (' V8'," E x S.
The constraints above represent the Bellman flow conditions any valid joint visitation distribution needs to satisfy. The mixture distribution matching objective preserves the fixed point of optimization d' (s, s', a') = d (s, s', a') irrespective of mixing parameter B3 thus serving as a principled objective for LfO. Mixture distribution matching has been shown to theoretically and practically effective way [28, 42] of leveraging off-policy data. Prior works [28, 42] dealing with state-action visitation in the context of imitation learning consider overconstrained objective resulting in complex min-max optimization. Our work departs by choosing constraints that are necessary and sufficient while giving us a dual objective that is action-free as well as simpler single-player optimization. The constrained objective is convex with linear constraints. An application of Lagrangian duality to the primal objective results in the following unconstrained dual objective we refer to as DILO:
DILO:
min 8(1 - 7) E ((s, s') + E, 8.5' Max (d p) 1 (7 [Vis', ( s') [V(ss
V(s,
- -(1 - 3) - [T a
- v(),
(5)
where V is the Lagrange dual variable defined as V : 8 R and f is variant of conjugate We derive DILO objective as Theorem 6.1 Appendix 6.1 where we also see that strong duality holds and the dual objective can recover the same optimal policy with the added benefit of being action-free. Moreover, we show that the solution to the dual objective in Equation 5, V*(s,s') represents the discounted utility of transitioning to a state from s' under the optimal imitating policy that minimizes the f-divergence with the expert visitation [43] (Appendix 6.1.)). Intuitively, this holds as the primal objective in Eq 4 can be rewritten as the reward maximization problem Exx [r(s, s', a')] with
f defined as f (x) = max(0, f (=)))) - f(
r(s, s', d') = (ix, pp) (ixe(dd",)) This reward function can be thought of as penalizing the
(xxa(d,p) f( (ixz(d,e)
policy every time it takes action leading to different next state-action than the expert's implied policy in agent's 's action space.
An empirical estimator for the DILO objective in Eq. 5 only requires sampling 8, 8, 8 under a mixture offline dataset and expert dataset and no longer requires knowing any of the actions that induced those transitions. This establishes DILO principled action-free free alternative optimizing the occupancy matching objective for offline settings.
4.3 Policy Extraction and Practical Algorithm
To instantiate our algorithm, we use the Pearson Chi-square divergence (f(x) (x (x-11 (x)) which
has been found to lead to stable DICE and Dual-RL. algorithms in the past [7]. With the Pearson chi-
square divergence, f takes the form (x) = x * (max (5 + 1). .0) ((max (5 + 1) 0) - 1)²
We outline the intuition of the resulting objective after substituting Pearson chi-square divergence in Appendix 1.4.4.
At convergence, the DILO objective does not directly give us the optimal policy x* but rather provides us with a utility function (*(ss s') that quantifies the utility of transitioning to state s' from s in visitation distribution matching. To recover the policy, we use value-weighted regression on the offline interaction dataset, which has been shown [44, 45, 46] to provably maximize the V function (thus taking action to minimize divergence with expert's visitation) while subject to distribution constraint of offline dataset:
[(bb) Ee,,,'' TVV (s,5)) log म. (a|s)
(6)
5

---

Choice of s')s Any distribution over state and next-state is implicitly dependent on the policy that induces the next-state. The initial distribution in Eq. forms the dis- tribution over states from which the learned policy will acquire effective imitation be- our work, we set do(s, to the uniform distri- bution over replay buffer {s,s'} pairs, ensuring that the learned policy is robust enough to imitate from any starting transition observed from all the transitions available to us. Practical optimization difficulty of dual objectives: Prior works in reinforcement learning that have lever- aged a dual objective based on Bellman-flow constraints suffer from learning instabilities under gradient descent. Intuitively, in our case, learning instability arises as the gradients from ss) and V(s', ((s','' s") can conflict if the network learns similar feature representations for nearby for t 1..2' iterations
havior to mimic the expert.
Algorithm 1: DILO
1: Init Vo,
Params: temperature т, mixture ratio 8 Let D أ {(s a, s')] an offline dataset and D {s, be expert
demonstrations dataset.
states due to feature co-adaptation [47]. Prior works [28] have resorted to using semi-gradient approaches but do not converge provably to the optimal solution [29]. To
Train V via Orthogonal gradient
update on Eq. 5
6:
Update Te by minimizing Eq. 6
end for
sidestep this issue, we leverage the orthogonal gradient update proposed by ODICE [29] for the offline RL setting that fixes the conflicting gradient by combining the projection of the gradient of V (ss',")) s")) on (ss, s') and the orthogonal component in principled manner. We refer to the ODICE work for detailed exposition. Our complete practical algorithm can be found in Algorithm
Experiments
In our experiments, first, we aim to understand where the prior LfO methods based on IDM or discriminator fail and how the performance of DILO compares to baselines under a diverse set of datasets. Our experiments with proprioceptive observations consider an extensive set of 24 datasets. The environments span locomotion and manipulation tasks, containing complex tasks such as 24-DoF dextrous manipulation. Second, we examine the simplicity of DILO objective indeed enables it to set of
scale directly mimic expert image observation trajectories. Finally, we test method real-robot manipulation tasks where consider learning from a few expert observations generated by human teleoperation as well as cross-embodied demos demonstrated by humans as videos.
5.1 Offline Imitation from Observation Benchmarking
We use offline imitation benchmark task from [28, 13] where the datasets are sourced from D4RL [19, 18]. For locomotion tasks, the benchmark uses an offline interaction dataset consisting of 1-million transitions from random or medium datasets mixed with 200 expert trajectories (or 30 expert trajectory in the few-expert setting). For manipulation environments, the suboptimal datasets comprise of 30 expert trajectories mixed with human or cloned datasets from D4RL. The expert demonstrates observation trajectory for all tasks. DILO uses single set of hyperparameters across all environments. Hyperparameters and additional experimental details be found in Appendix 6.2.1.
Baselines: We compare DILO against offline imitation from observations (LfO) methods such as ORIL [48], SMODICE [13] as well as offline imitation from action-labelled demonstration (LfD) methods like BC [49], IQ-Learn [50] and ReCOIL [28]. We choose these imitation learning methods as they represent the frontier of the Lf0 and LiD setting, outperforming methods like ValueDICE [42] and DemoDICE [34] as shown in prior works. Intuitively, the imitation from action-labeled demonstrations represents the upper bound of performance they have additional information on expert actions even though sometimes observe LiO algorithms surpass them in performance. ORIL and SMODICE first learn discriminator and, subsequently run downstream RL treating the discriminator as the expert rewudd-reward.
Table shows the cumulative return of different algorithms under the ground truth expert reward function that unavailable the learning agent during training. DILO demonstrates improved performance across wide range of datasets. Particularly in the setting of few-expert observations or high dimensional observations like dextrous manipulation, the performance of methods relying

---

Accrss expert action
No
Papert
Suboptimal Dassvrrt
Ernw
RCE
BC
BC
1Q-Laorm
ReCDEL
ORIL
SMODICE
DEUO
eaper dain
fell detsett
(edfine)
randome
hulkkhecihh
lopper
64.19:11.06
51,41::58.3
4.52=1.42
23:0.01
2.25-0.00
5664:435
1.85 42.19
4.8347.99
47.65-1695
108. 18:325
75,21+21:90
60,49:3.53
100.4610.644
85.16-3.62
91.18-0..4
97.57-9.11
111.33
88,83
expert
walker25
20.00:26.00
0.36-0.51
0.9100.5
0.57+0.09
102. 16+7.19
27.10--23.9
108.4 -0.47
10NA2-14 0.64
106.92
ard
5.172543
90.60:
42.25:30000
19:277700 70000
122.56:14.47
122.15
130.75
random+
hillcheeiah
hopper
25.31+18.99
2997:107
4.34-3.33
0.92+0.35
2:24-0001
3.0-0.54
1:14-1994
76992-7.55
29.86-122.00
25,76-9.52
4.10-125 1.50
3.00
52.32-1
93.73-7.89
111.33
expert
walker24
40.89:36.52
0.743:0:20
0
19.00
3.22
107.18:187
10842 0.25
100.92
15.02415.81
0.99
67.14
56.53 9.37
8.89-139.12
117:50:40
130.75
mediums
hallcheeiahh
34.01
13.822
16.09-1223 12:00
1.794022
62 0.42
25:33.31
12.90+24000
25 67 2.203322033
98.51
81.15-286
14,15 18.24
66.2 65.28-7..77
5691- 4.0
90.47 0..47-0.64
12.62
131.33
88,03
expert
waller2d
148.38 58 59
14.04
1.4
0.44
95
72.70 72.7633.82
56.571130.11 10.14
57. 15.12.41.15
108 14
120 12036617677
28 22
14. 1490
1050013344
2:11:241
17.16.66.6
10289-3.49
106.92
medium
hellcheetah
hopper
63. 19.33111.31
66. 15:35113116
1. 1.154006
1,3
058745.31
42.211005
11,05
36.27230244
75.664455
1001=110.3
59, 34314
14.2
4461:6608
46 46.662955
41.90:14811
6..71:6.35
111.33
59,83
few few-espert
walker28
85 $4.90
2.02
7042222 86
73 83301288
11225-1761
6.8115766
6,026.09
106.92
and
67
67.95.3667
10
$1.6115.77
35 17250.56
10.96
67. 18.5
98.03.9.01
130.55
pea
19:60=11.40
13.05:11004
34.9
2.15
45
0.92
13:29=13.57
101.3633.40
106,42
cloeedeepper
hammer
door
0.082 0.15
1.95.389
0:2211.00
2.4
001110000
5452
00010001
0.270000
95
101. 00.2551:000
0,26 0.01
01110001
0/001 0.10
1 2.25
112.58_22.10
005.0000.38
100.9
125.71
pea
+5.81:55.11
13.83=11176
90.76:25.00
1420422852
213-72422.0
3.7643
-3.33-0:12
:8.61:14.30
106.42
hamaneexpeer
hammer
0.05-0055
5.0055.64
0.39:0.14
103.71+1.22
122.61444.85
0.32+1138
3.647.29
125. 125.199
144.0000.55
0.32+0011
3.1140.04
0,40-0.45
117.52-8.55
101.51-0.99
100.94
125.71
patiileespert
kitchen
6.775-9.24
2:5:50
15.5
0.00000
60.0:5570
0
35.0:1 4.06
43.0-5.30
75.0
misedrexpert
kitchen
1.66
22:338
42.18:112
0.0:0000
0,00:00.00
48.35:
44.9:5.39
Table 1: The normalized return obtained by different offline IL (both provided with and without expert actions) methods on the D4RL suboptimal datasets with expert trajectory. The mean and std are obtained over random seeds. Methods with statistically significant improvement (1-test) over second best method are highlighted.
on learned discriminator falls sharply potentially due to overfitting of discrimination that results in compounding downstream errors. DILO gets rid of this intermediate step, completely reducing the problem of LfO to similar training setup as traditional actor-critic algorithm. BC methods, representing an upper bound to BCO [16] shows poor performance even without learned IDM.
5.2 Imitating from Expert Image Observations
Learning to mimic expert in the image observation space presents a difficult problem, especially in the absence of a pretrained representations. To evaluate our algorithm in this setting, we consider the Robomimic datasets [20] which gives the flexibility of choice to use image observations or the corresponding proprioceptive states for learning. Our suboptimal datasets comprises of Multi-Human (MH), Machine Generated (MG) datasets from Robomimic without access expert trajectories. We obtain 50 expert-observation trajectories from Proficient Human (PH) datasets. This setup is more complicated as the agent has to learn expert actions purely from OOD datasets and match expert visitations. We consider the most performant LfO baseline from the previous section SMODICE [13] along with a BCO [16] baseline as BCO has shown success in scaling up to image observations [51].
Fig 2 shows the result of these approaches on 4- different datasets using both state and image observations. SMODICE shows competitive results when learning from state-observations but does not scale up well to images likely due to the overfitting of the discriminator in a high-dimensional observation space. BCO fails consistently across both state and image exper- iments as learning an IDM is challenging in this contact-rich task, and any mistake by IDM can compound. DILO outperforms baselines and demonstrates improved performance across both state and image observations.
Lin-MG
LiftMH
Can-MGi
Can-MH
State Demos
BCO
0.00 0.000
0.00 0.00
0.00 0.00
0.0 0+ 0.00
SMODICE
0.41 :
0.46
0.54 0
0.28:
DILO
0.59 1 umo
0.97 1 9m2
0.53 : a,2
0.64 : am
Image
BCO
0.00 0.00
0.00 +.00
0.00 0.00
0.00 0.00
SMODICE
0.21 1 a.m
1.112
0.10
0.02 0.01
S0
DILO
0.76 : tus
0.94 a 902
0,25 0,82
0.15 : am
Figure 2: Side sidde comparison of LfO methods on state-only imitation vs image-only imitation. DILO shows noticeable improvement over existing LfO meth- ods without hyperparameter tuning. Columns denote different suboptimal datasets.
5.3 Imitating from Human Trajectories for Robot Manipulation
Setup: Our setup utilizes a UR5e Robotic Arm on tilted 1.93m x 0.76m Wind Chill air hockey table to hit a puck or manipulate tabletop objects. Puck detection utilizes an overhead camera, with additional environment details in Appendix 6.3. The set of tasks in this domain is designed to stress both 1) challenging inverse dynamics through complex striking motions and 2) partial state coverage
7

---

through the wide variety of possible paddle x puck positions and velocities. While baselines can struggle with compounding errors one or both of these settings DILO's ability to side-step learning one-step models allow it to scale gracefully to these complexities. Tasks and Datasets: We consider three tasks and datasets for real-world experiments. Our tasks are:
Obbyect Manigulttian
Puck-Striking expert
BCO
SMODICE
DILO
Few Trajectudess Fixed Sturt Few Uniferm
expest
BCO SMODICE DILO
3/10
7/10
6/0
7/11
4/11
2/10
1/10
010
5/11
4/11
8/10
9/10
5110
5/11
Safe Object Manipulation (Cross- Embodiieet)
Dynomic Pack liiting
Few Trajectntes Fixed Start Few Uniform
400 expert (trach)
400 expert
(hintin)
DCO
6/10
6/10
810
2/10
0/10
SMUDICE
1/10
1/10
1710
6/10
4/10
DILO
8/10
9/10
$/00
10/10
W/1
Figure 3: Real Robot Experiments: Table shows the (x/y) success rates x successes y trials for differ- ent methods on real-robot setup air-hockey. For the dynamic back-hitting task, we evaluate the number of touches made in addition to hitting behavior, which across algorithms: Puck's (red) gradient shows move- returns the puck in the opposite direction.
Figure 4: Example of learned hitting behavior
across time for Dynamics Puck Hitting.
1) Safe Objective Manipulation: Navigate object safely the goal without hitting obstacles. 2) Puck Striking: Hit a stationary puck 3) Dynamic Puck Hitting: A challenging task of hitting dynamically moving puck. For the safe manipulation task, we investigate three datasets a) Few-Trajectories: 15 expert trajectory observations are given with uniform initial state b) seeed-staat-trjecoorre expert observation trajectories are provided to the agent with fixed start state. c) Few Uniform: 300 transitions are provided to the agent uniformly in state space. For Puck Striking tasks, we consider two observation datasets, one with 20 experts and the other with 10 experts. For Dynamic Puck hitting, we consider a dataset of 400 expert trajectories. The suboptimal datasets for all tasks contain the same amount of transitions the expert dataset containing mix of successes and failures. The datasets for all tasks are obtained by teleoperation setup by humans, except for the cross-embodiment tasks where the humans demonstrate using their hands, and the state is detected using an overhead camera.
15
Analysis: Fig. compares the success rate of Learning from Observation algorithms in settings with varying dynamics. Safe Object Manipulation presents a task with easy inverse dynamics modeling since the arm restricts its motion to move through the workspace. Consequently, BCO performs well when provided with good coverage of expert observations (few-uniform), but is still outperformed by DILO result of ignoring offline datasets to learn recovery behaviors. SMODICE shows poor performance consistently in tasks with small datasets i.c. poor coverage. Puck striking presents both easy inverse dynamics and good state coverage, which may explain the comparable performance from BCO and SMODICE against DILO. On the other hand, Dynamic Puck Hitting is challenging both for inverse dynamies, because of the wide range of actions necessary to hit a moving puck, and for state coverage, where the range of possible paddle and puck positions is substantial. Fig. demonstrates example of learned puck hiting behavior. DILO handles both complexities gracefully, resulting in an impressive success rate over both baselines.
6 Conclusion
Offline Imitation from Observations provides solution for fast adaptation of the agent a variety of expert behaviors agnostic of the agent's action space. In this work, we propose a principled, computationally efficient, and empirically performant solution to this problem. Our work frames the problem as a particular ietrrbbttion-macciin objective capable of leveraging offline data. Using the principle of duality under a well-chosen but sufficient set of constraints, we derive an action-free objective whose training computational complexity is similar to an efficient offline RL algorithm. We show that the proposed method shows improved performance across wide range of simulated and real datasets, learning from proprioceptive image observations and cross-embodied expert demonstrations.

---

Acknowledgements
We thank Siddhant Agarwal, Chang Shi, Carl Qi, Max Rudolph, Haoran Xu and Shuozhe Li for valuable comments on this work. This work has taken place part the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst and Machine Intelligence and Decision Making Lab (MIDI) at The University of Texas. SCALAR research is supported in part by the NSF (IIS (IS-2323384), AFOSR -EA9550-20-1-00 (CAIS), and the Long-Term Future Fund. HS and A7. are funded by NSF 2340651. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, the Army Research Office or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
the Center for AI Safety
References
[1] S. Schaal. imitation learning the route to humanoid robots? Trends cognitive sciences, (6):233-242 1999.
[2] S. Vogt and R. Thomaschke. From visuo-motor interactions to imitation learning: behavioural and brain imaging studies. Journal of sports sciences, 25(5):497-511 2007.
[3] R. W. Byrne and A. E. Russon. Learning by imitation: A hierarchical approach. Behavioral and brain sciences, 21 2165)6677-684 1998.
[4] T. Ni, H. Sikchi, Y. Wang, T. Gupta, Lee, and B. Eysenbach. f-irl: learning via state marginal matching. In Conference on Robot Learning, pages 529-551. PMLR, 2021.
Inverse reinforcement
[5] C. Li, M. Vlastelica, Blaes, J. Frey, Grimminger, and Martius. Learning agile skills via adversarial imitation of rough partial demonstrations. In Conference on Robot Learning, pages 342-352. PMLR, 2023.
[6] H. Sikchi, A. Saran, W. Goo, and Niekum. ranking game for imitation learning. arXiv preprint arXiv:2202.0344:: 034811 2022.
[7] Al-Hafez, D. Tateo, O. Arenz, G. Zhao, and J. Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. arXiv preprint (iX.i:23333000999 2023.
[8] S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pages 1259-1277. 1199-1277.PMLR, 2020.
[9] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems,
2016.
[10] Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 1807.01807.0111 80..0000.8..... 2018.
[11] Z. Zhu, K. Lin, B. Dai, and Zhou. Off-policy imitation learning from observations. Advances Neural Information Processing Systems, 33:12402-122 12413, 2020.
[12] H. Hoshino, K. Ota, A. Kanezaki, and R. Yokota. Opirl: Sample efficient off-policy inverse reinforcement learning via distribution matching. In 2022 International Conference on Robotics and Automation (ICRA), pages 458-454. IEEE, 2022.
[13] Y. J. Ma, A. Shen, D. Jayaraman, and O. Bastani. Smodice: Versatile offline imitation learning via state occupancy matching, arXiv preprint axXiv:2202.024333 2022.
[14] L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 17X5::1906.05274, 2019.

---

[15] H. Sikchi, R. Chitnis, A. Touati, A. Geramifard, A. Zhang, and S. Nickum. Score models for offline goal-conditioned reinforcement learning. arXiv preprint arXiv:2311.02013 2023.
[16] Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint arXiv:/805.01994 2018.
[17] A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell. Imitating latent policies from observation.
In International Conference on Machine Learning, pages 1755-1763. PMLR, 2019.
[18] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[19] J. Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.0722 2020.
[20] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín- What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.0322 2021.
[21] R. Kidambi, J. Chang, and W. Sun. Mobile: Model-based imitation learning from observation
alone. Advances in Neural Information Processing Systems, 34, 2021.
[22] F. Torabi, G. Warnell, and P. Stone. Adversarial imitation learning from state-only demon- strations. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2229 2231, 2019.
[23] W. Sun, A. Vemula, B. Boots, and D. Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning, pages 6036-6045, PMLR. 2019.
[24] S. Ross, G. Gordon, and D. Bagnell. reduction imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artifi- cial intelligence and statistics, pages 627-635, JMLR Workshop and Conference Proceedings. 2011.
[25] G. Swamy, S. Choudbury, J. A. Bagnell, and S. Wu. Of moments and matching: A game- theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10032. PMLR, 2021.
[26] A. S. Manne. Linear programming and sequential decisions. Management Science, 6(3): 259-267, 1960.
[27] E. Denardo. On linear programming in markov decision problem. Management Science, (6(5):281-288 1970.
Unification and new methods for [28] H. Sikchi, Q. Zheng, A. Zhang, and S. Niekum. Dual rl: reinforcement and imitation learning. In Sixteenth European Workshop on Reinforcement Learning, 2023.
[29] L. Mao, H. Xu, W. Zhang, and X. Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update. arXiv preprint 2400002402.00348, 2024.
[30] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:19/2.02074, 2019.
[31] O. Nachum, Y. Chow, Dai, and L. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
10

---

[32] J. Lee, W. Jeon, B. Lee, J. Pineau, and K.-E. Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-61330 6130. PMLR, 2021.
[33] R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072. 2020.
[34] G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022.
[35] G.-H. Kim, J. Lee, Y. Jang. H. Yang, and K.-E. Kim. Lobsdice: Offline learning from observation via stationary distribution correction estimation. Advances in Neural Information Processing Systems, 35:8252-8264, 2022.
[36] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[37] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[38] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[39] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction, MIT press, 2018.
[40] Z. Liu, Y. Zhang, Z. Fu, Z. Yang, and Z. Wang. Provably efficient generative adversarial imitation learning for online and offline setting with linear function approximation. arXiv preprint arXiv:2108) 108 08765, 2021.
[41] B. DeMoss, P. Duckworth, N. Hawes, and I. Posner. Ditto: Offline imitation learning with world models. arXiv preprint arXiv:2302.03086, 2023.
[42] I. Kostrikov, O. Nachum, and Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXiv; 1912.0039,, 2019.
[43] O. Nachum and B. Dai. Reinforcement learning via femchel-rockafellar duality. arXiv preprint arXiv:2001.01866. 2020.
[44] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2/ 10.06/69, 2021.
[45] A. Nair, A. Gupta, Dalal, and Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0933: 09359, 2020.
[46] H. Sikchi, W. Zhou, and D. Held. Learning off-policy with online planning. Conference on Robot Learning, pages 1622-1633. PMLR, 2022.
[47] A. Kumar, R. Agarwal, T. Ma. A. Courville, G. Tucker, and S. Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint arXiv:21 2021.
[48] K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang. Y. Aytar, M. Denil, N. de Freitas, and S. Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:201 13885, 2020.
[49] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
[50] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-4030 4039,, 2021.
11

---

[51] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24465
2022.
[52] S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[53] B. Eysenbach, Levine, and R. Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification. Advances in Neural Information Processing Systems, 34: 1541-11552, 2021.
[54] Fujimoto and S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
[55] C. Chuck, C. Qi, M. J. Munje, S. Li, M. Rudolph, C. Shi, S. Agarwal, H. Sikchi, A. Peri, S. Dayal, et al. Robot air hockey: A manipulation testbed for robot learning with reinforcement learning, arViv preprint TrXiv:2405.0311 2024.
[56] G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
12

---

Appendix
6.1 Theory
6.1.1
Derivation for Action-free distribution matching
Theorem 6.1. The dual problem to the primal occupancy matching objective (Equation 4) is given by the DILO objective in Equation 5. Moreover, as strong duality holds from Slater's 's conditions the primal and dual share the same optimal solution d* for any offline transition distribution and any choice of mixture distribution ratio B.
We start with the primal objective that matches distributions between the agent's 's visitation d(s, s', a') and expert's visitation d (s, s', a'). As before denotes the visitation distribution of offline data. min D (Mix. (s, s','s,,,,,,,, ())),))((Mii)) M) (s. aa), p)).
(7)
where for any two distributions μ1 and Pz, Mixs( (M1 12) denotes the mixture distribution with coefficient 3 (0, 1] defined as ( 12) 3pm + (1 P2
Formulating the objective as a constrained objective in agent's 's visitation distribution allows us to create a primal objective that convex program. This is erucial in subsequently creating a dual
objective that is unconstrained and easy optimize.
max D, (Mix, p) Mixx(dd d
s.t E.. (1- + Eno'EsxA d(s, d')p(s" As', s" E S x S.
(8)
where the constraints above dictate the conditions that any valid visitation distribution d(s', s") needs to satisfy and are our proposed modifications to the commonly known bellman flow constraints.
Below we outline the derivation of how these specific constraints with the mixture distribution matching objective allows us to create dual objective that is independent of expert's s actions. Applying Lagrangian duality to the above constrained distribution matching objective, can convert it unconstrained problem with dual variables V(s, 8') defined for all 8, s' E S S:
max min (Mi (d. (s, s', a') || Mix p)) d30 V(8',"")
Σ v(s',s") (1 (1-
(''ss "d(s(
(9)
max min (1 - B) (8,888) + Ee, -d p( d'V(s - d30 ((s,s)
Dy (Mix (s s a') Mi (s, ,d))
(10)
where the last equation uses change of variable from s', 8" to 8, 8' without loss of generality. Using a simple algebraic manipulation below, we can get rid of the inner maximization. We add and subtract the terms shown below:
max min 3(1 - (1)-7)1E ((s,, d=0 V(r,s))
BE ad
-V(s,s)
+(1 B)E,
V(s,s')
-(1 B)
a'd -V(s,s))
- (M, (d
(11)
13

---

As strong duality holds using Slater's conditions [52] (see [43] for a detailed account of strong duality in RI. under visitation distributions). Using the fact that strong duality holds this problem we can swap the inner max and min and rewrite an equivalent maximization under the mixture distribution:
min V(s,s") Miza (e,e), of))
max
8(1 (L)-))Baots Boo(s, (s(sss))
Bl aadad ? E
V((,,) [
(1 -B)) a Σ p(s" D'ss" -v(ss''))))))))
V(s,5')
(1 B) at
a''"'sss'''V((ss'"""s) V(s,s')
D, (Mixg( (s, Mixx(d" (s. s',a')
(12)
In the following derivation, will show that the inner maximization in Eq 12 has closed form solu- tion even when adhering to the non-negativyy constraints. Let y(s, s', d') = E - ay [V(s', s"),s 88,
? d')(sss" V(s.s')
max (f a0)
Dy(Mixz(d, () 8, ,') Mix () dd)))
Now to solve this constrained optimization problem create the Lagrangian dual and study the KKT (Karush-Kautn Juhn--Tucker) conditions. Let w(s, s', a') A Mixa(d,pl(s,s'aa" Mix (), s', d') holds if and only if t',s,s', d') Vs,s',
Nixa(d" "')) then the constraint
max max B,,a', add s "'.ss''d)))ss....... ()) w(s,6', .a') A
Σ aw's -0)
(13)
susiaa
Since strong duality holds, use the KKT constraints to find the solutions w (s, and X* (s, s', s'a)).
Primal feasibility: w* a') 2 0 Va,a',a'
Dual feasibility: X 3 V s',,'a''
Stationarity: Mi p) . (dl + v(s.s',a') + (*
0 V s, s',
Complementary Slackness:
Using stationarity we have the following:
(s, a')) = y(s,s', a') + X* (s, Vs,s'd''
(14)
Now using complementary slackness, only two cases are possible w² (s, d') > Oor (s,8',a'))
Combining both cases arrive at the following solution for this constrained optimization:
** (s, s' a') = max f (y(s,
(
(15)
Using the optimal closed-fom solution (w*) for the inner optimization in Eq. (12) obtain
14

---

min 8(1 - (d) [V(s, V(a,a")
E "f.''.''Mixa (d ol [max (0, ( (u(s, ( ,')) ((), of (max (0, s',
-11-8)Bs,ae [ -(,,s')))))))))
(16)
For deterministic dynamics, this reduces to the following simplified objective:
min 8(1 - Bdo V(s.s))
[V(s,')] (Vs,,))
+ Es,s' a' "00, mmax (f') (u(s,s',a ( u(s,'','') (max (0. (
(1- 1-- -'sss(s))s)) {
(17)
where y(s, a, = (s', V(s,
6.1.2 What does the utility function (* s') represent?
Prior work [43] shows that for the regularized RL problem
Ed(s,a)[r(s - aD)(d(s,a) || (s,
d30 s.t EMEA d(s,a) = - 7)))))(()) + E(s d(s''d'sssse d')p(sls',,)) a'), Vs
(18)
the dual optimizes for a Langrangian variable V that represents regularized optimal value function. This insight directly extends to our work with reward function set to zero, our Lagrangian variable learns only the regularized visitation probabilities under optimal policy.
It is easy to why this is the case using the previous derivation. Following the derivation from the previous section, note that had rewritten the inner maximization w.r.t the visitation distribution d, thus effectively getting rid of manipulating visitation distributions in the final objective. Our derivation above uses the following substitution shown in Eq 15 that holds as part of the closed form solution w.r.t inner maximization:
Mix p)s,, max (vsss (ix R p)s
(19)
where y (V(S's V(s, s¹). For deterministic dynamics, convergence, the following holds for all 8, a' where d (s, 8,, a') 0:
Mixx (d* (s, Mix (d pp) (8)
(V -V"(,"")
(20)
implying:
(Mixx (dt ( Mixx(dE (dE (3,
(21)
The above relation makes the the interpretation of (*(8, 8') clear. (V*(s, (5) (s' de- notes the implied reward function (s, s', a') under which V computes the maximum cumulative expected return, where a' the action that leads to s". As shown above the the implied reward function (s, s',')) ff' "ix (aa(a,',,a)) ("...""", ()). the divergence between expert stationary visita-
tion distribution and agent's stationary visitation that is obtained after taking the action a' from s' and then acting optimally match the expert visitation distribution. Note that the function f is non-decreasing the function f is convex from definition of f-divergences.
15

---

6.1.3 Analytical form of fz for x² divergence
For x² divergence, the generator function f(z) = (x- 1)2. f'(x) 2(x 1) and correspondingly
(')(() Substituting (a) in definition of fz.
(x) max(0, (=)) - f(max((0
(x)))
(22)
Since a we substitute takes the form of residual residual = >b (V(( ((s,5))) the below pseudocode shows the implementation of f for DILO.
def f f_star_p (self, residual, type= "hi_e"cchsuuuee)
if type== squuuurrr':
omar torch.man (residual 2 1, torch. zeros_ residual))
return residual meega_star (omega_ star +*2
6.1.4 Intuitive understanding of DILO
To better understand this objective's s behavior we consider the last two terms from Eq 5 its expanded form below. We ignore the first states, prevent overestimation when learning from offine datasets.
simply pushing down Q-values initial distribution of
3E 8,d' 3" a0 V(s, + (1- - 3) * Es,st 5" ( - V(s,
(23)
-(1 - B-,,, a pt pt (V('' V(s, 8)}
Denote r(s, ((sss)) - s") as the implicit expert reward of under a learned Q- function. The objective presents a clear intuition when we study the objective's behavior different
situations individually: (a) For samples from p. the objective pushes down the implicit reward 0 as shown below:
if (1- B)r otherwise.
min [(r)
(24)
(b) For samples from the expert distribution d=, the objective ensures that reward is greater than equal 2
3( ( rf 2,
min [(r)
(25)
10 otherwise.
becomes clear now that DILO is implicitly learning valid reward function that ensures higher discounted return for the expert compared the suboptimal dataset by shaping Q-values directly.
6.2
Implementation
The algorithm for DILO can be found in Algorithm 1. We base the DILO implementation the official implementation of pytorch-IQL that is based IQL [44]. We keep the same network architecture as the original code and do not vary it across environments.
6.2.1 Imitation Learning with Proprioceptive Observations
Our experiment design is based the benchmark from [13, 28] but explain the setup here for completeness.
Environments: For the offline imitation learning experiments we focus on locomotion and manipulation environments from the MuJoCo physics engine [18] comprising Hopper, Walker2d, HalfCheetah, Ant, Kitchen, Pen, Door and Hammer make total of 24 datasets. The MuJoCo environments used in this work are licensed under CC BY 4.0 and the datasets used from D4RL are also licensed under Apache 2.0.
16

---

Suboptimal Datasets: We use the offline imitation learning benchmark from [28] that utilizes offline datasets consisting of environment interactions from the D4RL, framework [19]. Specif- ically, suboptimal datasets are constructed following the composition protocol introduced in SMODICE [13]. The suboptimal datasets, denoted as 'random+expert', 'random+few-epprrt', "medium+exper'' and "medium-ffw-expert combine expert trajectories with low-quality trajectories obtained from the "random-v2" and "medium-v2" datasets, respectively. For locomotion tasks, the rrndoom/meduum+expent dataset contains mixture of some number of expert trajectories (< 200) and = I million transitions from the "x" dataset. The "x*few-expert" dataset is similar to 'x+expert, but with only 30 expert trajectories included. For manipulation environments we consider only 30 expert trajectories mixed with the complete 'x' dataset of transitions obtained from D4RL. For the expert observation dataset we just expert observation trajectory with length as the horizon of environment for our experiments. The detailed suboptimal dataset composition for different datasets can be found in table 2 below:
Dataset
Data Points
All random+exppet
Im random transitions + 200 expert trajectories (horizon=
All medium+expert
Im medium transitions 200 expert trajectories (horizon= 1000)
All random=few-expet
Im random transitions + 30 expert trajectories (horizon=1000)
All mediun-few-eepent
Im medium transitions + 30 expert trajectories (horizon=1000)
Pen cloned+expert
5c6 cloned transitions + 30 expert trajectories (horizon= 100)
Pen human+expert
5000 human transitions + 30 expert trajectories (horizon=100)
Door cloned + expert
Im cloned transitions + 30 expert trajectories (horizon= 100)
Door human expert
6729 human transitions + 30 expert trajectories (horizon= 100)
Hammer cloned expert
Im cloned transitions + 30 expert trajectories (horizon=
Hammer human expert
11310 human transitions 30 expert trajectories (horizon= 100)
partial+expert
136950 partial transitions 30 expert trajectories (horizon=280)
mixed + expert
136950 partial transitions expert trajectories (horizon=280)
Table 2: Suboptimal dataset composition
Expert Observation Dataset: To enable imitation learning from observation, we use I expert observation trajectory obtained from the "expert-v2" dataset for each respective environment.
Baselines: To benchmark and analyze the performance of our proposed methods for offline imitation learning with suboptimal data, we consider different representative baselines in this work: BC [49]. SMODICE [13], RCE [53], ORIL [48], IQLearn [50], ReCOIL [28]. SMODICE has been shown to be competitive [13] to DEMODICE [34] and hence we exclude it from comparison. SMODICE an imitation learning method based on the dual framework, that optimizes an upper bound to the true imitation objective. ORIL adapts generative adversarial imitation learning (GAIL) [9] algorithm to the offline setting, employing an offline RI. algorithm for policy optimization. The RCE baseline combines RCE, an online example- based RL method proposed by Eysenbach al. [53]. RCE also uses a recursive discriminator to test the proximity of the policy visitations to successful examples. [53], with TD3-BC [54]. Both ORIL and RCE utilize a state-based discriminator similar to SMODICE, and TD3-BC serves as the offline RL. algorithm. All the compared approaches only have access to the expert state-action trajectory.
The open-source implementations of the baselines SMODICE, RCE, and ORIL provided by the authors [13] are employed in our experiments. We use the hyperparameters provided by the authors, which are consistent with those used in the original SMODICE paper [13], for all the MuloCo locomotion and manipulation environments.
In our set of environments, we keep the same hyper-parameters across tasks locomotion, adroit manipulation, and kitchen manipulation. We train until convergence for all algorithms including baselines and we found the following timesteps to be sufficient for different set of environments: Kitchen: le6, Few-expert-locomotion 500k, Locomotion: 300k, Manipulation: 500k
We keep a constant batch size of 1024 across all environments. For all tasks, we average mean returns over 10 evaluation trajectories and random seeds. Full hyper-parameters we used for experiments
17

---

are given in Table 3, For policy update, using Value Weighted Regression, we use the temperature to be 3 for all environments.
Hyperparameters for our proposed off- off-po imitation learning method DILO are shown in Table 3.
Hyperparameter
Value
Policy learning rate
3e-4
Value learning rate
3e-4
f f-divergence
X²
max-clip (Value clipping for policy learning)
100
MLP layers
(256,256)
3 (mixture ratio)
0.5
7) (orthogonal gradient descent)
0.5
7 (policy temperature)
3
Table 3: Hyperparameters for DILO in imitation from proprioceptive observations.
6.2.2 LIO with Image Observations
We use robomimic [20] for our imitation with image observations experiments. The following two environments are used here (the description taken from their paper and written here for conciseness):
Lift: Object observations (10-dim) consist of the absolute cube position and cube quaternion (7-dim), and the cube position relative to the robot end effector (3-dim). The cube pose is randomized at the start of each episode with a random z-rotation in small square region at the center of the table.
Can Object observations (14-dim) consist of the absolute can position and quaternion (7-dim), and the can position and quaternion relative to the robot end effector (7-dim). The can pose randomized the start of each episode with random z-rotation anywhere inside the left bin.
Robomimic provides three datasets and two modalities of observation (Proprioceptive, Images) for both environments above. The datasets are denoted by - MH (Multi-human), MG (Machine Generated), PH(Proficient-human We use the MG and MH datasets as the suboptimal datasets in our task and PH as the source of expert observations. MH and MG datasets consists of 200 trajectories of usually suboptimal nature and we use 50 observation-oni trajectory from PH datasets. This tasks is complex by the fact that expert-level actions mostly unseen in the suboptimal dataset and the agent needs to learn the best actions that matches expert visitation from the suboptimal dataset. We implement all algorithms in the Robomimic codebase without any change in network architecture, data-prey cnttapreprocessing or learning hyperparameters. We tune algorithm specific hyperparameters in course grid for BCO, SMODICE, and DILO to compare the best performance methods independent of hyperparameters. For BCO, we tune the inverse dynamics model learning epochs between [1.5,10]. For SMODICE, we tuned discriminator learning epochs between [1,5], and gradient penalty between [1,5,10,20]. To control overestimation due learning with offline datasets in DILO consider linear weighting À between the optimism and pessimism terms in Eq 5 inspired by prior work [28] as follows:
min(1 - A)3(1 Bj, (v s')] + AB "r q
- [V(s', V(s,
X(1 E"
[ sn)] V(sss')].
The hyperparameters used for DILD can be found in Table 4. For the architecture specific hyperpa- rameters we refer the readers to [20].
6.3 Robot Manipulation Experiments
Our setup for manipulation experiments inspired by the robot air hockey environment [55] for applying DILO to physical robotics settings. Our setup utilizes Universal Robotics 5 kilogram e-series (UR5e) 6-degree of freedom robotic arm fixed mount, Robotig parallel jaw gripper, 1.93m 0.76m Wind Chill air hockey table which is tilted 5.5 degree angle, and an overhead
18

---

Hyperparameter
Value
max-elip (Value clipping for policy learning)
100
À (pessimism parameter)
0.7
3 (mixture ratio)
0.5
(orthogonal gradient descent)
0.5
(policy temperature)
3
Tahle 4: Hyperparameters for DILO in imitation from image observations.
Sony Playstation Eye, high framerate camera, which gathers 640 480 frames 60 FPS, mounted the ceiling to have full view of the table. The paddle that is held by the robot end effector is 9.5cm in diameter and the puck is 6.3cm.
In this setup, the negative æ direction is oriented along the table, and the y across the table. The ac- tion space for the arm uti- lizes pose control in 2, through an inverse kine- matics controller accessi- ble through the Universal roboties real-time data exchange interface. Utilizing the serving command, the arm is controlled using delta positions clipped between 26cm in the x direction and 13em in the right direction. These control limits are specified prevent the robot from trigger- ing force limits, which results in emergency stop. Actions are taken at 20Hz fre- quency to allow for rapid response to dynamic elements, such as hitting a falling puck.
Figure Tasks: Left: Place object avoid obstacles. Center: Stationary Puck Striking. Right: Dynamic Puck Hitting
The position and velocity the end effector can be recovered through the real-time data ex- cahnge, but other objects like the puck the hand require identification. This work utilizes an overhead camera running at 60Hz to locate these objects using vision pipeline that relies on hue saturation value segmentation followed by object identification. This gives an Zes ye co- ordinate in camera space, which we convert OpenCV [56] homography robot coordinates. This homography is computed by mapping the end effector positions given by the robot sensors to visual locations from the camera.
Goal Position
Cups
Strawberry
BRUNSWICK
We apply imitation learning from observations on several tasks built on top of the above robot setup. The following experiments are described here:
Safe object manipulation: This task involves moving a strawberry to a bowl while avoiding four cups placed in the workspace of the robot. The bowl is placed in the top right corner of the workspace, and the cups placed in fixed loca- tions. The success metric is the robot stopping above the bowl while making no contact with any of the cups. The test set involves 10 random starting locations for the end effector that are fixed between assessments. The observation space is the 2D end effector position, and the strawberry is initialized inside the gripper. Our suboptimal dataset
Figure 6: Cross Embodiment Demonstration: Track- ing of the strawberry and obstacles for learning action- free.
19

---

consists of 50 trajectories of 100 time steps on average where the robot is initialized in random location, and the human moves the arm to a random different location, ignoring the positions of the cups or the bowl. In this setting, we investigated the following expert data, visualized in Figure 5:
Few Trajectories: The expert data is drawn from a set where the expert is initialized in a random location, sometimes touching an obstacle, and must use the teleoperation system to avoid the obstacle and reach the goal. In this setting we used 15 expert trajectories.
Fixed start: The expert is initialized at the opposite corner of the workspace, and navigates to the goal location following different paths using teleoperation. In this setting we used 15 trajectories.
Few Uniform: Uses the same expert data as the Few Trajectories setting, but the dataset consists of randomly samples 300 transitions, where one trajectory is approximately 60 transitions of data.
Cross Embodiment Few/Fixed/Unifor:: The expert is person holding the strawberry in his/her hand, visualized in Figure 6. They then move the strawberry tracked by the camera to the goal location while avoiding the obstacles, starting from random/fixed locations with 15 expert trajectories respectively or uniform with 300 transitions.
This task involves moving the end effector to strike a stationary puck. The success metric the robot touching the puck. The test set involves 10 initializations of the puck posi- tion across the length of the table. The ensure uniformity across evaluations, the set of initialization locations of the puck are fixed across methods. The end effector is initialized at 0.38m from the base in the center of the table, so success strike does not require backward motion. The observation space is the 2D end effector position and the tracked position of the puck. Our suboptimal dataset consists of 50 trajectories of 75 time steps on average where the robot is initialized at the start position, and the human moves the arm random, vaguely striking pattern.
Stationary Striking:
In this setting we used an expert dataset of 400 trajectories where the expert uses mouse teleoperation to strike the puck. The expert efficiently strikes the puck in single motion. We visualize the expert striking and the puck position in Figure 5. We show the learned action vectors for all algorithms and tasks fixed start (Figure 9), Few Uniform (Figure 8) and Few trajectories (Figure 10).
Dynamic Hitting:
This task involves hitting puck dropped from the top of the table. Because the table is set at angle, this will cause the puck fall with increasing acceleration towards the opposite side. The setup is visualized in Figure 5. The test set involves 10 initializations of the puck position dropped from positions across the length of the top of the table. The locations of the 10 puck drops are fixed using indicators across methods to give fair evaluation, and the arm is initialized in the center of the table, 0. 0.68m from the base. The observation space is the 2D end effector position and 2D end effector velocity and the history of the last tracked positions of the puck relative to the position of the end effector. Our suboptimal dataset consists of 50 trajectories of 200 time steps on average where the robot is initialized at the start position, and the human moves the around the puck without striking it.
We utilize two success metrics for this task: 1) touching: a trajectory is considered successful if the agent touches the puck. 2) hitting: the puck must have velocity in the opposite direction that it was dropped. This task is especially challenging for existing methods because of the long sequence of actions necessary to position the paddle properly, and the high level of both precision and timing: even few millimeters of error movement at the wrong time will result in a failure, especially for hitting. Previous work has observed that this task is challenging even for humans, who often require several tries of practice, and many dataset trajectories consist of many inaccurate hits. In this domain, Behavior cloning only achieves 30% success at touching the puck, and Implicit Q-learning, a popular offline RL method, can only achieve 60% success, even though it employs a hand-designed reward function.
We used the implementation details from the proprioception task with the difference that in all the real-robot tasks tune the following parameters across different methods: For BCO, we tune the
20

---

inverse dynamics model learning epochs between [1,5,10]. For SMODICE, we tuned discriminator learning epochs between [1,5], and gradient penalty between [1,5, 10,20.. For DILO we tune the conservatism parameter from previous section between (0.5.0.6.0.7.0.88))
In this domain, these challenges appear be empirically validated in the performance of the baseline methods. We hypothesize that the accumulation of error over long horizons in other learning from observation methods results in poor performance, visualized in Table 3 and Figure 4. For methods like BCO, learning the necessary inverse dynamics to interpolate the long sequence of actions for successful strike impractical, resulting in behavior that appears listless. On the other hand, while the SMODICE discriminator rewards are able to occasionally match the visitation distribution of the expert, there an exponential explosion of possible combinations of puck history and paddle positions, resulting in poor generalization: the left half of the table, SMODICE is unable hit the puck.
The hitting scenario involves two settings, both using mouse teleoperation to control the puck: one where the human strikes the puck only once and then the trajectory ends, which utilizes a dataset of 50 trajectories, and an expert dataset 850 trajectories where the human keeps hitting the puck repeatedly for up 2000 timesteps. The task challenging for human, so trajectories average only 500 timesteps and cleaned so that human mistakes are removed from the expert dataset. We visualize the expert hitting and the puck position in Figure 7.
0.4
0.0
-0.4
-2.0
-1.5
-1.0
-0.5
Figure 7: Expert Hitting: Visualization of one trajectory of puck tracking and hitting by the expert. Right: stacked frames of the environment. Left: puck position in robot coordinates
BCO
SMODICE
DILO
03
02
coooddrate
02
61
-03
-03
-0.6
-0.6
-0.6
-0.4
X coordinate
Xccordinate
ccordinate
Figure 8: Action Vectors qualitatively showing the next action for the safe manipulation with uniform sampled transitions. BCO generalizes incorrectly at number of locations producing policies that hit obstacles. DILO learns to mimic expert's intent better demonstrating signs has learned avoid obstacles by arrows around
6.4 Limitations
Learning from Observation is challenging setting, and while DILO makes some key assumptions in order to achieve good performance. First, matching distributions becomes exponentially more challenging in the dimensionality of the state space. In this work, while DILO outperforms basclines
21

---

BCO
SMODICE
DILO
0.1
0.1
Yoordinate
-0.1
4.1
-0.2
0.2
0..3
0.3
-0.0
-0.6
-0.4
-0.8
-0.6
-0.4
-0.8
X coordinate
X coordinate
X coordinate
Figure 9: Action Vectors qualitatively showing the next x-y action for the safe manipulation with fixed start state. BCO generalizes incorrectly number of locations producing policies that hit obstacles. DILO learns to mimic expert's intent better demonstrating signs that it has learned avoid obstacles by the around
BCO
SMODICE
DILO
0.3
02
011
0.1
coordinate
-0.1
-0.2
-022
-0.3
-0.3
-0.3
-0.8
-0.6
-0.4
-0.8
-0.6
-0.4
-0.8
-0.6
coordinate
X coordinate
X coorrdinate
Figure Action Vectors qualitatively showing the next x-y action for the safe manipulation with few trajectories. BCO generalizes incorrectly at a number locations producing policies that hit obstacles. DILO learns to mimic expert's intent better demonstrating signs that it has learned to avoid obstacles by the arrows around
in the Expert Image observations, it still shows limited performance. Second, while learning from observations opens the door for good performance without expert actions, the expert observation space must match that of the agent. In some video settings, this is not the case, ex. the agent might use a fixed camera when the human is egocentric, or vice versa. Finally, DILO utilizes the conservatism parameter т to regulate the degree of extrapolation from the algorithm. In some settings, the values can diverge, resulting in /* taking on values that might be too large be used for learning the downstream policy. Adaptively selecting to maximize extrapolation while avoiding divergence is area of active investigation.
6.4.1 Failure Cases
While DILD outperforms other methods in overall success rate, the failure modes can differ. In general, DILO tends be conservative in what actions it takes, learning motions that might be slower than BCO or may get stuck before arriving at the goal. In low-dim observation settings, DILO can also exhibit "dead zone" behavior, where the model becomes mostly unresponsive. Below we detail some of the exact error modes in particular tasks:
Safe Manipulation: While DILO and BCO have comparable success rates, the two algorithms fail in different ways. BCO tends take large actions while ignoring obstacles reach the goal, while DILO takes more conservative actions. Thus, while BCO might fail by knocking will tend to fail to reach the goal. Because this is low data setting, both algorithms BCO and DILO
cup, DILO
22

---

can end up coming close to the cups or brushing them gently. As a side note, SMODICE fails at even reaching the goal in most cases in this task, possibly because of this low data setting.
Striking: This domain challenging because of the narrow data regime, and all methods tend to struggle in similar ways. The most common consequence of low data arises through sensitivity to the x location of the puck (along the table). While intuitively, striking behavior should be relatively invariant for fixed y (horizontal position on the table), slight variation in r from the dataset can result in policy that moves the arm in the opposite direction of the puck, probably due to errors in extrapolation. In addition, striking is a dynamic behavior that requires a precise combination of forward and horizontal actions. Even a slight error in the ratio can result in a near miss. Finally, DILO tends to learn more conservative policies and, some locations, may not not strike the puck with much force. However, because of the low data coverage, this issue endemic all the learned policies.
Hitting: The primary challenge of achieving a hit in this task is the precise alignment of the paddle to the puck. While DILO performs well, it is not perfectly accurate, resulting in touches that bounce off the side of the paddle. This challenge endemic to all policies. Additionally, the conservatism of DILO actions appear when moves under the puck, where it tends to move slowly, and dropping result in DILO failing reach the puck. As result, while DILO is likely to succeed at the first hit, it can struggle generate multiple hits because this can require rapid side-to-side movement. These issues are largely endemic to all the learned policies, where SMODICE tends to be even less precise, and BCO struggles to learn to strike, though it can occasionally position under the puck.
the puck quickly
Visualizations of the failure modes can be seen in the project website.
6.5 Limitations of DILO
Our proposed method is limited by the assumption of matching visitation distributions in the observa- tion space of the agent and expert rather than a meaningful semantic space, but we hope that with improvement in universal representations, this limitation is lifted by distribution matching in compact representation space. Our work assumes that expert's optimality, but in reality, experts demonstrate a wide range of biases. We leave this extension to future work. Finally, we demonstrate the failure modes of our method and further limitations Appendix 6.4.
23


Reviews:
[{"review_id": "AhJV5pKuTu", "reviewer": "Reviewer_U3CE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel method for learning policies from expert observation trajectories without action information and from an action-labeled dataset of arbitrary quality. An action-free distribution matching objective is used to learn a utility function that quantifies the utility of transitioning between states based on the visitation distribution of an expert. The policy is extracted from the learned utility function using value-weighted regression on the action-labeled dataset. Experiments show improved performance over alternative methods that address the problem of learning from observations in a variety of settings.", "summary_of_recommendation": "After rebuttals, I would recommend accepting this paper for publication.", "technical_quality": 4, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* How different can the distribution of suboptimal data be from the expert data distribution while still being able to extract a policy from the learned utility function?\n* How well does DILO scale with increasing dataset size?\n* Could DILO be extended to a multi-task (or goal-conditioned) setting?\n* How can the performance of DILO with image observations be improved? What properties should a representation learned from visual inputs have to be easily applicable to DILO?\n* What theoretical properties of DILO you are referring to on lines 283-284?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nxaAhnTWZD", "reviewer": "Reviewer_5eEJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper works on the problem of learning robot manipulation skills from actionless expert demonstration videos. The proposed method first learns a utility function over the state transitions from the offline dataset, and then extracts the policy through maximizing the expected returns of the learned utility function. Experiments are performed in simulation and real world.", "summary_of_recommendation": "My rating is based on insufficient evaluation and weak performance", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Would the proposed method be sensitive to the quality of demonstrations? It would be valuable to analyze how policies learned from demonstrations of varying quality (e.g., using datasets from Robomimic[1]) perform on the tasks. \n\nThe performance for Robomimic benchmark seems to be not ideal, could authors give details analysis and explanations for this?  \n\nBesides the proposed tasks, demonstrating the effectiveness of the approach on other benchmarks (e.g., RLBench) would further strengthen the paper. \n\n[1] What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL’21. \n\n[2] RLBench: Robot Learning Benchmark, RA-L'20.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YgexKIHtrD", "reviewer": "Reviewer_2kPL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel approach for learning from observation (LfO) without access to a simulator. The authors start by motivating their approach, noting that many prior works fail in the offline LfO space. The authors then present their approach (Dual Imitation Learning from Observations or DILO), which directly learns a utility function that quantifies how state transitions impact the agent's divergence from the expert's visitation distribution. Importantly, the method is backed with several theoretical results, derivation, and ample citations, and explained both intuitively and in-depth. The authors conclude by displaying an abundance of results, both in synthetic domains and real-world, displaying that DILO presents a novel high-performing algorithm for learning robot behavior.", "summary_of_recommendation": "The paper presents a clear advancement for offline learning from observation, contains a strong mathematical basis behind the algorithm, and has ample results across synthetic and real-world domains.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Could you please address the weakness noted above?\n-  DILO achieves similar results for Safe Object Manipulation across the teleop and cross-embodiment demonstration. I would expect it would be easier to learn from the teleop demonstrations. Could you comment on why the performance is similar and if this was expected/designed for?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Before Rebuttal**

Strengths.
This paper studies learning from observations with action-free videos, an important problem to solve given the abundance of such data. The proposed method is novel, with strong theoretical motivations. Experimental results in both simulation and reality show the benefit of the method over alternative approaches. As well as comparing to alternative methods for learning from observation without action data, comparisons to methods which additionally have access to action data offer interesting insights.

Weaknesses. 
In some places, the paper is difficult to follow and understand due to low quality of the writing. There is insufficient information in the method and experiments to appreciate the intended contributions, and to understand what the key conclusions are from the results. The real-world experiments assume that low-dimensional state representations are available, which are challenging to obtain in practice, and so there is no evidence that the method performs well in the real world directly from visual observations.

---

**After Rebuttal**

Following the reviews, authors provided a revised paper which has now improved in clarity and addressed some of the concerns raised by the reviewers. Although the real-world experiments are limited due to the need for low-dimensional states, overall this is only a minor issue given the strong theoretical contribution of the paper. The AC and reviewers discussed the paper together, after which there is now unanimous agreement amongst reviewers that the paper should be accepted.
Decision: Accept