SEABO: A Simple Search-Based Method for Offline Imitation Learning
Authors: Jiafei Lyu
Xiaoteng Ma
Le Wan
Runze Liu
Xiu Li
Zongqing Lu
Venue: ICLR 2024 poster (2024)
Keywords: offline imitation learning; reward learning; reinforcement learning
Primary area: reinforcement learning
TL;DR: We propose a simple yet effective search-based method for offline imitation learning.

Abstract: Offline reinforcement learning (RL) has attracted much attention due to its ability in learning from static offline datasets and eliminating the need of interacting with the environment. Nevertheless, the success of offline RL relies heavily on the offline transitions annotated with reward labels. In practice, we often need to hand-craft the reward function, which is sometimes difficult, labor-intensive, or inefficient. To tackle this challenge, we set our focus on the offline imitation learning (IL) setting, and aim at getting a reward function based on the expert data and unlabeled data. To that end, we propose a simple yet effective search-based offline IL method, tagged SEABO. SEABO allocates a larger reward to the transition that is close to its closest neighbor in the expert demonstration, and a smaller reward otherwise, all in an unsupervised learning manner. Experimental results on a variety of D4RL datasets indicate that SEABO can achieve competitive performance to offline RL algorithms with ground-truth rewards, given only a single expert trajectory, and can outperform prior reward learning and offline IL methods across many tasks. Moreover, we demonstrate that SEABO also works well if the expert demonstrations contain only observations. Our code is publicly available at https://github.com/dmksjfl/SEABO.

Published as a conference paper at ICLR 2024
SEABO: A SIMPLE SEARCH-BASED METHOD FOR OFFLINE IMITATION LEARNING
Jiafei Lyu's ; Xiaoteng Ma², Le Wan Runze Liu', Xiu Li Zongqing Lu
Tsinghua Shenzhen International Graduate School, Tsinghua University
2 DDpaarment of Automation, Tsinghua University, 3IEG, Tencent "Scoool of Computer Science, Peking University 1vjf20 0ma 1 s.s.ssssss tsinghua..... edu. cr, cr.
edu. cn, zongqing. edu.
ABSTRACT
Offline reinforcement learning (RL) has attracted much attention due to its abil- ity in learning from static offline datasets and eliminating the need of interacting with the environment. Nevertheless, the success of offline RL relies heavily on the offline transitions annotated with reward labels. In practice, we often need hand-craft the reward function. which is sometimes difficult, labor-intensive or inefficient. To tackle this challenge, we set our focus on the offline imitation learning (II.) setting, and aim getting reward function based the expert data and unlabeled data. To that end, we propose a simple yet effective search- based offline method, tagged SEABO. SEABO allocates larger reward to the transi- tion that is close to its closest neighbor in the expert demonstration, and a smaller reward otherwise, all in an unsupervised learning manner. Experimental results on a variety of D4RL datasets indicate that SEABO can achieve competitive per- formance to offline RL algorithms with ground-truth rewards, given only single expert trajectory, and can outperform prior reward learning and offline IL methods across many tasks. Moreover, we demonstrate that SEABO also works well if the expert demonstrations contain only observations. Our code is publicly available at
INTRODUCTION
In recent years, reinforcement learning (RL) (Sutton & Barto, 2018) has made prominent achieve- ments in fields like video games (Mnih et al., 2015; Schrittwieser et al., 2020), robotics (Kober et al., 2013), nuclear fusion control (Degrave et al., 2022), etc. It is known that RL a reward- -oriened learning paradigm. Online RL algorithms typically require interactive environment for data col- lection and improve the policy through trial-and-error. However, continual online interactions are usually expensive, cone-consuming, or even dangerous in many practical applications. Offline RL (Lange et al., 2012; Levine et al., 2020), instead, resorts to learning optimal policies from previously gathered datasets, which are composed of trajectories containing observations, actions, and rewards.
A bare fact is that reward engineering is often difficult, expensive, and intensinte It is also hard to specify or abstract good reward function given some rules. To overcome this challenge in the offline setting, there are generally two methods. First, one can train the policy via the behavior cloning (BC) algorithm (Pomerleau, 1988), its performance is heavily determined by the per- formance of the data-collecting policy (a.ka., the behavior policy). Second, one can learn a reward function from some expert demonstrations and assign rewards to the unlabeled data in the dataset. Then, the policy can be optimized by leveraging the reward. This is also known offline imitation learning (offline IL). Note that in many real-world tasks, acquiring a few expert demonstrations is easy (e.g., ask human expert to operate the system) and affordable.
However, turns out that, similar to offline RL, offline IL also suffers from distribution shift issue (Kim et al., 2022b; DeMoss et al., 2023), where the learned policy deviates from the data-collecting policy, leading to poor performance during evaluation. Prior works concerning on distribution cor-
Work done while working as an intern Tencent. Corresponding Authors.

---

Published as a conference paper at ICLR 2024
expert trajectory
expert demonstration
SEABO
KD tree
reward
offline RL
query
unlabeled transition
expert sumples
query samples
Figure I: Left: The key idea behind SEABO. We assign larger rewards to transitions that are closer to the expert demonstration, and smaller rewards otherwise. The dotted lines connect the query samples with their nearest neighbors along the demonstration. Right: Illustration of the SEABO framework. Given an expert demonstration, we first construct KD-tree and then feed the unlabeled samples into the tree to query their nearest neighbors. We use the resulting distance to calculate the reward label. Then one can adopt any existing offline RL algorithm to train on the labeled dataset.
rection estimation (DICE family) address this by enforcing the learned policy to be close to the behavior policy via distribution divergence measure /.. (Ghasemipour al., 2019; Ke et al., 2019)). However, such distribution matching schemes can incur training instability (Ma et al., 2022) and werr-oonsevatiimm (Yu al., 2023), and they often involve training task task-specific discriminators. On the other hand, some works seek to decouple the processes reward annota- tion and policy optimization (Zolna et al., 2020; Luo et al., 2023). However, they involve solving complex optimal transport problems or contrasting expert states and unlabeled trajectory states.
In this paper, propose a simple yet effective alternative, BRArch-Based method for Offline im- itation learning, namely SEABO, that leverages search algorithms to acquire reward signals in an unsupervised learning manner. As illustrated in Figure (left), we hypothesize that the transition is near-optimal if it lies close to the expert trajectory, hence larger reward ought to be assigned to it, and vice versa. To that end, we propose to determine whether the sample approaches the expert trajectory via measuring the distance between the query sample and its nearest neighbor in the expert trajectory. practice, as depicted in Figure (right), SEABO first builds KD-tree upon expert demonstrations. Then for each unlabeled sample in the dataset, we query the tree to find its nearest neighbor, and measure their distance. the distance is small (i.e., close to expert trajectory), large reward will be annotated, while if the distance is large (i.e., stray away from the expert trajectory), the assigned reward is low. SEABO is efficient and easy to implement. It can be combined with any existing offline RI. algorithm to acquire meaningful policy from the static offline dataset.
Empirical results on the D4RL (Fu et al., 2020) datasets show that SEABO can enable the offline RI. algorithm to achieve competitive or even better performance against its performance under ground- truth rewards with only one expert trajectory. SEABO also beats recent strong reward annotation methods and imitation learning baselines on many datasets. Furthermore, we also demonstrate that SEABO can learn effectively when the expert demonstrations are composed of pure observations.
2 PRELIMINARY
We formulate the interaction between the environment and policy a Markov Decision Process (MDP) specified by the tuple (S,A, 7, Po). where S is the state space, A is the action space, x A + R is the scalar reward signal, Y E [0. 1]
A,,,,,
p S A + S is the transition dynamics, is the discount factor, Po is the initial state distribution. A policy ([ass) outputs the action based on the state s. We assume that the underlying MDP has a finite horizon. The goal of RL to maximize the discounted future reward: J(T) = Esgr Ea-a, a141 11pp wpppppp.... IET= 7fr(st- at)]. Whereas, in many scenarios, it is often hard for one get the reward signals. It is more common that the unlabeled trajectory, t = {sp, ag...., St. .... }r). is collected. This poses veritable challenges for applying offline RI. algorithms.
,,....,,....
2

---

Published as a conference paper at ICLR 2024
In this paper, focus on the offline IL setting. We assume that we have access the dataset of expert demonstrations D. = (T and a dataset of unlabeled data 2., = { ) where M and N are the sizes of the expert dataset and unlabeled dataset, respectively. The unlabeled trajecto- ries are gathered by some unknown behavior policy μs. Note that we allow the expert demonstrations to either contain actions or do not contain actions. We aim attaining the reward function by ex- tracting information from the expert trajectories and unlabeled trajectories, and assigning rewards to the unlabeled datasets, without any interactions with the environment. Then we can train the policy using any offline RI. algorithm.
Ji=1
3 RELATED WORK
Offline Reinforcement Learning. In offline RL (Lange et al., 2012; Levine et al., 2020), the agent is not permitted to interact with the environment, and can only learn policies from previously gathered dataset D {(5s, 02,, Ti, 864 ))+1))=- where N is the dataset size. Existing work on offline RL can be generally categorized into model-based (Yu et al., 2020; 2021; Kidambi et al., 2020; Lyu et al., 2022b; Rigter al., 2022; Lu et al.. 2022a; Chen al., 2021; Janner al., 2021: Uchara & Sun, 2022: Zhang et al., 2023) and model-free approaches (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2020; Kostrikov et al., 2022; Lyu et al., 2022c;a; Cheng et al., 2022; Zhou al., 2020; Ran et al., 2023; Bai et al., 2022: Yang et al., 2024). The success of these methods rely heavily on the requirement that the datasets must contain annotated reward signals.
Imitation Learning, Imitation Learning (IL) considers optimizing the behavior of the agent given some expert demonstrations, and no reward is needed. The primary goal of IL is to mimic the behavior of the expert demonstrator. Behavior cloning (BC) (Pomerleau, 1988) directly performs supervised regression or maximum-likellhonn compounding error and may result in performance collapse upon unseen states (Ross et al., 20. Another line of work, inverse reinforcement learning (IRL) (Arora & Doshi, 2021), first learns a reward function using expert demonstrations, and then utilizes this reward function train policies with RL algorithms. Typical IRL algorithms include adversarial methods (Ho & Ermon, 2016; Jeon et al., 2018; Kostrikov et al., 2019; Baram et al., 2017). maximum entropy approaches (Ziebart et al., 2008; Boularias et al., 2011). normalizing flows (Freund et al., 2023), etc. However, these methods often require abundant online transitions train a good policy. Imitation learning without online interactions, which is the focus of our work, is hence attractive and remains an active area. There are many advances in offline II., such as applying online IRL. algorithms in the offline setting (Zolna et al., 2020; Yue et al., 2023), using energy- based methods (Jarrett et al., 2020), weighting the BC loss with the output of the trained discriminator (Xu et al., 2022), etc. Among them, DICE (Nachum al., 2019) family receives much attention. Methods like ValueDICE (Kostrikov et al., 2020), DemODICE (Kim et al., 2022b), and LobsDICE (Kim et al., 2022a) can consistently drub BC in the offline setting. Notably, a recent work, OTR (Luo al., 2023), acquires the reward function in the offline setting via optimal transport. OTR decouples the processes of reward learning and policy optimization. Still, OTR needs to solve complex optimal transport problems. We, instead, explore to get the reward function via a search-based method.
on expert demonstrations. Yet, BC can suffer from
Search Algorithms. Search algorithms (Korf, 1999) are critical components in artificial intelli- gence. Typical search algorithms include brute-force search algorithms (Dijkstra, 1959; Stickel & Tyson, 1985; Korf, 1985; Taylor Korf, 1993), heuristic search approaches (Doran Michie, 1966; Hart et al., 1968; Pohl. 1970; Edelkamp & Schrödl, 201 1), etc. In this paper, we resort to the simple search approach, KD-tree (Bentley, 1975), for capturing the nearest neighbors of the unlabeled data in the expert demonstrations.
OFFLINE IMITATION LEARNING VIA SEARCH-BASER METHOD
In this section, we formally present our novel approach for offline imitation learning, DEArch-Bases Offline imitation learning (SEABO). We begin by analyzing the common formulation adopted in distribution matching IL methods (Ho & Ermon, 2016; Kim et al., 2020; Kostrikov et al., 2020), which attempt to match the state-action distribution of the agent p.r and the expert Pe, often by means of minimizing some f-divergence measurement Dy: argmin, (p(p Though these methods have promising results, they usually require training task-specific discriminators and suffer
3

---

Published as a conference paper at ICLR 2024
from training instability (Wang et al., 2020; Ma et al., 2022). A natural question arises, can we get the reward signals without training neural networks?
Instead of measuring the distribution of states state-action pairs, want to determine the op- timality of a single transition. Our idea is quite straightlorward. the closer the transition is to the expert trajectory, the more optimal this transition is. The agent ought to pay more attention those optimal transitions. This motivates us to measure how close the unlabeled transition is to the expert trajectories. We propose achieve this by finding the nearest neighbor of the query transition in the expert demonstrations, and then measuring their distance (e.g., Euclidean distance). If the distance large, then the transition is away from the expert demonstration. While if the distance is small, indicates that the transition is near-optimal, or even is exact expert data if the distance approaches O. Intuitively, this distance can be interpreted as a reward signal.
To that end, we construct function dubbed Nearest Ne NearestNeein ghbor (demo, query-sample sample) that returns the nearest neighbor of the query sample in the expert demonstrations. Suppose the expert trajectories are made up of state-action pairs, then for the query sample (s, s'), we have:
(ge, ae, &)) NearestNei (De, (s, a, s').
(1)
Then we measure their deviation using some distance measurement D:
d= (( a.. &²),, (s, 0, s)).
(2)
Afterward, following prior work (Cohen et al., 2022; Freund et al., 2023; Dadashi et al., 2021; Luo et al., 2023), get the rewards via squashing function: r = exp( 3 x d), where a and / are hyperparameters that control the scale of the reward and the impact of the distance, respectively.
Algorithm 1 BEArch-BBased Offline Imitation Learning (SEABO)
1: Require: expert demonstrations De, unlabeled dataset D.,
2: Initialize Diabel + O. Given distance measurement D
3: for (s, a, s') in Du do
4: Find its nearest neighbor, (5.., o,, 3k)) NarrestNeeighoooO..ssss (s...,
5: Measure the distance: d = D((3,, ae. 82), (8)),
6: Get the reward signal via Equation 3
7: Diabel + Diabel U (s, a, s.))
8: end for
We name the resulting method SEABO, and list its pseudo-code in Algorithm 1. For practical im- plementation of SEABO, we leverage KD-tree (Bentley, 1975) for searching the nearest neighbors, and adopt Fuclidean distance (Torabi et al., 2019) as the distance measurement for simplicity (i.e., the default setting of KD-tree). We also slightly modify the aforementioned formula the reward function to make it better adapt to different tasks with one set of hyperparameters, which gives:
exp (-3+) (-1AA
B x d
[A|
where |A| is the dimension of the action space. Note that this technique is also adopted in Dadashi et al. (2021). We choose to use (s, a, s') to query since the magnitude of states and actions may be different. One possible solution is query the demonstrations via (E x s, a), & RR, but introduces an additional hyperparameter that may need to be tuned per dataset. We empirically find that involving in the query sample can ensure good performance across many tasks. The above procedure (as specified in Figure 1 (right)) also applies when the expert demonstrations contain only observations, because it is feasible that we find the nearest neighbors using only observations.
SEABO enjoys many advantages over prior reward learning methods or offline imitation learning algorithms. First, SEABO does not require any additional processing upon the offline ' The unlabeled dataset can have different trajectory lengths, and the unlabeled trajectories can be fragmented, or even scattered, since SEABO computes the rewards only using the single transition instead of the entire trajectory. Second, SEABO does not require training reward models discrim- inators, hence getting rid of the issues of training instability and hyperparameter tuning of the neural networks. Third, SEABO is easy to implement and can be combined with any offline RL algorithm.
'Methods like OTR (Luo et al., 2023) need zero padding of the unlabeled trajectories to ensure that they have identical length the expert trajectories.
4

---

Published as a conference paper at ICLR 2024
hos hiafichhetal--- -wt.---ddd--------222----- (swadke)
falf:reeeehhereedam-eeeeee (SPAAO))
hopper-meduum replay (oracie)
hooper meduummrrppaa replay (SEABO)
danaty
donatty
derstty
2.13
E
rowardd
rowardd
roward
noward
Figure 2: Density plots of ground-truth rewards and rewards acquired by SEABO. Note that oracle indicates the ground-truth rewards are plotted.
To show the effectiveness of our method, we plot the distribution of ground-truth rewards (oracle) and rewards given by SEABO. We choose two datasets, hal Echeetah-nee ah-medi -m-expert-ve and hopper-mess
um-v2 from D4RL (Fu al., 2020) examples, and use a 1, B 0.5, which is the same as our hyperparameter setup in Section 5. The results are summarized in Figure
We find that the reward distributions of SEABO resemble those of oracle. Notably, SEABO suc- cessfully gives two peaks in halfcheetat aheeeet -eemmmiiiu www-um--eeeepee wepppeer-eeettt...eeeme indicating that it can distinguish samples of different qualities. These reveal that SEABO can serve good reward labeler, which validates its combination with off-the-shelf offline RL algorithms.
5 EXPERIMENTS
In this section, we empirically evaluate SEABO on D4RL datasets. We are targeted at examining, given only one single expert demonstration, whether SEABO can make different base offline RL algorithms recover or beat their performance with ground-truth rewards across varied tasks. We are also interested in exploring how SEABO competes against prior reward learning and offline imitation learning methods. We further investigate whether SEABO can work well if the expert demonstrations are composed of pure observations. Moreover, we check how different choices of search algorithms affect the performance of SEABO.
We discard reward signals in the D4RL datasets to form unlabeled datasets. For expert demonstra- tions, we follow Luo et al. (2023) and utilize the trajectory with the highest return in the raw dataset for ease of evaluation. One can also use a separate expert trajectory. All of the experiments this paper are run for 1M gradient steps over five different random seeds, and the results are averaged at the final gradient step. We report the mean performance in conjunction with the corresponding standard deviation. We adopt the same squashing function for tasks under the same domain. Unless specified, we use the number of expert demonstrations K = for evaluation. It is worth noting that SEABO is computationally efficient since there is only single expert trajectory, and the time com- plexity of KD-tree gives (dly log [Del). where dj is the feature dimension size. It takes SEABO about minute to annotate million unlabeled transitions using merely CPUs. Hence, believe the overall computation overhead from SEABO is minor and tolerable. We defer the experimental details and hyperparameter setup for all of our experiments Appendix A.
5.1 MAIN RESULTS
SEABO upon different base algorithms. We first explore whether SEABO can aid different of- fline RL algorithms. We verify this by incorporating SEABO with two popular offline RL algo- rithms, TD3_BC (Fujimoto & Gu, 2021) and IQL (Kostrikov et al., 2022). We conduct experiments on 9 medium-
(medium, medium-reples tion "-v2" datasets (hal fcheetah, hopper, walker2a) and summarize the results in Table 1. One can see that IEL+SEABO beats IQL. with ground-truth rewards on 6 out of 9 datasets, and TD3 DD3_BCO outperforms TD3_BC with raw rewards on 5 out of 9 datasets. On other datasets, SEABO can achieve competitive performance against the oracle performance. The overall scores of SEABO with IQL and TD3_BC exceed those of ground-truth rewards. This evidence indicates that SEABO can generate high-qualing rewards and benefit different offline RL algorithms.
medium-expert) D4RL MuJoCo locomo-
SEABO competes against baselines. To better illustrate the effectiveness of SEABO, compare IQL+SEABO against the following strong reward learning and offline IL baselines: ORIL (Zolna al., 2020), which learns the reward function by contrasting the expert demonstrations with the
5

---

Published as a conference paper at ICLR 2024
Table 1: Results of SEABO upon different base algorithms. Mmax denotes the normalized return of the highest return trajectory in the specific dataset, IQL and TD3 BBC indicate that they are trained upon the ground-truth reward labels, while +SEABO indicates the algorithm trained on the reward signals provided by SEABO. The normalized average scores at the final 10 episodes of evaluations are reported, along with standard deviations. We bold the mean score and highlight the cell SEABO outperforms algorithms trained on ground-truth rewards.
Task Name
Mmax
IQL
IQL+SEABO
TD3_BC
TD3_BC+SEABO
halfcheetah medium
45.0
47.4=0.2
44.8+0..3
48.0+0.7
45.9=0.3
99.5
66.2:5.77
80.913.2
60.7+12.5
76.1
walker2d meddiu
92.0
78.348.7
80.910.6
83.745.3
76.6=0.4
malfchectah-mediiu--rpllla
42.4
4.22+
42.3++0.1
44.4+-0.8
43.0+004
hupper-medium-reppe
98.6
94.7+-8.6
92.7++2.9
64.8+22.5
96.3+3.0
faikker2d-meduum--eee
89.9
73.8+7..1
74.0
87.4++8.4
73.1+2.2
halfcheetah-men weaeriaaa--miiimepeeeeeeee
92.8
86.7.5..3
89.3
93.5112.0
95.7
hopper-mediume medium-expert
116.0
91.5114433
97.5.15.8
100.2.1.0.0
107.113.3
walker2d walker2d-mediun----ee medium--expert
109.0
109.6.1..0
110.9
109.510.55
109.710.2
Total Score
785.2
692.4
713.3
692.3
723.5
Comparison of SEABO against recent baselines. We report the mean normalized scores and the corresponding standard deviations. We bold and highlight the mean score cell if is close to beats IQL. trained the raw rewards.
Table 2:
Task Name
10%BC
IQL
IQL+ORIL
IQL+UDS
IQL+OTR
IQL+SEABO
halfehcetah-mediie
42.6
42.5
47.4
49.0
42.4+0.3
43.2+:..2
44.8=00.3
hopper-medium
52.9
56.9
66.2.15.7
47.0
54.513..0
74.2.15.1
80.913.2
walker2d-medium
75.3
75.0
78.3.18.7
61.916.6
68.916.2
78.7112.2
80.9
36.6
40.6
44.21:1.2
44.1-10.6
37.94:2.4
41.810..3
42.3.10.1
hupper-mediung replay
18.1
75.9
94.7188.6
82,4=1.7
49.3:122.7
85.4+0.8
92.7=2.9
walker2d medium meplum
26.0
62.5
73.847.1
76.314.9
17.749.6
67.216.0
74.0=2.7
halfcheetah- tetthhettth-mmddum-t -edeum-expprrrt expert
55.2
92,9
86.715..3
87.5+3.9
63.0+5..7
874144
89.3+2.5
hopper-medium-expe
52.5
110.9
91.5+14.3
29.7++222
53.9+2.55
88.4++1.6
97.5+
walker2d-medium-eexpe
107.5
109.0
109.6+1.0
110.6+0.6
107.5+1.7
109,5+00.3
110.9
Total Score
466.7
666.2
692.4
588.5
495.1
675.8
713.3
unlabeled trajectories; UDS (Yu al., 2022), which keeps the rewards in the expert demonstrations and simply assigns minimum rewards to the unlabeled data; OTR (Luo et al., 2023), which learns a reward function using the optimal transport to get the optimal alignment between the expert demonstrations and unlabeled trajectories. For fair comparison, all these methods adopt IQL as the base algorithm. We additionally compare against BC, and 10%BC (Chen et al., 2021). We take the results of IQL+ORIL and IQL+UDS directly from the OTR paper. As OTR computes rewards using pure observations (and SEABO uses (8, a, 8') to query the reward), we modify its way of solving optimal coupling by involving actions, and run IQL+OTR on these datasets with its official codebase. We summarize the comparison results in Table 2. It can be found that, though methods like ORIL and OTR can lead competitive better performance trained with raw rewards, SEABO beats them numerous tasks. Mcanwhile, SEABO is the only method that can surpass IQL with ground- truth rewards in terms of the total score.
some of the datasets than IQL
SEABO evaluation on wider datasets. We further evaluate IQL+SEABO on two challenging do- mains from D4RL,, AntMaze and Adroit. We run IQL with ground-truft rewards to obtain the IQL performance. We take the results of IQL+OTR from its paper directly. Table 3 demonstrates the detailed comparison results. We find that IQL+SEABO beats IQL and IQL+OTR on out of 6 datasets on Ane and outperforms baselines on 6 out of 8 datasets on Adroit, often by large margin. IQL+SEABO incurs a performance improvement of 6.0% and 32.0% beyond IQL with vanilla rewards AntMaze and Adroit tasks, respectively. These indicate that SEABO with one single expert trajectory can handle datasets with diverse behavior, and work as a good and promising proxy to the hand-crafted rewards.

---

Published as a conference paper at ICLR 2024
Table Experimental results on the AntMaze-vO and Adroit-v domains. SEABO and OTR use IQL as the base algorithm. IQL denotes that IQL uses the ground-trute reward for policy learning. We report the mean normalized scores and the corresponding standard deviations. We bold and highlight the best mean score cell.
Task Name
IQL
IQL+OTR
IQL+SEABO
Task Name
IQL
IQL+OTR
IQL+SEABO
pen-human
70.748.6
66.8411.2
94.3112000
uniaee
87.54:2.6
33,41333
90.01:188
pen cloneleeed
37.2:1..3
26.9:
48.71:15.3
diverse
62.2.1138
66.2.17.2
door deor-cloned
5.1
5.112..0
mediun-divvere
70.0:110.9
70.4=4.8
72.2::4.1
16:00.5
0.0:0.00
0,40088
medium play
71.2.17.3
70.5.16.6
71.6115.4
relocute human
0.1.10.0
0.1
0.410.5
large-diverse
47.54.9.5
45.5
50.0
relocate-cloned
-0.2=20.0
-0.21.000
-0.2-0.0
largo-play
39.645.8
45.3=6.9
8..8
bammer-human
1.6+0.66
1.8114
2.7
bammer-cloned
2.1+1.0
0.9.0.3
2.240.88
Total Score
378.0
384.0
400.8
Soore
1164
122.2
Comparison of SEABO against imitation learning algorithms. We use IQL as the base algorithm for SEABO and PWIL. PWIL-action means that we concatenate state and action to compute rewards in PWIL. We report the mean performance at the final 10 episodes of evaluation for each algorithm, + captures the standard deviation. We highlight the best mean score cell.
Table 4:
Task Name
SQIL
DemoDICE
SMODICE
PWIL-action
SEABO
halfcheetah-medium
31.3+
42.54:177
41.7+
44.4+0..2
44.8
hopper-medium
44.74-20.1
55.1+3.3
56,3+22.3
6.
80.9+33.2
malker2d-mediue
59.617.5
73.412.6
13.31992
72.6:6.33
80.9
halfchhetal-meediu--eelle.
medium-replay
29.31:2.2
38.1122.7
38.71:2.4
42.6.10.5
42.311.11
hopper-medium- medium-replay replay
45.21123.1
39.01115.4
44.3119.7
94.017.0
92.7+2.9
walker2d-medium medium-replayy
36.3+13.2
52.2+13.1
44.6+23.4
41.9+6..0
74.0+2.7
halfcheetah-mediing
40.1+6.4
85.8+5.77
87.9+-.88
89.5+3.6
89.3+2.5
hopper-mediumg expert
49.8+5.88
92.3+14.2
76.0+8,6
70.9+35.1
97.5+5.8
walker2d medimm-edeeeeeeeeee eexpprtrre-tttt
35.9112.2
106.911.9
47.8131.1
109.810.22
110.9
Total Score
372.2
585.3
450.6
626.1
713.3
5.2 COMPARISON AGAINST OFFLINE IL ALGORITIIMS
To further show the advantages of SEABO, we additionally compare it against recent strong offline imitation learning approaches, including DemoDICE (Kim et al., 2022b) and SMODICE (Ma al., 2022). We also convert two online II. algorithms into the offline setting, SQIL (Reddy et al., 2020) and PWIL (Dadashi al., 2021), where replace the base algorithm in SQIL with TD3_BC and utilize IQL as the base algorithm for PWIL. All algorithms are run using their official implemen- tations under the identical experimental setting as SEABO (i.e., one single expert demonstration). For fair comparison, we involve actions when training discriminators in SMODICE and measur- ing the distance in PWIL. We use IQL as the base algorithm for SEABO. The empirical results in Table 4 show that IQL+SEABO achieves the best performance on out of 9 datasets, and has the highest total score (surpassing PWIL some datasets, it significantly beats PWIL on tasks like hopper cepperr-mdiii um-v2. Note that SMODICE behaves poorly on many tasks, which is also observed in Li et al. (2023).
second highest one by 13.9%). Though SEABO underperforms
5.3 STATE-ONLY REGIMES
We now examine how SEABO behaves when the expert demonstrations consist of only observa- tions, i.e., D. {rí { f=il where M is the size of the demonstration and {sp, 81 $1...., ST). In principle, SEABO can also calculate rewards by querying the KD-tree with only states, ($ 8
HearestNeighbor (De, (s)))). The distance can then be calculated with some distance metric D,d= {,), s')), and the rewards can be computed accordingly, via Equation 3. For base- lines, since DeCE and ValueDICE are inapplicable to state-only regimes (Zhu et al., 2021), compare against LobsDICE (Kim et al., 2022a), which is a state-of-the offline IL algorithm that learns from expert observations. We also involve SMODICE, PWIL, and OTR for comparison, and train them using only expert observations. All baselines
with their official implementa-

---

Published as a conference paper at ICLR 2024
Table Experimental results on the state-only regime. SEABO, PWIL, and OTR utilize IQL as the base offline RL algorithm. PWIL-state denotes that PWIL only uses observations to compute re- wards. The results are averaged over the final 10 evaluations, and + captures the standard deviation. We highlight the cell with the best mean performance.
Task Name
SMODICE
LobsDICE
PWII--state
OTR
SEABO
malfchectahh-mdding
41.1#2.1
41.5++18
0.1+0.6
43.3+-0.2
45.0++0.2
hopper-medium
56.5
16.9+1.4
1.440.5
78.7+5.5
74.7+
melkiumd-medium
18.5
69.3+554
0.2+0.2
79.4+1.4
81.3++1.3
aalfcheetah-mediu--pphhee
39.2.13.1
39.913.11
-2.40.22
41.310066
42.4.00.6
hopper-medium-reppe
55.312214
41.6.116.8
0.7.0.22
84.8.12.6
88.011..7
walker2d-meediun-reee
37.8110.2
33.2.1..0
-0.2.1.2
66.016.77
76.4
halfcheetahh medium- expert
88.0#4.0
89.4+3.2
0.0
89.6#3.0
91,8
hopper-medium expert
15.1
53,443.2
2.742.1
93.24200.6
97.5+6.4
salker2d-medium-e expeett
32.3+14.7
106.6:2.77
0.2
109.3+0.8
110.5=0.3
Total Score
440.8
531.8
2.7
685.6
707.6
Table 6: Comparison of different choices of search algorithms in SEABO. We report the mean
normalized scores with standard deviations. We highlight the best score cell except for IQL.
Task Name
IQL
SEABO (KD-tree)
SEABO (Ball-tree)
SEABO (HNSW)
halfcheetah-meete meddiu
47.4
44.8.1..3
44.910.3
42.1110.6
hopper-medium
66.2.15.7
80,913.2
80.7.13.7
47.241..9
walker2d -mediimm
78.3=8.7
80,9:0.6
80.8+0.6
30.7+19.9
halfcheetah medium replay
44.2=1.2
42.3+0.1
42.5+:0.3
26.9+4.22
hopper- medium replay
94.7=2.6
92.712.99
92.1#2.3
25.8
walker2d- meeer2dd-mddimmrppppaa
73.8+7.11
74.0+-2.7
74.3+2.0
10.1
halfcheetah-medium-e-ppe
86.7+-5.3
89.
89.2+2.44
34.5+-2.2
hopper-medium-expe
11.5
97.5++5.8
96.7++6.2
41.5++..7
walker2d-medium-eet exppett
109.6:11.0
110.910.2
110.910.1
108.6110.8
Total Score
692.4
713.3
712.1
386.4
tions and single expert demonstration. The results in Table suggest that SEABO outperforms other methods on 8 out of 9 tasks, achieving a total score of 707.6, while LobsDICE and OTR only have total score of 531.8 and 685.6, respectively. It indicates that SEABO can work quite well regard- less of whether the expert demonstrations contain actions, further demonstrating the advantages of SEABO. Note that the failure of PWIL state-only regimes is also reported in Luo et al. (2023).
5.4 COMPARISON DIFFERENT SEARCH ALGORITHMS
The most critical component in SEABO is the nearest neighbor search algorithm. It interesting to check how SEABO performs under different search algorithms. To that end, we build SEABO on top of Ball-tree (Omohundro, 1989; Liu et al., 2006), and HNSW (Hicrarchical Navigable Small World graphs, Malkov & Yashunin (2018)). These are widely applied nearest neighbor algorithms, where Ball-tree partitions regions via hyper-spheres and HNSW is fully graph-based search structure. We allow the single expert demonstration to involve actions (i.e., query with (s, a, s')), and run all of the variants of SEABO using the same set of hyperparameters for fair comparison. Empirical results on 9 D4RL locomotion datasets are shown in Table 6. It is interesting to see that SEABO with Ball- is competitive with SEABO with KD-tree (their performance differences are minor), while SEABO with HNSW exhibits poor performance on many datasets. This means that the choice of the search algorithm counts in SEABO, and simply employing KD-tree can already guarantee good performance. Please see more discussions in Appendix C.
5.5 PARAMETER STUDY
is vital to examine how sensitive SEABO is to the introduced hyperparameters. Due to the space limit, we only report part of the experiments here and defer more experiments Appendix B.3.
8

---

Published as a conference paper at ICLR 2024
a
a=10
haalschoeiaa-mmdiii---eeia---
hopper-medium--2
wallker22-modduu--rpaayy--
Normalized score
8
Normalized score
8
10
18
Million Steps
Million Steps
Million Steps
Figure 3: Parameter study on the reward scale. The shaded region denotes the standard deviation.
P- 0.1
0.5.5
A
hattchestah-meddm-rrss N
N 10
haffcheetah-meduume seppueett2
hoppermeokummi2
hopper-meddum-2
Normaizodd soore
Normatzed score
Normalzed score
Normatoed
Mlllon
Stops
Millinn
(a) Weighting coefficient 8.
(b) Number of neighbors N.
Figure 4: Parameter study of (a) weighting coefficient 3, (b) number of neighbors N. The shaded region captures the standard deviation.
Reward scale a. o controls the scale of the resulting rewards. To check its influence, conduct experiments on three datasets from D4RL locomotion tasks and sweep a across {1, 5, 10]. Results in Figure 3 demonstrate that the best may depend on the dataset while smaller preferred.
Weighting coefficient 3. 3 is probably the most critical hyperparameter which decides the scale of the distance. In Figure 4(a), we vary |3 across {0.1, 0.5, 1,5), and find that the performance drops with too small or too large 3. It seems that B = 0.5 or B = 1 can achieve a good trade-off.
Number of neighbors N. To see whether the number of neighbors N matters, we run IQL+SEABO with N E {1, 5, 103. Results in Figure 4(b) show that SEABO is robust to this hyperparameter.
Number of expert demonstrations K. We investigate whether increasing the number of expert demonstrations can further boost the performance of SEABO and baselines by running experiments of these methods on MuJoCo locomotion tasks. We report the aggregate performance (i.e., total score) in Table 7. One can see that all methods enjoy performance improvement when K = 10, while none of them can outperform SEABO (there still exists a large performance gap).
Table Comparison of SEABO against baseline algorithms under different amounts of expert demonstrations. We report the aggregate performances and bold the best one.
# demo
DemoDICE
IQL+ORIL
IQL+UDS
IQL+OTR
IQL+PWIL
IQLASEABO
K 1
585.3
588.5
495,1
685.6
626.1
713.3
K 10
589.3
618.3
575.8
694.2
638.0
716.1
6 CONCLUSION
In this paper, we propose a novel search- offline imitation lcarning method, dubbed SEABO, that annotates the unlaheled offline trajectories in an unsupervised learning manner. SEABO builds a KD-tree using the expert demonstration(s) and searches the nearest neighbors of the query sample. We then measure their distance and output the reward signal via a squashing function. SEABO is easy to implement and can be incorporated with any offline RL algorithm. Experiments on D4RL datasets show that SEABO can incur competitive or even better offline policies than pre-defined reward functions. SEABO can also function well if the expert demonstrations observations. For future work, it is interesting to apply SEABO in visual offline RL datasets (e.g., Lu et al. (2022b)), or adapt SEABO cross-domain offline imitation learning tasks.
made up of only

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the STI 2030-Major Projects under Grant 2021ZD0201 and NSFC under Grant 62250068. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Saurabh Arora and Prashant Doshi. A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress. Artificial Intelligence, 297:103500, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic Bootstrapping for Uncertainty-Drive Driven Offline Reinforcement Learning. In International Conference on Learning Representations, 2022. URL https: /tttps://ppeee //oprreeee//er.eee..
net/ forum?
33
Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-End Differentiable Adversarial Imitation Learning. In International Conference Machine Learning, 2017.
Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communi- cations of the ACM, 18(9):509-517, 1975.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative Entropy Inverse Reinforcement Learning. In International Conference on Artificial Intelligence and Statistics, 2011.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. ArXiv, abs/2106.01345, 2021.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially Trained Actor Critic for Offline Reinforcement Learning. In Proceedings of the 39th International Conference Ma- chine Learning, 2022. URL
mlr .ww.sppess..... vv162/ cheng22b.
html.
Kamil Ciosek. Imitation Learning by Reinforcement Learning. In International Confer- ence on Learning Representations, 2022. URL http::/s////ppreeieieee 1zwleytEpYx.
Samuel Cohen, Brandon Amos, Mare Peter Deisenroth, Mikael Henaff, Eugene Vinitsky, and Denis Yarats. Imitation Learning from Pixel Observations for Continuous Control, 2022. URL https:
/ /poenreeveew net/ /tte//foriee fooommmiii www.mmmdd-Jb.
Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal Wasserstein Imitation Learning. In International Conference on Learning Representations, 2021. URL
htttp::/cppereeee htt.://ewww...ttooooooooootttsS t
net
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414- 419, 2022.
Branton DeMoss, Paul Duckworth, Nick Hawes, and Ingmar Posner. DITTO: Offline Imitation Learning with World Models. ArXiv, abs/2302.03086, 2023,
Edsger W. Dijkstra. A note two problems in connexion with graphs. Numerische Mathematik, 1: 269-271, 1959.
James E Doran and Donald Michie. Experiments with the graph traverser program. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 294((437):235-255) 1966.
Stefan Edelkamp and Stefan Schrödl. Heuristic search: theory and applications. Elsevier, 2011.
10

---

Published as a conference paper at ICLR 2024
Gideon Joseph Freund, Elad Sarafian, and Sarit Kraus. A Coupled Flow Approach to Imitation Learning. In International Conference on Machine Learning, 2023.
Justin Fu, Aviral Kumar, Ofrr Nachum, G. Tucker, and Sergey Levine. D4RL: Datasets for Deep
Data-Driven Reinforcement Learning ArXiv, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference on Machine Learning. 2019.
Seyed Kamyar Seyed Ghasemipour, Richard S. Zemel, and Shixiang Shane Gu. A Divergence Minimization Perspective on Imitation Learning Methods. In Conference on Robot Learning, 2019.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay Pol- icy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, ArXiv, 0bs//191.11196, 2019.
Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation. ArXiv, 0tss/2305.12822. 2023.
Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. In Advances in Neural Information Processing Systems, 2016.
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal M. P. Be- hbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Alexander Novikov, Sergio Gomez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie. Ziyun Wang. Bilal Piot, and Nando de Freitas. Acme: Re- search Framework for Distributed Reinforcement Learning. ArXiv, abs/2006.0099 2020. URL www.nticscheeeccom www.nntcccchlaarr wwwpeaarrrrrrrerreeeeeeeeeeeeeee Cttp://orpssIiI-C 221116617777
https: /ppi.
Michael Janner, Qiyang Li, and Sergey Levine. Offline Reinforcement Learning as One Big Se- quence Modeling Problem. In Advances in Neural Information Processing Systems, 2021.
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strietly Batch Imitation Learning by Energy- based Distribution Matching. In Advances in neural information processing systems, 2020.
Wonseok Jeon, Seokin Seo, and Kee-Bung Kim. Bayesian Approach Generative Adversarial Imitation Learning. In Neural Information Processing Systems, 2018.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha S. Srinivasa. Imitation Learning as f-Divergence Minimization. In Workshop on the Algorithmic Foundations of Robotics, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- Based Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2020.
Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Kee-Eung Kim. Lobs- DICE: Offline Learning from Observation via Stationary Distribution Correction Estimation. In Advances in Neural Information Processing Systems, 2022a. URL URLhhtisss//pperreviee net/Eorum?in Bit.ummmddBBUU56KK MtV.
Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. DemoDICE: Offline Imitation Learning with Supplementary Imper- feet Demonstrations. In International Conference on Learning Representations, 2022b. URL https: htttps://operreview net fo http:://ggggeeeeeeeeeeeeeeeeeeea

---

Published as a conference paper at ICLR 2024
Kuno Kim, Akshat Jindal, Yang Song, Jiaming Song, Yanan Sui, and Stefano Ermon. Imitation with Neural Density Models. In Neural Information Processing Systems, 2020.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representation, 2015.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-1277 2013.
Richard Korf. Depth-first iteratie-ddeepennng: An optimal admissible tree search. Artificial intel- ligence, 27(1):97-109, 1985.
Richard F. Korf. Artificial Intelligence Search Algorithms. In Algorithms and Theory of Computa- tion Handbook, 1999.
Iya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Perrminnttor-Acto--Citte Addressing Sample Inefficiency and Reward Bias in Adversar- ial Imitation Learning. International Conference on Learning Representations, 2019. URL
net wtt.://wwwweeeeouuiiiiiieee... www.mmidddtteeo ( fpoA5Km.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation Learning via Off-Policy Distribution Matching. In International Conference on Learning Representations, 2020. URL https:// openreview net/ forum Ho
Iyy Kostrikov, Ashvin Nair, and Sergey Levine Offiine Reinforcement Learning with Implicit
Q- Q-Learning, In International Conference on Learning Representations, 2022. URL https:
//wppneeeieww...o////////......e wwtt/errtiteeeeee www.rmmmmmdddddmmmmeemmmmm 92
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative Q-Leaning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 2020.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch Reinforcement Learning. In Rein- forcement Learning, 2012.
Youngwoon Lee, Edward S. Hu, Zhengyu Yang, Alexander Yin, and Joseph J. Lim. IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks. 2021 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 6343-63444 2019.
Youngwoon Lee, Joseph Lim, Anima Anandkumar, and Yuke Zhu. Adversarial Skill Chaining for Long-Horizzl Robot Manipulation via Terminal State Regularization. In Conference on Robot Learning. 2021.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. ArXiv, abs/2005.01643, 2020.
Anqi Li, Byron Boots, and Ching-An Cheng. MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations. In International Conference on Machine Learning, 2023.
Ting Liu, Andrew W Moore, Alexander Gray, and Claire Cardie. New Algorithms for Efficient nigh--Diennsionll Nonparametric Classification. Journal of Machine Learning Research, 7(6), 2006.
Cong Lu, Philip Ball, Jack Parker-Holder, Michael Osborne, and Stephen J. Roberts. Revisit- ing Design Choices Offline Model Based Reinforcement Leamning In International Confer- ence on Learning Representations, 2022a. URL http://s://///eeeeie///// net /etee/forumm forum?id= zz9hXVhf4C.
Cong Lu, Philip Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osbore,, and Yee Whye Teh. Challenges and Opportunities Offline Reinforcement Learning from Visual Observations. arXiv preprint XxXiv:2206.04779 2022h.
12

---

Published as a conference paper at ICLR 2024
Yicheng Luo, zhengyao jiang, Samuel Cohen, Edward Grefenstette, and Mare Peter Deisen- roth. Optimal Transport for Offline Imitation Learning. In The Eleventh International Confer- Learning Representations, 2023. URL https: https/////ppeeeeiieeeeeee net. / forum?id- MhuF2FsrFi
Jiafei Lyu, aicheng Gong, Le Wan, Zongqing Lu, and Xiu Li. State Advantage Weighting for Offline RL. In 3rd Offline RL Workshop: Offline RL as "Launchpad", 2022a. URL https:
//openreview. wet./oormmidd22000011110
Jiafei Lyu, Xiu Li, and Zongqing Lu. Double Check Your State Before Trusting It: Confidence- Aware Bidirectional Offline Model-Based Imagination. In Advances Neural Information Pro- cessing Systems, 2022b. URL httpa::/opeereve//e
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2022c. URL
net/ (errumiddVYYYGG67777
Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile Offline Imita- tion from Observations and Examples via Regularized State-Occupancy Matching. In Interna- tional Conference on Machine Learning, 2022.
Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions pattern analysis and machine intelligence, 42(4))824-836, 2018.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level Control through Deep Reinforcement Learning. Nature, 518:529-533, 2015. URL https: https:////p..aeeeeee wwweeanniicchhee org/ CorpusID:20524221
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections. In Advances neural information processing systems, 2019.
Stephen M Omohundro. Five balltree construction algorithms. Technical report, International Com- puter Science Institute, 1989.
Jyothish Pari, Nur Muhammad (Mahi) Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effecttvenness of representation learning for visual imitation. ArXiv, abs/2112.01511, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Neural Information Processing Systems, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leamn Machine learning in Python. Journal of Machine Learning Research, 2022825-2830 2011.
Ira Pohl. Heuristic search viewed as path finding in a graph. Artificial intelligence, 1(3-4):193-244, 1970.
Dean Pomerleau. Alvinn: autonomous land vehicle in a neural network. In Advances in Newral Information Processing Systems, 1988.
Yuhang Ran, Yichen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy Regularization with Dataset Constraint for Offline Reinforcement Learning. In International Conference on Machine Learning 2023.
13

---

Published as a conference paper at ICLR 2024
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. {SQIL): Imitation Learning via Reinforce- ment Learning with Sparse Rewards. In International Conference on Learning Representations, 2020. URL https: www.eeneeieee.ee/rrrreee..
wB.
Mare Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2022.
URL bttt:p:://ppereeee net.//eeeeeeeeeeeeeeeeeeee.eeeee forum? id=nrk
Stéphane Ross, Geoffrey J. Gordon, and Andrew Bagnell. Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In International Conference on Artificial Intelligence and Statisties, 2011.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon
Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
Mark Stickel and Mabry Tyson. An analysis of consecutively bounded depth-first search with applications in automated deduction. In IJCAI, pp. 1073-1075, 1985.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Larry A. Taylor and Richard E. Korf. Pruning Duplicate Nodes in Depth-First Search. In AAAI Conference on Artificial Intelligence, 1993.
Emanuel Todorov, Tom Frez, and Yuval Tassa. MuJoCo: A Physics Engine for Model-based Con- trol. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent Advances in Imitation Learning from Obser- vation. In International Joint Conference on Artificial Intelligence, 2019.
Masatoshi Uehara and Wen Sun. Pessimistic Model-based Offline Reinforcement Learning under
Partial Coverage. In International Conference on Learning Representations, 2022. URL https:
// //oppeeeview..eeeeeee net htt//formm/// wwtorummdd-tty... wow.mmidttyyssaaeeeee.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272. 2020.
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Y. Demiris. Support-weighted Adver- sarial Imitation Learning. ArXiv, abs/2002.08803, 2020.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator- Weighted Offline Imita- tion Learning from Suboptimal Demonstrations. In International Conference on Machine Learn- ing, 2022.
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-exploration with distributional random network distillation. ArXiv, abs/2401.09750, 2024.
Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching. In AAAI Confer- ence on Artificial Intelligence, 2023.
Tianhe Yu, Garrett Thomas. Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. In Advances in Neural Information Processing Systems, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative Offline model-Bassed Policy Optimization. Advances in Neural Infor- mation Processing Systems, 2021.
14

---

Published as a conference paper at ICLR 2024
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to Leverage Unlabeled Data in Offline Reinforcement Learning. International Conference on Machine Learning. 2022.
Sheng Yue, Guanbo Wang, Wei Shao, Zhaofeng Zhang, Sen Lin, Ju Ren, and Junshan Zhang. CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learn- ing. In The Eleventh International Conference on Learning Representations, 2023. URL
https:// http::://ppeneeviie mott:foruurii--aaaaaaoaoooo
Junjie Zhang, Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Jun Yang, Le Wan, and Xiu Li. Uncertainty-driven Trajectory Truncation for Model-based Offline Reinforcement Learning. ArXiv, abss2230400466,, 2023.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent Action Space for Offline Rein- forcement Learning. In Conference on Robot Learning, 2020.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou, Off-Policy Imitation Learning from Obser- vations. In Neural Information Processing Systems, 2021.
Brian D. Ziehart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum Entropy Inverse Reinforcement Learning. In AAAI Conference Artificial Intelligence, 2008.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyun Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott E. Reed. Offline Learning from Demonstrations and Unlabeled Experience. ArXiv, abs/201 13858 2020.
15

---

Published as a conference paper at ICLR 2024
A HYPERPARAMETEE SETUP
In this section, detail the hyperparameter setup utilized in our experiments. We conduct exper- iments on 9 MuJoCo locomotion "-v2" medium-level datasets, AntMaze "-v0" datasets, and Adroit "-v0" datasets, yielding total of 23 tasks. We list the hyperparametter setup for IQL and TD3_BC on MuJoCo locomotion tasks in Table 8. We keep the hyperparameter setup of the base offline RL algorithms unchanged for both IQL and TD3 BC. IQL. do not rescale the rewards in the datasets by 1000/max return min return, as we have an additional hyperparameter a to control the reward scale. In practice, we find minor performance differences if we rescale the rewards. We generally utilize the same formula of squashing function for most of the datasets, except that we set
= 1 in hopper-medium-repkaa v2, and a 10, 3 0.1 in hopper medium-expert--> for better performance. Note that using
1, 0.5 these tasks can also produce good performance (e.g. setting a = 1,3 = 0.5 on hopper-medium wwdi-m-ddmmepppeeeeeee -erdiiimrrpla-2222 leads to an average performance of 87.2, still outperforming strong baselines like OTR), while slightly modifying the hyperparameter setup can result in better performance. We divide the scaled distance by the action dimension of the task strike balance between different tasks (as also adopted in PWIL paper (Dadashi et al., 2021). For TD3 BC, we use the same type of squash- ing function as IQL on the locomotion tasks, with a 1, 0.5, except that use 10 and weakkeedd-mmdiu--reppeee -22 for slightly better performance. We use
of hyperparameters) This is
for walker2d-medium- the official implementation TD3 TD3_BC (https://github.. com/s/fjjm/TT3.BBB. and adopt the PyTorch (Paszke et al. (2019)) version of IQL for evaluation.
Table 8: Hyperparameter setup of SEABO on locomotion tasks, with IQL and TD3 BC as the base
offline RL algorithms.
Hyperparameter
Value
Shared Configurations
Hidden layers
(256, 256)
Discount factor
0.99
Actor learning rate
3 10-4
Critic learning rate
3 x 10-4
Batch size
256
Optimizer
Adam (Kingma & Ba, 2015)
Target update rate
10-3
Activation function
ReLU
IQL
Value learning rate Temperature
10-4
3.0
Expectile
0.7
TD3_BC
Policy noise Policy noise clipping Policy update frequency
0.2
(-0.5, 0.5)
Normalization weight
2.5
SEABO
Squashing function
r= exp( 0.5xd)
[A]
Distance measurement
Euclidean distance
Number neighbors
Number of expert demonstrations
We summarize the hyperparameter setup of SEABO (using IQL as the underlying algorithm) on the AntMaze domain and Adroit domain in Table 9 and Table 10, respectively. We only list the different hyperparameters in these tables and the other hyperparameters follow those presented in Table 8. Note that we filter the highest return trajectory as the expert demonstration in the Adroit domain, while selecting the goal-reached trajectory as the expert demonstration in the AntMaze domain, which is also adopted in OTR paper (Luo et al., 2023). We adopt comparatively large 3 = 5 on AntMaze tasks. We also follow the IQL paper (Kostrikov et al., 2022) to subtract from the rewards, which find can result in better performance. For Adroit tasks, remove the action dimension in the squashing function, since these tasks have large action space dimensions. If one insists involving [A], much larger 3 than 0.5 is then necessary mitigate its influence. We find
16

---

Published as a conference paper at ICLR 2024
Table 9: Hyperparameter setup of SEABO on AntMaze tasks, with IQL as the base offline RL algorithm.
Hyperparameter
Value
IQL
Temperature
10.0
Expectile
0.9
SEABO
Squashing function
r exp(
5xd
[A]
Table 10: Hyperparameter setup of SEABO Adroit tasks, with IQL, as the base offline RL. algo- rithm.
Hyperparameter
Value
IQL
Temperature
0.5
Expectile
0.7
Actor dropout rate
0.1
SEABO
Squashing function
exp(-0.5 x d)
that simply removing A can ensure quite good performance on all of the evaluated Adroit datasets. Note that OTR (Luo et al., 2023) also adopts different forms of squashing functions for different domains. We query with (s, s') for Adroit tasks and (s, a, s') for other domains.
To acquire expert demonstrations, we use the trajectory with the highest return as expert demon- strations on MuloCo locomotion tasks and Adroit tasks, and filter the goal-reached trajectory in AntMaze tasks. For all of the baseline reward learning and offline imitation learning algorithms, we follow this setting and run them with their official codebases' over five different random seeds. We use the PWIL implementation from Acme (Hoffman et al., 2020)².
In SEABO, we use the KD-tree implementation from the seipy library (Virtanen al., 2020), i.e. scipy spatial.KDTaa.. default hyperparameterrs in KD-tree. Note that we can directly get the desired distance by query- its implementation in the scikit-learn package (Pedregosa ing the KD-tree. For Ball-tree, wwthhbbrs..aarrree..eeeeeeee.eee BaleTree. We also keep its original hyperparameters et al., 2011), i..,, sklearn ne i ghbors. unchanged. For HNSW, we its implementation in hnswlib' We use the suggested hyperpa- rameter setting in its GitHub page and set e f_constttttommeeeooce ruct construct on=200 (which defines a construction time/accuracy trade-off) and M=1 (which defines the maximum number of outgoing connections in the graph). All these search algorithms adopt the Euclidean distance as the distance measurement.
We set the number of nearest neighbors N 1, and keep other
(Todorov et al., 2012) with Gym version 0 3, PyTorch (Paszke et al., 2019) version 1. 8. We use the normalized score metric recommended in the D4RL paper (Fu et al., 2020), where corresponds to random policy, and 100 corresponds to an expert policy. Formally, suppose we get the average return J by deploying the learned policy the test environment, normalized score gives:
In our experiments, we MuJoCo
J Jr
Normalized score
x 100,
(4)
J. J,
where J, is the return of random policy, and Je is the return of an expert policy.
B MISSING EXPERIMENTAL RESULTS
In this section, present the missing experimental results from the main text due to the space limit.
2TTR official codehase:
SMODICE off-
cial codebase: http:///hhbbb.cooo///smmmm//2/// DemoDICE and LobsDICE official codebase: https//giiubbbooooegoooohyy//nnm hhtp:://gthhh.com/mmmihhhhnn
17

---

Published as a conference paper at ICLR 2024
Table 11: Comparison SEABO against baseline algorithms under 10 expert demonstrations.
We use IQL as the base algorithm for SEABO, PWIL, and OTR. We report the mean performance at the final 10 evaluations for each algorithm, and captures the standard deviation.
Task Name
IQL
Ptate
PWIL--cction
OTR-state
OTR-action
SEABO
meaccheetah-mddinee
47.4110.
1.61122
47.5+1..2
43.1
43.4:10.3
44,410.2
hopper-medium
66.215,7
2.1
70,414.2
80.0
75,41:466
81.413.5
walker2d-medium
78.3+8.7
0.9+1.3
81,9+1.0
79.2+1..3
79.7+1122
81.1+00.7
halfchhcctah-meddu--rrppll
4..2+11.2
-2.3+0.5
44.6+1111
41.6++0.3
41.9+0..
43.9-=0.2
hopper-modiim--eellyy
94.74+8.6
1.4
89.7+4499
84.4+1.8
85.3+1.1
86.4+1.4
waller2dd-medium--rplla
73.8.7.11
-0.11102
72.2.110.6
71.8.3.8
69.11:4.6
78.0
hallchhetab-mmediu---eooo
86.715.3
-0.311.5
88.6.14.3
87,913.4
88.315..1
90.5
hopper--nediun-eepert
91.5.114.3
1.510.6
32.9125.00
96.6111.5
86,6.122.9
100.0.7..0
109.6
1.01.9
110.2
109.6=0.55
109.2:00.5
110.4:00.6
Total Score
692.4
5.8
638.0
694.2
678.9
716.1
B. NUMERICAL COMPARISON UNDER TEN EXPERT DEMONSTRATIONS
In Section 5.5, we present the comparison results of SEABO and baseline reward learning and offline IL algorithms under different numbers of expert demonstrations K E {1, 10). However, only report the aggregate performance (i.e., the total score) on the MuJoCo locomotion medium- level tasks (medi um, um, medi um-replay, medi em-expert) in Table 7. To make the comparison
clearer, we present the detailed normalized scores of these methods under = 10 on different datasets in Table 11, where mainly compare SEABO against different variants of PWIL and OTR. SEABO computes the rewards with actions involved in the single expert demonstration here.
The results reveal that SEABO outperforms baseline methods 5 out of datasets and is com- petitive with baselines on the rest of the datasets. SEABO achieves a total score of 716.1, surpass- ing the second best method (OTR-state) by 3.2%. Though we observe that PWIL-action beats SEABO on datasets like halfcheeter aeeeeeett--meiiuee um- it can perform poorly on datasets like hopper-med -n-expert-vee22 We also note that the performance of PWIL deteriorates in the state-only regimes, i.e., learning from pure expert observations. This phenomenon is also reported Dadashi et al. (2021); Luo et al. (2023). SEABO, instead, is flexible and can be applied regardless of whether the expert demonstrations contain actions.
Furthermore, we show in Table the results SEL+O on AntMaze and Adroit datasets when 10 expert demonstrations are provided. We compare IQL++EABO against IQL+OTR and IQL. with raw rewards (denoted as IQL). The results demonstrate that SEABO can recover the performance of the offline RL algorithm with ground-truth rewards and sometimes yield better performance. This advantage is agnostic to the number of expert demonstrations K.
IQL+SEABO matches the performance of IQL+OTR many AntMaze tasks and outperforms IQL+OTR on out datasets from the Adroit domain. On both the AntMaze domain and Adroit domain, OTR underperforms SEABO in terms of the total score. One may notice that the perfor- mance of IQL+SAABO BO decreases with more expert demonstrations, mainly on the Adroit datasets. This is caused by the performance drop on ben-human-vee which dominate the total score (the magnitude of its score is much larger than those of other datasets). One can also observe that the performance of IQL+OTR declines many Adroit tasks, given 10 expert demonstrations (see Table 7 in Luo et al. (2023)). Still, IQL+SEABO exhibits strong performance across numerous datasets.
B.2 COMPARISON OF TD3 BC+OTR AND TD3 BC+SEABO
Since the majority of the experiments in the main text are conducted using IQL as the base offline RL algorithm, is interesting to see how SEABO competes against baseline methods with another offline RL algorithm the base method. To that end, we choose TD3. _BC and incorporate it with the strong baseline method, OTR. We follow the experimental setting utilized in the main text, filter single expert demonstration with the highest return in the offline dataset, and deem it as the pert demonstration. We run TD3 BC+SEABO and TD3 BC+OTR on D4RL MuJoCo locomotion datasets. We follow experimental setup specified in Appendix A, and use the default hyperpa-
18

---

Published as a conference paper at ICLR 2024
Table Experimental results SEABO on the AntMaze-vO and Adroit-vO domains with 10 expert demonstrations. SEABO and OTR use IQL as the base algorithm. The average normalized scores along with the corresponding standard deviations are reported. We bold and highlight the best mean score cell.
Task Name
IQL
IQL+OTR
IQL+SEABO
Task Name
IQL.
IQL+OTR
SEABO
pen-buman
70.74886
69.4221.5
858:116.1
untuuse
87.54:2.6
88.7
87.61.2.0
pen: pen-cloned
37.211..3
42.7::25.0
49.2:1:2.2
umaze umaze-diverse
62.2.113.8
18.2
70.0
door human
3.3
4.2
6.8.5..6
medium-divvese
10.0++00,9
70.5-669
70.2:45.4
door-cloned
1.6+:055
0.0:0.0
0.1
mediun-play
71.2
72.7=6.2
72.8=116
relocate-human
0.1+:0.0
0.1
0.1=0.1
large-diverse
47.54.9.5
50.7:26.9
50.0::7.9
relocate-cloned
-0.2--0.0
-0.2:0.00
-0.2:000
large play
39.6.15.8
51.217.1
48.6.1..8
hammer humnn
1.6.1.66
141002
1.710.3
bammer-cloned
2.1
1.320.7
1.7:0.5
Score
399.2
Total Score
1164
118.9
145.2
Table 13: Comparison SEABO against OTR using TD3 BC the base algorithm. We report the average normalized scores and their standard deviations. We bold and highlight the mean score cell except for TD3_BC We adopt one single expert demonstration for OTR and SEABO.
Task Name
BC
10%BC
TD3 TD3BCC BC
DD3_BC+OTR
1D3SBCO
halfcheetah-mediimm
42.6
42.5
48.0
42.6.11.0
45.9
hopper-medium
52.9
56.9
60.7+12..5
66.4410..3
76.1
walker2d-medium
75.3
75.0
83.715..3
76,915.4
76.6=10.4
halfcheetah- medium- replay
36.6
40.6
44.4
39.411.3
43.0:1..4
18.1
75.9
64.8+12.5
74.91:28.8
96.3+3.0
walker2dd-ecduu--relly
26.0
62.5
87.4+8.4
69.7+16.4
73.1+2.2
halfchhecta--mdiiu--xxe
55.2
92.9
93.5+2.0
74.8+-0011
95.7++0.4
hopper-medium-exper
52.5
110.9
100.2+20.00
103.2+13.9
107.1+3.3
107.5
109.0
109.510.55
109.0110.6
109.7.0.22
Total Score
466.7
666.2
692.3
656.9
723.5
rameter setup of OTR suggested by the authors. We summarize the comparison results in Table 13. It turns out that TD3_ BBCBSEBBOOOO outperforms TD3 [D3_BC+OTR on out of datasets, often by large margin, surpassing it by 10.1 % in terms of the total score. TD3 TT3_BC+SEABO is the only algorithm that even beats TD3_BC learned with raw rewards in total score. We observe that the standard de- viation of TD3 BC+OTR is large datasets like ha ww..eeeeehhhmne medi mediim-expees cemm-expertttee v2, while the standard deviation of TD3.BC D033BC+SEABO is much smaller. This evidence indicates that SEABO superior to OTR when acting as the reward labeler, and consistently aid different base offline RL algorithms recover its performance under ground-truth rewards or achieve better performance.
B.3 HYPERPARAMETER SENSITIVITY
In Section 5.5, are only able to attach the results small proportion of datasets from D4RL,, e.g. halfcheet ah aah-mediieeee um-replay-v2 due to the space limit. In this part, we include wider experimental results in of the reward scale a, weighting coefficient B, and number of neighbors N. Again, we use IQL as the base offline RL algorithm for SEABO. The expert demonstrations utilized here contain actions. We follow the hyperparameter setup specified Section
Reward scale a. The reward scale controls the magnitude of the computed rewards. In Figure 3 of the main text, we find that smaller a seems be better (especially on hopper-medi um- We further conduct experiments on three additional tasks, hal fcheetah ieeeettah--eeiue umm rep um-rep ay-v2, and wa ker2d-meat um-v2 by varying a E [1,5, 10). The results shown in Figure 5, where actually do not find much performance difference of a these three tasks. That indicates that IQL+SEABO is robust to a on most the datasets. In practice, can simply set 1, which find can already yield very good performance on MuJoCo tasks, AntMaze tasks, and Adroit tasks.
hopper r- um-
Weighting coefficient B. As commented in the main text, the weighting coefficient is per- haps the most important hyperparameter SEABO, since controls the weights of the measured distance and this may have significant influence the final rewards. For specific domain,
19

---

Published as a conference paper at ICLR 2024
a=1
a=
Q=10
halicheeai--mediuu-e-ppp-ttz-
hopper-meduun-eellay22
walkerzo-medium-22
103
Normalized score
Normalizzd score
103
Normalized score
80
60
0.0
02
0.4
0.6
0.8
1.0
00
0.2
0.4
0.6
0.8
1.0
0.0
62
0.4
0.6
0.8
10
Million Steps
Million Steps
Million Steps
Figure 5: Additional experiments on the influence a. The shaded region captures the standard
deviation. All other hyperparameters are kept unchanged except a.
0.1.1
8=
B
B ==3 5
hallcheebah-mediiu-eeppoo expert-v2
hoppee--meium-replee
wallker2d-meduu-reppple
103
Normalized score
Normalized score
Normalized score
%
60
90
60
60
0.0
0.4
0.8
0.2
0.6
1.0
0.0
0.
0.6
1.0
Million Steps
Million Steps
Million Steps
Figure 6: Additional experiments the effect of B. We choose three additional datasets from
D4RL, and plot their mean normalized score curve. The shaded area denotes the standard deviation.
we mostly adopt fixed B as we do not want to bother tuning this hyperparameter. However, we believe is vital to examine how influences the performance of SEABO in wider experi- ments. We additionally conduct several experiments on halfcheetah- from D4RL locomotion tasks. We sweep 3 across (0.1, 0.5, 1,5). and summarize the results in Figure 6. It can be clearly seen that a large 3 results poor performance on hal fchfeteet ah-medi -im-meddut--x and malkerrdd-medi on ueppeer-medium--eplle In the hyperparameter setup part, we state that we set B = due to the fact that SEABO comparatively stable with 3(0.5, 1). We do not doubt that the best / is task-dependent, and one can get higher performance by carefully tuning this hyperparameter. However, empirically show that using fixed is also feasible, and we believe this is appealing since the users can get rid of the work of tedious hyperpa- rameter search.
um expert-v2
22,, while setting 3 5 results in the best performance on
Number of neighbors N. The number of neighbors N is hyperparameter introduced in the nearest neighbor algorithms. For all of our main experiments, we simply adopt N = 1, i.e., searching for the nearest neighbor. In Figure 4(b), we see that SEABO is robust to this hyperparameter. To exam- ine whether this conclusion applies to wider range of datasets, we conduct experiments on three additional datasets, halfcheetah--eee wall ker2d aederdd-eeee 2--medi um repaaypay-v2 The results are summarized in Figure 7, where we also ob- serve that SEABO is robust to this hyperparameter, indicating the effectiveness and generality of SEABO.
v2, hal aalfcheetah-ee ah-medi um-uum-eeee um-expert-v22 and
B.4 PERFORMANCE OF SEABO UNDER LONG-HORIZON MANIPULATION TASKS
In this part, we investigate how SEABO behaves under long-horizon manipulation tasks. To that end, we evaluate SEABO in Kitchen datasets (Fu et al., 2020). The kitchen environment (Gupta et al., 2019) consists of a DoF Franka robot interacting with a kitchen scene that includes an openable microwave, four turnable oven burners, oven light switch, freely movable kettle, two hinged cabinets, and a sliding cabinet door. kitchen, the robot may need to manipulate different components, e.g., it may need to open the microwave, move the kettle, turn the light, and slide open the cabinet (precision required). We run IQL+SEABO on three kitchen datasets using the hysrommmmnndddyyyypraaaeeetteee
hyperparameters of IQL. the kitchen environment. We set reward
20

---

Published as a conference paper at ICLR 2024
==1
=
N=
halfcheetah--mediun-22
haitcheda-mndddu-eeerrtv
walker22-mmedium--plaayyy
100
Normalized score
46.0
Normalized score
Normalized score
80
62.5
60
40
40
20
825
300
02
0.4
0.5
08
1.0
00
0.2
0.4
0.5
0.8
1.0
0.0
02
0.4
0.5
0.8
10
Million Steps
Million Steps
Million
Steps
Figure 7: Additional experiments on examining the influence of the number of neighbors in
KD-tree. The shaded region represents the standard deviation.
Table 14: Comparison of SEABO against baselines in the Kitchen tasks. We report the average normalized scores and the corresponding standard deviations. We bold and highlight the best mean score cell.
Task Name
BC
CQL
IQL
IQL+SEABO
Clichen-complete-v0 v0
65.0
43.8
62.5
67.5+t4.2
kitchen-pand kitchen-partial-v00 partial-v0
38.0
49.8
46.3
71.0:t:.1
kitchen-mixed-v0
51.5
51.0
51.0
55.0+-3.5
Average Score
51.5
48.2
53.3
64.5
scale a L, coefficient 3 = 0.5 for SEABO, We compare IQL+SEABO against some baselines taken from the IQL paper and summarize the results Table 14. We find that SEABO exhibits superior performance, surpassing IQL. with raw rewards by 21.0 %. We believe these results show that SEABO can aid some long-horizon manipulation tasks.
However, we experimentally find that SEABO does not exhibit strong performance for some tasks that require high precision, e.g., the IKEA Furniture assembly benchmark (Lee et al., 2019; 2021; Heo et al., 2023). We leave the open problem of how to enable SEABO to successfully address such benchmarks future work.
B.5 LEARNING CURVES
In this section, we provide the detailed training curves of IQL+SEABO on the locomotion tasks, AntMaze tasks, and Adroit tasks. We also provide learning curves of TD3 TD3TTD3BBBBBC.C BCCBEABO on locomo- tion tasks. We summarize the results of IQL+SEABO on D4RL MuJoCo locomotion tasks in Figure 8, the performance of IQL+SEABO on AntMaze tasks in Figure and the curves IQL+SEABO on Adroit tasks in Figure 10. The results of TD3_BC+SEABB are depicted in Figure 11.
From all these results, we find that both IQL+SEABO and TEBABO have stable and strong performance on the evaluated tasks, indicating the advantages of our method.
C DISCUSSIONS ON DIFFERENT SEARCH ALGORITTMSS
The success of SEABO can be largely attributed to the adopted search algorithm (i.e. KD-tree). In Section 5.4 of the main text, we compare different design choices for the underlying search algorithm. It is not surprising to find that Ball-tree results in a similar performance as KD-tree, as Ball-tree shares many similarities with KD-tree. However, we find that HNSW incurs quite poor performance on many datasets using its default hyperparameter setup (see Appendix A). HNSW builds multi-layer structure made up of a hierarchical set of proximity graphs for nested subsets of the stored elements while employing a heuristic for selecting proximity graph neigh- bors. HNSW is a graph-based based search algorithm. Based on the empirical results in Table 6 in the main text, find that HNSW leads to quite poor performance for the base offline RL algo- rithm, only achieving competitive performance against KD-tree on halfcheet aneeeethhmeeeeeee sech-medium--e and wa ker2d-med fum -xm-exeet--22.
21

---

Published as a conference paper at ICLR 2024
haffcheetah-mediumm22
halliheetah-mmddin--ppla---
haffcheiiah-meddumeeppeet
Normalized score
65.0
46
42.5
40
60.0
25.0
525
QL-SEABO
25
IQL+SEABO
20
IQL-SEABO
300
20
Gradient Steps
Gradient Steps
Gradient Steps
hopper-mediie 2apper--edium-222
hoppe-medium-rellay-vv
hogper-mmedum-eepeerttz
Normalzzen zed score
100
50
76
25
40
QEL-SEBOO
IQL-SEABO
IQL-SE
Gradient Steps
Gradient Steps
Gradient Steps
wallker2d-medium-22
wa filkerdd-mmdium--xppe-----o
Normalized score
80
60
20
80
IELAS
IQL+SEABO
IELAS
40
0.4 0.8
0.0
0.8
0.0
0.8
Gradient Steps
Gradient Steps
Gradient Steps
Figure 8: Full learning curves of IQL+SEABO on D4RL MuJoCo datasets. We plot the average performance and the shaded region captures the standard deviation.
antmaze-umazzo-v0
arttmazz-ummaoo-iivrrs-vv-
anlrmazemmddum-divvrro-vv
Normalized score
103
20
19
80
40
40
20
IQL+SEABO
IQL+SEABO
IQL+SEABO
-20
0.0
0.2
0.4
26
1.0
6.0
02
C.4
ne
1.0
00
0.2
0.4
06
on
1.
Gradient Steps
Gradient Steps
Gradient Steps
antmaze-mede mattmaze-meddum-playyvv
antmaze Saarmazee-argg-eeo wwwpe--aaagg--eeeeeeeeeee diverse-vO
antmaze-largg-paae plage-
100
Normalized score
80
80
80
40
40
40
20
IQL+SEABO
IQLYSEABO
IQLAOEABO
Gradient Steps
Gradient Steps
Gradient Steps
Figure 9: Full learning curves of IQL+SEABO on AntMaze tasks. The mean performance in conjunction with the standard deviations are plotted.
22

---

Published as a conference paper at ICLR 2024
pen-human--0
pen-sooed-d
doo--huan--d
door-dooned-00
Normelized scone
ILL-EEAO
100 SEABO
Gradient
Gradiont Slops
Gradiont
Gradiont
relocalo-hummnn-000
raboate-doond--0
hammar-human--d
Pammes-Clanad--C
-09
Nomalzze
+030
25
Cradient Stops
Gradient Slops
Gradiont Steps
Gradient Stops
Figure 10: Full learning curves of IQL+SEABO on Adroit datasets. We report the mean perfor- mance along with its standard deviation.
hallcheotah--eddium-y
hallchhobbh-mmdiuu--rpaaa--
Normlizzdd score
66.0
42.5
60.0
60
20
10
96.0
TD3 BC-SEABO
TD3, BC-SEABO
20
BC-SEABO
0.2
2.4
0.6
0.5
0.0
0.2
04
0.8
0,8
1.0
0.0
0.2
04
0.8
0.8
1.2
Gradient Steps
Gradient Steps
Gradient Steps
hopper-mediu--replayy-22
hopper-mediie ceeptot--mduum-exeeetteeee
Normalized score
100
60
60
TD3. BC-SEABO
20
TD8 BC-SEABO
TD3, BC-SEABO
0.4
0.6
0.0
0.6
0.8
1.0
0.0
0.2
0.8
1,0
Gradient Steps
Gradient Steps
Gradient Steps
walker2d-medium--2
walker22-medduu--eelayy
wa walker2d- -ellur22dmmduuumeeeeetr--/
Normalized zed score
28
60
100
40
20
66
TD3 TD3_BC- BC-SEABO
TDS_ BC-SEABO
TD3, BC-SEABO
0.0
Gradient Steps
Gradient Steps
Gradient Steps
Figure 11: Full learning curves of TD3 SEABOO D4RL MuJoCo datasets. The average performance as well as its statistical significance is depicted.
23

---

Published as a conference paper at ICLR 2024
In this subsection, try to understand why HNSW fails through some empirical evidence. We choose some subsets, halfccheetab-mmmdiu-aaaa halfcheetah- -eweeeeiieeeeeeeeeeeeeeeee and wa htw.auuuua..eepaaaaaaaoooaeeeeee utp--eeettttdtt---eeeeee----- walker2d um-v2, from D4RL. MuloCo datasets and plot the reward den- sity of ground-truth rewards, rewards computed using KD-tree, and rewards acquired via HNSW. We summarize the results in Figure 12. It is clear that SEABO with KD-tree can produce a similar reward structure the ground-truth reward distribution, while SEABO with HNSW tends assign large rewards to only a small proportion of samples and small rewards to the majority of transitions. We believe this explains the unsatisfying performance of IQL+SEABO with HNSW as the base search algorithm, indicating that a graph-based search mechanism may not be suitable for D4RL datasets. Another possible explanation that the hyperparameters of HNSW need to be tuned to adapt to different tasks. We do not doubt that a careful tuning of hyperparameters (e.g., the maxi- mum number of outgoing connections in the graph, the number of neighbors, etc.) has the potential of making SEABO with HNSW work in D4RL datasets. However, to do that considering the fact that adopting KD-tree with its default hyperparameters can already result in quite good performance across different datasets. Hence, it is recommended that uses KD-tree (or Ball-tree) as the base search algorithm.
do not think it is necessary
D COMPUTE INFRASTRUCTURE
In Table 15, we list the compute infrastructure that we use to run all of the algorithms.
Table 15: Compute infrastructure.
CPU
GPU
Memory
AMD EPYC 7452
RTX3090x8
288GB
E LIMITATIONS
Despite the simplicity and effectiveness of our proposed algorithm, SEABO, we have to admit hon- estly that there may exist some potential limitations. First, SEABO is slightly sensitive to the weight- ing coefficient/ 3 on some datasets (not all datasets), and one may need to manually tune it so as to find the best-suited hyperparameter setup for specific task. While based one can find the best 39 E (0.5. 1.5] using grid search, It is not difficult to conduct experiments since SEABO is computationally efficient (and can be applied with only CPUs). Second, it may take more time for SEABO to annotate the unlabeled trajectories with visual input, as images are hard to pro- cess. Whereas, ImageNet pretrain models) to obtain low- www-dimensional representations of the high-dimensional im- age. Note that we build KD-tree upon expert demonstrations which usually contain small amount of expert transitions. Thus, it should not be time-consuming to annotate the visual trajectories.
empirical results,
can preprocess the visual images using some pre-trained image encoder (e.g.,
We hope this work can provide some new insights to the community and inspire future work on offline imitation learning.
F ADDITIONAL REWARD PLOTS ON ADROIT AND ANTMAZE TASKS
In this section, we provide reward distribution plots of the ground truth rewards, rewards obtained by SEABO, and rewards output from HNSW on some Adroit-v0 and AntMaze-v0 tasks, fanmer-human-vo, hammer-clored-eee wtwpaaeeeemiimmmdiieeeeeee-
door-human-v0, doer-cloned-v0,
antmaze waazzze-umzzeeeaee v0, and togram plot of rewards in antmaze-ment um diverse-v0 datasets have quite similar reward signals, making it difficult to draw the density plot. We sum- marize the results in Figure 13. It can be that with KD-tree, SEABO outputs similar reward density as vanilla rewards (e.g., SEABO successfully gives three peaks in hammer r -uman-vv and door-human-vO).
Note that we provide the his- -ediimddiieeeee---v as most of the samples in this
24

---

Published as a conference paper at ICLR 2024
hellcheelatt-medleee http:hheellllmmllll-eeeerrr--eee (oracle)
(KD-tree)
hellcheetalh-meduunee hallehheeal--mdddu-eeeeeeev (HNSW)
0.25
0.20
25
12.5
densiyy
density
2.5
7.5
0.30
0.5
05
25
halfcheetaah-e 2tthheeetah-meiiu--222 (oracle)
hailccheetah-meediu--22 (KD-tree)
halfcheetah- -atiheeetth--mdium-222 (HNSW)
2.0
densiyy
0.3
density
2.0
density
0.2
1.5
0.1
0.0
30
0.0
0.5
0.0
11.5
1.0
3.5
-05
0.0
0.5
1.0
15
reward
reward
reward
hopper-meddun--elllyyye (oracle)
happer-medium-replaayyy (KD-tree)
hosper-meddum-repllayy (HNSW)
04
20
density
0.3
density
density
2.5
10
0.1
reward
reward
reward
hogper-medlum- www.rmedduu---eeee--Z (oracle)
hopper-medium-exxeett (KD-tree)
hogper-medium-expertwy (HNSW)
0.8
60
density
demsity
density
40
0.4
20
0.0
-2
10
3.0
0.5
1.0
15
-0.5
0.0
0.5
1.0
1.
reward
reward
reward
wallker26-medlum-22 (oracle)
walker22d-meduum-22 (KD-tree)
walker22d-mdiimme (HNSW)
0.6
3.5
60
3.0
density
0.4
density
2.5
density
0.2
0.5
10
0.0
3.
0.0
0.5
1.0
1.5
-0.
0.0
0.5
1.0
15
reward
reward
wallkerzo--meduu---epaayyyy (oracle)
walker226-mediim--eeaaayyz (KD-tree)
walkeer2d-mmdddum--epaayyya (HNSW)
1.75
30
density
0.15
3.00
28
0.10
10
0.66
0.00
-10
30
15
8:0
0.0
0.5
1.0
1.5
-05
0.0
2.5
1.0
1.5
reward
reward
rewarr
Figure 12: Density plot comparison groundd-ttuth rewards and rewards acquired by different
search algorithms. The right two columns show reward distributions of two SEABO variants.
25

---

Published as a conference paper at ICLR 2024
hammer-human--0 (KD-tree)
Soammer-human-- (HNSW)
hammer-hure hommmer-uuman-- (oracle)
3.5
0.190
0.123
2.5
density
density
0.775
4.6
1.0
0025
5000
100
100
0.0
30
0.5
16
-0.
0.0
0.
1.
15
reward
reward
reward
hammer-dloned-vo (oracle)
hammer--doned-vi (KD-tree)
hammer-cloned-v (HNSSW
0..00
14
0.25
density
density
0.10
0.68
8.00
-00
50
100
150
0.0
0.5
1.0
15
0
-05
0.0
0.5
1.0
15
reward
reward
reward
door-human-vo (oracle)
dloo-human--v (KD-tree)
door-human-v0 (HNSW)
0.6
27.5
density
0.4
density
density
0.1
9..5
1.0
reward
reward
reward
door-cloned-vO (oracle)
Noor-cloned-vv (KD-tree)
door-cloned-vo (HNSW)
12.5
density
0.0
density
0.6
6.0
20
00.c5
-0.
reward
antraaze-umaze-v (oracle)
untmazz--uazze-ve (KD-tree)
commaze-umazer v- (HNSW)
density
density
30
100
0
-15
-1.0
-0.5
0.0
0.5
-15
-1.0
-0.3
0.
+1.0
00
reward
reward
antmaze-neddummddwrree---e (oracle)
anfiaae--medlundddeess--- (KD-tree)
antmaze wammaazze-edduuuddivee-----ee (HNSW)
donsity
demsity
-10
-0.5
-0.6
-0.4
-0.2
2.0
11.5
12.5
114
02
0U
-0.4
-0.2
0.0
0.2
044
reward
reward
reward
Figure 13: Density plot comparison of ground-truth rewards and rewards acquired by different search algorithms. The results are on selected datasets from Adroit and AntMaze tasks.
26

---

Published as a conference paper at ICLR 2024
G DISCUSSIONS ON SEABO AND ILR
There are some previous studies that use nearest neighbor-based methods for imitation learning, e.g., Pari al. (2021). Among them, the most relevant to our work Ciosek (2022). In this section, we discuss the connections and differences between our method and prior work, ILR (Ciosek, 2022), which can be summarized below:
The motivations are varied. The practical reward formula in ILR is given by r 1 - ma dig ((s, a), (8', a²))², which is relaxation of its theoretical version. There exists a gap between the theory and the resulting reward formula. The authors claim that the relaxation is upper bound on the scaled theoretical reward and inter- pret L = min E dig ((s, a), (s', as the l2-diameter of the state-action space. The primary goal of doing so is reduce imitation learning RL with stationary reward for deterministic experts. However, the motivation of SEABO is that we would like to de- termine the optimality of the single transition (instead of examining whether the transition comes from the expert trajectory or performing relaxation to the rewards). We assume that the transition is near-optimal if it lies close the expert trajectory. Hence, we assign larger reward to the transition if it is close to the expert trajectory and a smaller reward oth- erwise. Meanwhile, SEABO does not require that the expert is deterministic (and also does not require that the environment is deterministic). We aim to adopt SEABO to annotate unlabeled samples in the dataset and train off-the-shelf offline RL algorithms.
The methods are different but connected. The reward formula adopted in ILR is a special case of SEABO with Euclidean distance. SEABO does not interpret L as the diameter of the state-action space. SEABO can adopt N nearest neighbors and use their average distance to compute the reward (ILR simply finds the smallest Euclidean distance between sample (s, a) and the expert trajectory). Meanwhile, SEABO not restricted to Euclidean distance. Our procedure is, that we first find the nearest neighbor of the query sample, and then utilize some distance measurements (different distance measurements can be used here) to decide the distance between the query sample and its nearest neighbor, and finally get the reward by adopting a squashing function. Furthermore, SEABO strongly relies on the nearest neighbor methods (e.g., KD-Tree), and one can different types of nearest neighbor algorithms in SEABO, while ILR does not emphasize search algorithms. Note that different search algorithms with different hyperparameter setups can result in different final rewards. For example, in scipy.spatial. KDTree. query, setting eps larger than O enables approximate nearest neighbors search and ensures that the k-th returned value is no further than (1 + eps) times the distance to the real k-th nearest neighbor. This may incur different results from ILR even under Euclidean distance. Moreover, SEABO can also work in state- only regimes, which is both a more general and challenging setting, while ILR strongly relies on the assumption that state-action pairs are present the expert trajectory in its theory and practical implementation. Finally, one can query with (s,a,'') (s, a) (s, s') in SEABO (ILR limited to (s, a)), and SEABO adopts a different choice of squashing function.
The settings are varied. SEABO is targeted at the offline imitation learning setting while ILR addresses the online setting. It also turns out that the experiment setup (e.g., number of expert trajectories) is different between SEABO and ILR.
27


Reviews:
[{"review_id": "gmt4PrHTuS", "reviewer": "Reviewer_KA4c", "summary": "This paper proposes a method for Offline Imitation Learning (IL) that defines a reward function based on the Euclidean distance to the nearest neighbor expert state. The method, called SEABO, uses a KD-tree to efficiently query expert states and compute rewards for all transitions. The resulting problem can then be optimized using an arbitrary offline RL algorithm. The experimental results demonstrated improved performance in several tasks of the D4RL benchmark.", "questions": "1. Can SEABO utilize alternative distance metrics in place of the Euclidean distance? If so, how much modification is required?\n2. The comparison between ground-truth rewards and the rewards obtained by SEABO (as in Figures 2 and 12) is intriguing. Have you also conducted similar reward comparisons in additional environments, such as AntMaze-v0 and Adroit-v0?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is both novel and simple, and its implementation is efficient due to the use of a KD-tree, without the need for training an extra discriminator.\n2. This paper focuses on the context of single-expert-demonstration IL tasks, which is an area of growing interest in the field.\n3. I find the discussion on using different search algorithms in Section 5.4 and Appendix Section C interesting.\n4. The Limitations section in the appendix is highly appreciated, as it provides valuable guidance on tuning hyperparameters and applying SEABO on visual input.", "weaknesses": "1. I'm concerned about the use of Euclidean distance and would suggest that the authors include references justifying the use of this distance metric. This is crucial because there might be scenarios where states that are close in Euclidean distance are, in fact, far apart when accounting for the transitions within the Markov Decision Process (MDP). This particular challenge doesn't arise in discriminator-based methods, mainly due to the use of an additional neural network during training.\n2. I believe that \"(oracle)\" should be omitted from Table 1-3 and 6. For instance, consider \"IQL (oracle)\": it utilizes the ground truth reward but doesn't rely on expert demonstrations. Removing the ground truth reward and integrating an additional expert demonstration does not necessarily make the task more challenging.\n3. The experiments currently compare with only two Offline RL methods (IQL and TD3_BC). It would be better to include more recent baselines such as Trajectory Transformer [[1]], Diffuser [[2]], or other methods.\n\n[1]: https://arxiv.org/abs/2106.02039\n[2]: https://arxiv.org/abs/2205.09991", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qc7N4XfuXW", "reviewer": "Reviewer_m3fB", "summary": "The paper proses an approach for imitation learning using nearest-neighbor-based reward computation and offline RL. Given a dataset of unlabeled environment interactions and a single demonstration, they use a KD-tree to compute the distance of each transition to it's nearest neighbor in the demo (euclidean distance) and use this as a pseudo-reward which they can optimize with offline RL. The method is evaluated extensively in D4RL locomotion (and few manipulation) tasks and shows strong performance over prior works.", "questions": "--\n\n# Post Rebuttal Comments\n\nThank you for answering my review.\n\nI appreciate the new experimental results -- performance on Kitchen seems strong and should be included in the paper.\n\nRegarding the furniture assembly benchmark: if offline imitation algorithms struggle in this benchmark, would it be feasible to apply your algorithm in an online context? Also, note that there is a new version of the benchmark (IKEA Furniture Bench, https://clvrai.github.io/furniture-bench/) -- while its focus is on real-world manipulation, it also comes with a simulated counterpart and offline dataset -- you could check that one out as well!\n\nIn any case, thank you for adding the experiments and I maintain my recommendation of acceptance. I also skimmed the other reviews and it seems that all reviewers are in agreement that the paper should be accepted.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The idea is simple and well-explained in the paper. The empirical validation is thorough, with results across many D4RL locomotion tasks and a small number of manipulation tasks. A representative set of baselines is used and the method shows strong empirical performance. The simple proposed method beats more complex alternatives that need to use optimal transport etc.", "weaknesses": "I don't have many issues with the paper. The method has some limitations (see below), but I don't think this invalidates the contributions of the current paper. \n\nOne minor weakness is that the approach is mostly evaluated on locomotion tasks for which precision is not the most critical. It could be great to evaluate it on a challenging, long-horizon manipulation task to test the limits of the method. For example the IKEA Furniture assembly benchmark could provide a nice test bed and I would be curious how well the proposed method performs.\n\nAnother minor point is that the paper lacks explanation how the euclidean distance is computed on the transition tuple. Do you compute euclidean distance on state, action and next state separately and then sum them? is there a weighting? Providing some details on this would be great!\n\n\n### Commentary on Limitations\n\nThe proposed method seems to have two limitations: \n\n(1) it's unclear how well it would work with visual observations where computing distances and nearest neighbors is a lot more challenging -- the authors mention this as a direction for future work and I agree that this is a limitation shared by many similar works so I wouldn't hold it against this paper. It should be mentioned though that the proposed approach with it's reliance on an analytic distance metric my be more troublesome in such scenarios than e.g. methods that use discriminator-based \"distances\".\n\n(2) it's unclear how well this would work in tasks that require very precise manipulation, since in such scenarios the proposed method would still assign high rewards to states that are only \"nearly\" demonstration states but may fail to perform the task in practice. It may be that methods that match trajectory segments instead of individual transitions fare better here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QrE7JPWxW5", "reviewer": "Reviewer_u2Fn", "summary": "This paper proposes the \"SEABO\" algorithm, which is an imitation learning algorithm that utilizes an expert dataset along with an unlabeled dataset. SEABO annotates the unlabeled dataset with rewards based on the distance between each state and the closest state in the expert dataset, and then runs an offline RL algorithm on the annotated dataset. The authors show improvements on the D4RL benchmark compared to prior baselines.", "questions": "\"we hypothesize that the transition is near-optimal if it lies close to the expert trajectory\" -> Is this strategy always good or does it have weaknesses? I would imagine that certain environments with discontinuities or discrete events (e.g. a car crash) would not favor this strategy.\n\nThere is another simple baseline commonly used in offline RL, which is sometimes referred to as percent-BC, which is to imitate the top N% of trajectories in the offline dataset (such as N=10 or N=25). As it is somewhat similar in spirit to SEABO, it would be good to see comparisons to this approach.\n\nIs there any effect of the \"curse of dimensionality\" for higher dimensional states?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a very simple algorithm that achieves good performance relative to more complex methods. I think this type of work has good value to the community in that it introduces easy-to-reproduce results and discourages over-engineering of methods.\n\n- The paper is clear in presentation and mostly well written.", "weaknesses": "- There is only empirical analysis of the proposed method. I believe there are certain tasks where SEABO would perform poorly, such as a cliff-walking type of task where there is a precise boundary between what is accetable and what is a failure. \n\n- I hypothesize that the approach will also only work in lower dimensional control environments. This is because the method relies heavily on a  distance function, and this could suffer from the curse of dimensionality in more complex environments.\n\nMinor:\nSearch in the context of RL and planning typically has a slightly different connotation, which is using some type of tree-based or trajectory-shooting method that optimizes some cost function. In this work search is only used to find states close to an expert. I'm not sure what can be done with this in the writing, but I was expecting a method in the former category after reading the abstract. It may be better to replace \"search\" with \"nearest neighbors\" to be more specific.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3RTKcBiSPq", "reviewer": "Reviewer_LUNW", "summary": "In this paper, the authors introduce an offline imitation learning algorithm called SEBO, which is centered on the task of learning a reward function from a dataset of expert demonstrations and applying it to unlabeled data to facilitate offline reinforcement learning.\n\nThe key innovation in SEBO lies in its use of a straightforward metric for generating the reward function, without the use of any neural networks. The algorithm employs search algorithms to determine the reward function. Specifically, SEBO constructs a KD-tree based on the expert demonstrations. For each unlabeled data point, the algorithm queries the KD-tree to identify its closest neighbor in the expert dataset and assesses the proximity between them. If the distance is minimal (indicating similarity to the expert trajectory), a high reward is assigned; conversely, if the distance is substantial (indicating deviation from the expert trajectory), a low reward is assigned.\n\nThe paper evaluates the SEBO algorithm across a range of MuJoCo environments and demonstrates its performance.", "questions": "1. Is there an inherent assumption being made between the distribution of state-action pairs in the expert demonstrations and the unlabeled dataset? \n2. Why are the experiments on D4RL restricted to medium level task and not conducted on expert and random versions of it? \n3. Does the stochasticity of environments affect the performance of SEBO? \n4. When using point wise matching to determine the reward are you making some inherent assumptions on the transition kernel?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well written, and the algorithm is studied well on different MuJoCo environments. The authors also conduct a sensitivity analysis with respect to the two parameters $\\alpha$ and $\\beta$\n2. SEBO is efficient, and easy to implement. It costs a minimal overhead over existing Offline RL algorithms, and does not involve training of a new neural network.  \n3. The authors also evaluate it in a scenario where there is only access to observations, a case that might be of real world importance.", "weaknesses": "**Evaluations restricted to deterministic environments**\n\nAll evaluations performed in this paper are conducted on deterministic environments (transitions in MuJoCo are completely deterministic), the claim that just one expert trajectory is sufficient might not be true if the environments are stochastic. For example, you may have a good (s,a,s’) pair in the dataset, but you may assign a lower reward to it as its not present in the expert demonstration.  I think there is an inherent correlation between the number of expert trajectories needed and the stochasticity of the environment. \n\n**The paper lacks theoretical justifications, which makes understanding some parts a little difficult. For instance,**\n\nThis method might not work in situations where there are multiple ways to solve the same task. \nConsider the following example of a navigation problem, where the goal is to navigate to the destination from start position, and there are two ways to solve this task. One that goes left and reaches the goal, the other that goes right and reaches the goal.  Suppose the expert demonstrations takes the left path, and the unlabeled demonstrations (say from an expert policy as well) are from the right path, SEBO will assign low rewards to all transitions and not learn a good policy while BC will probably work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)