RLIF: Interactive Imitation Learning as Reinforcement Learning
Authors: Jianlan Luo
Perry Dong
Yuexiang Zhai
Yi Ma
Sergey Levine
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement learning
imitation learning
online learning
Primary area: reinforcement learning
TL;DR: An RL approach runs under DAgger setting without making optimal expert or task reward assumption.

Abstract: Although reinforcement learning methods offer a powerful framework for auto-
matic skill acquisition, for practical learning-based control problems in domains
such as robotics, imitation learning often provides a more convenient and accessible
alternative. In particular, an interactive imitation learning method such as DAgger,
which queries a near-optimal expert to intervene online to collect correction data for
addressing the distributional shift challenges that afflict naïve behavioral cloning,
can enjoy good performance both in theory and practice without requiring manually
specified reward functions and other components of full reinforcement learning
methods. In this paper, we explore how off-policy reinforcement learning can
enable improved performance under assumptions that are similar but potentially
even more practical than those of interactive imitation learning. Our proposed
method uses reinforcement learning with user intervention signals themselves as
rewards. This relaxes the assumption that intervening experts in interactive imita-
tion learning should be near-optimal and enables the algorithm to learn behaviors
that improve over the potential suboptimal human expert. We also provide a uni-
fied framework to analyze our RL method and DAgger; for which we present the
asymptotic analysis of the suboptimal gap for both methods as well as the non-
asymptotic sample complexity bound of our method. We then evaluate our method
on challenging high-dimensional continuous control simulation benchmarks as
well as real-world robotic vision-based manipulation tasks. The results show that it
strongly outperforms DAgger-like approaches across the different tasks, especially
when the intervening experts are suboptimal. Additional ablations also empirically
verify the proposed theoretical justification that the performance of our method is
associated with the choice of intervention model and suboptimality of the expert.
Code and videos can be found on the project website: https://rlif-page.github.io

Published as a conference paper at ICLR 2024
RLIF: INTERACTIVE IMITATION LEARNING AS REIN- FORCEMENT LEARNING
Jianlan Luo" Perry Dong" Yuexiang Zhai Yi Ma Sergey Levine
UC Berkeley; { (jianlanluo, perzydong) @bbrrneeyeeeeeeeeeeee..
ABSTRACT
Although reinforcement learning methods offer powerful framework for auto- matic skill acquisition, for practical learning-basee control problems domains such as robotics, imitation learning often provides alternative. In particular, an interactive imitation learning method such as DAgger, which queries near-optimal expert to intervene online collect correction data for addressing the distributional shift challenges that afflict naïve behavioral cloning, can enjoy good performance both in theory and practice without requiring manually specified reward functions and other components of full reinforcement learning methods. In this paper, explore how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imita- tion learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert. We also provide a uni- fied framework to analyze our RL method and DAgger; for which we present the asymptotic analysis of the suboptimal gap for both methods as well as the non- asymptotic sample complexity bound of our method. We then evaluate on challenging high-dimensional continuous control simulation benchmarks as well real-world robotic vision-hased manipulation tasks. The results show that it strongly outperforms DAgger-like approaches across the different tasks, especially when the intervening experts are suboptimal. Additional ablations also empirically verify the proposed theoretical justification that the performance of our method is associated with the choice of intervention model and suboptimality of the expert. Code and videos can be found on the project website: rlif-ppi elif-page. gi github..... io
convenient and accessible
method
INTRODUCTION
Reinforcement learning methods have exhibited great success in domains where well-specified reward functions are available, such as optimal control, games, and aligning large language models (LLMs) with human prefer- ences (Levine et al., 2016; Kalash- nikov et al., 2018; Silver et al., 2017; Figure 1: RLIF uses RL to learn without ground truth rewards, with data collected with suboptimal human interventions.
Jumas
humin
r.-0
n==
r==1
r==0
r
N=0
Nesative
incervention
Hunan Mikoove
Ouyang et al., 2022). However, imita- tion learning methods are still often preferred in some domains, such robotics, because they often more convenient, accessible, and easier to use. An often-cited weakness of naïve behavioral cloning is the compounding distributional shift induced by accumulating errors when deploying learned policy, Interactive imitation learning methods, like the DAgger family of algorithms (Ross al., 2011; Kelly et al., 2018; Hoque al., 2022; Menda al., 2018; Ross & Bagnell, 2014), address this issue by querying expert actions online and retraining the model iteratively in supervised learn- ing fashion. This performs well in practice, and in theory reduces the quadratic regret of imitation
Equal contributions.

---

Published as a conference paper at ICLR 2024
learning methods to be linear in the episode horizon. One particularly practical instantiation of this idea involves a human expert observing a learned policy. and intervening to provide corrections (short demonstrations) when the policy exhibits undesirable behavior (Kelly et al., 2018; Spencer et al., 2020). However, such interactive imitation learning methods still rely on interventions that near-optimal, and offer no means to improve over the performance of the expert. Real human demonstrators are rarely optimal, and in domains such as robotics, teleoperation often does not afford the same degree of grace and dexterity as a highly tuned optimal controller. Can we combine the best parts of reinforcement learning and interactive imitation learning, combining the wwward-mxxiiian behavior of RL, which can improve over the best available human behavior, with the accessible assumptions of interactive imitation learning?
The key insight we leverage in this work is that the decision to intervene during an interactive imitation episode itself can provide reward signal for reinforcement learning, allowing us to instantiate RL. methods that operate under similar but potentially weaker assumptions as interactive imitation methods, learning from human interventions but not assuming that such interventions are optimal. Intuitively, for many problems, it's easier to detect a mistake than it is to optimally correct it. Imagine an autonomous driving scenario with safety driver. While the driver could intervene when the car deviates from good driving behavior, such interventions themselves might often be relatively uninformative and suboptimal for example, if the human driver intervenes right before a collision by slamming on the breaks, simply teaching the policy to slam on the breaks is probably not the best solution, as it would be much better for the policy to learn to avoid the situations that necessitated such an intervention in the first place.
Motivated by this observation, we propose a method that runs RL on data collected from DAgger-style interventions, where a human operator observes the policy's 's behavior and intervenes with suboptimal corrections when the policy deviates from optimal behavior. Our method labels the action that leads to an intervention with negative reward and then uses RL to minimize the occurrence of intervention by maximizing these reward signals. We call our method RLFF: Reinforcement Learning via Intervention Feedback. This offers a convenient mechanism to utilize non-expert interventions: the final performance of the policy would not be bottlenecked by the suboptimality of the intervening expert, but rather the policy would improve to more optimally avoid interventions happening at all. Of course, the particular intervention strategy influences the behavior of such a method, and we require some additional assumptions and evaluate their effect on performance, finding that several reasonable strategies for selecting when to intervene lead to good performance. We also provide theoretical justification for the proposed method, via both an asymptotic analysis of the suboptimality gap that generalizes the theoretical framework of DAgger (Ross et al., 20, and non-asymptotic analysis on learning an e-optimal policy with finite samples using the intervention rewards.
when interventions occur. We formalize several such assumptions
Our main contribution is a practical RL algorithm that can be used under assumptions that closely resemble interactive imitation learning, without requiring ground truth reward signals. We provide a theoretical analysis that studies under which conditions we expect this method to outperform DAgger- style interactive imitation techniques. Empirically, we evaluate our approach in comparison to DAgger on variety of challenging continuous control tasks, such as the Adroit dexterous manipulation and Gym locomotion environments (Fu et al., 2020a). Our empirical results show that our method is on average 2-3x better than best tess-feffrrmng DAgger variants, and this difference is much more pronounced as the suboptimality gap expands. We also demonstrate method scales to challenging real-world robotic task involving an actual human providing feedback. Our empirical results are well justified by our theory: suboptimal experts can in principle deteriorate the performance both imitation learning and RL, but RL is generally more powerful than imitation learning. This is because RL can still recover the optimal policy मO with additional samples, while imitation learning methods perform poorly due to the introduced suboptimality.
2 RELATED WORK
Interactive imitation learning. Imitation learning extracts policies from static offline datasets via supervised learning (Billard et al., 2008; Argall et al., 2009; Ho & Ermon, 2016; Osa et al., 2018; Laskey et al., 2017). Deploying such policies incurs distributional shift, because the states seen at deployment-time differ from those seen in training when the learned policy doesn't perfectly
2

---

Published as a conference paper at ICLR 2024
match the expert, potentially leading to poor results (Ross & Bagnell, 2010; Ross et al., 201 1). Interactive imitation learning leverages additional online human interventions from states visited by the learned policy to address this issue (Hoque et al., 2022; Kelly et al., 2018; Hoque et al., 2021; Menda et al., 2018; Ross et al., 2011). These methods generally assume that the expert interventions near-optimal. Our method relaxes this assumption, by using RL to train data collected in this interactive fashion, with rewards derived from the user's choice of when to intervene.
Imitation learning with reinforcement learning. Another line of related work uses RL to improve on suboptimal human demonstrations (Vecerik et al., 2017; Sun et al., 2017: Cheng et al., 2018; Sun al., 2018; Nair al., 2018; Ainsworth al., 2019; Rajeswaran et al., 2018; Kidambi et al., 2020; Luo et al., 2021; Xie et al., 2022; Xue et al., 2023; Song et al., 2022: Ball et al., 2023). These methods typically initialize the RL replay buffer with human demonstrations, and then improve upon those human demonstrations by running RL with the task reward. In contrast to these methods, our approach does not require any task reward, but rather recovers a reward signal implicitly from intervention feedback. Some works use RL with interventions but assume the expert is optimal (Li et al., 2022b), which our method does not assume. Other works incorporate example high-reward states specified by a human user in place of demonstrations (Reddy et al., 2019; Eysenbach et al., 2021). While this is related to our approach of assigning negative rewards at intervention states, our interventions are collected interactively during execution under assumptions that match interactive imitation learning, rather than being provided up-front. Closely related to our approach, Kahn et al. (2020) proposed a robotic navigation system that incorporates disengagement feedback, where model is trained to predict states where a user will halt the robot, and then avoids those states. Our framework model-free and operates under general interactive imitation assumptions, and utilizes more standard DAgger-style interventions rather than just disengagement signals.
3 PRELIMINARIES AND PROBLEM SETUP
In this section, we set up the interactive imitation learning and RL. formalism, and then introduce our problem statement.
Behavioral cloning and interactive imitation learning. The most basic form of imitation learning is behavioral cloning, which simply trains policy (aas)) dataset of demonstra- tions D, conventionally assumed to be produced by an optimal expert policy (als). with dz (8) its state marginal distribution. Then, for each (s,a) E D, behavioral cloning assumes that 8 ~ (s) and a ~ T² (als). Behavioral cloning then chooses मे arg min Ex a(s,a,T), where a(s,a, some loss function, such as the negative log- likelihood (i.e., E(s, a, x) = log (a]ss). Naïve behavioral cloning known to accumulate re- gret quadratically in the time horizon H: when म differs from x- even by small amount, er- roneous actions will lead to distributional shift in the visited states, which in turn will lead to larger errors (Ross et al., 2011). Interactive imitation learning methods, such as DAgger and its variants (Ross et al., 2011; Ross & Bagnell, 2010; Kelly et al., 2018; Hoque et al., 2022: Menda et al., 2018; Hoque et al., 2021), pro- pose address this problem, reducing the error to be linear in the horizon by gathering addi- tional training data by running the learned policy
Algorithm Interactive imitation
Require: т, π oxp, D
1: for trial i 1= 1 to N do
2: Train π on D via supervised learning
t, essentially adding new samples (s, a) where
3: for timestepp t = 1 to T do
(s), and a nT" (a)ss) Different inter-
4:
if TND intervenes at then
active imitation learning methods prescribe dif-
5:
append (st, (s*)) to D, end if
ferent strategies for adding such labels. Classic
6:
DAgger (Ross et al., 2011) 1) runs â and then asks a human expert to relabel the resulting states with a (a]s). This is often unnatural in Lime-sensitive control settings, such as robotics and driving, and more user-friendly alternative such HG-DA and its variants (Kelly et al., 2018) instead allows a human expert to intervene, taking over control from á and overriding it with an expert action. We illustrate this in Algorithm
7:
end for
8:
D + DUDU
9: end for
3

---

Published as a conference paper at ICLR 2024
Although this changes the state distribution, the essential idea of the method (and its regret bound) remain the same. However, as we will analyze further in Sec. 6, when the expert actions are not optimal (i..., they come from a policy πhP that is somewhat worse than x*), the regret gap for DAgger-like methods expands. Our aim in this work will be to apply RL to this setting to address this issue, potentially even outperforming the expert.
Reinforcement learning. RL algorithms aim to learn optimal policies in Markov decision pro- cesses (MDPs). We will use an infinite-horizon formulation in our analysis. The MDP de- fined as M = {S,A, 7) M comprises: S, a state space of cardinality S, A, an ac- tion space with size A, P : S x A + A(S), representing the transition probability of the MDP, r x A- [0, 1] is the reward function, and 7 (0,1) represents the discount fac- tor. We use π S + ((4). We introduce the value function (*)ss and the Q-function (=(s,a) associated with policy as:
(*(() 7 as)lso s; IT) Vs E S and V(s,a) E S x A (*(s,a) == E 7rr(,, (1)ss,,))laa az)sa 8.00 = a;; r], as is standard in rein- forcement learning analysis. We assume the initial state distribution is given by μc: So ~ μ., and H E A(S) and we slightly abuse the notation by using VT ((()) to denote V (s). The goal of RL is to learn an optimal policy ''' in the policy class IT that maximizes the expected cumu- lative reward within the horizon H: '' = arg max V*(u) (Bertsekas, 2019). Without loss of generality, we assume the optimal policy x* to be deterministic (Bertsekas, 2019; Li et al., 2022a). We slightly abuse the notation by using V*.Q* dz = - 7) Ei= 7'p' s|50 ~ (u), denote the state occupancy distribution under policy ı on the initial state distribution 8q ~ μ.. We also slightly abuse the notation by using di. E IR$ to denote a vector, whose entries are d7 (s).
E
Q" to denote V=', ,Q Additionally, we use
Problem setup. Our aim will be to develop a reinforcement learning algorithm that operates under assumptions that resemble interactive imitation learning, where the algorithm is not provided with reward function, but instead receives demonstrations followed by interactive interventions, as discussed above. We will not assume that the actions in the interventions themselves are optimal, but will make additional mild assumption that the choice of when to intervene itself carries valuable information. We will discuss the specific assumption used in our analysis in Section 6, and we utilize several intervention strategies in our experiments. but intuitively we assume that the expert is more likely to intervene when ø takes a bad action. This in principle can provide an RI. algorithm with signal to alter its behavior, as it suggests that the steps leading up to this intervention deviated significantly from optimal behavior. Thus, we will aim to relax the strong assumption that the expert is optimal in exchange for the more mild and arguably natural assumption that the expert's choice of when to intervene correlates with the suboptimality of the learned policy.
4 INTERACTIVE IMITATION LEARNING AS REINFORCEMENT LEARNING
The key observation of this paper is that, under typical interactive imitation learning settings, the intervention signal alone can provide useful information for RL. to optimize against, without assuming that the expert demonstrator actually provides optimal actions.
Algorithm 2 RRLF
Require: n, exp, D
1:
for trial i to N do
2: Train π on D via reinforcement learning.
3:
for tortestepp t to T do if exp intervenes at then
4:
Our method, which we refer as RLIF (rein- forcement learning from intervention feedback), follows a similar outline as Algorithm 1, but using reinforcement learning in place of super- vised learning. The generic form of our method is provided in Algorithm 2. On Line 2, the pol- icy is trained on the aggregated dataset D using RL, with rewards derived from interventions. The reward is simply set to 0 for any transition where the expert does not intervene, and -1 for the previous transition where an expert inter- venes. After intervention, the expert can also optionally take over the control for few steps;
5:
label (st-1, 0,,,,,,,,, 8t) with -1 reward,
append to D;
6:
else
7:
label (s. (s.-1, 8t) with 0 reward,
append to D,
8:
end if
9:
end for
10:
D DU D2
11: end for
4

---

Published as a conference paper at ICLR 2024
after which it will be released to the RL agent. This way, an expert also intervene multiple times during one episode. All transitions are added to the dataset D. not just those that contain the expert reward; in practice, we can optionally initialize D with small amount of offline data to warm-start the process. An off-policy RL algorithm can utilize all of the data and can make use of the reward labels to avoid situations that cause interventions. This approach has a number of benefits: unlike RL, it doesn't I require the true task reward to be specified, and unlike interactive imitation learning, it does not require the expert interventions to contain optimal actions, though it does require the choice of when to intervene to correlate with the suboptimality of the policy (as we will discuss later). Intuitively, we expect to be less of burden for experts to only point out which states are undesirable rather than actually act optimally in those states.
Practical implementation. To instantiate Algorithm in a practical deep RL framework, it is important to choose the RL. algorithm carefully. The data available for RL consists of a combination of on-policy samples and potentially suboptimal near-expert interventions, which necessitates using a suitable off-policy RL algorithm that can incorporate prior (near-expert) data easily but also can efficiently improve with online experience. While variety of algorithms designed for online RL with offline data could be suitable (Song et al., 2022; Lee et al., 2022; Nakamoto et al., 2023), we adopt the recently proposed RLPD algorithm (Ball et al., 2023), which has shown compelling results on sample-efficient robotic learning. RLPD is off- actor-critic reinforcement learning algorithm that builds on soft-actor critic (Haarnoja et al., 2018), but makes some key modifications to satisfy the desiderata above such as a high update-to-data ratio, layer-norm regularization during training, and using ensembles of value functions, which make it more suitable for incorporating offline data into online RL. For further details this method, though we emphasize that our method is generic and in principle could be implemented relatively easily top of variety of RL algorithms. The RL algorithm itself is not actually modified, and our method can be viewed as a meta-algorithm that simply changes the dataset on which the RL algorithm operates.
refer readers to prior work (Ball et al., 2023),
5 EXPERIMENTS
Since our method operates under standard interactive imitation learning assumptions, our experiments aim to compare RLIF to DAgger under different types of suboptimal experts and intervention modes. We seek to answer the following questions: (1) Are the intervention rewards sufficient signal for RL to learn effective policies? (2) How well does our method perform compared to DAgger, especially with suboptimal experts? (3) What are the implications of different intervention strategies on empirical performance?
5.1 INTERVENTION STRATEGIES
Human Foodback
Policy Update
In order to learn from interventions, we need the interven- ing experts to convey useful information about the task through their decision about when to intervene. Since real human experts likely to be imperfect in making this decision, we study variety of intervention strate- gies in simulation empirically to validate the stability of our method with respect to this assumption. The baseline strategy, which we call Random Intervention, simply intervenes uniformly at random, with equal probability at every step. The more intelligent model, Value-Based Intervention, strategy assumes the expert intervenes with probability 3 when there is a gap between actions from the expert and agent w.r.t. a reference value function QT"" This model aims to capture an uncertain and stochastic expert, who might have particular policy cxp that they use to choose actions (which could be highly suboptimal), and a separate value function n with its corresponding policy re that they to determine if the robot is doing well or not, which they use to determine when to intervene. Note that re might be much better than "xp for example, human expert might correctly determine the robot is choosing poor actions for the
Policy
Pebhor prrkees soleed
RJF
an regurtt prouddes introaatiin
foodbeck
Figure 2: A human operator supervises policy training and provides intervention with 3D mouse.
5

---

Published as a conference paper at ICLR 2024
Figure 4: Tasks in our experimental evaluation: Benchmark tasks Walker2d, Pen, and Hopper and two vision-based contact-rich manipulation tasks on a real robot. The benchmark tasks require handling complex high-ding sigh-dimensional dynamics and underactuation. The robotic insertion task requires additionally addressing complex inputs such as images, non-differentiable dynamics such as contact, and all sensor noise associated with real-world robotic settings.
task, if their own policy TP is not good enough to perform the task either. ó represents the confidence level of the intervening expert: the smaller á is, the more willing they are to intervene on even slightly suhoptimal robot actions. We formalize this model in Eq. 5.1. In practice, choose value for B close to I, such as 0.95.
B, 1-
if "F"""" (s, n (s)) > Q=
(s, (s,x(s)) +8
P(Intervention|s)
(5.1)
otherwise.
This model may not be fully representative of real human behavior, so we also evaluate our method with real human interventions, where an expert user provides intervention feedback to real-word robotic system performing peg insertion task, shown in Figure 2. We discuss this further in Sec. 5.3.
In Appendix 1, we also present a didactic experiment with the Grid World environment where we use value iteration as the underlying RL algorithm, verify that RLIF does indeed converge the optimal policy under idealized assumptions when remove approximation and sampling error of RL method from consideration.
5.2 PERFORMANCE ON CONTINUOUS CONTROL BENCHMARK TASKS
First, we evaluate RLIF in comparison with various interactive imitation learning methods on several high-dimensional continuous control benchmark tasks. These experiments vary both the optimality of the expert's policies and the expert's 's intervention strategy.
Simulation experiment setup. We use Gym locomotion and Adroit dexterous manipulation tasks in these experiments, based on the D4RL environments (Fu et al., 2020b). The Adroit en- viromments require controlling 24-DoF robotic hand to perform tasks such as opening door or rotating pen to randomly sampled goal loca- tions. The Gym locomotion task (walker) re- quires controlling a planar 2-legged robot to walk forward. Both domains require continu- ous control relatively high frequencies, where approximation errors and distributional shift can accumulate quickly for naïve imitation learning methods. Note that although these benchmarks are often used to evaluate the performance of RL algorithms, we do not assume access to any reward function beyond the signal obtained from the expert's 's interventions, and therefore our main point of comparison are DAgger variants, rather than prior RI. algorithms.
Rith aupart 33% supart
Secoer Bate
10% auper
0.
0.2
2.5
2.5
Figure 3: Average success rate and intervention rate for the Adroit-Pen task during training, as the agent improves, the intervention decreases.
The experts and QT" associated with intervention strategies are obtained by training policies on subsampled datasets induce the desired level suboptimality. Further details on initial policy and intervention expert training can be found in Appendix. A.3.
Results and discussion. We report our main results in Table, 4, with additional learning curves for analyzing learning progress in terms of the number of iterations presented in Appendix. A.5 and an IQL baseline in Appendix. A.7. Each table depicts different expert performance levels different tasks (rows), and different methods with different intervention strategies (columns).

---

Published as a conference paper at ICLR 2024
Figure 5: Sequential steps of robot manipulation for the peg insertion and cloth unfolding tasks on a real robot.
Domain
Expert Level
RLIF with RLIF
with Random Intevvnsion
HG- DAgger
HG- BAG-DAgger
DAgger
DAgger with 85% Random
BC
Value Rased
with 85%
Ramdom Intervention
adroit-pen
909
3.04
12.87:12.86
7344: 19
74.27=5577 7.79
78.13433244
79.07::9
29.53=66.56
35.734:7.99
28.67:22.06
14.23
~000
17.54
28.33=4441
28.53:17.66 66
9.47::4.09
8.93-41443
12846625
77.791889
35
$413881
37.9
40:33:005
43.1153.7
54.15E14244
Toonnot Fon-
~1
108.99
106.1140.7
$3,55:9
112.722511
$7.04:8.00
134332
walker2d
~700
99.66:1..9
75 53.62:550002
44,75 46
69.73=3.99 5.99
20.49:1::15
43.591:..56
14/6113.599
102.85-2.22
19.111-24.08
01,9420.88
19.66:3.69
12.37:42.6
20.14:2.11
mvgrage
103.83
67.0831-3.85
36.75:21.7
606
302271-1193
16.61
14/6611339
locomotion Ton-
1100
001.17=0.16
93.76=787
80.3214.74
4693=:85
70.58:99:9 98
61661111136
hopper
40.66:13.35
19.63-1:.33
64.77::10225
~155%
108.01 0.64
75.
12::2222
25.24:3.58
24.37=2226
19:54::2.14
10.29:1.24
average
0.4453:0.477
90.93
48.72::7.22
51.32:3.32
42.91:5527
30.46::4997
64.77::10.25
Table 1: A comparison of RLIF, HG-DAgger, DAgger, and BC on continuous control tasks. RLIF consistently performs better than HG-DAgger and DAgger baselines for each individual expert level as well as averaged over all expert levels.
Note the "expert level" in the table indicates the suboptimality of the expert; for example, "90%" expert means can achieve 90% the reference optimal expert score a particular
task. This score is normalized across environments to be 100.0. Since trained such perts using our curated datasets, the normalized score can exceed 100.0. For the rest of the table, report the performance of different methods w.r.t. this normalized reference score.
To start the learning process, we initialize the replay buffer with small number of samples. The
Rate
Cloth Unfolding
Evaluation Success Rate
Peg Insertion
E E
details of the datasets can be found in Appendix.
A. We
compare RLIF with DAgger and HG-DAgger under different in- tervention modes and expert lev-
Evaluation
els. Specifically, we run DAgger under two intervention modes: 1) the expert issues an interven- tion if the difference between the agent's action and the expert's tion is larger than a threshold, as used in Kelly et al. (2018); Ross et al. (2011). 2) the expert issues random intervention uniformly with given probability each step of the episode. The results suggest several takeaways: (1) We see that regardless of the suboptimality of the experts, RLIF with value-based interventions can indeed reach good performance, even the suboptimality gap is very large, such as improving a 15% expert to a score over 100.0 in the 2D-walker task. (2) RLIF with value-based simulated interventions outperforms RLIF with random interventions, especially when the suboptimality is large. This confirms that the interventions carry meaningful signal about the task, providing implicit rewards to RLIF. We also observe that the value-based intervention rate goes down as the agent improves, shown in Fig. 3. This confirms proposed intervention model does carry useful information about the task, and works reasonably as the agent learns. (3) RLIF outperforms both DAgger and HG- HG-DAgger consistently across all tasks, and the performance gap more pronounced when the expert is more suboptimal, and can be as large as 5x: crucially, this resonates with motivation: the performance of DAgger-like algorithms will be subject to the
Iterations
teeraions
HG-DAgger 33% Optimal Expert
HG-DAgger 100% Ostimal Expert
HG DAgger 50% Optimal Expert
RUF
Figure 6: RLIF on the real-world robotic manipulation task.
7

---

Published as a conference paper at ICLR 2024
suboptimality of the experts, while our method can reach good performance with suboptimal experts by learning from the expert's 's decision of when to intervene.
Ablations on Intervention Modes. As stated in Sec. 5.1, the performance of RLIF critically
depends on when the expert chooses to intervene. Now we analyze the effect of different QT value functions on the final performance of RLIF We report the numbers in Table. 3 of Appendix. A.6. We can observe that the particular choice of QT does heavily influence our algorithm's performance: as the x ref deteriorates, Q becomes increasingly inaccurate, making the intervention decision more "uncalibrated." This translates to worse policy performance.
5.3 REAL-WORLL USSON-BASED ROBOTIC MANIPULATION TASK
While we have shown that RLIF works well under reasonable models of the expert's intervention strategy, to confirm that this model actually reflects real human interventions,
next conduct an experiment where a human operator supplies the interventions directly for a real-world robotic manipulation task. The first task, shown in Fig. 2, involves controlling a 7-DoF robot arm to fit peg into its matching shape with a very tight tolerance (I (1.5mm), directly from image observations. This task is difficult because involves dealing with discontinuous and non-differentiable contact dynamics, complex high-dingn gigh-dimensional inputs from the camera, and an imperfect human operator. The second task involves controlling the same robot arm to unfold piece cloth by hooking onto and pulling it open. This task is difficult because it requires manipulation of deformable object: specifying rewards programmatically for such a task is very challenging, and the perception system trained via RL must be able to keep track of the complex object geometry. Filmstrips of the tasks are shown in Fig. 5 to visualize the sequential steps of both the peg insertion and cloth unfolding tasks. More details
the setup be found in Appendix. A.2. We report the results in Fig. 6. Our method can solve the insertion task with a 100% success rate within six rounds of interactions, which corresponds 20 minutes in wall-clock time, including robot resets, computation, and all intended stops. It solves the unfolding task with a 95% success rate in seven rounds of interaction. This highlights the practical usability of our method in challenging real-world robotic tasks, We refer the training and evaluation videos of the robotic tasks to our website: rliffapee github. io
6 THEORETICAL ANALYSIS
In this section, we provide a theoretical analysis of RLIF in terms of its suboptimality gap. We introduce our theoretical settings and goals in Sec. 6.1, and quantify the suboptimality gap in Sec. 6.2.
6.1 THEORETICAL SETTINGS AND ASSUMPTIONS
We consider the setting where we cannot access to the true task reward r, and we can only obtain rewards ïs through interventions. For simplicity of analysis. we define this reward function as is (Not intervened] (i.c., for each step when an intervention does not happen), which differs from the reward Algorithm 2 by a constant and thus does not change the result, but allows us to keep rewards in the range [0. 1]. We aim to apply RL using the intervention reward As induced by RLIF, and quantify the suboptimality gap with respect to the true reward r. particular, let ă denote the policy learned by RLIF for maximizing the overall return on reward function is:
2'F a [
π E 11 S.L. IIPpE_ :- arg max IE (s, TEll
(6.1)
Our goal is to bound the the suboptimality gap: SubOpt V*( V*().. Based the intervention strategy specified in Eqn. 5.1, we introduce our definitions of iss as follows. We leave the discussion of the property of the function space 11o0t to Appendix E.
II, the reward function is induced by Alg. 2 is defined as Pa = (Not intervened). The probability of intervention is similarly defined as Eqn. 5.1, for a constant 3 > 0.5. We assume the expert may refer to reference policy nh e 11 for determining intervention strategies as mentioned in Sec. 5.1. In addition, we assume the expert's 's own policy will not be intervened exp r
Assumption 6.1. For policy
8

---

Published as a conference paper at ICLR 2024
The å that appears in i's for determining the intervention condition is similarly defined in Eqn. 5.1 could be referred as the "confidence level" of the expert. Note that Assumption 6. is a generalized version of the DAgger, as in the DAgger case one can treat n- xPP..
6.2
MAIN RESULT
To provide comparable suboptimality gap with DAgger (Ross et al., 2011), we introduce the similar behavioral cloning loss on =T
Definition 6.2 (Behavior Cloning Loss (Ross al.,, 20) We use the following notation to denote the 0-1 loss on behavior cloning loss w.E.t TXP and nr as: ((8, = 1 (m(() + AT (0))(s)} (s)}, E E, (s, T (s)) 1 (T()) + π (s)} Vs
With Assumption 6.1, and Def. 6.2, we now introduce the suboptimality gap of RLIF.
Theorem 6.3 (Suboptimality Gap of RLIF). Let म E ITg denote an optimal policy from maximiz- ing the reward function is generated by RLIF. Let e = max aF (s. x (s)), Ba (s, n (8))
(Def. 6.2). Under Assumption 6.1, when Vaa is known, the RLIF suboptimality gap satisfies:
de
SubOptRLIF = (*()) (x) S min Vn (u) (u)-v="' (4), V (e) - (m)
Note that our Assumption 6.1 is generalized version of DAgger with the additional V=*ef we can use the same analysis framework of Thm. 6.3 to obtain a suboptimality gap DAgger and obtain the following suboptimality gap. Using similar proof strategy to Thm. 6.3, we provide suboptimality gap for DAgger as follows.
Corollary 6.4 (Suboptimalityy Gap of DAgger), Let π E 1155 denote an optimal policy from maximizing the reward function is generated by RLIF. Let L E ((s, (s)) (Def. 6.2). Under
Assumption 6.1, when V unknown, the DAgger suboptimality gap satisfies:
8c
SubOptpAgger VV) V*(N) S V (p) V= (n)
(6.2)
1 1-7
direct implication of Thm. 6.3 and Cor. 6.4 is that, under Assumption 6.1, RLIF is at least as good as DAgger, since SubOpiRLIF S SubOpt
We provide bandit example in Appendix. B.3 to show that our upper bounds in Thm. 6.3 and Cor. 6.4 are tight. We leave the proof of Thm. 6.3 and Cor. 6.4 analysis under our framework to Appendix B. For completeness, we also provide a non-asymptotic sample complexity analysis for learning an that maximizes V (p) in Appendix C. Our non-asymptotic result in Cor. C.3 suggests SC
that the total sample complexity does not exceed the sample complexity Ö (1-ン);>) ). where Cosp is the concentrability coefficient w.r.t. e (formally defined in Def. C.1).
7 DISCUSSION AND LIMITATIONS
We presented a reinforcement learning method that learns from interventions in a setting that closely resembles interactive imitation learning. Unlike conventional imitation learning methods, our ap- proach does not rely strongly on access to an optimal expert, and unlike conventional reinforcement learning algorithms, it does not require access to the ground truth reward function, instead deriving a reward signal from the expert's decision about when to intervene. However, our approach does have a number of limitations. First, we require an RL method that can actually train on all of the data collected by both the policy and the intervening expert. This is not necessarily an easy RL. problem, as it combines off-policy and on-policy data. That said, we were able to use an off-the-shelf offline RL algorithm (RLPD) without modification. Second, imitation learning is often preferred precisely because it doesn't 't require online deployment at all, and this benefit is somewhat undermined by interactive imitation learning methods. While in practice deploying a policy under expert oversight might still be safer (c.g., with a safety driver in the case of autonomous driving), investigating the safety challenges with such online exploration is important direction for future work.

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENTS
This research was partly supported by ARO W911NF-21-110099,, and ONR N00014-21-1-2838 and N00014-20-1-2383, the Berkeley Research Computing Platform (BRC), NSF Cloud Bank program.
REFERENCES
Samuel Ainsworth, Matt Barnes, and Siddhartha Srinivasa. Mo'states mo' problems: Emergency stop mechanisms from observation. Advances in Neural Information Processing Systems, 32, 2019,
Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey robot learning from demonstration. Robot. Auton. Syst., 57(5):469-483, may 2009. ISSN 0921-8890. doi: 10, 1016/j.robot.2000 200..00. URL https: / /ooi. org/10. 1016/ ( robot 2008. 10. 024.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for rein- forcement learning. In International Conference Machine Learning, pp. 263 272-272. PMLR, 2017.
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 1577-1594. PMLR, 23-29 Jul 2023. URL ht httpas:///rreeeeeeeeeeeeeeeeeeoe httpp/ppcceedengeeeeeeeeeeeeestt mlr. prwss wttps/////////aaeeeeeaaaaaaa html.
Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.
Aude Billard, Sylvain Calinon, Rüdiger Dillmann, and Stefan Schaal. Robot Programming by Demonstration, pp. 1371-1394. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 978- 3-540-30301-5. doi: 10.
-660. URL https: http://oooooo//1000/0/00
978-3-2 540- 30001-301---0
50.
Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots. Fast policy learning through imitation and reinforcement. 2018.
Ben Eysenbach, Sergey Levine, and Russ R Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification. Advances in Neural Information Process- ing Systems, 34:11541-11555 11552, 2021.
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. In arXiv, 2020a. URL
htt.:/orggp//d/////eeee// 07219.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020b.
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai. Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems, 15:15281-12295, 2022.
Haamoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In arXiv, 2018. URL http:tt::/// //arxivav vww.aaivvoooo org/ pdf/1801. 01090.00..00.. .pdf.
J. Ho and S. Ermon. Generative adversarial imitation learning. In Neural Information Processing Systems (NIPS), 2016.
Ryan Hoque, Ashwin Balakrishna, Ellen R. Novoseller, Albert Wilcox, Daniel S. Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning. 2021.
Ryan Hoque, Lawrence Yunliang Chen, Satvik Sharma, K Dharmarajan, Brijen Thananjeyan, P. Abbeel, and Ken Goldberg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Conference on Robot Learning, 2022.
10

---

Published as a conference paper at ICLR 2024
Gregory Kahn, P. Abbeel, and Sergey Levine. Land: Learning to navigate from disengagements. IEEE Robotics and Automation Letters, 6: 6:1872-1879, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning, pp. 651-673, 2018.
Michael Kelly, Chelsea Sidrane, K. Etaggs-Campbell, and Mykel J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. 2019 International Conference Robotics and Automation (ICRA), pp. 8077-8083, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05993 2020.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning, 2017.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-onling reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1377 2016.
Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.0577. 2022a.
Qiyang Li, Yuexiang Zhai, Yi Ma, and Sergey Levine. Understanding the complexity gains of single- task rl with a curriculum. In International Conference on Machine Learning, pp. 20412-20451. PMLR, 2023.
Quanyi Li, Zhenghao Peng, and Bolci Zhou. Efficient learning of safe driving policy via human-ai copilot optimization, 2022b.
Jianlan Luo, Oleg O. Sushkov, Rugile Pevceviciute, Wenzhao Lian, Chang Su, Mel Vecerík, Ning Ye, Stefan Schaal, and Jonathan Scholz. Robust multi-modal policies for industrial assembly via reinforcement learning and demonstrations: A large-scale study. ArXiv, abs/2103.11512, 2021.
Kunal Menda, K. TriggsCCampbell, and Mykel J. Kochenderfer. Ensembledagger: bayesian approach to safe imitation learning. 2019 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pp. 5041-5044 2018
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 6292-6299) 2018. doi: 10.1109/1CRA,2018
8463162.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, arXiv preprint arXiv:2303.05475 2023.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2): 1-179, 2018. ISSN 1935-8253. 1561/23000000
doi: 00.1561/2300000053. URL http: / doi. org/10.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with buman feedback, 2022.

---

Published as a conference paper at ICLR 2024
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning; A tale of pessimism. arXiv preprini arXiv:2103.
2021.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv: Learning, 2019.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In International
Conference on Artificial Intelligence and Statistics (AISIATS), pp. 661-668, 2010.
Stephane Ross and J. Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret
learning. Advances in Neural Information Processing Systems, 2014.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 627-635, 2011.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-353 359, 2017.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint 1rX1v:2210.06718. 2022.
Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang Peter Ramadgel, and Siddhartha Srinivasa2. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In Proceedings of Robotics: Science and Systems, 2020.
Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, and J. Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In Proceedings of Machine Learning Research, 2017.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning and imitation learning. 2018.
Mingxing Tan and Quoe Le. EfficientNe: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Research, pp. 6114 PMLR, 09-15 Jun 2019. URL https: pttpp/eeeceee...eeeeeepppcc...se mlr.
Machine Learning, volume 97 of Proceedings of Machine Learning
press/v97 h////aa19/ 9a html.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 17X7v:1707.088/ 2017.
Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-274077 2021.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. 2022.
Zhenghai Xue, Zhenghao Peng, Quanyi Li, Zhihan Liu, and Bolei Zhou. Guarded policy optimization with imperfect online demonstrations, 2023.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENT RESULTS AND DETAILS
A.1 GRID WORLD NAVIGATION AS DIAGNOSTIC TAsK
We start with didactic task in Grid World environment
E
to visualize the behavior of our method over the course
of training. The MDP is 6x6 grid, where the task is to
navigate from location (1, 1) to (6, 6). The (unobserved)
optimal policy is produced by optimizing a reward that is
sampled from [-0.1, 0] at each point on an optimal route, and from |-1, [-1] for any state that is not on this route,
such that a good policy must follow the route precisely. We employ the Value-Based Intervention strategy. We col- lect five intervention trajectories per round following this strategy. We use value iteration as the RL algorithm since this MDP can be computed exactly. We plot the results in Fig. 7, visualizing the value function as well as the actions learned by method. As the expert applies interventions on successive iterations, RLIF determines a reasonable value function, with high (red) values along the desired route and low (blue) values elsewhere, arrow indicating the action actions; despite not actually observing the true task reward. We also plot the true value function of this task in Fig. 7(f). While of course the true value function differs from the solution found by RLIF, the policy matches on the path from the start to the goal, indicating that RLIF can successfully learn the policy from only intervention feedback when we abstract away sampling error.
(a) N=2 Rounds
(b) N=10 Rounds
(c) N=20 Rounds (d) True task values
Figure 7: Learned value function at suc- cessive rounds of RLIF (red is higher). Over the course of the algorithm, the value function and policy converge on the optimal path through the grid world, only using intervention-bate rewards.
A.2 ROBOT EXPERIMENT DETAILS
Task Description. For real robot experiments, we perform the task of peg insertion into 3D 3D-printed board and cloth unfolding with velcro hooks using a 7-DoF Franka Research 3 robot arm. The RL agent stream control commands to the robot at 10 HZ, each episode is at maximum of 50 timesteps, which converts to 5 seconds. The robot obtains visual feedback from the Intel Realsense D405 cameras mounted on its end-effectors. In setup, the human operator provides interventions by using a 3D mouse. Whenever the operator touches the mouse, they take over control of the robot, and return control the RI. agent by disengaging with the mouse. We mark the transition of the previous time step of such an intervention as negative reward. We use an ImageNet pre-trained EfficientNet-B3 (Tan Le, 2019) as vision backbone for faster policy training. Two cameras mounted on the robot end-effector provide continuous visual feedback. For the insertion task, trial counted as a success if the peg is inserted into its matching hole with certain tolerance, and for the unfolding task, trial is counted as success if the cloth is successfully unfolded all the way. All success rates are reported based on 20 trials.
Collecting Trajectories. For RLIF, DAgger-based baselines, and BC, five suboptimal trajectories are used to initialize the replay buffer. During training, human interventions are given when the policy appears to be performing suboptimally. The BC results are trained on the same five trajectories that initialize the RLIF buffer.
A.3 EXPERIMENT SETUP DETAILS
Success Rates. For the Adroit tasks, success rate is used a measure of performance since whether the agent learns a policy that can success the task can be more informative how well expert interventions are incorporated than reward. The evaluation is done on the Gymnasium sparse environments AdroitHandPenSparee. determined to be success if at the end the agent receives a reward of 10 corresponding success in the Gymnasium environments. Walker2d and Hopper uses average normalized returns since success rate is hard to definitively measure for the locomotion environments.
and AdroitHanddoorSpareeeey where a trajectory is
13

---

Published as a conference paper at ICLR 2024
Offline Datasets. To perform controlled experimentation of RLIF against DAgger and HG-DAgger we prepare offline datasets to use for pretraining on DAgger and HG-DAgger and to initialize to replay buffer for RLIF. The initial datasets of all simulation tasks subsets of datasets provided in d4rl. The specific dataset used to subsample and sizes the initial dataset for each task are listed in Table 2. The rewards for all timesteps for all of the initial datasets are
Tasks
Dataset
Subsampled Size
Adroit Pen
pen-expert-vl
50 trajectories
Locomotion Hopper
hopper-expert-v2
50 trajectories
Locomotion Walker2d
walker2d-expert-v2
10 trajectories
Real-World Robot
manually collected suboptimal
5 trajectories
Experiments
trajectories
Table 2: Offline datasets for each task.
Expert Training. Experts of varying levels are trained for all tasks on the dataset with either BC, IQL, SAC, or RLPD depending on the task. The various levels of the experts are obtained by training on the human dataset or expert datasets subsampled to various sizes. For each level and task, the same expert is used to intervene across all intervention strategies for both RLIF, DAgger, and HG-DAgger.
A.4 INTERVENTION STRATEGIES
Random Intervention. For the random intervention strategy. in our experiments interventions are sampled uniformly at random. We consider 30%, 50%, and 85% intervention rates, where the intervention rate is computed the number of steps under intervention over the total number of steps. We define as the particular timestep where an expert choose intervene, and k as the number of steps expert takes over after such intervention. At any given timestep that not intervened by the expert, we can then define the uniformly random intervention strategy with different probabilities below:
30% Intervention Rate: ~ 1((t + 1, 10) and k: ~ ((1,5)
50% Intervention Rate: I ~ L((( + 1, and ie U(3, 7)
85% intervention Rate: I ~ L(( + 1, t + 2) and k ~ U(12,16)
Value-based Intervention. For value-based intervention strategy, compare the value of the current state and policy actions to the value of the current state and expert actions under the reference Q=!af function.
Specifically, we follow the intervention strategy below where an intervention likely to occur if the expert values are greater than the policy values by threshold.
B, if 17 (s, (xP (F'"((ss + 1 B,
(s,
P(Intervention(s)
(A.1)
otherwise.
In practice, we found relative threshold comparison effective, which is stated as below:
P(Intervention(s) [, if @F*T" TxP()) > Q=TH (8, (s))
(A.2)
1- otherwise.
We choose value for /3 close to 1 such 0.95 and value of a close to such as 0.97.
A.5 EXPERIMENT PLOTS
We present some representative training plots below to describe the learning process RLIF and HG-DAgger. The plots cover variety of expert levels and intervention strategies.
14

---

Published as a conference paper at ICLR 2024
RLIF Pen With Value Based Intervention 98% expert
RLIF Pen With Random Intervention
10t expert
espert
expert
expert
expert
Success Putte
Rape
0.5
0.6
0.3
0.2
0.2
Step
Stop
200k
400k
cook
BC0k
200k
400k
000k
b0ck
1M
Pen HG-DAgger With Random Intervention
Pen Original HG-DAgger
-45%% expert
10% export
export
expeitt
export
90% expert
Rato
011
0.6
0.4
0.2
100
Pen DAgger With Random Intervention
Pen Original DAgger
10% expert
48% expert
90% expert
166% expert
40% expert
30% expert
0.8
0.6
0.6
0.4
0.2
0.2
100
Walker2d RLIF With Value Based Intervention
Walker2d RLIF With Random Intervention
40000 Riturnss
26 expert
113 expert expe nt
5000
70 expert
29 expert 110 expert
garage hetums
4000
4000
3000
3000
2000
2000
1000
1000
Step
200k
400k
GOOk
B0ok
IM
Walker2d HG-DAgger With Random Intervention
Walker2d Original HG- DAgger
70 expert
20 expert 113 expert
78 expert
28 expert 113 expert
5000
MeC-90 Retums
2500
Retumss
4000
2000
3000
1500
2000
1000
1000
Step
Step
20
120
20
60
100
Walker2d DAgger With Random Intervention
Walker2d Original DAgger
74 espert
20 expert 110 expert
70 expert
28 espert
118 expert
30000
No0Dgg Retumo m5
3000
2500
Retu
2000
1500
1900
1000
1000
Step
15

---

Published as a conference paper at ICLR 2024
Hopper RUF With Value Baxed Intervention
Hopper RLIF With Random Intervention
11 expert
40 exspert
15 expe 1t
3000
3000
2000
1000
1000
Step
2004
Hopper HG-DAgger With Random Intervention
Hopper Original HG-DAgger
15 expert
expert
110 expert
15 expert
48 expert 110 expert
Retumss
3000
3000
2500
2000
2000
1800
1500
1000
1000
300
300
Step
20
60
100
20
40
60
80
100
Hopper DAgger With Random Intervention expert
Hopper Original DAgger
40 export
118 expert
15 expert
48 expert
110 export
2500
2500
1800
1500
1000
1000
500
Stop
Step
0
20
40
60
80
100
20
40
543
100
300
Figure 9: Training Plots of RLIF, DAgger, and HG-
A.6 ABLATION EXPERIMENT RESULTS
Domain
Expert Level
10% qr!"!
60% Q
110% Q
loconation
wi10t
78.9:149.97
85.91
108.99:5.228
<400
38.78:120.3
71.21
99.6625.99
~208
33.55:19633
66.8::::22
102.85:2.266
Table 3: An ablation of RI.F on walker2d with different Q and expert levels. A 10% QT"" means that it was trained with the offline dataset generated by a 10% expert.
A.7 ADDITIONAL BASELINES
Domsin
Expert Level
IQL with
IQL
with Random
Value Based Intervention
Intervention
adroit-pen
990%
42,5422.2 68.83433.75
46.58
25.92:43.52
~11
40.67=5.16
12.08:4:.39
average
43.25=1.41
35.61::3.89
locomation-
~110
20.37=5.188
80.22:#12.8
walkes2d
47.97::14.44
56.33:4:.45
~200%
47.02=1.83
14.7:::192
average
38.45=9.42
50.42 d:7.45
locemotioon-
~0100
26.37=2.04
37.91:4:.448
hopper
<404
28.98-4.11
27.88:11.84
25.32=4811
16.64:1:18
average
26.89_3655
27.48:42.37
Table 4: A comparison of RLIF, HG-DAgger, DAgger, and BC on continuous control tasks. RLIF consistently performs better than HG-DAgger and DAgger basclines for each individual expert level as well averaged over all expert levels.
16

---

Published as a conference paper at ICLR 2024
A.8 EXPERIMENT HYPERPARAMETERS
Training Parameters. We set the number of rounds to N = 100 and the number of trajectories collected per round to 5. We also use the number of pretraining epochs and pretraining train steps per epoch to 200 and 300, and the epochs and train steps per epoch for each round to 25 and 100 to achieve consistent training.
Tasks
Parameters
Values
Adroit Pen
UTD Ratio
Locomotion Hopper
Ratio
Locomotion Walker2d
Ratio
Tasks
Batch
256
Learning Rate
3e-4
Weight Decay
1e:3
Discount
0,99
Hidden Dims
(256, 256)
DAgger & DAgger Pretrain
60,000
Steps
DAgger & HG-DAgger Steps Per
2500
Iteration
Table 5: RLIF and HG-DAgger parameters for each simulation task. The parameters specified under All Tasks are for both BC and RLIF.
Tasks
Parameters
Values
Insertion and Unfolding Tasks on Franka Robot
Pretrained Vision Backbome
EfficientNer-
B3
Learning Rate
3e-4
ML.P Dims
(256, 256)
Layer Norm
true
Discount
0.99
UTD Ratio
4
Batch Size
256
Table 6: RLIF and HG-DAgger parameters for insertion task on Franka robot.
17

---

Published as a conference paper at ICLR 2024
B SUBOPTIMALITY GAP
B.1 SUROPTIMALITY GAP OF RLIF
Theorem B.1 (Suboptimality Gap of RLIF, Thm. 6.3 restated). Let i. note an optimal policy from maximizing the reward function ñs generated by RLIF. Let == c=max (Bowsz ((,)))sss(s) az (s.()))) (Def. 6.2). Under Assumption 6.1, when V is known, the RLIF suboptimality gap salisfies:
11 de-
{ (v- (w) V* (p), V (n) Va (m)) + dc 1-7
SubOpt = (p) V (p) S min
Proof. Notice that ने denote the optimal policy w.r.t. the RLIF reward function is:
f E arg max (e, xell
(1)(0, ~
(B.1)
Let E denote the following event:
E =
(s, 1 (s)) Qr' (s, (s)) + 8 or Q² (x))) >
By Assumption 6.1, we can write the reward functions as random variable as follows:
P(Fo(s, = 0[5) = { 3, if E happens,
{1 - B, otherwise.
(B.2)
33,
if & happens, otherwise.
P(rs(s, (s)) 1|s) 1 B,
Taking an expectation of the equations above, we have
Ba)t (s)) = 1 B, when & happens,
(B.3)
E (ss (s)) = 3, when & happens.
And since we assume B > 0.5, we know that (s, x(s) achieves maximum 3, when & happens. Hence, in order to maximize the overall return in Eqn. 6.1, म should satisfy Q="( (s) S QTt (s, (+)) + and QT (x))))(())) S (s))(s)) + 6, Vs S. Since
this result holds for all states S, then for V= (K) - -V²(N)) have
y=' (4) v=(m]
|I p R 9TTTM (5, (s)) Q7 (s)a(s))
E a Qaaal (s, (o. Banay Q (s, (s.
+ E Eenn Q (s,)(s)) E Bendz ( (s. (*, a())
S de + y (aNP(())()))
[ Bandgro Qu (8', xT)( Banaz (8(,5(8)))))
de + yoc + 7280 +.. dc
1-
Applying similar strategy | (1)) - V (1))) can also get V - V=(+]] s de Hence, we conclude the following
1-7
max | (11) - v* (10) (w). (m) - v2
de
(B.4)
1-7
Hence, we conclude that
V*(H)= - V*(N)
< min V= (n)) V.00 (1) + () (1)) - v* /.y (14) - vavc (n)
p (1) v* (*
dc
< min (v=' (n) V* (11), V (nr) (mm
+
7-
18

---

Published as a conference paper at ICLR 2024
where the last inequality holds due to Eqn. B.4. Hence conclude the results.
B.2 SUBOPTIMALITY GAP OF DAGGER (Ross ET AL., 2011)
Corollary B.2 (Suboptimality Gap of DAgger, Cor. 6.4 restated). Let π E 11 denote an optimal
policy from maximizing the reward function Ta generated by RLIF. Let = Essay ((, [()) (Def. 6.2).
Under Assumption 6,1, when Vx is unknown, the DAgger suboptimality gup satisfies:
de
SubOptpAgger V*) (*) S V (N) V (p)
(B.5)
Proof. Notice that ह denote the optimal policy w.r.t. the RLIF reward function is:
0c
iT E arg max ( #ell
(Fs(s),a))los ~
(B.6)
Let denote the following event:
E f ( (s) > Q (s, x())) + 8).
By Assumption 6.1, we can write the reward functions as a random variable as follows:
P(Fa(s,T(s)) = 0|s) = { 3,
if happens,
{1 - B, otherwise.
(B.7)
[ 33
if happens,
((fs(s, xs)) =||8]=
1 B,
otherwise.
Taking an expectation of the equations above, we have
3, when & happens, E (x(( = 3, when E happens.
(B.8)
And since we assume B > 0.5, we know that (es (()) achieves maximum B when E happens. Hence, in order to maximize the overall return in Eqn. 6.1, # should satisfy Q (s, a (s)) < Q (s, (s)) + 5, Vss E S. Since this result holds for all states s E S, then for yauu (4) V (p)), we have
y (w) - (H()) ( = BBandg Q (s, (5,Te(((s) - B, Brndz (
(s, (s,)(ss))
B Bond Q R0D (xp - B, Bandy (8, (s))
+ E Eowdy Q Q7T07(1000000 (s, (s)) Eawdy Q (s, (()))
(B.9)
< de + γ (xrnP(s)A(s))
BandterD Quaaa (s', x (x)((s - Banas Q² (s)(
Sc + ydc + 280 +o++-..
1 1-
Hence, we conclude the following
V*(u) - (*))
V= (n) V= (4) + (m) (n) - V$ (m))
< V (p) Va (u) + dc
1-7
where the last inequality holds due to Eqn. B.9.
19

---

Published as a conference paper at ICLR 2024
B.3 TIGHT SUBOPTIMALITY GAP
Example B.3 (Lower Bounds of RLIF). Consider a bandit problem where we only have one state S == {8}, two actions A {ai, and, and the reward function is given as r(s, 01) and r(s, a2) = O. Assume the policy space II C IR2 is vvo-dimensional Euclidean space, where each policy x E II satisfies such that (a1) + π (a2) = 1. Let ne ,TxD E II be any two policies in the policy space, and π 11g be the optimal policy from maximizing the reward function i& generated
by RLIF, e max [Bewag f(s, (s)), Eo (s)) (Def. 6.2). Then we have
SubOptRLIP min { (V"(()) V (1)), V") (n) va [m) + de
1-
de
SubOptDAgger = V (4) - -- + 1-7
Proof. Note that in the bandit case constructed above, the optimal policy should satisfy x²' [1,0]T. Let re π exp be that
ref
[x1, 1 xr1]" exp [x2, 1 x2 T-
(B.10)
Hence, we know that
Vale (s) 1 --7
21 / (s) -
3/2
(B.11)
Let denote the following event:
E f (s, (s)) QT"
(s, T (s))
+ or Q Q7 "050""1 (x))) > Q' (s,
By Assumption 6.1, we can write the reward functions as random variable as follows:
P(Fa(s, = 0[8) = f 3,
if E happens,
{1 B, otherwise.
(B.12)
((Fs(s,<(t)) = 1|s) = [3.
if E happens, otherwise.
1 B,
Taking an expectation of the equations above, we have
(=s0,,)) 1- - 3, when & happens, B (()) = 3, when & happens.
(B.13)
And since assume 3 > 0.5, know that (s (s)) achieves maximum B, when & (ra, xr] should satisfy Q= (s, ( ne (x)) s QT (s, i (x)) + and Q= (s, n (8)) S Q" "D (x)) + 8, vis E S, which implies that
happens. Hence, in order to maximize the overall return in Eqn. 6.1, ग
ral
It + 2 T1, 23 + 8 2 Z2-
(B.14)
Without loss of generality, assume क 2 x2. Let x3 I1 + 8, then we have
1 33
1 - Φ
8e
SubOptRuıe = EE [E( (s)} = E Exnds
= 1-
(B.15)
min { | (5) ya (8), V (s) - ya (s) +
8E
(B.16)
1-7
In the DAgger case, by setting gee = Texp we can similarly obtain
1 33
1 1 Φ
8e
SubOptRLIF = =E(V= - -V*(s)} = E, Exnds
(B.17)
1
1 1-7
8e
= = (s) - /= (s) + 1 7
(B.18)
Hence, we conclude results.
20

---

Published as a conference paper at ICLR 2024
C NON-ASYMPTOTIC ANALYSIS
For the non-asymptotic analysis, we adopt the LCB-VI (Rashidinejad et al., 2021; Xie et al., 2021; Li al., 2022a) framework, because we warm-start the training process by adding a small amount of offline data, and later on we gradually mix d da into the replay buffer due to the intervention. The LCB-VI VI framework provides a useful tool for studying the distribution shift in terms of the concentrability coefficient when incorporating offline data. While RLPD (Ball et al., 2023) is not an LCB-VI algorithm, our method is generic with respect to the choice of RL subroutines, such as the online setting (Azar et al., 2017; Gupta et al., 2022) or the hybrid setting (Song et al., 2022). Hence we believe this analysis is useful for characterizing our approach.
We first present the single-police concentrability coefficient that has been widely studied in prior literature (Rashidinejad et al., 2021; Li et al., 2022a) as the following.
Definition C.1 (Single-Policy Concentrability Coefficient). We define the concentrabilliy coefficient d (s,0)
of d= with respect to an offline visitation distribution p is defined as Cp := maX(x,))ESxA p(s,a)
Similar to Li et al. (2022a), we adopt the conventional setting of di (8,a)/p(s,a) = 0, when dy (s, a) 0 for state-action pair (s, a).
Note that the concentrability coefficient plays crucial role in the final non-asymptotic bound, we present an upper bound on the concentrability coefficient with RLIF
Lemma C.2 (Concentrablitt of a Intervention Probability). Suppose we construct pun by mixing d' tet with probability 3 (0 < 3 < 1) and distribution p such that p(s, a) > 0, V(, a) E S x A: p
then the concentrability coefficient satisfies Of S min 1- C 1 Cexp } (1 - p) + 3d rot
3
Proof. Notice that when the optimal policy x² is deterministic (Bertsekas, 2019; Li et al., 2022a),
we can apply a similar decomposition technique shown Li al. (2023), we have
d5 (8. a)
d= (s)1 {a x" (s)}
max 6,0
max
paint (8.
plin(() (8)
(C.1)
dx
min
C+
C
piut
B)p + 3dg
1
3
Hence, conclude that
d= (s, n) pinn((s (s, (s a)
1 B
Cj
max 8,0
< min
C" pr
Coxp
(C.2)
1 3
With Lemma C.2, we present our main non-asymptotic sample complexity bound in the following.
Corollary C.3 ASym-tsypi Sample Complexity). Suppose the conditions in Assumption / holds, then there exists an algorithm that returns 6-optimal ît for V (1i) such that V (14
V (1) S 6, with sample complexity o (1-7)²))
SC
Proof. For any policy क, since RLIF will induce a dataset with the distribution of
int (1 B)d "d,
(C.3)
We overload the notation let O C. denote the concentrability coefficient w.r.t. the state visitation distribution d Now applying Lemma C.2 with int defined in Eqn. C.3, obtain concentrability coefficient of
Cimn & min
1 C+
coz
1
C
}
(C.4)
1
3
3
21

---

Published as a conference paper at ICLR 2024
Before applying such concentrability coefficient to Thm. D.2, will first provide an error bound induced by the stochastic reward F... Ve O, suppose we want to achieve a statistical error of e/2 induced by the stochastic is [0. An 0 Proposition D.1 implies that
Σ
(a(s,
(s (s)))
- exx N), As E
(C.5)
N
where (s, T (s)) is the tt sample of the reward function is(ss, sTs)). Taking union bound of the probability above overall E yields
I max SES
(o((s Eis(s, Nn 28 exp (-2
(C.6)
Hence, by setting " = (1 - 7e/22, we know that with probability at least 1 - pn. the sub- optimality gap induced by the stochastic reward is most c/2, given enough sample size
N (ーा log
25 60
(ー-)) for one state S, which leads to the overall In the deterministic reward case, Thm. D.2 implies that one
sample complexity of Ō find an algorithm that achieves suboptimality gap of e//2 with the sample complexity of
(1-7)8-2)
δ SC (1-7)32
SC )
SCixp
=Ō 3(1 ((1-2)882) =
(1-
(C.7)
where the above equation holds due Eqn. C.4. Hence, we conclude there exists algorithm that can achieve an optimal policy w.r.t. V (4) at the total sample complexity of
δ SC (1 73-2
(0
S = =Ō (11-
SCexp
(1- (
+0
(C.8)
Cor. C.3 indicates that the total sample complexity for learning á that maximizes ViA the total sample complexity does not exceed the sample complexity [1-7)<>] with true reward r.
SCrep
of solving original RL problem
D SUPPORTING THEORETICAL RESULTS
Proposition D.1 (Hoeffding Bound, Proposition 2.5 of Wainwright (2019)). Suppose that variables
Ie = 1,2,...,? n are independent, and x; [a, b), Vi = 1,2,..., Then for all > O, we have
nt
[r. Exa)
2
exp
(D.1)
(b
a)
Theorem D.2 (Convergence of Offline RL, Thm. of Li et al. (2022a)). Suppose E [1/2, E (0, | and the concentrability coefficient C is defined in Def. C.I. With high probabiliry, there exists an algorithm that learns an algorithm & with concentrability coefficient C that î can achieve an e-optimal policy with sample complexity of N = Ō ( (リ-ンパp>))
SC:
in terms of the tuples
{st O4, s where (st p. E Sx
E PROPERTIES OF ToP
Lemma E.1 (Properties of (T 48 0, dg is a metric over TIpt. Let
opt = max
ds( x
(E.1)
N. .are
then [T is monotonic decreasing in 5.
22

---

Published as a conference paper at ICLR 2024
Proof. To verify that m is metric, need to show the following properties:
Distance to itself is zero: VT E I have that
do( n²) max [|+(s (-||))] O.
(E.2)
sES
Positivity: VT, n' g such that # n', we have that d8 (T, x²)
max .ES
[-|ss) - x' x'(+s)]l, O.
(E.3)
Symmetry: Vn, x' " we have that
do( m² = max |||(-|) - x (-1+s)11 = ||7((-1)) x-116]11 = do(T''
(E.4)
SES
Triangle inequality: VT,, n2, m3 E _1 we have
do( T3) = max | (-1s) nT(- ses
max [l|m (-18) m2(118)l1 + || (s) m=
SES
(E.S)
max
|| (-|s) m2(+1)]l1 + max || <+(+s)
SEB
dg (T1, T2) + d5( T3).
Hence conclude that dæ is metric over Ig Next, will show [ is monotonic decreasing. We will show this result by proving 11 C CTT v0 < 8 < 8.. Notice that Væ mT we have that
Vi, (1) = max aaa
(E.6)
max (f)
Σ
2 (St, a.) ~
This results implies that E To Therefore, know that |T we conclude the our results.
when d S 8'. Hence,
23


Reviews:
[{"review_id": "meAFnB3DxJ", "reviewer": "Reviewer_9AYW", "summary": "This paper tackles how to enable sub-optimal human interventions (sub-expert samples) to improve a RL agent. It falls in two sub-domains in the field: interactive imitation learning, and reinforcement learning from human feedback. \n\nThe authors explore the assumption that the decision to intervene provides an effective supervision signal. Accordingly, a solution based on augmentation the reward with intervention penalty, and off-policy learning is used to minimize such a penalty (maximize the accumulative negative reward), so that the RL agent achieve 3X performance gain compared to DAgger, and BC (bahevior cloning) baselines. Ablation studies on sub-optimality of expert samples are also provided.", "questions": "Please check the two questions the reviewer proposed in the Weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty:\n========\nThe assumption of “when an sub-optimal expert to intervene” provides useful learning signals is novel to me.\n\nSoundness:\n========\nAccordingly, the authors provide both empirical results in simulation and real-world robotic task, and theoretical justifications by proposing a probabilistic model of “when human expert will intervene”.", "weaknesses": "- Though the theoretical analysis does not answer the key problem, how the level of sub-optimality results in what sample complexity of the offline RL agent. The reviewer still enjoys reading the analysis framework proposed here. The reviewer may miss some key contents, but would the authors explain more in the proposed Corollary 6.7 regarding the above key problem?\n- Baselines:\nThere is no offline-RL baselines provided. Is it because that the true reward function is unknown in the evaluation results? Otherwise, please add at least 1 offlineRL baseline in all the simulation based experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nn0ZbGfR0G", "reviewer": "Reviewer_RarS", "summary": "This paper introduces a novel off-policy RL method that uses user intervention signals as rewards, allowing learning beyond the limitations of potentially suboptimal human experts. This approach is less reliant on near-optimal expert input compared to DAgger. The authors provide a unified analytical framework for their method and DAgger, including asymptotic and non-asymptotic evaluations. They validate their method with high-dimensional simulations and real-world robotic tasks, showing its practical effectiveness in surpassing traditional imitation learning methods in complex control scenarios.", "questions": "I am wondering what the difference is between RLHF and your work RLIF when you choose the Space Mouse to give signals by hand. No related works are mentioned and no experiments are conducted for this. For example, what about using some foundation models to give signals?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The motivation behind this paper is natural and valuable.\n\n(2) The theoretical analysis is sufficient.\n\n(3) The authors conduct some experiments on real robot arms, which proves the effectiveness of the algorithm.", "weaknesses": "(1) The author should include more baselines in Table 1, where the mentioned methods only contain DAgger. For example, some IRL methods could also be applied in the same settings.\n\n(2) Does this method require high frequencies to intervene? Can you do some ablations for this? If RLIF cannot work under high frequent intervenes, it can be hard to apply in more complicated real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rz8Jz5RHU2", "reviewer": "Reviewer_guRK", "summary": "The paper tackles the problem of learning from suboptimal experts in interactive settings. It proposes to add a negative reward term in reinforcement learning whenever an expert requires being invoked in a RL DAgger setup, and it shows performance gains with different expert accuracy levels compared to supervised learning baselines.", "questions": "- What is the advantage of the proposed paper versus using DAgger in RL?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I appreciate the effort to bring the model to a real-world setting.\n\nEdit:\n- The paper has a through theoretical justification and a sensible set of simulated baselines, showing the power of the proposed method.", "weaknesses": "- The paper only compares with DAgger in supervised learning setups, even though DAgger is often used in RL and should have been included as a baseline in such configuration.\n- The proposed technique seems to reduce to a heuristic to define a reward function.\n\nEdit:\n- After a very thorough set of responses from the authors, I admit I misinterpreted the actual scope of the paper, so these weaknesses are actually obsolete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iu7c5UjGKs", "reviewer": "Reviewer_48gS", "summary": "This paper provides a methodology for learning a policy given human intervention. Unlike other imitation learning frameworks such as Dagger, the method does not directly clone the actions taken by the expert, but rather uses the expert’s decision to intervene as a negative reward signal. The paper claims that by doing so, the learned policy has the potential to exceed the performance of a suboptimal expert, and does not need to assume the expert is optimal. The method is demonstrated experimentally in simulation to outperform imitation learning baselines, and is demonstrated to work in a real world experiment. Theoretical analysis is provided to justify the method.", "questions": "- If the expert intervenes for 5 steps, would all 5 transitions be labeled -1? Wouldn’t this result in actions taken by the expert being labeled -1 too? \n- Could the else statement in line 7 of Algorithm 2 set the reward to the RL reward signal if it were available? \n- I’m very confused by this sentence: “the performance of DAgger-like algorithms will be subject to the suboptimality of the experts, while our method can reach good performance even with suboptimal experts by learning from the expert’s decision of when to intervene.“ If no task reward is provided, how could the policy end up outperforming the expert?\n- How do you end up with a 110% expert if the expert level is w.r.t an expert trained to near optimality?\n- It seems unfair that Dagger variants are given 100 episodes to learn from while RLIF gets 1 million. It is also unrealistic to expect that we could have 1 million episodes in any real-world setting. How would RLIF perform if also restricted to a more realistic 10-100 episodes?\n- In experiments, do you warm start the dataset for RLIF? If so, with what reward? And does this pose an unfair advantage to your algorithm?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The key strengths of the paper are as follows:\n- The method is simple and easy to implement, thereby making it useful.\n- The method does not require access to a task reward signal and makes only the benign assumption that an expert would intervene if the policy were performing badly. This is realistic and gives the method a broad scope.\n- The paper includes a thorough justification of the method, which with some additional clarifications could be compelling (see the questions I have).", "weaknesses": "The main weakness of this paper is that I am suspicious of the results due to some confusion. Please answer the questions below, and I would be happy to revise my rating upward if they are satisfactory. [Edit: updated rating since questions were addressed]\n\nAside from the questions, below are a few recommendations for clarity improvement and a few grammatical errors caught. \n\nClarity recommendations:\n- In the intro, add a more intuitive explanation for how RLIF is able to exceed the expert’s performance without access to a task reward signal.\n- In the intro, you describe the theoretical analysis performed, but don't state the bottom line. What does the analysis tell us? \n\nGrammar:\n- “…for selecting when to intervene lead to good performance…”\n- “We leave the of DAgger analysis under…”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)