SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
Authors: Siddharth Reddy
Anca D. Dragan
Sergey Levine
Venue:  (2020)
Keywords: Imitation Learning
Reinforcement Learning

Abstract: Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated actions, it can drift away from demonstrated states due to error accumulation. Recent methods based on reinforcement learning (RL), such as inverse RL and generative adversarial imitation learning (GAIL), overcome this issue by training an RL agent to match the demonstrations over a long horizon. Since the true reward function for the task is unknown, these methods learn a reward function from the demonstrations, often using complex and brittle approximation techniques that involve adversarial training. We propose a simple alternative that still uses RL, but does not require learning a reward function. The key idea is to provide the agent with an incentive to match the demonstrations over a long horizon, by encouraging it to return to demonstrated states upon encountering new, out-of-distribution states. We accomplish this by giving the agent a constant reward of r=+1 for matching the demonstrated action in a demonstrated state, and a constant reward of r=0 for all other behavior. Our method, which we call soft Q imitation learning (SQIL), can be implemented with a handful of minor modifications to any standard Q-learning or off-policy actor-critic algorithm. Theoretically, we show that SQIL can be interpreted as a regularized variant of BC that uses a sparsity prior to encourage long-horizon imitation. Empirically, we show that SQIL outperforms BC and achieves competitive results compared to GAIL, on a variety of image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo. This paper is a proof of concept that illustrates how a simple imitation method based on RL with constant rewards can be as effective as more complex methods that use learned rewards.

Published as a conference paper at ICLR 2020
SQIL: IMITATION LEARNING VIA REINFORCEMENT LEARNING WITH SPARSE REWARDS
Siddharth Reddy, Anca D. Dragan, Sergey Levine Department of Electrical Engineering and Computer Science University of California, Berkeley {sgr, anca, svlevine] (bbrreleey we..rkeeleeee.uuu...
ABSTRACT
Learning to imitate expert behavior from demonstrations can be challenging, es- pecially in environments with high-dimensional. continuous observations and un- known dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated actions, it can drift away from demonstrated states due to error accumulation. Re- cent methods based on reinforcement learning (RL), such inverse RL and gen- erative adversarial imitation learning (GAIL), overcome this issue by training an RL agent to match the demonstrations over long horizon. Since the true reward function for the task is unknown, these methods learn a reward function from the demonstrations, often using complex and brittle approximation techniques that in- volve adversarial training. We propose a simple alternative that still uses RL, but does not require learning a reward function. The key idea is to provide the agent with an incentive to match the demonstrations over a long horizon, by encourag- ing it to return to demonstrated states upon encountering new, out-of-distribution states. We accomplish this by giving the agent a constant reward of r +1 for matching the demonstrated action in a demonstrated state, and a constant reward of r = 0 for all other behavior. Our method, which we call soft Q imitation learn- ing (SQIL), can be implemented with a handful minor modifications to any standard Q-learning
off-cc actor-critic algorithm. Theoretically, we show that SQIL. can be interpreted as a regularized variant of BC that uses a sparsity prior encourage long-horizon imitation. Empirically, performs BC and achieves competitive results compared to GAIL,, on a variety of image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo. This pa- per is a proof of concept that illustrates how a simple imitation method based on RL with constant rewards can be as effective as more complex methods that use learned rewards.
show that SQIL out-
INTRODUCTION
Many sequential decision-making problems can be tackled by imitation learning: an expert demon- strates near-optimal behavior to an agent, and the agent attempts to replicate that behavior in novel situations (Argall et al. 2009). This paper considers the problem of training an agent to imitate an expert, given expert action demonstrations and the ability to interact with the environment. The agent does not observe a reward signal query the expert, and does not know the state transition dynamics.
Standard approaches based on behavioral cloning (BC) use supervised learning to greedily imitate demonstrated actions, without reasoning about the consequences of actions (Pomerleau) 199T). As result, compounding errors cause the agent to drift away from the demonstrated states (Ross et al.) 2011]. The problem with BC is that, when the agent drifts and encounters out-of-distribution states, the agent does not know how to return to the demonstrated states. Recent methods based on in- verse reinforcement learning (IRL) overcome this issue by training an RL agent not only to imitate demonstrated actions, but also to visit demonstrated states (Ng al. 2000 Wulfmeier al. 2015 Finn et al., 2016b, Fu et al.) (017). This is also the core idea behind generative adversarial imi- tation learning (GAIL) (Ho & Ermon 2016). which implements IRL using generative adversarial

---

Published as a conference paper at ICLR 2020
networks (Goodfellow et al. 2014 Finn et al. (016a). Since the true reward function for the task is unknown, these methods construct a reward signal from the demonstrations through adversarial training. making them difficult to implement and use in practice (Kurach et al. 2018).
The main idea in this paper is that the effectiveness of adversarial imitation methods can be achieved by much simpler approach that does not require adversarial training, or indeed learning a reward function at all. Intuitively, adversarial methods encourage long-horizon imitation by providing the agent with (1) an incentive to imitate the demonstrated actions in demonstrated states, and (2) an incentive to take actions that lead it back to demonstrated states when it encounters new, out-of- distribution states. One of the reasons why adversarial methods outperform greedy methods, such as BC, is that greedy methods only do (I), while adversarial methods do both (1) and (2). Our approach intended to do both (1) and (2) without adversarial training, by using constant rewards instead of learned rewards. The key idea is that, instead of using a learned reward function to provide a reward
signal to the agent, we can simply give the agent a constant reward of r = +1 for matching the demonstrated action in demonstrated state, and constant reward of r 0 for all other behavior.
We motivate this approach theoretically, by showing that it implements a regularized variant of BC that learns long-horizon imitation by (a) imposing a sparsity prior the reward function implied by the imitation policy, and (b) incorporating information about the state transition dynamics into the imitation policy. Intuitively, our method accomplishes (a) by training the agent using ex- tremely sparse reward function - +1 for demonstrations, O everywhere else and accomplishes (b) by training the agent with RL instead of supervised learning.
We instantiate our approach with soft Q-learning (Haarnoja et al. 2017) by initializing the agent's experience replay buffer with expert demonstrations, setting the rewards the demonstration experiences, and setting rewards to a constant r = 0 in all of the new experiences the agent collects while interacting with the environment. Since soft Q-Jearning is an off- off-policy algorithm, the agent does not necessarily have to visit the demonstrated states in order to experience positive rewards. Instead, the agent replays the demonstrations that were initially added its buffer. Thus, our method can be applied in environments with stochastic dynamics and continuous states, where the demonstrated states are not necessarily reachable by the agent. We call this method soft Q imitation learning (SQIL).
constant r +1 in
The main contribution of this paper is SQIL: a simple and general imitation learning algorithm that effective in MDPs with high-dimensional, continuous observations and unknown dynamics. We run experiments in four image-based based environments - Car Racing, Pong, Breakout, and Space Invaders - and three cow-dimensional environments - Humanoid, HalfChectah, and Lunar Lander- to compare SQIL to two prior methods: BC and GAIL. We find that SQIL outperforms BC and achieves com- petitive results compared to GAIL. Our experiments illustrate two key benefits of SQIL: (1) that it can overcome the state distribution shift problem of BC without adversarial training or learning re- ward function, which makes easier use, e.g., with images, and (2) that is simple implement using existing Q-learning or off-policy actor-critic algorithms.
2 SOFT Q IMITATION LEARNING
SQIL. performs soft Q-learning HHarrojj et al. 2017) with three small, but important, modifications: (1) it initially fills the agent's 's experience replay buffer with demonstrations, where the rewards are set to a constant r = +1; (2) as the agent interacts with the world and accumulates new experiences, it adds them to the replay buffer, and sets the rewards for these new experiences to a constant r = 0; and (3) it balances the number of demonstration experiences and new experiences (50% each) in each sample from the replay buffer These three modifications are motivated theoretically in Section 3 via an equivalence to a regularized variant of BC. Intuitively, these modifications create a simple reward structure that gives the agent an incentive to imitate the expert demonstrated states, and to take actions that lead it back to demonstrated states when it strays from the demonstrations.
SQIL resembles the Deep Q-lcarning from Demonstrations (DQrD) Hester et al. 2017 and Normalized Actor-Critic (NAC) algorithms (Gao et al. 2018)) in that initially fills the agent's 's experience replay buffer with demonstrations. The key difference is that DQID and NAC are RL. algorithms that assume access to a reward signal, while SQIL. is an imitation learning algorithm that does not require extrinsic reward signal from the environment. Instead, SQIL automatically constructs reward signal from the demonstrations.
2

---

Published as a conference paper at ICLR 2020
Algorithm Soft Q Imitation Learning (SQIL)
1:
Require Asamp E R2D, Pazmo
Initialize Dsamp
3:
while Qo not converged do
4: - 0 19-10 --yoo ( 1) x 0)) {See Equation
5:
Sample transition (s, a, s') with imitation policy x[ass) exp (Qo(s,a)
6: DDump Damp U (,)
7:
end while
Crucially, since soft Q-learning is off-policy algorithm, the agent does necessarily have to visit the demonstrated states in order to experience positive rewards. Instead, the agent replays the demonstrations that were initially added to its buffer. Thus, SQIL can be used in stochastic environments with high-dimensional, continuous states, where the demonstrated states may never actually be encountered by the agent.
SQIL is summarized in Algorithm the squared soft Bellman error,
where Qe is the soft Q function, Doomo are demonstrations, 82
(ools.a) r r y-gylog Σ exp (Qo(s', AoE
(D(Drr)
Σ
(I)
D
(S,0,8")ED
and r {0 is a constant reward The experiments Section use a convolutional neural network multi-layer perceptron model Qo. where the weights of the neural network. Section A.3] in the appendix contains additional implementation details, including values for the hyperparameter Asamp' note that the simple default value of Asamp works well across variety of environments.
As the imitation policy in line of Algorithm T learns behave more like the expert, a growing number of expert-like transitions get added to the buffer Dsamp with an assigned reward of zero. This causes the effective reward for mimicking the expert to decay over time. Balancing the number of demonstration experiences and new experiences (50% each) sampled for the gradient step in line 4 eessures that this effective reward remains at least 1/(1 + Asamp), instead of decaying zero. In practice, we find that this reward decay does not degrade performance if SQIL is halted once the squared soft Bellman error loss converges to a minimum (e.g., see Figure g in the appendix). Note that prior methods also require similar techniques: both GAII. and adversarial IRL. (AIRL) (Fu et al.) 2017) balance the number of positive and negative examples in the training set of the discriminator, and AIRL tends to require early stopping avoid overfitting.
3 INTERPRETING SQIL AS REGULARIZED BEHAVIORAL CLONING
To understand why SQIL works, we sketch surprising theoretical result: SQIL is equivalent to variant behavioral cloning (BC) that uses regularization to overcome state distribution shift.
BC is simple approach that seeks to imitate the expert's actions using supervised learning in particular, greedily maximizing the conditional likelihood of the demonstrated actions given the demonstrated states, without reasoning about the consequences of actions. Thus, when the agent makes small mistakes and enters states that are slightly different from those in the demonstrations, the distribution mismatch between the states in the demonstrations and those actually encountered by the agent leads to compounding errors (Ross et al. 2011). We show that SQIL is equivalent to augmenting BC with regularization term that incorporates information about the state transition dynamics into the imitation policy, and thus enables long-horizon imitation.
3.1 PRELIMINARIES
Maximum entropy model of expert behavior. SQIL is built on soft Q-learning, which assumes that expert behavior follows the maximum entropy model (Ziebart et al. 2010 Levine 2018). In
Equation assumes discrete actions, but SQIL. can also he used with continuous actions, shown in
Section 4
3

---

Published as a conference paper at ICLR 2020
an Markov Decision Process (MDP) with continuous state space S and discrete action space A the expert assumed to follow policy я that maximizes reward R(s, a). The policy и forms a Boltzmann distribution over actions,
exp (Q(s, a))
(als) Ex'EA exp (Q(s,
where Q is the soft Q function. The soft Q values are a function of the rewards and dynamics, given by the soft Bellman equation.
(ss e R(s, + yEs
log Σ exp (Q(('
Na'EAA
In our imitation setting, the rewards and dynamics are unknown. The expert generates fixed set of demonstrations Daemo by rolling out their policy π in the environment and generating state transi- tions (8, a, 8') Damo-
Behavioral cloning (BC). Training imitation policy with standard BC corresponds to fitting parametric model Te that minimizes the negative log-likelihood loss,
BB (0)
Σ
log To (a)s))
(4)
(s.a)EDarm
In our setting, instead of explicitly modeling the policy ma. we can represent the policy ı in terms of soft Q function Qo via Equation
exp (Qols,))))
(aas)) Ex'ed exp (Q((s,a)))
(5)
Using this representation of the policy, train Qo via the maximum-likelihenn objective in Equation
Σ -(eotss)- log Σ exp (Qo(s, a'Ed
(0)
A
(6)
(s.0) E Daam
However, optimizing the BC loss in Equation 6 does not in general yield valid soft function Qe - i.e., a soft Q function that satisfies the soft Bellman equation (Equation 3 with respect to the dynamics and some reward function. The problem is that the BC loss does not incorporate any information about the dynamics into the learning objective, so Qo learns to greedily assign high values demonstrated actions, without considering the state transitions that occur consequence of actions. As a result, Qo may output arbitrary values in states that are off-distribution from the demonstrations Docmo-
In Section 3.2 we describe a regularized BC algorithm that adds constraints to ensure that Qe is valid soft Q function with respect to some implieitly-representedd reward function, and further regularizes the implicit rewards with sparsity prior. In Section 3.3 we show that this approach recovers an algorithm similar to SQIL.
3.2 REGULARIZED BEHLAVIORAL CLONING
Under the maximum entropy model described in Section 3.1 function, a soft Q function that computes expected future returns, and a policy that takes actions with high soft Q values. In the previous section, we used these assumptions policy in terms of a model of the soft Q function Qo (Equation 5). In this section, we represent the reward function implicitly in terms of Qe, shown in Equation 7 This allows us to derive SQIL as a variant of BC that imposes sparsity prior on the implicitly- represented rewards.
expert behavior is driven by a reward
represent the imitation
Sparsity regularization. The issue with BC that, when the agent encounters states that are out- of-distribution with respect to Daemo- Qo may output arbitrary values. One solution from prior work
A ssuming discrete action space simplifies our analysis. SQIL. can be applied to continuous control tasks using existing sampling methods (Haarnoja al. 2017 2018), illustrated in Section 4.3
4

---

Published as a conference paper at ICLR 2020
(Poo et al., 2014) is to regularize Qe with sparsity prior on the implied rewards in particular, penalty on the magnitude of the rewards ExesueEA (s, implied by Qo via the soft Bellman
equation (Equation B). where
a) & Qo(s,a) - ylE'' log Σ exp (Q((s (
Note that the reward function Ry is explicitly modeled in this method. Instead, directly minimize the magnitude of the right-hand side of Equation which is equivalent to minimizing (s.al.
The purpose of the penalty on (s, a)] is two-fold: (1) it imposes sparsity prior motivated by prior work (Piot et al. 2201)) and (2) incorporates information about the state transition dynamics into the imitation learning objective, since (s, a) is function of an expectation over next state s'. (2) is critical for learning long- nag-hooizon behavior that imitates the demonstrations, instead of greedy maximization of the action likelihoods in standard BC. For details, see Piot et al. (2014).
Approximations for continuous states. Unlike the discrete environments tested in Piot et al. (2044), we assume the continuous state space S cannot be enumerated. Hence, we approximate the penalty E.ES.neA (s, (s, a) by estimating from samples: transitions (s, a, s') observed in the demonstrations D Demmo as well additional rollouts sampp periodically sampled during training using the latest imitation policy. This approximation, which follows the standard approach to con- straint sampling (Calafiore & Dabbene 2006), ensures that the penalty covers the state distribution actually encountered by the agent, instead of only the demonstrations.
To make the penalty continuously differentiable, we introduce an additional approximation: instead of penalizing the absolute value (s, a)), we penalize the squared value (R,, a))² Note that since the reward function Rg is not explicitly modeled, but instead defined via Qo in Equation 7
the squared penalty (R, a))² is equivalent to the squared soft Bellman error 8² (Daema Dasamp 0) from Equation
Regularized BC algorithm. Formally, we define the regularized BC loss function adapted from Piot et al. 2014) as
(+) - (0) + >6 (Ddemoo U Dssapp 0),
(8)
where À E Rxo is a constant hyperparameter, and 8² denotes the squared soft Bellman error de- fined in Equation [ The BC loss encourages Qo to output high values for demonstrated actions at demonstrated states, and the penalty term propagates those high values to nearby states. In other words, Qo outputs high values for actions that lead to states from which the demonstrated states are reachable. Hence, when the agent finds itself far from the demonstrated states, it takes actions that lead back to the demonstrated states.
The RBC algorithm follows the same procedure as Algorithm 1 except that in line RBC takes gradient step on the RBC loss from Equation g instead of the SQIL loss.
3.3 CONNECTION BETWERN SQIL AND REGULARIZED BEHAVTORAL. CLONING
The gradient of the RBC loss in Equation g is proportional to the gradient of the SQIL loss in line
of Algorithm T plus an additional term that penalizes the soft value of the initial state 8o (full derivation in Section of the appendix):
(9) In other words, SQIL. solves a similar optimization problem to RBC. The reward function in SQII. also has clear connection to the sparsity prior in RBC: SQIL imposes the sparsity prior from RBC, by training the agent with an extremely sparse reward function r = 1 at the demonstrations, and = 0 everywhere else. Thus, SQIL can be motivated as practical way to implement the ideas for regularizing BC proposed in Piot et al. (2014).
VolRuc(O) x Vo (D2emo 1) + Asamp 5² (D samp- 0) + V(su))
The main benefit of using SQIL instead of RBC is that SQIL is trivial implement, since it only requires a few small changes to any standard Q-learning implementation (see Section ().. Extending SQIL. to MDPs with continuous action space is also easy, since we simply replace Q-learning
5

---

Published as a conference paper at ICLR 2020
with off- off-policcy actor-critic method (Haarnoja et al. (0118) (see Section 4.3 Given the difficulty
of implementing deep RL algorithms correctly (Henderson et al. 2018), this flexibility makes SQIL more practical to use, since it can be built top of existing implementations of deep RL algorithms. Furthermore, the ablation study in Section 4. 4.4 that SQII. actually performs better than RBC.
4 EXPERIMENTAL EVALUATION
Our experiments aim to compare SQIL. to existing imitation learning methods on a variety of tasks with high-dimensional, continuous observations, such as images, and unknown dynamics. We benchmark SQIL against BC and GAIl on four image-based games - Car Racing, Pong, Break- out, and Space Invaders and three state-based tasks Humanoid, HalfCheetah, and Lunar Lander (Brockman et al., 20116; Bellemare et al.) 2013: Todorov et al.) 2012). We also investigate which components of SQIL contribute most to its performance via an ablation study on the Lunar Lander game. Section A.3 in the appendix contains additional experimental details.
4.1 TESTING GENERALIZATION IN MAGE-BASED CAR RACING
The goal of this experiment is to study not only how well each method can mimic the expert demon- strations, but also how well they acquire policies that generalize to states that are not seen in the demonstrations. To do so, we train the imitation agents in an environment with a different initial state distribution Sjain than that of the expert demonstrations Symmo, allowing to system- atically control the mismatch between the distribution of states in the demonstrations and the states actually encountered by the agent. We run experiments on the Car Racing game from OpenAI Gym. To create Sgmin the car is rotated 90 degrees so that it begins perpendicular to the track, instead of parallel to the track as in St This intervention presents a significant generalization challenge to the imitation learner, since the expert demonstrations do not contain any examples of states where the car is perpendicular to the road, or even significantly off the road axis. The agent must learn to make a tight turn to get back on the road, then stabilize its orientation so that it is parallel to the road, and only then proceed forward to mimic the expert demonstrations.
The results in Figure show that SQIL and BC perform equally well when there is variation in the ini- tial state. The task is easy enough that even BC achieves high reward. Note that, in the unperturbed condi- tion (right column), BC substantially outperforms GAIL, despite the well- known shortcomings of BC. This in- dicates that the adversarial optimiza- tion in GAIL can substantially hinder learning, even in settings where standard BC is sufficient. SQIL performs much better than BC when starting from Shain showing that SQIL is capable of generalizing to a new initial state distribution, while BC is not. SQIL learns to make tight turn that takes the car through the grass and back onto the road, then stabilizes the car's orientation so that it is parallel to the track, and then proceeds forward like the expert does in the demonstrations. BC tends drive straight ahead into the grass instead of turning back onto the road.
Domain Shift (Stti))
No Shift (Sdemo)
Random BC
21 i 56
-68 1 4
-15 + 18
698 +
GAIL-DQL.
-97 + 3
-66 1 8
SQIL (Ours)
375 + 19
704 + 6
Expert
180 + 11
704 + 79
Figure 1: Average reward on 100 episodes after training. Standard error three random seeds.
4 For all the image-based tasks, we implement version of GAIL that uses deep Q-learning (GAIL-DQL) instead of TRPO in the original GAIL paper (Ho & Ermon 2016). since Q-leaming performs better than TRPO in these environments, and because this allows for head-to-head comparison of SQIL and GAIL: both algorithms use the same underlying RL algorithm, but provide the agent with different rewards- SQIL provides constant rewards, while GAIL provides learned rewards. We use the standard GAIL-TRPO method as a baseline for all the low- low-dimensional tasks, since TRPO performs better than Q-learning in these environments.
The original GAIL method implicitly encodes prior knowledge- namely, that terminating an episade is either always desirable or always undesirable. As pointed out in Kostrikov al. (2019), this makes comparisons to alternative methods unfair. We implement the unbiased version of GAIL proposed by Kostrikov et al. (2019). and use this in all of the experiments. Comparisons to the biased version with implicit termination knowledge are included in Section A.2 in the appendix.

---

Published as a conference paper at ICLR 2020
Breakout
Pong
Space Invaders
300
20
Revord
SQIL (Ours)
Revord
Rewwrd
403+
200
GAIL-DQL
BC (P'91)
100
Expert
-00
2000
1500
1000
6000
Number of On-Policy Rollouts
Number On-Folicy Rellouts
Number of On-Palicy Rolluuts
Figure 2: Image-based Atari. Smoothed with a rolling window of 100 episodes. Standard error on three random seeds. X-axis represents amount of interaction with the environment (not expert demonstrations).
SQIL outperforms GAIL in both conditions. Since SQIL and GAIL both use deep Q-learning for RL in this experiment, the gap between them may be attributed the difference in the reward functions they use to train the agent. SQIL benefits from providing a constant reward that does not require fitting a discriminator, while GAIL struggles train discriminator to provide learned rewards directly from images.
4.2 IMAGE-BASED EXPERIMENTS ON ATARI
The results in Figure 2 show that SQIL outperforms BC on Pong, Breakout, and Space Invaders additional evidence that BC suffers from compounding errors, while SQIL does not. SQIL also outperforms GAIL on all three games, illustrating the difficulty of using GAIL. to train an image- based discriminator, in Section
4.3 INSTANTIATING SQIL FOR CONTINUOUS CONTROL IN LOW-DIMENSIONAL MuJoCo
HalfCheetah-v1
The experiments in the previous sections evaluate SQIL on MDPs
with a discrete action space. This section illustrates how SQIL can be adapted to continuous actions. We instantiate SQIL using soft actor-critic (SAC) an off- off-policy RL algorithm that can solve con- tinuous control tasks (Haarnoja al. 2018). In particular, SAC modified in the following ways: (1) the agent's experience re- play buffer is initially filled with expert demonstrations, where re-
1000
Rowerd
(4F1))
BC (FR11)
SALL (Ouss
Number Rallots
wards are set to r +1, (2) when taking gradient steps to fit the agent's soft Q function, balanced number of demonstration ex- periences and new experiences (50% each) are sampled from the replay buffer, and (3) the agent observes rewards of r during its interactions with the environment, instead of an extrinsic reward signal that specifies the desired task. This instantiation of SQIL is compared to GAIL on the Humanoid (17 DoF) and HalfCheetah (6 Dol) tasks from MuJoCo.
Humanoid-vv
800
0C0-
Ronard
800
200-
40 Number of Demonstration Rollouts
The results show that SQIL outperforms BC and performs compa- rably to GAIL on both tasks, demonstrating that SQIL can be suc- cessfully deployed problems with continuous actions, and that SQIL can perform well even with a small number of demonstra- tions. This experiment also illustrates how SQIL be top of SAC or any other pollpy value-based RL. algorithm.
Figure SQIL: best per- formance on 10 consecutive training episodes. BC, GAIL; results from Dhariwal et al. 2017).
4.4 ABLATION STUDY ON LOW-DIMENSIONAL LUNAR LANDER
We hypothesize that SQIL works well because it combines information about the expert's policy from demonstrations with information about the environment dynamics from rollouts of the imi- tation policy periodically sampled during training. We also expect RBC to perform comparably to SQIL, since their objectives are similar. To test these hypotheses, we conduct ablation study using the Lunar Lander game from OpenAI Gym. As in Section 4.1
we control the mismatch between the
7

---

Published as a conference paper at ICLR 2020
distribution of states in the demonstrations and the states encountered by the agent by manipulating the initial state distribution. To create Sp the agent is placed in a starting position never visited in the demonstrations.
In the first variant of SQIL, Aump is set to zero, to prevent SQIL from using additional samples drawn from the environment (see line 4 of Algorithm T). This comparison tests if SQIL really needs to interact with the environment, or if it can rely solely on the demonstrations. In the second condition, y is set to zero to prevent SQIL from accessing information about state transitions (see Equation and line 4 of Algorithm|| D. This comparison tests if SQIL is actually extracting information about the dynamics from the samples, or if it can perform just as well with a naive regularizer (setting to zero effectively imposes a penalty on the L2-norm of the soft Q values instead of the squared soft Bellman error). In the third condition, a uniform random policy is used to sample additional rollouts, instead of the imitation policy To (see line 6 of Algorithm I This comparison tests how important it is that the samples cover the states encountered by the agent during training. In the fourth condition, we use RBC to optimize the loss in Equation g instead of using SQIL to optimize the loss in line of Algorithm I RBC vs. SQIL (see Equation 9
This comparison tests the effect of the additional V Vssu) term in
The results in Figure show that all methods perform well when there is no variation in the initial state. When the initial state is varied, SQIL per-
Domain Shift (S)
No Shift (S))mm)
Random
0.10 = 0.30
0.04 + 0.02
BC
0.07 + 0.03
0.93 + 0.03
GAIL-TRPO
0.67 + 0.04
0.93 = 0.03
forms significantly better than BC,
SQIL (Ours)
0.89 + 0.02
0.88 + 0.03
GAIL, and the ablated variants of SQIL. This confirms our hypothesis that SQIL needs to sample from the environment using the imitation pol- icy, and relies on information about the dynamics encoded in the samples. Surprisingly, SQIL outperforms RBC by a large margin, suggesting that the penalty on the soft value of the initial state V (so), which is present in RBC but not in SQIL (see Equation 9), degrades performance.
Ablation
Asmp
0.12 = 0.02
0,87 0.02
= 0
0.41 + 0.02
0.84 0.02
Unif
0.47 + 0.02
0.82 + 0.02
RBC
0.66 + 0.02
0.89 + 0.01
Expert
0.93 - 0.03
0.89 + 0.31
Figure 4: Best success rate on 100 consecutive episodes dur- ing training. Standard error on five random seeds. Perfor- mance bolded if at least within one standard error of expert.
5 DISCUSSION AND RELATED WORK
Related work. Concurrently with SQIL, two other imitation learning algorithms that use constant rewards instead of a learned reward function were developed (Sasaki et al. 2019] Wang et al.) 2019). We our paper contributing additional evidence to support this core idea, rather than proposing a competing method. First, SQIL. is derived from sparsity-regularized BC, while the prior meth- ods are derived from an alternative formulation of the IRL objective (Sasaki et al.. (019) and from support estimation methods (Wang et al., 2019). showing that different theoretical approaches inde- pendently lead to using RL with constant rewards as an alternative to adversarial training - sign that this idea may be a promising direction for future work. Second, SQIL is shown to outperform BC and GAIL in domains that were not evaluated in Sasaki et al. 2019) or Wang et al. (2019) - in particular, tasks with image observations and significant shift in the state distribution between the demonstrations and the training environment.
Summary. We contribute the SQII. algorithm: a general method for learning to imitate an expert given action demonstrations and access to the environment. Simulation experiments on tasks with gigh-dimensional, continuous observations and unknown dynamics show that our method outper- forms BC and achieves competitive results compared to GAIL, while being simple to implement on top of existing off- policy RL algorithms.
Limitations and future work. We have not yet proven that SQIL matches the expert's state oceu- pancy measure in the limit of infinite demonstrations. One direction for future work would be to rigorously show whether or not SQIL has this property. Another direction would be to extend SQIL. recover not just the expert's 's policy, but also their reward function; e.g., by using parameterized reward function to model rewards in the soft Bellman error terms, instead of using constant rewards. This could provide simpler alternative to existing adversarial IRL algorithms.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Brenna D Argall, Sonia Chemova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5)469-4483, 2009.
Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Giuseppe Calafiore and Fabrizio Dabbene. Probubilliticc and randomized methods for design under uncertainty. Springer, 2006.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alee Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. /attpa://gitub.. com/ com/ openai / ines 2017.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversar- ial networks, inverse reinforcement lcarning, and energy-based models. arXiv preprint arXiv:1611.03852 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abheel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference Machine Learning, pp. 49-58, 2016b.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint ar00: 12448, 2017.
Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell, et al. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv: 180..0330 2018.
Ian Goodfellow, Jean Pouget-Aboee Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information process- ing systems, pp. 2672-2680, 2014.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. arXiv preprint arXiv:1809.01999 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abbecl, and Sergey Levine. Reinforcement leamning with deep energy- based policies. arXiv preprint arXiv:/702.08166, 2017.
Tuomas Haarnoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, arXiv preprint arXiv:1801.01290 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelie Pincau, Doina Precup, and David Meger. Deep rein- forcement learning that matters. In Thirty- AAAI Conference on Artificial Intelligence, 2018.
Todd Hester. Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan. John Quan, Andrew Sendonaris. Gabriel Dulac-Arolld, et al. Deep q-learing from demonstrations. arXiv preprint 174.0:17 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, pp. 4565 4573, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method stochastic optimization. arXiv preprint
arXiv: 1412.6980, 2014.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-acto--cti Addressing sample inefficiency and reward bias in adversarial imitation learn- ing. In International Conference on Learning Representations, 2019.
ex.
net bet/ffrumm d fpoA5km
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. The gan landscape: Losses, architectures, regularization, and normalization. arXiv preprint 1xXv:1807.0411 2018.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint 805.0:1805.000009, 2018.

---

Published as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukeuoglu. David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 51 518(7540):529, 2015.
Andrew Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, pp. 663-670, 2000.
Bilal Piot, Matthieu Geist, Olivier Pietquin. Learning from demonstrations: it worth estimating a reward function? In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 17-32. Springer, 2013.
Bilal Piot, Matthicu Geist, and Olivier Pictquin. Boosted and reward- reghaariegl classification for apprentice- ship lcarning. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp. 1249 1256. International Foundation for Autonomous Agents and Multiagent Systems, 2014.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Compu- tation, 3(1):88-97, 1991.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell, A reduction of imitation learning and structured pre- diction no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 627-635, 2011.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. In International Conference on Learning Representations, 2019. URL /ttttpts///opeeeeeii..
met/fooumm Eefrummid-BkNee 1d-BBN500AQF7
Emanuel Tadorov, Tom Frez, and Yuval Tassa. Mujoco: physics engine for model-hased control. In In-
telligent Rohots ond Systems (IROS), 2012 IEEF/RSJ International Conference on, pp. 5026-5033. IEFE, 2012.
Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. arXiv preprint 1057::900..06750, 2019.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement Ieam- ing. arXiv preprint arXiv: 04888, 2015.
Brian D Zichart, Andrew Bagnell, and Anind Dey. Modeling interaction via principle maximum causal entropy. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp- 1255-1266 1262. Omnipress, 2010.
10

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 DERIVATION OF RBC GRADIENT
Let (5o, ao. $1,..., ST) denote rollout, where ST is absorbing state. Let V denote the soft value function,
V(s) - log Σ exp (Qo(s.)))
(10)
Splitting up the squared soft Bellman error terms for Demmo and D Psump in Equation
T-1
V (O) =
-
TE Deno
1=0
T-1
+ Ademo Σ
Σ
v(Qo(sn,.)) - (V(6++1)) + >>samp (8²(Dsamp: 0)
TEDarn
1=
Σ
Σ - (V((s))>VV((sH))
DEDam 1=0
+ Ademo Vd² (Daemos, 2ddemo
+ Xump VD2 0).
(11)
Setting 7 - turns the inner sum in the first term into telescoping sum:
Σ
(VV(so) (8T))) + Atemm V Deemos 2d
Asamp Vo² (Dsmapp (amp: 0). (12)
TEDam
Since ST is assumed to be absorbing, V(ST) is zero. Thus,
12
VV()s) + Ademo V8² (Dasmo: 2ddemo
+ Asamp VO² samp. 0),
(13)
E Camo
In our experiments, we have that all the demonstration rollouts start at the same initial state so Thus,
(Demmo 1) + >samp 62 6' 0) + V(so))
(14)
where Asamp E R2o constant hyperparameter.
A.2 COMPARING THE BIASED AND UNBIASED VARIANTS OF GAIL.
As discussed in Section 4 correct the original GAIL method's biased handling of rewards at absorbing states, we implement the suggested changes to GAIL. in Section 4.2 of Kostrikov et al. (2019): adding transition to an absorbing state and a self-loop at the absorbing state the end of each rollout sampled from the environment, and adding binary feature to the observations indicat- ing whether or not state is absorbing. This enables GAIL to learn a non-zero reward for absorbing states. We refer the original, biased GAIL method GAIL-DOL-B and GAIL-TRPO-BB and the unbiased version as GAIL-DQL-U and GAIL-TRPO-U.
The mechanism for learning terminal rewards proposed in Kostrikov et al. (2019) does not apply to SQIL., since SQII. does not learn a reward function. SQIL. implicitly assumes a reward of zero at absorbing states in demonstrations. This is the case in all our experiments, which include some envi- ronments where terminating the episode is always undesirable (e.g., walking without falling down) and other environments where success requires terminating the episode (e.g., landing target), suggesting that SQIL is not sensitive to the choice of termination reward, and neither significantly benefits nor significantly harmed by setting the termination reward to zero.
Demoonttation rollouts may still vary due to the stochasticity of the expert policy.
11

---

Published as a conference paper at ICLR 2020
Domain Shift (SQ
No Shift (S)
Random
-21 56
68 4
BC
-45 18
698 + 10
GAIL-DOL-B
-91 i 4
-34 1 21
GAIL-DQL-U
-97 +
-66 +
SQIL (Ours)
375 19
704 +
Expert
480 1 11
704 79
Figure 5: Image-based Car Racing. Average reward 100 episodes after training. Standard error on three random seeds.
Breakout
Pong
Space Invaders
4041
SGIL [Ours]
GAIL-DQI-B
500
Reword
GAIL DQLLU
Revard
Raword
BC (P 91]
2000
Fxpertt
-10
200-
-20
2000
1000
6000
5000
500
1000
1500
2000
20000 Number
4000
6000
Number Os-Palicy Rallouts
Number
On-Poliry Rollouts
On-Policy Rollauts
Figure 6: Image-based Atari. Smoothed with a rolling window of 100 episodes. Standard error on three random seeds. X-axis represents amount of interaction with the environment (not expert demonstrations).
Car Racing. The results in Figure show that both the biased (GAIL-DQL-B) and unbiased (GAIL- DQL-U) versions of GAIL perform equally poorly. The problem of training an image-based discrim- inator for this task may be difficult enough that even with an unfair bias toward avoiding crashes that terminate the episode, GAIL-DQL-B does not perform better than GAIL-DOL-U.
Atari. The results in Figure 6 show that SQIL outperforms both variants of GAIL on Pong and the unbiased version of GAIL (GAIL-DOL-U) on Breakout and Space Invaders, but performs compara- bly to the biased version of GAIL (GAIL-DQL-B) on Space Invaders and worse than it on Breakout. This may be due the fact that in Breakout and Space Invaders, the agent has multiple lives five in Breakout, and three in Space Invaders and receives a termination signal that the episode has ended after losing each life. Thus, the agent experiences many more episode terminations than in Pong, exacerbating the bias in the way the original GAIL method handles rewards at absorb- ing states. Our implementation of GAIL-DQL-B in this experiment provides learned reward of r(s, a) = - log (1 details). The learned reward is always positive, while the implicit reward at an absorbing state is zero. Thus, the agent is inadvertently encouraged to avoid terminating the episode, For Breakout and Space Invaders, this just happens to be the right incentive, since the objective to stay alive as long possible. GAIL-DQL-B outperforms SQIL in Breakout and performs comparably to SQIL in Space Invaders because GAIL -DLL-B is accidentally biased in the right way.
a)), where D is the discriminator (see Section A.3 in the appendix for
Lunar Lander: The results in Figure 7 show that when the initial state is varied, SQIL outperforms the unbiased variant of GAIL (GAIL-TRPO-U), but underperforms against the biased version of GAIL (GAIL-TRPO-B), The latter result is likely due to the fact that the implementation of GAIL- TRPO-B we used in this experiment provides a learned reward r(s, a) = log (D(s, a)), where D is the discriminator (see Section A.3 in the appendix for details). The learned reward always negative, while the implicit reward at an absorbing state is zero. Thus, the agent is inadvertently encouraged terminate the episode quickly. For the Lunar Lander game, this just happens to be the right incentive, since the objective is to land on the ground and thereby terminate the episode. As in the Atari experiments, TRPO-B performs better than SQII. in this experiment because GAIL-TRPO-B is accidentally biased in the right way.
12

---

Published as a conference paper at ICLR 2020
Domain Shift (So
No Shift (S0
Random
0.10 0.30
0.04 +
BC
0.07 + 0.03
0.93 + 0.03
GAIL-TRPO-E (HI 16)
0.98 + 0.01
0.95 + 0.02
GAIL-TRPO-U
0.67 + 0.04
0.93 0.03
SQIL (Ours)
0.89 + 0.02
0.88 + 0.03
Ablation
Aump
0.12 1 0.02
0.87
0
0.41 + 0.02
0.84
Unif
0.47 + 0.02
0.82 0.02
RBC
0.66 0.02
0.89 0.01
Expert
0.93 + 0.03
0.89 + 0.31
Figure 7: Low-dimensional Lunar Lander. Best success rate on 100 consecutive episodes during
training. Standard error on five random seeds. Performance bolded at least within one standard emror of expert.
Humanoid-v1
600
Squared Bellman Error
150-
400
Reward
100
200
50-
50000
100000
150000
Number of Training Steps
Figure 8: Standard error over two random seeds. No smoothing across training steps.
A.3 IMPLEMENTATION DETAILS
To ensure fair comparisons, the same network architectures were used to evaluate SQIL, GAIL, and BC. For Lunar Lander, we used network architecture with two fully-connected layers containing 128 hidden units each to represent the Q network in SQIL, the policy and discriminator networks in GAIL, and the policy network in BC. For Car Racing, we used four convolutional layers (following
(Ha & Schmidhuber 2018)) and two fully-connected layers containing 256 hidden units each. For Humanoid and HalfCheetah, we used two fully colle-ceneeeted layers containing 256 hidden units each. For Atari, used the convolutional neural network described in (Mnih et al. 2015) to represent the Q network in SQII,, as well as the Q network and discriminator network in GAIL.
To ensure fair comparisons, the
demonstration data used train SQIL, GAIL, and BC. For Lunar Lander, we collected 100 demonstration rollouts. For Car Racing, Pong, Breakout, and Space Invaders,
generated from scratch for Lunar Lander using DQN (Mnih et al. 2015). and collected from open-source pre- trained policies for Car Racing (Ha & Schmidhuber) 2018) well as Humanoid and HalfCheetah (Dhariwal et al. 2017). The Humanoid demonstrations were generated by stochastic expert policy, while the HallCheetah demonstrations were generated by deterministic expert policy; both experts were trained using TRPO 0 We used two open-source implementations of GAIL: (Fu et al. 2017] for Lunar Lander, and (Dhariwal et al. (007)) for MuJoCo. We adapted the OpenAl Baselines implementation of GAIL to use soft Q-learning for Car Racing and Atari. Expert demonstrations were generated from scratch for Atari using DQN.
collected 20 demonstration rollouts. Expert demonstrations
For Lunar Lander, we set Asamp 10-6, For Car Racing, we set Asamp = 0.01. For all other environments, we set Asamp 1.
13

---

Published as a conference paper at ICLR 2020
SQIL was not pre-trained in any of the experiments. GAIL pre-trained using BC for HalfChee- tah, but was not pre-trained in any other experiments.
In standard implementations of soft Q-learning and SAC, the agent's experience replay buffer typ- ically has a fixed size, and once the buffer is full, old experiences are deleted to make room for new experiences. In SQIL,, never delete demonstration experiences from the replay buffer, but otherwise follow the standard implementation.
We use Adam (Kingma & Ba 2014) take the gradient step in line of Algorithm
The BC and GAIL performance metrics in Section 4.3 are taken from (Dhariwal et al. 2017
The GAIL and SQIL policies in Section 4.3 are be deterministic during the evaluation rollouts used to measure performance.
ga
14


Reviews:
[{"review_id": "rkl_u9hhYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe authors propose SQUIL, an off-policy imitation learning (IL) algorithm which attempts to overcome the classic drift problems of behavioral cloning (BC). The idea is to reduce IL to a standard RL problem with a reward that incentivizes the agent to take expert actions in states observed in the demonstrations. The algorithm is tested on both image-based tasks (Atari) and continuous control tasks (Mujoco) and shown effective against GAIL and a simple supervised BC approach.\n\nComments\n--------\nOvercoming the limitations of BC algorithms is a relevant problem and this work presents a simple and interesting solution. The idea is to use any off-policy RL algorithm (assuming access to the true environment) with a reward that favors imitation. The paper is well-organized and easy to read. The experiments are also quite convincing and demonstrate that the proposed methodology is, in fact, more robust to distribution shift (i.e., less prone to overfitting the train set) than standard supervised BC and GAIL. Some comments/questions follow.\n\n1. The considered settings (e.g., MDPs, IL/IRL problems) are never formalized in the paper (which directly starts by describing the method in Section 2). It would be good to add some preliminaries to allow even readers unfamiliar with this problem to easily go through the paper.\n\n2. Though the approach solves one of the main problems of many BC/non-IRL algorithms, it still has some of the other limitations. For instance, the recovered policy is non-transferable to different dynamics, while a reward function is. Furthermore, it cannot recover optimal behavior if the expert is sub-optimal.\n\n3. In many practical cases, the assumption that we can access the true environment is not reasonable. Suppose, for instance, that we have a dataset of driving demonstrations from a real car, and that is all we can obtain. Of course, we cannot start interacting with the real car to imitate the driver's behavior and we need to rely on simulators, which poses additional difficulties due to the inevitable approximations. Do the authors think that a batch RL algorithm could be adopted to solve this task (i.e., by solely using the expert data)? The ablation study in section 4 seems to answer this question negatively, at least for large distribution shifts.\n\n4. Below equation (1), why \"r does not depend on the state or action\"? It should be R(s,a) = INDICATOR{(s,a) \\in D_{demo}}, so it does depend on (s,a).\n\n5. D_{demo} is missing from the inputs to Algorithm 1 \n\n6. I did not understand why, in Section 3.2 (above Equation 8), the authors mention that (7) squared is equivalent to the squared Bellman error of Equation 1. Where is r? It should be equivalent only from (s,a) \\in D_samp, for which r=0, but for (s,a) \\in D_demo we have r=1 and that term is missing in (7). Have I misunderstood something?\n\n7. In Figure 1, why does GAIL perform better with shift than no-shift?\n\n8. Why are some curves in Figure 3 (bottom) above the expert performance?\n\n9. It would be interesting to show how the proposed method compares to IRL-based approaches for continuous states/actions such as [1] or [2]. These algorithms should be applicable to the considered domains (at least Mujoco ones).\n\n[1] Finn, C., Levine, S., & Abbeel, P. (2016, June). Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (pp. 49-58).\n[2] Boularias, A., Kober, J., & Peters, J. (2011, June). Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (pp. 182-189).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgvIf2kcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an imitation learning approach via reinforcement learning. The imitation learning problem is transformed into an RL problem with a reward of +1 for matching an expert's action in a state and a reward of 0 for failing to do so. This encourages the agent to return to \"known\" states from out-of-distribution states and alleviates the problem of compounding errors. The authors derive an interpretation of their approach as regularized behavior cloning. Furthermore, they empirically evaluate their approach on a set of imitation learning problems, showing strong performance. The authors also stress the easy implementability of their approach within standard RL methods.\n\nI think this is a good paper which shows strong empirical results based on simple but effective idea. I would welcome an extended discussion of certain aspects of the experiments though -- for instance the trends in Figure 3. Furthermore, I think the huge gap in performance between SQIL and RBC warrants a more detailed discussion/analysis. Also the papers by Sasaki et al. (which was accepted at last year's ICLR) [and (maybe) the paper by Wang et al.] deserve to be discussed earlier in the paper (introduction?) and make it to the experimental results. \n\nThe paper could be improved considering the following minor suggestions:\n* Introducing the important bits of soft-Q learning formally.\n* Defining demonstrations formally (not only in the algorithm).\n* Showing the necessity to balance demonstrations and new experiences in the algorithm.\n* Removing implementation details from Section 2 and adding them to the experiments.\n* There is a bunch of repetitive paragraphs, that could be cleaned up.\n* Equation (2) and (5) are the same. Use $\\pi^*$ and $Q^*$ in (2)?\n* I disagree with paragraph following equation (6). If we phrase BC as an optimization problem over Q then Q should satisfy consistency. Otherwise, we are not optimizing over Q but some other object. \n* Please specify the initial state distribution.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xFrZXx9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple method for imitation learning that is competitive with GAIL.  The approach, Soft Q Imitation Learning (SQIL), utilizes Soft Q Learning, and defines high rewards as faithfully following the demonstrations and low rewards as deviating from them.  Because SQIL is off-policy, it can utilize replay buffers to accelerate sample efficiency.  One can also interpret SQIL as a regularized version of behavioral cloning.\n\nThe authors really play up the \"surprising\" connection of SQIL to a regularized behavioral cloning.  But this isn't really that surprising in the general sense (although I applaud the authors for rigorously defining the connection).  SQIL is basically adding \"pseudo-data\" and the idea of using pseudo-counts as regularization has been around for a long time.   I think it's a stretch to hype up the \"surprise\" factor so much, and that it does a disservice to the paper overall.\n\nThe experiments are sound, showing competitive performance to GAIL, and also having a nice ablation study.  \n\nI wonder if there was a quantitative way to demonstrate that SQIL is \"simpler\" than GAIL.  It's certainly conceptually easier.  Is the implementation in any concrete sense easier as well?  Or sensitivity to hyperparameters or initial conditions?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a simple alternative to adversarial imitation learning methods like GAIL that is potentially less brittle, and can skip learning a reward function, instead learning an imitation policy directly.  Their method has a close relationship with behavioral cloning, but overcomes some of the disadvantages of BC by encouraging the agent via reward to return to demonstration states if it goes out of distribution.  The reviewers agree that overcoming the difficulties of both BC and adversarial imitation is an important contribution.  Additionally, the authors reasonably addressed the majority of the minor concerns that the reviewers had.  Therefore, I recommend for this paper to be accepted.
Decision: Accept (Poster)