Support-guided Adversarial Imitation Learning
Authors: Ruohan Wang
Carlo Ciliberto
Pierluigi Amadori
Yiannis Demiris
Venue:  (2020)
Keywords: Adversarial Imitation Learning
Reinforcement Learning
Learning from Demonstrations

Abstract: We propose Support-guided Adversarial Imitation Learning (SAIL), a generic imitation learning framework that unifies support estimation of the expert policy with the family of Adversarial Imitation Learning (AIL) algorithms. SAIL addresses two important challenges of AIL, including the implicit reward bias and potential training instability. We also show that SAIL is at least as efficient as standard AIL. In an extensive evaluation, we demonstrate that the proposed method effectively handles the reward bias and achieves better performance and training stability than other baseline methods on a wide range of benchmark control tasks.

Under review as a conference paper at ICLR 2020
SUIDDRT-GUIDED ADVERSARIAL IMITATION LEARN- ING
Anonymous authors Paper under Houble-blind review
ABSTRACT
We propose Support-guided Adversarial Imitation Learning (SAIL), a generic imi- tation learning framework that unifies support estimation of the expert policy with the family of Adversarial Imitation Learning (AIL) algorithms. SAIL addresses two important challenges of AIL, including the implicit reward bias and potential training instability. We also show that SAIL. is at least as efficient as standard AlL.. In extensive evaluation, the reward bias and achieves better performance and training stability than other baseline methods on a wide range of benchmark control tasks.
demonstrate that the proposed method mitigates
INTRODUCTION
The class of Adversarial Imitation Learning (AIL) algorithms learns robust policies that imitate an expert's actions from small number of expert trajectories, without further access the expert or environment signals. AIL iterates between refining a reward via adversarial training, and rein- forcement learning (RL) with the learned adversarial reward. For instance, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) shows the equivalence between some settings of inverse reinforcement learning and Generative Adversarial Networks (GANs) (Goodfellow al., 2014), and recasts imitation learning as distribution matching between the expert and the RL agent. Similarly, Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al.] 2017) modifies the GAIL discriminator to learn reward function robust to changes in dynamics or environment properties.
AIL mitigates the issue of distributional drift from behavioral cloning (Ross al. 201 a classical imitation learning algorithm, and demonstrates good performance with only small number of expert demonstrations. However, AIL has several important challenges, including implicit reward bias (Kostrikov et al., 2219)), potential training instability (Salimans et al.) [206]] Brock et al. (2018), and potential sample inefficiency with respect to environment interaction (Sasaki et al. 2019). this paper, we propose principled approach towards addressing these issues.
Wang al. (209)) demonstrated that imitation learning is also feasible by constructing fixed reward function via estimating the support of the expert policy. Since support estimation only requires expert demonstrations, the method sidesteps the training instability associated with adversarial training. However, show in [Section 4.2 that the reward learned via support estimation deteriorates when expert data sparse, and leads to poor policy performances.
Support estimation and adversarial reward represent two different yet complementary RL signals for imitation learning, both learnable from expert demonstrations. We unify both signals into Support- guided Adversarial Imitation Learning (SAIL), generic imitation learning framework. SAIL leverages the adversarial reward to guide policy exploration and constrains the policy search to the estimated support of the expert policy. It compatible with existing AIL algorithms, such GAIL and AIRL. We also show that SAIL is at least as efficient as standard AIL. In an extensive evaluation, demonstrate that SAIL mitigates the implicit reward bias and achieves better performance and training stability against baseline methods over series of benchmark control tasks.
2 BACKGROUND
We briefly review the Markov Decision Process (MDP), the context of our imitation learning task, followed by related works imitation learning.

---

Under review as a conference paper at ICLR 2020
Markov Decision Process We consider an infinite-horizon discounted MDP (S, P,r. Po: 7) where is the set of states, A the set of actions, 5 A S + [0, the transition proba- bility, r S x A + R the reward function, Po S [0. 1] the distribution over initial states, and 7 E (0, 1) the discount factor. Let ι be a stochastic policy л S x A - [0, 1] with ex-
pected discounted reward E (r(s,a) a)) - EE (rr(
7 az)) where so ~ po. ~ π (-|ss),, and 8141 P(-(st, at) for i 2 O. We denote TH the expert policy.
Behavioral Cloning (BC) learns policy 5 - A directly from expert trajectories via supervised learning. BC is simple to implement, and effective when expert data is abundant. However, BC is prone to distributional drift: the state distribution of expert demonstrations deviates from that of the agent policy, due to accumulation of small mistakes during policy execution. Distributional drift may lead to catastrophic errors (Ross et al. (011). While several methods address the issue (Ross & Bagnell 2010 Sun et al.) 2017), they often assume further access to the expert during training.
Inverse Reinforcement Learning (IRL) first estimates reward from expert demonstrations, fol- lowed by RL using the estimated reward (Ng & Russell 22000) AAbeel & Ng 2004). Building upon a
maximum entropy formulation of IRL (Ziebart al. 2008), Finn al. (2016) and Fu et al. (2017)
explore adversarial IRL and its connection to Generative Adversarial Imitation Learning (Ho &
Ermon 2016).
Imitation Learning via Distribution Matching Generative Adversarial Imitation Learning (GAIL) (Ho& & Ermon| (016) frames imitation learning distribution matching between the expert and the RL agent. The authors show the connection between IRL and GANs. Specifically, GAIL imitates the expert by formulating a minimax game:
min max Ex (log D(s, a)) + E - D(s, a))), DE(0,1)
(1)
where the expectations Ex and ETg denote the joint distributions over state-actions of the RL agent and the expert, respectively. GAIL is able to achieve expert performance with a small number of expert trajectories on various benchmark tasks. However, GAIL is relatively sample inefficient with respect to environment interaction, and inherits issues associated with adversarial learning, such as vanishing gradients, training instability and overfitting to expert demonstrations (Arjovsky & Bottou 2017 Brock al. 2018).
Recent works have improved the sample efficiency and stability of GAIL. For instance, Generative Moment Matching Imitation Learning (Kim & Park 2018) replaces the adversarial reward with non-parametric maximum mean discrepancy estimator to sidestep adversarial learning. BBaram et al.
(011)) improve sample efficiency with model-based RL algorithm. Kostrikov et al. (2019) and
Sasaki et al. (2019) demonstrate sigmificant gain in sample efficiency with off oollyy RL algorithms. In addition, Generative Predecessor Models for Imitation Learning (Schroecker al. 2019) imitates the expert policy using generative models to reason about alternative histories of demonstrated states.
Our proposed method closely related to the broad family of AIL algorithms including GAIL, and adversarial IRL. It is also complementary to many techniques for improving the algorithmic efficiency and stability, as discussed above. In particular, we focus on improving the quality of the learned reward by constraining adversarial reward to the estimated support of the expert policy.
Imitation Learning via Support Estimation Alternative to All., Wang et al.) (2019) demonstrate the fcasibility of using fixed RL reward via estimating the support of the expert policy from expert demonstrations. Connecting kernel-based support estimation (De Vito et al.) 2014) to Random Network Distillation (Burda al., 2018), the authors propose Random Expert Distillation (RED) to
learn reward function based on support estimation. Specifically, RED learns the reward parameter 0 by minimizing:
min (f (s. a) - ff(ssa)]13
where fo : S x A 4 RKKK projects (s, a) from expert demonstrations to some embedding of size K, with randomly initialized 0. The reward then defined as:
(rea(s, a) = exp( exp(-ollfg(s,a) (aan))-a(ssss (ss..))l)).
where a is a hyperparameter. As optimizing Eg. (2) only requires expert data, RED sidesteps adversarial learning, and casts imitation learning
standard RL task using the learned reward.
While RED works well given sufficient expert data, we show in the experiments that its performance suffers in the more challenging setting of sparse expert data.
2

---

Under review as a conference paper at ICLR 2020
3 METHOD
Formally, we consider the task of learning a reward function f(s,, a) from a finite set of trajectories {rr) sampled from the expert policy TE within a MDP. Each trajectory is a sequence of state- action tuples in the form of T; = {$1. 01, 82, a2...., sr, ar). Assuming that the expert trajectories are consistent with some latent reward function (*, a), we aim to learn policy that achieves good performance with respect to rs(s, a) by applying RL on the learned reward function f(s,a).
In this section, we first discuss the advantages and shortcomings of AlL to motivate our method. We then introduce Support-guider Adversarial Learning (SAIL), and present theoretical analysis that compares SAIL with the existing methods, specifically GAIL.
3.1 ADVERSARIAL IMITATION LEARNING
clear advantage of AIL resides in its low sample complexity with respect to expert data. For instance, GAIL requires little as 200 state-action tuples from the expert achieve imitation. The reason that the adversarial reward may be interpreted as an effective exploration mechanism for the
RL agent. To see this, consider the learned reward function under the optimality assumption. With
the optimal discriminator to Eq. (1) D* (s. (..)) Pag D, (8.0)' a common reward for GAIL, is
P=(s,))
(gust(, (8, a) = (s,(D")),a)) = log (1 (an(s,a) = log(1 + (68,
(4)
Pa (8.
P- (s.a) Intuitively, Tgast incentivizes the RL agent towards under-visited state-actions, where Φ(s, > 1. and away from over-visited state-actions, where o(s, a) < 1. When TE and π match exactly, Tyail converges
Eq. (4) shows that the adversarial reward only depends the ratio ((s, a) Prg (s,a)
to an indicator function for the support TE, since Φ(s, a) = 1 A (s, a) E supp(Te) (Goodfellow et al.] 2014). In practice, the adversarial reward is unlikely converge, Præ estimated from finite set of expert demonstrations. Instead, the adversarial reward continuously drives the agent to explore by evolving the reward landscape.
However, AlL also presents several challenges. Kostrikov et al. (019)) demonstrated that the reward log (ss a)) suffers from implicit survival bias, the non-negative reward may lead to sub- optimal behaviors in goal-oriented tasks where the agent Iearns to move around the goal to accumulate rewards, instead of completing the tasks. While the authors resolve the issue by introducing absorbing states, the solution assumes extra RL signals from the environment, including access to the time limit of an environment detect early termination of training episodes. In [Section 4.1 we empirically demonstrate the survival bias on Lunar Lander, common RL benchmark, by showing that agents trained with GAIL often hover over the goal location We also show that our proposed method able to robustly imitate the expert.
Another challenge with AIL is potential training instability. Wang et al. (2019) demonstrated empirically that the adversarial reward could be unreliable in regions where the expert data is sparse, causing the agent to diverge from the intended behavior. When the agent policy is substantially different from the expert policy, the discriminator could differentiate them with high confidence, resulting in very low rewards and significant slow down in training, similar to the vanishing gradient problem in GAN training (Arjovsky & Bottou 2017).
3.2 SUPPORT-GUIDER ADVERSARIAL IMITATION LEARNING
We propose novel reward function by combining the standard adversarial reward Tgait with the corresponding support guidance Tred-
(s, ()).a))=))))))) ) (s, a) TTgui a).
(5)
SAIL is designed to leverage the exploration mechanism offered by the adversarial reward, and to constrain the agent the estimated support of the expert policy. Despite being simple modification, support guidance provides strong reward shaping to address the challenges discussed in the previous The agents still learn land for some initial conditions.
3

---

Under review as a conference paper at ICLR 2020
Algorithm SUPPORT-GUIDED ADVERSARIAL IMITATION LEARNING
1: Input: Expert trajectories TE = (()),)))) i=1 function models, initial policy Twp, initial discriminator parameters wo, learning rate Ip.
2: Tred RED(O,,TT)
3: for 0, I...
4: sample trajectory T;
5: Wi+1 w/ Ln (E,, (V log Dw: (8, (V Drog (s, (s))))))
6: Igail (s. () + 1 - D Dwats (8, 0)
7:
Tw:-1 -TRPO(rred Tgails Twss
8:
def RED(O, T)
9:
Sample A E +
10: 0 = Je.
11: return Tred (s, + exp(-o)l fo(s, a)
section. As both support guidance and adversarial reward are learnable from expert demonstrations, method requires no further assumptions that standard AlL.
SAIL. addresses the survival bias in goal-oriented tasks by encouraging the agent to stop at the goal and complete the task. In particular, Tred shapes the adversarial reward by favoring stopping the goal against all other actions, as stopping at the goal is on the support of the expert policy, while other actions are not. We demonstrate empirically that SAIL assigns significantl higher reward towards completing the task and corrects for the bias in Section 4.1] To improve training stability, SAIL constrains the RL agent to the estimated support of the expert policy, where r'gail provides more reliable RL signal (Wang et al. 2019). As fred tends to be very small (ideally zero) for (8, a) é supp(Tx),, Tsait discourages the agent from exploring those state-actions by masking away the rewards. This is desirable property as the quality of the RI. signals beyond the support of the expert policy can't 't be guaranteed. We demonstrate in Section 4.2 the improved training stability on the Mujoco benchmark tasks.
We provide the pseudocode implementation of SAIL in Algorithm 1 The algorithm computes Tred by estimating the support of the expert policy, followed by iterative updates of the policy and goail- We apply the Trust Region Policy Optimization (TRPO) algorithm (Schulman et al. 2015) with the reward rail for policy updates.
Reward Variants In practice, we observe that constraining the range of the adversarial reward generally produces lower-variance policies. Specifically, we transform r'gail in Fiq. (5) from - logg ((s, a) E [0, xo] to 1 a(s,a) E [o, 1]. For ease of notation, we refer to the bounded variant as SAIL-b, and the unbounded variant SAIL. Similarly, we denote the bounded GAIL reward as GAlL-b. We include the comparison between the reward variants in the experiments.
3.3 COMPARING SAIL WITH GAIL
In this section, we show that SAIL is at least as efficient as GAIL in its sample complexity for expert data, and provide comparable RL signals on the expert policy's support. We note that our analysis could be similarly applied to other AIL methods, suggesting the broad applicability of our approach.
We begin from the asymptotic setting, where the number of expert trajectories tends to infinity. In this case, both GAIL's,, RED's and SAIL's discriminators ultimately recover the expert policy's support at convergence (see Ho & Ermon (2016) for GAIL and Wang et al. (2019) for RED; SAIL follows from their combination). Morcover, for both GAIL and SAIL, the expert and agent policy distributions match exactly at convergence, implying successful imitation learning. Therefore, it is critical to characterize the rates of convergence of the two methods, namely their relative sample complexity with respect to the number of expert demonstrations.
Formally, let (s, a) & supp(TT)) Prototypical learning bounds for an estimator of the support f 0 provide high probability bounds in the form of TP(((s,)) S clog(1/8)) 10/00/0m999 1 8 for
4

---

Under review as a conference paper at ICLR 2020
any confidence 8 E (0. 1], with constant not depending ò or the number of samples (i.e., expert state-actions). Here, a > represents the learning rate, namely how fast the estimator is converging to the support. By choosing the reward in Eq. (5) we leveraging the faster learning rates between Ored and Ogail, with respect to support estimation. At the time being, no results are available to characterize the sample complexity of GAIL (loosely speaking, the and introduced above). Therefore, we proceed by focusing on a relative comparison with SAIL. In particular, we show the following (see appendix for proof).
Proposition 1. Assume that for any (s, a) & supp(TT) the rewards for RED and GAIL have the following learning rates in estimating the support
Gred log
Cgail log 8
IP (Frea(s, a)
S
P a)
P
3 6.
(6)
Matred
18
Then, for any ā E (0, 1] and any (8, & supp( the following holds
(Grallgai nrod
Cgail Cgait
Tsnil (s, a) S min
log 8-
(7)
with probability at least respectively.
8, where Rred and Ryust are the upper bounds for Tred and rgail.
Eq. (7) shows that SAIL is at least as fast as the faster among RED and GAIL with respect to support estimation, implying that SAIL is at least as efficient as GAIL in the sample complexity for expert data. Eq. (7) also indicates the quality of the learned reward, as state-actions outside the expert's support should be assigned minimum reward.
Proposition 2. For any (s, a) E supp(=E) and any 5 E (0, 1]. we assume that
P (Frea(s,a) 1|> Gred log 1 )
8.
(8)
naved
The following event holds with probability at least 5 that
[reau(s a) Tgaai( (s, a)) S Cred
1 log 8"
(9)
Eq. shows that on the expert policy's 's support, Fsail close to Tgail up to precision that improves with the number of expert state-actions. SAIL thus provides RL signals comparable to GAIL on the expert policy's support.
It is also worth noting that the analysis could explain why Fred + Tgast is a less viable approach for combining the two RL signals. The analogous bound to Eq. (7) would be the sum of errors from the two methods, implying the slower of the two learning rates, while Eq. (9) would improve only
by a constant, as Rgadl would be absent from Eq. (9) Our preliminary experiments indicated that
Tred + Tgail performed noticeably worse than Eq.
Lastly, we comment on whether the assumptions in Eqs. (6) (8) are satisfied in practice. Following the kernel-based version of RED (Wang et al., 2019), we can borrow previous results from the set learning literature, which guarantee RED to have a rate of ared = 1/2 (De Vito et al., 2014 Rudi et al. 2017). These rates have been shown to be optimal. Any estimator of the support cannot have imposed. Learning rates for distribution
faster rates than n 1/2, unless additional assumptions matching with GANs are still an active area of research, and conclusive results characterizing the convergence rates of these estimators are not available. We refer to Singh et al. (2018) for in-depth analysis of the topic.
4 EXPERIMENTS
We evaluate the proposed method against BC, GAIL and RED on Lunar Lander and six Mujoco control tasks including Hopper, Reacher, HalfChhetahh Walker2d, Ant, and Humanoid. We omit evaluation against methods using off policy RL algorithms, as they are not the focus of this work. We also note that support guidance is complementary such methods.
5

---

Under review as a conference paper at ICLR 2020
Default
No-terminal
BC
100.38 - 130.91
100.38 130.91
RED
13.75 53.43
-39.33 = 24.39
GAIL
258.30 = 28.98
169.73 - 80.84
GAIL-b
250.53 = 67.07
-69.33 = 79.76
SAIL
257.02 = 20.66
237.96 + 49.70
SAIL-bb
262.97 = 18.11
256.83 20.99
Expert
253.58 = 31.27
253.58 31.27
The task of Lunar Lander requires landing the spacecraft between the flags without crashing.
Figure 1:
Table I: Average environment reward and standard de-
viation on Lunar Lander, evaluated over 50 runs for the default and no-terminal environment.
4.1 LUNAR LANDER
We demonstrate that SAIL variants mitigate the survival bias in Lunar Lander (Fig. from OpenAl Gym (Brockman et al. 2016), while other baseline methods imitate the expert inconsistently. this task, the agent is required to control a spacecraft to safely land between the flags. A human expert provided demonstrations for this task as an imitation target.
We observe that even without the environment reward, Lunar Lander provides natural RL signal by terminating episodes early when crashes are detected, thus encouraging the agent to avoid crashing. Consequently, all methods are able to successfully imitate the expert and land the spacecraft appropriately. SAIL variants perform slightly better than GAIL variants on the average reward, and achieve noticeably lower standard deviation. The average performances and the standard deviations evaluated over 50 runs are presented in TTall
To construct more challenging task, disable all early termination feature of the environment, thus removing the environment RL signals. In this no-terminal environment, a training episode only ends after the time limit. We present each algorithm's 's performance for the no-terminal setting in Table I SAIL variants outperform GAIL variants. Specifically, we observe that GAIL learns to land for some initial conditions, while exhibit survival bias in other scenarios by hovering the goal. In contrast, SAIL variants are still able to recover the expert policy [
To visualize the shaping effect from support guidance, plot the average learned reward for GAIL, SAIL-b and RED at goal states. The goal states are selected from the expert trajectories and satisfy two conditions: 1) touching the ground (the state vector has indicator variables for ground contact), and 2) has "no op" as the corresponding action. As the adversarial reward functions are dynamic, we snapshot the learned rewards when the algorithms obtain their best policies, respectively, Fig. 3 shows the average rewards for each available action, averaged across all the goal states. Compared against the other algorithms, SAIL-b assigns a significantly higher reward to "no op", which facilitates the agent learning. Though GAIL and RED still favor "no op" to other actions, the differences in reward are much smaller, causing less consistent landing behaviors.
We further observe that all evaluated AIL methods oscillate between partially hovering behavior and landing behavior during policy learning. The observation suggests that our method only partially addresses the survival bias, limitation we will tackle in future works. This is likely caused by SAIL's non-negative reward, despite the beneficial shaping effect from support estimation. For additional experiment results and discussion on Lunar Lander, please refer the appendix.
4.2 MUJoCO TASKS
Mujoco control tasks have been commonly used as the standard benchmark for AIL. We evaluate SAIL against GAIL, RED and BC Hopper, Reacher, HalfCheetah, Walker2d, Ant and Humanoid. We adopt the same experimental setup presented in Ho & Ermon (2016) by sub-sampling the expert trajectories every 20 samples. Consistent with the observation from Kostrikov et al. (2019), our preliminary experiments show that sub-sampling presents a more challenging setting, as BC is competitive with AIL when full trajectories are used. In our experiments, we also adopt the minimum
experimennnnn
?A illustrative video available wwhttps::///
835881

---

Under review as a conference paper at ICLR 2020
S4L
73 60
0.
25-
0.2
5-
2000 9200
1000
10000
Figure Performance histogram of 50 evaluation runs on Humanoid for RED, GAIL, and SAIL-b. SAIL-b imitates the expert consistently. GAIL has undesirable failure cases, with rewards of less than 1000 (bottom left corner). RED is consistent though sub-optimal.
Figure 3: Average reward assignment at the goal states by diffcrent algorithms. SAIL-b assigns signif- icantly higher reward "no op", enabling the agent to learn the appropriate landing behaviors. Other algorithms fail to imitate the expert consistently.
Hopper
Reacher
Cheetah
Walker
Ant
Humanoid
BC
312.3 + 34.5
-8.8 + 3.3
1892.0 + 206.9
248.2 + 117.8
1752.0 + 434.8 539.4 + 185.7
RED
1056.5 + 0.5
99.1 4.1
-0.2 = 0,7
2372.8 - 8.8
1005.5 8.6
6012.0 + 434.9
GAIL
3826.5 3.2
-9.1
4604.7 + 77.6
5295,4 44.1
1013.3 = 16.0
8781.2 + 3112.6
GAIL-b
3810.5 1 8.1
-8.3 1 2.5
4510.0 68.0
5388.1
161.2
3413.1 + 744.7
10132.5 + 1859.3
SAIL
3824.7 1 6.6
-7.5
2.7
4747.5
43.4
5293.0
590.9
3330.4 1 729.4
9292.8 1 3190.0
SAIL-b
3811.6 1 3.8
-7.4 1
2.5
4632.2
59.1
5438.6 18.4
4176.3 1 203.1
10589.6 + 52.2
Table 2: Episodic reward and standard deviation on the Mujoco tasks by different methods evaluated over 50 runs. SAIL-b achieves overall the best performance, with significantly lower standard deviation, indicating the robustness the learned policies.
number of expert trajectories specified in Ho & Ermon (2016) for each task. More details on experiment setup are available in the appendix.
We apply each algorithm using different random seeds in all Mujoco tasks. Table shows the performance comparison between the evaluated algorithms. We report the mean performance and standard deviation for each algorithm over 50 evaluation runs, choosing the best policies obtained for each algorithm of the random seeds.
The results show that SAIL-b is comparable to GAIL on Hopper, and outperform the other methods on all other tasks. We note that RED significantly underperforms in the sub-sampling setting, while WWng itt al. (2019) used full trajectories in their experiments. Across all tasks, SAIL-b generally achieves lower standard deviation compared other algorithms, in particular for Humanoid, indicating the robustness of the learned policies.
We stress that standard deviation also a critical metric, it indicates the robustness of the learned policies when presented with different states. For instance, the large standard deviations in Humanoid are caused by occasional crashes, which may be highly undesirable depending on the intended applications. To illustrate robustness of the learned policies, we plot the histogram of all 50 evaluations in Humanoid for RED, GAIL-b and SAIL-b Fig. The figure shows that SAIL-b performs consistently with expert performance. Though GAIL-b appears be only slightly worse in average performance, the degradation is caused by occasional and highly undesirable crashes, suggesting incomplete imitation of the expert. RED performs the worst in average performance, but consistent with no failure modes detected. The result suggests that the proposed method combines the advantages of both support guidance and adversarial learning.
Comparing SAIL against SAIL-b, we observe that the bounded variant generally produces policies with smaller standard deviations and better performances, especially for Ant and Humanoid. This
7

---

Under review as a conference paper at ICLR 2020
2008
1548
1060
244
254
1008
Ttaraaiion
Itaratioo
(a) Hopper
(b) Reacher
(c) HalfCheetah
4000
red
2003
1008
L008
8008
-008
2004
thecation
Deeratioon
Ieecatioon
(d) Walker2d
(e) Ant
(f) Humaaooi
Figure 4: Training progress for RED, GAIL, GAIL-b, SAIL, and SAIL-b. Consistent with our thcoretical analysis, SAIL-b (blue) is more stable and sample efficient in Reacher, Ant and Humanoid, and comparable to other algorithms for the remaining tasks.
is likely due to the fact that SAIL-b receives equal contribution from both support guidance and adversarial learning, as Tred and Tgoit have the same range in this formulation. In addition, we note that GAIL fails to imitate the expert in Ant, while GAIL-b performs significantly better. The results suggest that restricting the range the adversarial reward could improve performance.
4.3 TRAINING STABILITY AND SAMPLE EFFICIENCY
To assess the sensitivity with respect random seeds, we plot the training progress against number of iterations for the evaluated algorithms in Fig. 4) Each iteration consists of 1000 environment steps. The figure reports mean and standard deviation of each algorithm, across the random seeds.
Fig. 4 shows that SAIL-b bi is more sample efficient and stable in Reacher, Ant and Humanoid tasks; and is comparable the other algorithms the remaining tasks. Consistent with our analysis in Section 3.3, SA appears at least efficient GAIL. even when the support guidance (i.e., the performance of RED) suffers from insufficient expert data in Hopper, HalfCheetah and Walker2d. In Reacher, Ant and Humanoid, SAIL-b benefits from the support guidance and achieves better performance and training stability. In particular, we note that without support guidance, GAIL fails to imitate the expert in Ant (Fig. 4e). Similar failures
also observed in Kostrikov al. (2019). GAIL is also more sensitive to initial conditions: in Humanoid, GAIL converged to sub-optimal policies in 2 out 5 seeds. Lastly, while RED improves noticeably faster during early training in Humanoid, it converged to a sub-optimal policy eventually.
5 CONCLUSION
In this paper, we propose Support-guided Adversarial Imitation Learning by combining support guidance with adversarial imitation learning. Our approach is complementary to existing adversarial imitation learning algorithms, and addresses several challenges associated with them. More broadly, our results show that expert demonstrations contain rich sources information for imitation learning. Effectively combining different sources of reinforcement learning signals from the expert demon- strations produces more efficient and stable algorithms by constraining the policy search space; and appears be a promising direction for future research.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-first international conference on Machine learning, pp. 1. ACM, 2004.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. stat, 1050:17, 2017.
Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-end differentiable adversarial imitation learning. In International Conference on Machine Learning, pp. 390-399, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1809: 11096, 2018.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606vv/606..15555 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894, 2018.
Ernesto De Vito, Lorenzo Rosasco, and Alessandro Toigo. A universally consistent spectral estimator for the support of distribution. Appl Comput Harmonic Anal, 37:185-217 2014.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between generative adversarial networks, inverse reinforcement learning, and energy baasyd models. arXiv preprint arXiv:1611.03852 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv: 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural informa- tion processing systems. pp. 2672-2682 2014.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances Neural Information Processing Systems, pp. 4565 4565-4573, 2016.
Kee-Eung Kim and Hyun Soo Park. Imitation learning via kernel mean embedding. AAAI, 2018.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-acton arttierrrii:t.tt.iiiiii Addressing sample inefficiency and reward bias in adversarial imitation learning. International Conference on Learning Representation. 2019.
Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 663-670. Morgan Kaufmann Publishers Inc., 2000.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourleenth international conference on artificial intelligence and statistics, pp. 627-635, 2011.
Alessandro Rudi, Ernesto De Vito, Alessandro Verri, and Francesca Odone. Regularized kernel algorithms for support estimation. Frontiers in Applied Mathematics and Statistics, 3:23, 2017.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234-2242. 2016.
Fumihiro Sasaki. Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. International Conference Learning Representation, 2019.

---

Under review as a conference paper at ICLR 2020
Yannick Schroecker, Mel Vecerik, and Jonathan Scholz. Generative predecessor models for sample- efficient imitation learning. International Conference On Learning Representations, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
Shashank Singh, Ananya Uppal, Boyue Li, Chun-Liang Li, Manzil Zaheer, and Barnabas Poczos. Nonparametric density estimation under adversarial losses. In Advances in Neural Information Processing Systems, pp. 10225-10236, 2018.
Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggre- vated: Differentiable imitation learning for sequential prediction. arXiv preprint arXiv:1703.01030. 2017.
Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, and Yiannis Demiris. Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation. arXiv e-prints, pp. 1rXiv:190.006750, 2019.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. AAAI, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
A PROOF FOR PROPOSITION AND 2
Observe that for any (s, E
ssait (s, a) (s, a) Igaat (s, a) min(rrea(s,, a) Rgait a) Rred).
(10)
of the two following events holds with probability at least 1 - ö, for any (s, a) & supp g and (0,1]
By the assumption the learning rate in Eq. (6)
Gred log 1
(s a) Cgait log !
Trea (8, a)
(11)
Plugging the above upper bounds into Eq.
the desired result in Eq. (7)
By assumption in Eq. (8) the following event holds with probability at least - for (s, a) e
supp(Fr).
Gred log
(rea(s,, (s, a) 11 S
(12)
12
Plugging this inequality in the definition of Tsaal, we obtain
(s (s, a) Tyaal (s, a)l =
Irgai((s, () a) 1)]
(13)
Irgais(s, (s, (rllrred(ss
(14)
Cred Cred log
(15)
n
B EXPERIMENT DETAILS
The experiments are based on OpenAl's baselines and the original implementation of RED We adapted the code from RED for our experiments, and used the accompanying dataset of expert trajectories. 4 Nvidia GTX1070 GPUs were used in the experiments.
Table shows the environment information, number of environment steps and number of expert trajectories used for each task. Each full trajectory consists of 1000 (s, a) pairs. They are sub-sampled during the experiments.
https://github.rr httra:///gfthub..cc////oolaaal/ ines
10

---

Under review as a conference paper at ICLR 2020
Task
State Space
Action Space
Trajectories
Env Steps
Exp Performance
Hopper-v2
11
3
X 106
3777.8 + 3.8
Reacher-v2
11
2
4
106
-3.7 - 1.4
HalfCheetah-v2
17
6
4
3 x 106
4159.8 + 93.1
Walker2d-v2
17
6
x 106
5505.8 + 81,4
Ant-v2
111
8
5 10t
4821.0 + 107.4
Humanoid-v2
376
17
80
x 10?
10413.1 = 47.0
Table 3: Environment information, number of expert trajectories and environment steps used for cach task
B. NETWORK ARCHITECTURE
The default policy network from OpenAl's baselines are used for all tasks: two fully- fully-connected layers of 100 units each, with tanh nonlinearities. The discriminator networks and the value function networks use the same architecture.
RED and SAIL RND Burda et al. (2018) for support estimation. We use the default networks from RED We set a following the heuristic in Wang et al. (2019) that (s, a) from the expert trajectories mostly have reward close to
B.2 HYPERPARAMETERS
For fair comparisons, all algorithms shared hyperparameters for each task. We present them in the table below, including discriminator learning rate !p. discount factor number of policy steps per iteration ng. and whether the policy has fixed variance. All other hyperparameters are their default values from OpenAl's baselines.
Task Name
7
Ip
ng
Fixed Variance
Hopper
0.99
0.0003
3
False
Reacher
0.99
0.0003
3
False
HalfCheetah
0.99
0.0003
False
Walker2d
0.99
0.0003
3
False
Ant
0.99
0.0001
3
False
Humanoid
0.99
0.0001
5
False
Table 4: Hyperparameters used for each tasks
C ADDITIONAL RESULTS ON LUNAR LANDER
In the default environment, Lunar Lander contains several terminal states, including crashing, flying of view, and landing the goal. In the no-terminal environment, all terminal states are disabled, such that the agent must solely rely on the expert demonstrations for training signals.
To compare our method with the technique of introducing virtual absorbing state (AS) (Kostrikoy et al. 2019), we also construct goal-terminal environment where the only terminal state is suc- cesstul landing at the goal, because the AS technique cannot directly applied in the no-terminal environment. We present the results in Appendix
The results suggest that AS overall improves both the mean performance and standard deviations for both GAIL and SAIL. Specifically, the technique is able mitigates the survival bias in GAIL signif- icantly. However, SAIL still compares favorably to the technique in the goal-terminal environment. Further, since AS and support guidance is not mutually exclusive, also combine them and report the performances. The results suggest that support guidance compatible with AS, and achieves overall the best performance with low standard deviations.
11

---

Under review as a conference paper at ICLR 2020
Default
Goal-terminal
No-terminal
GAIL
258.30 = 28.98
-7.16 + 31.64
-69.33 79.76
GAIL-b
250.53 + 67.07
4.16 + 107.37
169.73 + 80.84
SAIL
257.02 + 20.66
261.07 + 35.66
237.96 + 49.70
SAIL-b
262.97 = 18.11
252.07 67.22
256.83 = 20.99
GAIL+ + AS
271.46 11.90
110.22 119.25
GAIL-b AS
269.97 = 16.48
186.02 + 98.27
SAIL AS
274.89 = 12.82
254.58 25.40
SAIL-b + AS
270.33 = 15.86
258.30 + 20.75
Table Average environment reward standard deviation on Lunar Lander, evaluated over runs default, goal-terminal and no-terminal environment.
The results also suggest that both AS and support guidance partially mitigate the reward bias, but don't fully solve it. We will further explore this issue in future work.
12


Reviews:
[{"review_id": "rJxtJOspFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an imitation learning algorithm that combines support estimation with adversarial training. The key idea is simple: multiply the reward from Random Expert Distillation (RED) with the reward from Generative Adversarial Imitation Learning (GAIL). The new reward combines the best of both methods. Like the GAIL reward, the new reward encourages exploration and can be estimated from a small number of demonstrations. Like the RED reward, the new reward avoids survival bias and is more stable than the adversarial reward.\n\nI have a concern regarding the Lunar Lander experiment. Were the demonstrations generated in the modified environment? If they were generated in the original environment (with early termination), this may have unintentionally created a state distribution mismatch between the demonstration environment and training environment that unfairly hurts the GAIL baseline's performance. If the demonstrations were instead generated in the modified environment (without early termination) where the agent is actually trained, the demonstrations would contain many self-loop transitions at the goal state, and GAIL would likely not exhibit survival bias.\n\nI am also a bit concerned about the MuJoCo results. The stochasticity of the demonstrations and the evaluation trajectories may have a significant effect on the standard deviation of rewards. Was a stochastic policy or a deterministic policy used to generate the demonstrations? Were the evaluation trajectories generated by rolling out the stochastic imitation policy, or by rolling out a deterministic version of the imitation agent? Also, could the authors provide the mean and standard deviation of rewards in the demonstrations in Tables 1-2 and Figure 4? It would be nice to establish a rough upper bound on the performance of the imitation methods.\n\nUpdate: \nAfter reading the author response, I have increased my score from 3 to 6.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1ex1UxPKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of the paper: \nThe paper proposes an IL method named support-guided adversarial IL (SAIL), which is based on generative adversarial IL (GAIL) (Ho and Ermon, 2016) and random expert distillation (RED) (Wang et al., 2019). The key idea of SAIL is to construct a reward function by multiplying reward functions learned by GAIL and RED. This multiplication yields two benefits; 1) it handles the issue of biased reward in GAIL, since state-action pairs outside the expert’s support are assigned low reward values. 2) SAIL’s reward is more reliable than RED’s reward for state-action pairs inside the expert’s support. The authors show that SAIL is at least as fast as than GAIL in terms of the sample complexity. Experiments on continuous control benchmarks show that SAIL is overall more stable than GAIL. \n\n**Rating: \nThe paper proposes a simple but effective combination of existing methods. The proposed method is well motivated and performs well on benchmarks. Still, the paper has some issues regarding justification, clarity, and evaluation, which should be addressed (see below). I vote for weak acceptance. \n\n**Major comments/questions: \n- No guarantee of the optimality of the learned policy.\nCan it be guaranteed that SAIL learns the expert policy? (assuming the expert policy is realizable). Propositions 1 and 2 show the convergence of the support estimation, but these results are not related to the optimality of a policy learned with the reward function. This is an important point for justifying SAIL, since SAIL does not perform distribution matching to learn the expert policy, and it also does not perform IRL to learn the reward function. Therefore, SAIL lacks the optimality guarantee from both distribution matching and IRL perspectives. Please address and clarify this point. \n\n- Clarity in the theoretical analysis.\nIn the theoretical analysis, the paper assumes a rate of GAIL for support estimation. This is quite confusing, since GAIL performs distribution matching and does not estimate the support. Also, given that r_gail = -log D(s,a), the reward’s upper-bound (R_gail) is infinity and the bound in Eq. (9) is not informative. \n\n- The reward r_red is constant at the optimal.\nEq. (2) and Eq. (3) imply that, for state-action pairs from the expert’s state-action distribution, r_red is constant at the optimal. Specifically, the optimal solution of Eq. (2) is \\hat{\\theta} = \\theta, which yields to a constant value of r_red(s,a) in Eq. (3). In this scenario, SAIL is equivalent to GAIL for the expert state-action distribution. This means that Eq. (2) should not be optimized until optimal, and some early stopping criteria are required. Does this scenario (constant value of r_red) occur in the experiments?\n\n- IRL baseline methods.\nThe paper should compare SAIL to methods which aim to handle the bias in reward function, e.g., DAC (Kostrikov et al. 2019). While DAC requires the time limit, this time limit is known in the benchmark tasks. Also, IRL methods such as AIRL (Fu et al., 2018) should be compared, since IRL methods are better than GAIL at handling bias in reward function (Kostrikov et al. 2019). \n\n**Minor comments/questions: \n- Typos: \"offline RL algorithms\" should be \"off-policy RL algorithms\". Line 5 of Algorithm 1 should perform gradient ascent instead of gradient descent. An expectation over state-action distribution of expert is missing from Eq. (2). \n\n- What are the bold numbers in table 1 and 2 indicating? Why does the Hopper task have two bold numbers, but the other tasks have only one?\n\n--After author response--\nI have read the author response and other reviews. I thank the authors for including additional experiments. However, the authors' arguments regarding optimality do not fully address my comments (see below). I will keep the vote of weak acceptance.  \n\nThe authors argue that \"In this asymptotic case, SAIL is equivalent to performing distribution matching via GAIL with the additional *constraint* that candidate distributions need to have the same support of the expert distribution\". However, the support of the expert distribution may coincide with the entire state-action space, which makes the additional constraint uninformative in the asymptotic case. Specifically, the expert distribution coincides with the state-action space when the expert policy has an infinite support (e.g., the expert policy is Gaussian). Assuming the asymptotic case, the support estimation in RED will give an indicator function over an entire state-action space, and the support constraint in SAIL is always satisfied. In other words, SAIL is exactly equivalent to GAIL in this case. For these reasons, the authors' arguments regarding optimality do not fully address my comments. I think additional assumptions are required, e.g., the expert policy needs to have a finite support or be deterministic.  \n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlhjasptB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach for improving adversarial imitation learning, by combining it with support-estimation-based imitation learning. In particular, the paper explores a combination of GAIL (Ho and Ermon, 2016) and RED (Wang et. al., 2019), where the reward for the policy-gradient is a product of the rewards obtained from them separately. The motivation is that, while AIL methods are sample-efficient (in terms of expert data) and implicitly promote useful exploration, they could be unreliable outside the support of the expert policy. Therefore, augmenting them by constraining the imitator to the support of the expert policy (with a method such as RED) could result in an overall better imitation learning algorithm. \n\nWhile the motivation and intuition are clear to me, I have reservations about the claims made in the abstract and the experimental sections:\n\n1.\tSAIL is an effective method for solving the reward bias in AIL.\nThe reward in SAIL is “always” non-negative (product of 2 non-negative terms), making the method a very ad-hoc way of getting around the reward bias problem, especially when compared to other methods such as those which estimate the value function of the absorbing state (Kostrikov et. al. 2019). Consider a simple chain MDP with 3 states A, B and a terminal state T. The actions are left/right from each state. Let the expert trajectory be A->B->T. Also, for SAIL, consider perfect support estimation with an optimal RED-network. When at B, the agent can terminate with a right-action and collect some reward. But taking left and collecting 0 reward (due to perfect support estimation) makes it land in A, from where it can now achieve a positive reward for the A->B transition, and repeat the process. Hence, one could always create MDPs where the Q value of B->A is higher than B->T.\n\n\tThe Lunar-Lander environment (with certain parameters) in Section 4.1 appears to present a scenario where SAIL get arounds the reward bias, but this doesn’t remove my doubts over the generalization of this approach. Also, in Table 1, why does GAIL not hover above the landing spot even in the default case? If the reward bias is strong there, with sufficient exploration, the agent should converge to the same policy as in the modified case.\n\nFigure 3 is concerning for the same reason as above. It shows the immediate reward at the goal state, and points that SAIL has large reward for no-op action. The issue is that RL optimizes for actions that have the maximum Q value, not the action with the maximum immediate reward.\n\n\n2.\tI would recommend that the authors refer to the original GAIL algorithm as “GAIL” in the experiments section, and their practical stabilization trick as “GAIL-bounded” (or something to that effect). Referring to original algorithm as GAIL-log, and the modification as GAIL could be misleading to readers. \n\n\n3.\tThe authors claim that SAIL has better training stability, leading to more robust policies. If this is due to the algorithmic contribution of combining AIL and Support-Estimation-IL, then GAIL-log and SAIL-log in Table 2. should show this in the standard deviation numbers. This doesn’t appear to be the case. Also, Figure 4 (Half-Cheetah) has unusually large variance for SAIL-log.\n\n\n4.\tFigure 4 and Table 2 numbers are very different. Take Humanoid for instance. From Figure 4, it seems that SAIL is way better than GAIL. But if you look at Table 2, they both achieve mean-score in excess of 10k. What’s the difference between Table 2. and final performance in Figure 4?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a method for adversarial imitation learning that combines two previous approaches - GAIL and RED - by simply multiplying their reward functions. The claim is that this adaptation allows for better learning - both handling reward bias and improving training stability. 

The reviewers were divided in their assessment of the paper, criticizing the empirical results and the claims made by the authors. In particular, the primary claims of handling reward bias and reducing variance seem to be not well justified, including results which show that training stability only substantially improves when SAIL-b, which uses reward clipping, is used. 

Although the paper is promising, the recommendation is for a reject at this time. The authors are encouraged to clarify their claims and supporting experiments and to validate their method on more challenging domains.
Decision: Reject