What Matters for Adversarial Imitation Learning?
Authors: Manu Orsini
Anton Raichuk
Leonard Hussenot
Damien Vincent
Robert Dadashi
Sertan Girgin
Matthieu Geist
Olivier Bachem
Olivier Pietquin
Marcin Andrychowicz
Venue: NeurIPS 2021 Poster (2021)
Keywords: imitation learning
GAIL
continuous control

Abstract: Adversarial imitation learning has become a popular framework for imitation in continuous control. Over the years, several variations of its components were proposed to enhance the performance of the learned policies as well as the sample complexity of the algorithm. In practice, these choices are rarely tested all together in rigorous empirical studies.
It is therefore difficult to discuss and understand what choices, among the high-level algorithmic options as well as  low-level implementation details, matter. To tackle this issue, we implement more than 50 of these choices in a generic adversarial imitation learning framework
and investigate their impacts in a large-scale study (>500k trained agents) with both synthetic and human-generated demonstrations. We analyze the key results and highlight the most surprising findings.

What Matters for Adversarial Imitation Learning?
Manu Orsini; Anton Raichuk; Léonard Hussenot"; Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, Marcin Andrychowicz-
Google Research, Brain Team
Abstract
Adversarial imitation learning has become a popular framework for imitation continuous control. Over the years, several variations of its components were proposed to enhance the performance of the learned policies as well as the sample complexity of the algorithm. In practice, these choices rarely tested all together in rigorous empirical studies. what choices, among the high-level algorithmic options well low-level im- plementation details, matter. To tackle this issue, we implement more than 50 of these choices in generic adversarial imitation learning framework and investigate their impacts in large-scale study (>500k trained agents) with both synthetic and human- comnan-generater demonstrations. We analyze the key results and highlight the most surprising findings.
therefore difficult to discuss and understand
Introduction
Reinforcement Learning (RL) has shown its ability perform complex tasks in contexts where clear
reward functions can be set-up (e.g. +1 for winning a chess game) [15] 37 40 43 but for many
real-world applications, designing a correct reward function is either tedious or impossible (20). while
demonstrating a correct behavior is often easy and cheap. Therefore, imitation learning (IL, [4] 7
might the key to unlock the resolution of more complex tasks, such autonomous driving, for which reward functions are much harder to design.
The simplest approach to IL is Behavioral Cloning (BC, (2) which uses supervised learning predict the expert's is action for any given state. However, BC is often unreliable as prediction errors compound in the course of an episode. Adversarial Imitation Learning (AIL, (14) aims to remedy this using inspiration from Generative Adversarial Networks (GANs, 99) and Inverse RL 3 S 6: the policy is trained generate trajectories that are indistinguishable from the expert's ones.
GANs, this
is formalized as a two-player game where a discriminator is co-trained to distinguish between the policy and expert trajectories (or states). See App. [or a brief introduction AlL.
myriad of improvements over the original AIL algorithm were proposed over the years 17
31
41.44 44 46). from changing the discriminator's loss function 117 to switching from on-policy to off-policy agents (31]. However, their relative performance is rarely studied in a controlled setting, and never these changes have never been compared simultaneously. The performance of these high-level choices may also depend on low-level implementation details which might be silenced in the original publications [19 29 36 42 well as the hyperparameters (HPs) used. Thus, assessing whether the proposed changes are the reason for the presented improvements becomes extremely
41
"Equal contribution.
Univ. de Lille, CNRS, Inria Scool, UMR 9189 CRISIAL. "Corresoondig author. E-mail: marcinatgoogle.com com.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

difficult. This lack of proper comparisons slows down the overall research in imitation learning and the industrial applicability of these methods.
We investigate such high- and low-level choices in depth and study their impact on the algorithm performance. Hence, as key contributions, we (1) implement highly-configurable generic AIL algorithm, with various axes of variation (>50 HPs), including 4 different RL algorithms and 7 regularization schemes for the discriminator, (2) conduct a large-scale (>500k trained agents) experimental study on 10 continuous-control tasks and (3) analyze the experimental results to provide practical insights and recommendations for designing novel and using existing AlL algorithms. We release this generic AlL agent, implemented in JAX (25) as part of the Acme
[49]
framework: https:///githbb www//deepmnnd////eee/e//eeoe acme http://bbbbaaat/aaaaaaaaeeeeeeee haallo///sstrrr//c////////////// agaata//aa/aaaiii
Most surprising finding #1: regularizers. While many of our findings confirm common practices in AIL research, some of them are surprising or even contradict prior work. In particular, we find that standard regularizers from Supervised Learning - dropout [10] and weight decay (T] often perform similarly to the regularizers designed specifically for adversarial learning like gradient penalty (18). Moreover, for easier environments (which were often the only ones used in prior work), we find that it is possible to achieve excellent results without using any explicit discriminator regularization.
Most surprising finding #2: human demonstrations. Not only does the performance of AIL heavily depend on whether the demonstrations were collected from human operator generated by an RI. algorithm, but the relative performance of algorithmic choices also depends on the demonstra- tion source. Our results suggest that artificial demonstrations that the very common practice of evaluating II. algorithms only with synthetic demonstrations may lead to algorithms which perform poorly in the more realistic scenarios with human demonstrations.
not good proxy for human data and
2 Experimental design
Environments. We focus on continuous-control tasks as robotics appears as one of the main potential applications of IL and a vast majority of the IL literature thus focuses on it. In particular, we run experiments with five widely used environments from Open Gym [13]: HalfCheetah-v2, Hopper-ve Walker2d-v2, Ant-v2, and Humanoid-vz v2 and three manipulation environments from Adroit 21]: ven-v0, door-v0, and hammer The Adroit tasks consist in aligning a pen with a target orientation, opening a door and hammering nail with 5-figerred hand.
Demonstrations. For the Gym tasks, we generate demonstrations with SAC [28 agent trained on the environment reward. For the Adroit environments, we use the "expert" and "human" datasets from D4RI. [45], which are, respectively, generated by an RI. agent and collected from a human operator. As far as we know, our work the first to solve these tasks with human datasets in the imitation setup (most of the prior work concentrated on Offline RL). For all environments, we use 11 demonstration trajectories. Following prior work [14] 31 46). we subsample expert demonstrations by only using every 20th state-action pair to make the tasks harder.
Adversarial Imitation Learning algorithms. We researched prior work on AlL. algorithms and made a list of commonly used design decisions like policy objectives or discriminator regularization techniques. We also included a number of natural options which we have not encountered in literature (e.g. dropout [10] the discriminator or clipping rewards bigger than threshold). All choices are listed and explained in App. D Then, we implemented single highly-configurable AIL agent which exposes all these choices as configuration options in the Acme framework (49) using JAX (25) for automatic differentiation and Flax [47) for neural networks computation. The configuration space is so wide that it covers the whole family of AIL algorithms, in particular, it mostly covers the setups from AIRL 17) and DAC 31] We plan to open source the agent implementation.
Experimental design. We created a large HP sweep (57 HPss swept, >1 >120k agents trained) in which each HP is sampled uniformly at random from a discrete set and independently from the other HPs. We manually ensured that the sampling ranges of all HPs are appropriate and cover the optimal values. Then, we analyzed the results of this initial experiment (called wide, detailed description and results in App. G). removed clearly suboptimal options and ran another experiment with the pruned sampling

---

ranges (called main, 43 HPs swept, >250k agents trained, detailed description and results in App. The latter experiment serves as the basis for most of the conclusions drawn in this paper but we also run a few additional experiments to investigate some additional questions (App. aand App. D
(App.Dand
This pruning of the HP space guarantees that we draw conclusions based training configurations which are highly competitive (training curves can be found in Fig. 24) while using a large HP sweep (including, for example, multiple different RL algorithms) ensures that our conclusions are robust and valid not only for single RL algorithm and specific values of HPs, but are more generally applicable. Moreover, many choices may have strong interactions with other related choices, for example we find surprisingly strong interaction between the discriminator regularization scheme and the discriminator learning rate (Sec. 4). This means that such choices need to be tuned together (as it is the case in our study) and experiments where only a single choice is varied but the interacting choices are kept fixed may lead to misleading conclusions.
Performance measure. For each HP configuration and each of the 10 environment-dataset pairs we train a policy and evaluate it 10 times through the training by running it for 50 episodes and computing the average undiscounted return using the environment reward. We then average these scores to obtain single performance score which approximates the area under the learning curve. This ensures we assign higher scores to HP configurations that learn quickly.
Analysis. We consider two different analyses for each choice
Conditional 95th percentile: For each potential value of that choice (c.g., RL Algorithm = PPO), we look at the performance distribution of sampled configurations with that value. We report the 95th percentile of the performance as well as error bars based on bootstrapping This corresponds to an estimate of the performance one expect if all other choices were tuned with random search and limited budget of roughly 13 HP configuration T0 All scores are normalized so that O corresponds to random policy and to the expert performance (expert scores can be found in App.
Distribution of choice within top 5% configurations. We further consider for each choice the distribution of values among the top 5% HP configurations. In particular, we measure the ratio of the frequency of the given value in the top 5% of HP configurations with the best performance to the frequency of this value among all HP configurations. If certain values over-represented in the top models (ratio higher than I), this indicates that the specific choice is important for good performance.
We release the raw results of our experiments" along with a Notebook allowing to load and study i
3 What matters for the agent training?
Summary of key findings. The AIRL reward function perform best for synthetic demonstrations while - Imn(1 - D) is better for human demonstrations. Using explicit absorbing state is crucial in environments with variable length episodes. Observation normalization strongly affects the performance. Using an off-policy RL algorithm is necessary for good sample complexity while replaying expert data and pretraining with BC improves the performance only slightly.
Implicit reward function. In this section, we investigate choices related to agent training with AIL, the most salient of which is probably the choice of the implicit reward function. Let D(s, a) be the probability of classifying the given state-action pair expert by the discriminator. In particular, we run experiments with the following reward functions: r(s, a) = log(1 D(s, a)) (used in the original GAIL paper [14) ),)), r(s, a) log D(s, a) log(1 D(s, a)) (called the AIRL reward
17 ). r(s, a) = log (ss, a) (a natural choice we have not encountered in literature), and the FAIRL 46
reward function r(s, a) = -hss a) ch(,.a), where h(s, a) is the discriminator logit. It can be
"This analysis is based on a similar type of study focused on on-policy RL algorithms (42)
'We compute each metric 20 times based randomly selected half of all training runs, and then report the mean of these 20 measurements while the error bars show mean-si std and mean+std.
'The probability that all 13 configurations score worse than the 95th percentile is equal 0.95 ~ 50%. wwt.///pppp.......tttttt..aa
www./o///ttte.e matteesttttttestse imi tation www.ring/aaaaaaaaa.a/eeaea htg.://see...a.atoooooooooeaeaaa
ison
com/ what matters in-imi inc--mmitit tation-
hagp://attttttt....ooaaaaaaaaaaa Learning/analys....
colab.ipynb
3

---

shown that, under the assumption that all episodes have the same length, maximizing these reward functions corresponds the minimization of different divergences between the marginal state-action distribution of the expert and the policy. See (46) for in-depth discussion on this topic. We also consider clipping the rewards with absolute values bigger than a threshold which is a HP.
all
variable episode length
variable episode length
0.5
0.9
0.8
0.4
0.4
0.2
0.7
0.1
0.2
0.0
-In(1-D) FAIRL In(D)
-In(i-D)
AIRL
In(D)
-In(1-D)
In(D)
(a) wide HP search
(b) no absorbing state
(c) absorbing state
Figure 1: Comparison of different reward functions. The bars show the 95th percentile across HPs sampling of the average policy performance during training. Plot (a) shows the results averaged across all 10 tasks. Plots (b) and (c) show the performance on the subset of environments with variable length episodes when the absorbing state is disabled (b) or enabled (c). See Fig. and Fig. 72 for the individual results in all environments.
The FAIRL reward performed much worse than all others in the initial wide experiment (Fig. aa) and therefore was not included in our main experiment. This is mostly caused by its inferior performance with off-policy RL algorithms (Fig. 22). Moreover, reward clipping significantly helps the FAIRL reward (Fig.
while it does not help the other reward functions apart from some small gains D) (Fig. 82 Therefore, we suspect that the poor performance of the FAIRL reward function may be caused by its exponential term which may have very high magniudes. Moreover, the FAIRL paper (46) mentions that the FAIRL reward is more sensitive to HPs than other reward functions which could also explain poor performance in our experiments.
for In 1m
Fig. 25 shows that the In(D) reward functions performs a bit worse than the other two reward functions in the main experiment. Five out of the ten tasks used in our experiments have variable length episodes with longer episodes correlated with better behaviour (Hopper. Walker2d, Ant, Humanoid, pen) on these tasks we can notice that r(s, a)
In(1 D(s, a)) often performs best and r(s, a) = In D(s, a) worst. This can be explained by the fact that - In(1 D(s, a)) 0 which means that the former reward encourages longer episodes and the latter one shorter ones (3T). Absorbing state (described in App. D.2) is a technique introduced in the DAC paper (3T) to mitigate the mentioned bias and encourage the policy to generate episodes of similar length to demonstrations. In Thee we show how the performance of different reward functions compares in the environments with variable length episodes depending on whether the absorbing state is used. We can notice that without the absorbing state r(s, a) - In(1 D(s, a)) > 0 performs much better in the environments with variable episode length which suggests that the learning is driven large extent by the reward bias and not actual imitation of the expert behaviour 3TJ. This effect disappears when the absorbing state enabled (Fig. [T)..
and In (s,, a)
Fig. 72 shows the performance of different reward functions in all environments conditioned on whether the absorbing state used. If the absorbing state used, the AIRL reward function performs best in all the environments with RL- RL-generated demonstrations, and In performs only marginally worse. The (1 D) reward function underperforms on the Humanoid and pen tasks while performing best with human datasets. We provide some hypothesis for this behaviour in Sec. where we discuss human demonstrations in more details.
Observation normalization. We consider observation normalization which is applied to the inputs of neural networks involved AIL (policy, critic and discriminator). The normalization aims to transform the observations so that that each observation coordinate has mean 0 and standard deviation In particular, we consider computing the normalization statisties either using only the expert demonstrations so that the normalization fixed throughout the training, using data from the policy

---

being trained (called online). See App. D.6 for more details. Fig. 26 shows that input normalization significantly influences the performance with the effects on performance being often much larger than those of algorithmic choices like the reward function or RI. algorithm used. Surprisingly, normalizing observations can either significantly improve or diminish performance and whether the fixed or online normalization performs better is also environment dependent.
Replaying expert data. When demonstrations as well as external rewards are available, it is common for RL algorithms to sample batches for off-policy updates from the demonstrations in addition to the replay buffer (30] 39 We varied the ratio of the policy to expert data being replayed some benefits, it is usually best to replay 16-64 times more policy than expert data. On some tasks (Humanoid) replaying even a single expert transitions every 256 agent ones significantly hurts performance. We suspect that, in contrast to RL. with demonstrations, we see little benefit from replaying expert data in the setup with learned rewards because (1) replaying expert data mostly helps when the reward signal is sparse (not the case for discriminator based rewards), and (2) discriminator may overfit to the expert demonstrations which could result in incorrectly high rewards being assigned to expert transitions.
but found only very minor gains (Fig. 83 Moreover, in the cases when
Pretraining with BC. We also experiment with pretraining a policy with Behavioral Cloning (BC, (2) at the beginning of training. Despite starting from much better policy than a random one, usually observe that the policy quality deteriorates quickly at the beginning of training (see the pen task in Fig. 3 due being updated using randomly initialized critic and discriminator networks, and the overall gain from pretraining is very small in most environments (Fig. 27
RL algorithms. We run experiments with four different RL algorithms. three of which are off-policy algorithms (SAC (28). TD3 (26) and D4PG (24)), as well as PPO (22) which is nearly on-policy. Fig. 7 shows that the sample complexity of PPO significantly worse than that of the off-policy algorithms while all off-policy algorithms perform overall similarly.
RL algorithms HPs. Fig. 8 shows that the discount factor is one of the most important HPs with the values of 0.97 - 0.99 pertorming well on all tasks. Fig. 29 shows that in most environments it better not to erase any data from the RL replay buffer and always sample from all the experience encountered so far. It is common in RL to use noise-free version of the policy during evaluation and we observe that it indeed improves the performance (Fig. 30)). The policy MLP size does not matter much (Figs. 31 32) activation functions, relu performs on par or better than tanh in all environments apart from door in which tanh is significantly better (Fig. 33). Our implementation of TD3 optionally applies gradient clipping but it does not affect the performance much (Fig. 34). D4PG can use n-step returns, this improves the performance the Adroit tasks but hurts on the Gym suite (Fig. 35)
while bigger critic networks perform significantly better (Figs. 9
TO). Regarding
4 What matters for the discriminator training?
Summary of key findings. MLP discriminators perform on par or better than AlL-specific archi- tectures. Explicit discriminator regularization is only important in more complicated environments (Humanoid d and harder ones). Spectral norm is overall the best regularizer but standard regularizers from supervised learning often perform on par. Optimal learning rate for the discriminator may be 2-2.5 orders of magnitude lower than the one for the RL agent.
Discriminator input. In this section we look at the choices related the discriminator training. Fig. [4]]999/ssw how the performance depends on the discriminator input. We can observe that while it is beneficial to feed actions as well as states to the discriminator, the state-only demonstrations perform almost as well. Interestingly, on the door task with human data, it is better to ignore the expert actions. We explore the results with human demonstrations in more depth in Sec. 5
Discriminator architecture. Regarding the discriminator network, our basic architecture is an MLP but we also consider two modifications introduced AIRL [17]: a reward shaping term and a log rTlls) logit shift which introduces dependence on the current policy (only applicable to RL algorithms with stochastic policies, which in our case are PPO and SAC). See App. [D.3 for a detailed description of these techniques. Fig. B shows that the logit shift significantly hurts the performance.
5

---

This is mainly due to the fact that it does not work well with SAC which is off-policy (Fig. 21) Fig. shows that the shaping term does not affect the performance much. While the modifications from AIRL does not improve the sample complexity in our experiments, it is worth mentioning that they were introduced for another purpose, namely the recovery of transferable reward functions.
Regarding the size of the discriminator MLP(s), the best results on all tasks are obtained with single hidden layer (Fig. (l). while the size of the hidden layer is of secondary importance (if it is not very small) with the exception of the tasks with human data where fewer hidden units perform significantly better (Fig. 52). All tested discriminator activation functions perform overall similarly while sigmoid performs best with human demonstrations (Fig. 53
Discriminator training. Fig. 54 shows that it is best to use as large as possible replay buffers for sampling negative examples (i.e. agent transitions). Prior work has claimed the initialization of the last policy layer can significantly influence the performance in RL [42], thus we tried initializing the last discriminator layer with smaller weights but it does not make much difference (Fig 55).
Discriminator regularization. An overfit or too accurate discriminator can make agent's 's training challenging, and therefore it is common to use additional regularization techniques when training the AIL discriminator (or GANs in general). We run experiments with number of regularizers commonly
used with AIL, namely Gradient Penalty (18) (GP, used e.g. in (31), D. spectral norm (35 (e.g. in [44) ).
Mixup 23 (e.g. in (52). D. well as using the PUGAIL loss [41] instead of the standard cross entropy loss to train the discriminator. Apart from the above regularizers, we also run experiments with reg- ularizers commonly used in Supervised Learning, namely dropout (10). the weight decay (T] variant from Adam W (33 as well as the entropy bonus of the discriminator output treated as a Bernoulli distribution. The detailed description of all these regularization techniques can be found in App. D.5
gym
adroit
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
GP
Mixup
No regularizer
PUGAIL
dropout
entropy
spectral norm weight decay-
Mixup
No regulanzer
PUGAIL
dropout
entropy
spectral norm
weight decay-
Mixup
No regularizer
PUGAIL
dropout
entropy
spectral norm weight decay wese
Figure 2: The 95th percentile of performance for different discriminator regularizers. The central plot shows the average performance across 5 tasks from OpenAl Gym and the right one the average performance across 5 tasks from the Adroit suite. See Fig, for the plots for individual environments.
Fig. Sisowss how the performance depends the regularizer. Spectral normalization performs overall best, while GP, dropout and weight decay all perform on par with each other and only a bit worse than spectral normalization. We find this conclusion to be quite surprising given that have not seen dropout or weight decay being used with AlL. in literature. We also notice that the regularization generally more important on harder tasks like Humanoid or the tasks in the Adroit suite (Fig. 566).
Most of the regularizers investigated in this section have their own HPs and therefore the comparison of different regularizers depends on how these HPs are sampled. As we randomly sample the regularizer-speefii HPss in this analysis, our approach favours regularizers that are not too sensitive to their HPs. At the same time, there might be regularizers that sensitive their HPs but for which good settings may be easily found. Fig. 67 shows that even if we condition on choosing the optimal HPs for each regularizer, the relative ranking of regularizers does not change.
Moreover, there might be correlations between the regularizer and other HPs, therefore their relative performance may depend on the distribution of all other HPs. In fact, have found two such

---

surprising correlations. Fig. 73 shows the performance conditioned on the regularizer used as well as the discriminator learning rate. We notice that for PUGAIL, entropy and no regularization, the performance significantly increases for lower discriminator learning rates and the best performing discriminator learning rate (10-9) is in fact 2-2.5 orders of magnitude lower than the best learning rate
On the other hand, the remaining regularizers are not too sensitive to the discriminator learning rate. This means that the performance gap between PUGAIL, entropy and no regularization and the other regularizers is to some degree caused by the fact that the former ones are more sensitive to the learning rate and may be smaller than suggested by Fig. if we adjust for the appropriate choice of the discriminator learning rate. We can notice that PUGAIL and entropy are the only regularizers which only change the discriminator loss but do not affect the internals of the discriminator neural network. Given that they are the only two reg- ularizers benefiting from very low discriminator learning rate, we suspect that it means that a very low learning rate can play regularizing role in the absence of an explicit regularization inside the network.
for the RI. algorithm (0.0001-0.0005) -00000000, Figs. 14 38 39
41
47
Another surprising correlation is that in some environments, the regularizer interacts strongly with ob- servation normalization (described App. (D6) employed on discriminator inputs (see Fig. [74 for an ex- ample on Ant). These two correlations highlight the difficulty of comparing regularizers, and algorith- mic choices more broadly, as their performance significantly depends on the distribution of other HPs.
We also supplement our analysis by comparing the performance of different regularizers for the best found HPs. More precisely, we choose the best value for each HP in the main experiment (listed in App. E and run them with different regularizers. To account for the mentioned correlations with the discriminator learning rate and observation normalization, we also include these two choices in the HP sweep and choose the best performing variant (as measure by the area under the learning curve) for each regularizer and each environment.
HatCChetthh
Higgorr
Walkerdd
Humeanos
1.0
0.0
0.8
0.8
0.3
D.5
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0,0
300000
200000
0
100806 Door expert
750000
0
106n0c
700000
100036 human
703500
1C000C
20060D
Pen expert
Hammer expert
numan
1.0
1.0
2.0
0.8
10
100
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
200000
400000
400000
200000
400000
200030
4000C0
400000
Figure 3: Learning curves for different discriminator regularizers when the other HPs are set to the best performing value across all tasks. The y-axis shows the average policy return normalized so that O corresponds to a random policy and 1 to the expert. See App. for the HPs used. The plots shows the averages across 30 random seeds. Best seen in color.
While it is not guaranteed that the performance is going to be good at all because greedily choose the best performing value for each HP and there might be some unaccounted HP correlations, we find that the performance is very competitive (Fig. 33.. Notice that we use the same HPs in all environments and the performance can be probably improved by varying some HPs between the environments, or at least between the two environment suites.
We notice that on the four easiest tasks (HalfCheetah, Hopper. Walker2d, Ant), investigated discriminator regularizers provide no, or only minor performance improvements and excellent results can be achieved without them. On the tasks where regularization is beneficial, we usually see that there are multiple regularizers performing similarly well, with spectral normalization being one of the best regularizers in all tasks apart from the two tasks with human data where PUGAIL performs better.
Regularizers-specine HPs. For GP, the target gradient norm of is slightly better in most environ- ments but the value of 0 is significantly better in hammer -human (Fig. 57). while the penalty strength of 1 performs best overall (Fig. 58). For dropout, it is important to apply it not only to hidden layers
9 'The optimal learning rate for those regularizers was the smallest one included in the main experiment. We also additional sweep with smaller rates but found that even lower ones do not perform better (Fig 81).
7

---

but also to inputs (Fig. 59) and the best results are obtained for 50% input dropout and 75% hidden
activations dropout (Figs. 59 60 and 67p. For weight decay, the optimal decay coefficient in the AIL setup is much larger than the values typically used for Supervised Learning, the value À = 10 performs best in our experiments (Fig. 6T). For Mixup, a = outperforms the other values on almost all tested environments (Fig. 62). For PUGAIL, the unbounded version performs much better on the Adroit suite, while the bounded version is better on the Gym tasks (Fig. (3). and positive class prior of 1) = 0.7 performs well on most tasks (Fig. 64). For the discriminator entropy bonus, the values around 0.03 performed best overall (Fig. 65 All experiments with spectral normalization enforce the Lipschitz constant of 1 for each weight matrix.
How to train efficiently? So far we have analysed how HPs affect the sample complexity of AIL algorithms. For the analysis of the HPs which influence sample complexity well as the computational cost of running an algorithm see App. A In particular, we describe there a simple code optimization relying on processing multiple batches once which makes training 2-3x faster in wall clock time without affecting the sample complexity (Fig. 5).
5 Are synthetic demonstrations a good proxy for human data?
Summary of key findings Human demonstrations significantly differ from synthetic ones. Learn- ing from human demonstrations benefits more from discriminator regularization and may work better with different discriminator inputs and reward functions than RL-generated demonstrations.
Using a dataset of human demonstrations comes with a number of additional challenges. Compared to synthetic demonstrations, the human policy can be multi-modal in that for a given state different decisions might be chosen. A typical example occurs when the human demonstrator remains idle for some time (for example think about the next action) before taking the actual relevant action: we have two modes in that state, the relevant action has a low probability while the idle action has a very high probability. The human policy might not be exactly markovian either. Those differences significant enough that the conclusions on synthetic datasets might not hold anymore.
In this section, we focus on the Adroit door and hammer environments for which we run experiments with human as well as synthetic demonstrations. 10 Note that on top of the aforementioned challenges, the setup with the Adroit environments using human demonstrations exhibits few additional specifics. The demonstrations were collected letting the human decide when the task is completed: said in a different way, the demonstrator is offered
additional action to jump directly to a terminal state and this action is not available to the agent imitating the expert. The end result is a dataset of demonstrations of variable length while the agent can only generate episodes consisting of exactly 200 transitions. Note that there was no time limit imposed on the demonstrator and some of the demonstrations have a length greater than 200 transitions. Getting to the exact same state distribution as the human expert may be impossible, and imitation learning algorithms may have to make some trade-offs. The additional specificity of that setup is that the reward of the environment is not exactly what the human demonstrator optimized. In the door environment, the reward provided by the environment is the highest when the door is fully opened while the human might abort the task slightly before getting the highest reward. However, overall, we consider the reward provided by the environment as a reasonable metric assess the quality of the trained policies. Moreover, in the hammer environment, some demonstrations have low return and we suspect those are not successful demonstrations
Discriminator regularization. When comparing the results for RL-genneaater androit-expen and human demonstrations (adroit-humall we can notice differences on number of HPs related to the discriminator training. Human demonstrations benefit more from using discriminator regularizers (Fig. (5)) and they also work better with smaller discriminator networks (Fig. (2) trained with lower learning rates (Fig. 66 The increased need for regularization suggest that it is easier to overfit to the idiosyncrasies of human demonstrations than to those of RL policies.
For pen, we only use the "expert" dataset, the "human" one consists of a single (yet very long) trajectory. "D4RL datasets (45) contain only the policy observations and not the simulator states and therefore it is not straightforward to visualize the demonstrations.
We do not include pen in the -expert plots so that both adroit- expeert and adroit- show the results averages across the door and hammer tasks and differ only in the demonstrations used.
8

---

adroit expert
adroit human
adroit expert
adrait human
0.7
0.7
0.7
0.7
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.6
0.6
0.6
0.6
-In(1-D) AIRL
In(D)
-n|1-DD A/RL
In(D)
(a) discriminator input
(h) reward functions
Figure 4: Comparison of discriminator inputs (a) and reward functions (b) for environments with
human demonstrations. See Fig. 49 and Fig. 25 for the individual results in all environments.
Discriminator input. Fig. 4 shows the performance given the discriminator input depending on the demonstration source. For most tasks with RL--generated demonstrations, feeding actions well as states improves the performance (Fig. 49). Yet, the opposite holds when human demonstrations are used. We suspect that it might be caused by the mentioned issue with demonstrations lengths which forces the policy to repeat similar movement but with a different speed than the demonstrator.
Reward functions. Finally, we look how the relative performance of different reward functions depends on the demonstration source. Fig. 4 shows that for RI-generated demonstrations the best reward function is AIRL while - I1 D) performs better with human demonstrations. Under the assumption that the discriminator is optimal, these two reward functions correspond the minimization of different divergences between the state (or state-action depending the discriminator input) occupancy measures of the policy (denoted π) and the expert (denoted E).
The reward function performing best with human demonstrations In(1 D)) corresponds to the minimization of the Jensen-Shannow Shannon divergence (proof in (14). Interestingly, this divergence Das
{s)(E|(m)) [E][=)) and bounded (0 < D (T) In(2)). For AlL, the symmetry means that it penalizes the policy for doing things the expert never does with exactly the same weight for not doing some of the things the expert does while the boundedness means that the penalty for not visiting single state is always finite. We suspect that this boundedness is beneficial for learning with human demonstrations because it may not be possible exactly match the human distribution for the reasons explained earlier.
is symmetric (s [=|[=
In contrast (proof in (46)) is neither symmetric, nor bounded - penalizes the policy much more heavily for doing the things the expert never does that for not doing all the things the expert does and the penalty for visiting a single state the expert never visits is infinite (assuming perfect discriminator).
the DKL (x](E) divergence which is optimized by the AIRL reward
While it is hard to draw any general conclusions only from the two investigated environments for which we had access human demonstrations, our analysis shows that the differences between synthetic and human-generated demonstrations can influence the relative performance of different algorithmic choices. This suggests that RL-generated data are not good proxy for human demon- strations and that the very common practice of evaluating IL only with synthetic demonstrations may lead to algorithms which perform poorly the more realistic scenarios with human demonstrations.
Related work
The most similar work to ours is probably [44] which compares the performance of different discriminator regularizers and concludes that gradient penalty is necessary for achieving good performance with off-policy AIL algorithms. In contrast to (44), which uses a single HP configuration, we run large-scale experiments with very wide HP sweeps which allows to reach more robust conclusions. particular, we are able to achieve excellent sample complexity on all the environments used in 44 without using any explicit discriminator regularizer (Fig.) 3).

---

The methodology of our study is mostly based 42 which analyzed the importance of different choices for on-policy actor-critic methods. Our work is also similar other large-scale studies done in other fields of Deep Learning, e-g. model-based RI (38). GANs (34). NLP (5), disentangled
representations [3] and convolution network architectures [5].
Conclusions
In this empirical study, we investigate in depth many aspects of the AIL framework including discriminator architecture, training and regularization as well as many choices related to the agent training. Our key findings can be divided into three categories: (1) Corroborating prior work, e.g. for the underlying RL problem, off- off-policy algorithms are more sample efficient than on-policy ones; (2) Adding nuances previous studies, e.g. while the regularization schemes encouraging Lipschitzness improve the performance, more classical regularizers like dropout or weight decay often perform on par: (3) Raising concerns: we observe high discrepancy between the results for RL-generated and human data. We hope this study will be helpful to anyone using or designing AlL algorithms.
Additionally we released the [I] unified AlL agent we implemented in JAX within the Acme framework as well as [2] the raw data of our experiment, along with [3] Notebook that allows to load and study them.
[3]
http:://ww.........a.....aaaaaaa colab.ipynb
Acknowledgments
We thank Kamyar Ghasemipour for the discussions related the FAIRL reward function and Lucas Beyer for the feedback on an earlier version of the manuscript.
10

---

References
[1] Stephen Hanson and Lorien Pratt. "Comparing biases minimal network construction with back-propagation" In: Advances in neural information processing systems 1 (1988), pp. 177-
185.
[2] Dean Pomerleau. "Efficient training of artificial neural networks for autonomous navigation". In: Neural computation 3.1 (1991), pp. 88-97.
[3] Stuart Russell. "Learning agents for uncertain environments". In: Conference Computational
learning theory. 1998.
[4] Stefan Schaal. "Is imitation learning the route to humanoid robots?" In: Trends in Cognitive Sciences 3.6 (1999). pp. 233-242. ISSN: 1364-6613. DOI: https: :///:/ org / 10 1016
S1364 6613 (99) 01327-3 URL: http: www. sciencedirect. cow ccwm///cieencce/attiicl///e///e
$13646613990112733.
[5] Andrew Ng. Stuart J Russell, et al. "Algorithms for inverse reinforcement learning." In: Ioml. Vol. 1. 2000, 2.
[6] Brian D Ziebart et al. "Maximum entropy inverse reinforcement learning." " In: Aaai. Vol. 8. Chicago, II. USA. 2008, pp. 1433-1438.
[7] Brenna D Argall et al. "A survey of robot learning from demonstration". In: Robotics and autonomous systems 57.5 (2009), pp. 469-483.
[8] Brian Ziebart. "Modeling purposeful adaptive behavior with the principle of maximum causal entropy". " In: (2010).
[9] lan Goodfellow et al. "Generative adversarial networks". " In: arXiv preprint 10Xiv:1406.2661 (2014).
[10]
Nitish Srivastava et al. "Dropout: a simple way to prevent neural networks from overfitting". In: The journal of machine learning research 15.1 (2014), pp. 1929-1958.
[11] Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL: http::/tt///tt .rrg www.oorgg//...........e..e httg//bbs/412... 69640.66.666)))
[12] John Schulman et al. "High-dimensiona" continuous control using generalized advantage estimation". In: arXiv preprint -xXivv/506 024482438 (2015).
[13] Greg Brockman al. "Openai gym"" In: arXiv preprint 1606.1160. (2016).
[14] Jonathan Ho and Stefano Ermon. "Generative adversarial imitation learning". In: arXiv preprint
a606 03476 (2016).
[15] David Silver et al. "Mastering the game of Go with deep neural networks and tree search". In: nature 529.7587 (2016), p. 484.
[16] Mare G Bellemare, Will Dabney, and Rémi Munos. "A distributional perspective on reinforce- ment learning". In: International Conference on Machine Learning. PMLR. 2017, pp. 449- 458.
[17] Justin Fu, Katie Luo, and Sergey Levine, "Learning robust rewards with adversarial inverse reinforcement learning". In: arXiv preprint arXiv: 1224 (2017). gans". In: arXiv preprint
[18] Ishaan Gulrajani et al. "Improved training of wasserstein
arXiv:/704 /ttv://70400033 (2017).
[19] Riashat Islam et al. "Reproducibility of benchmarked deep reinforcement learning tasks for continuous control". In: arXiv preprint arXiv: 04133 (2017).
[20] Ivaylo Popov et al. "Data-efficient deep reinforcement learning for dexterous manipulation". In: arXiv preprint arXiv: 1704.03073 (2017).
[21] Aravind Rajeswaran al. "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations". In: arXiv preprint urXiv: 10087 (2017).
[22] John Schulman et al. "Proximal policy optimization algorithms". In: arXiv preprint
arXiv://07.06344 (2017).
[23] Hongyi Zhang al. "mixup: Beyond empirical risk minimization". In: arXiv preprint
/71 (2017).
[24] Gabriel Barth-Maron et al. "Distributed distributional deterministic policy gradients". In: arXiv preprint arXiv: 08617 (2018).
[25] James Bradbury et al. JAX: composable transformations of Python+ NNmmy programs. Ver-
sion 0.2.5. 2018. URL: httt:://gtthii com/ google/jax

---

[26] Scott Fujimoto, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods", In: International Conference on Machine Learning. PMLR. 2018,
pp. 1587-1599.
[27] Tuomas Haarnoja et al. "Soft actor-critic algorithms and applications". In: arXiv preprint arXiv:18/2.05900 (2018).
[28] Tuomas Haarnoja et al. "Soft Actor-Critie: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor". In: International Conference on Machine Learning. 2018,
pp. 1861-1870.
[29] Peter Henderson et al. "Deep reinforcement learning that matters". In: Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
[30] Todd Hester et al. "Deep q- q-learning from demonstration"". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. 1. 2018.
[31] Ilya Kostrikov et al. "Discriminator-actor--e bias in adversarial imitation learning". In: arXiv preprint 1rXiv:1809.000 (2018).
Addressing sample inefficiency and reward
[32] Francesco Locatello et al. "Challenging common assumptions in the unsupervised learning of disentangled representations". In: arXiv preprint /rXivv/8/1 1811.2239 (2018).
[33] Ilya Loshchilov and Frank Hutter. "Fixing weight decay regularization in adam". In: (2018). Mario Lucic et al. "Are gans created equal? a large-scale study". In: Advances in neural information processing systems. 2018, pp. 700-709.
[34]
[35] Takeru Miyato et al. "Spectral normalization for generative adversarial networks". In: arXiv preprint arXiv:/8022 000000555770 (2018).
[36] George Tucker al. "The mirage of action-dependent baselines in reinforcement learning". In: arXiv preprint arXiv: /802. 1003/ (2018).
[37] Christopher Berner et al. "Dota 2 with large scale deep reinforcement learning". In: arXiv preprint X9:: 06680 (2019).
[38] Eric Langlois et al. "Benchmarking model-based reinforcement learning". In: arXiv preprint arXiv:1907. 97.00007770077 (2019).
[39] Tom Le Paine et al. "Making efficient use of demonstrations to solve hard exploration prob- lems". In: arXiv preprini arXiv:1909.01 31777 (2019).
[40] Oriol Vinyals et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learn-
ing". In: Nature $75.7782 (2019), pp. 350-354.
[41] Danfei Xu and Misha Denil. arX14:1911.00459 (2019).
reward learning". In: arXiv preprint
[42] Marcin Andrychowicz al. "What matters in on-policy reinforcement learning? a large-scale empirical study". In: arXiv preprint urXiv:2006.05999 (2020).
[43] OpenAl: Marcin Andrychowicz et al. "Learning dexterous in-hand manipulation". In: The International Journal of Robotics Research 39.1 (2020), pp. 3-20.
[44] Lionel Blondé, Pablo Strasser, and Alexandros Kalousis. "Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning". In: arXiv preprint arXiv:2006. 16785 (2020).
[45] Justin Fu et al. "D4rl: Datasets for deep data-driven reinforcement learning". In: arXiv preprint arXiv:2004.0 07219 (2020).
[46] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. "A divergence min- imization perspective on imitation learning methods". In: Conference Robot Learning.
PMLR. 2020, pp. 1259-1277.
[47] Jonathan Heek et al. Flax: A neural network library and ecosystem for JAX. Version 0.3.3.
2020. URL: http:://gtttiii
com/ wtw.:oom//oooeeee hamm//ooole/faa
[48] Matteo Hessel et al. Optax: composable gradient transformation and optimisation, in JAX!
Version 1. 2020. URL: htttp://githan cwm/ com//eepmind. optax
[49] Matt Hoffman et al. "Acme: A research framework for distributed reinforcement learning". In: arXiv preprint arXiv:2006.009:
(2020).
[50] Jared Kaplan et al. "Scaling laws for neural language models"". In: arXiv preprint
arXiv:2001. 0836/ (2020).
[51] Ilija Radosavovic et al. "Designing Network Design Spaces". In: arXiv preprint arXiv:2003.13677 (2020).
12

---

[52] Annie Chen et al. "Batch exploration with examples for scalable robotic reinforcement learn- ing". In: IEEE Robotics and Automation Letters (2021).
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We study general purpose imitation algorithms which are not related to any particular application.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] D4RL demonstrations are available under Apache License 2.0
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "ykVnN4Qeaf-", "reviewer": "Reviewer_YvcC", "summary": "The work conducts a detailed empirical analysis of a range of design decisions in the domain of adversarial imitation learning. The imitation learning framework is sufficiently general to encompass prior work like GAIL, AIRL, various extensions to these algorithms (e.g. regularizers), different RL algorithms used for the generator training. These decisions are tested in a comprehensive hyperparameter sweep involving over 500k trained agents across eight continuous control tasks. The results include many observations that will be useful to practitioners, such as the importance of observation normalization. It also in some cases casts doubt on received wisdom, such as the various AIL-specific methods for discriminator regularization, which seem to be little better than standard SL approaches like dropout.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "With the proliferation of adversarial imitation learning algorithms and the sadly often not-so-careful evaluation in these papers, testing what actually works is timely and important. The paper is clearly written and the experiments are carefully conducted. I expect practitioners to find this a helpful aid to tuning their imitation learning algorithm. For those active in AIL research, this paper serves both as a useful wake-up call (with, in effect, several negative results contradicting prior work) and as a guide to areas where further research is needed (e.g. more human data benchmarks).\n\nAlthough I am overall excited by this paper, I did find some of the experiments hard to interpret and potentially confounded. In particular:\n  1. The performance measure that approximates area under the learning curve differs from the metric used in much prior work, of episode return at convergence. I think both of these metrics could be appropriate depending on the context. Perhaps you could also report episode return at the end of the training, at least for the experiments where this most significantly changes results? In particular, this would allow the reader to establish whether seemingly negative results about prior work are due to the differing performance measure (perhaps prior methods are better at convergence but worse during early training) or a true replication failure.\n  2. I am glad that you highlighted the issue that variable-length episodes cause in evaluation, an issue that has plagued this field. I was also pleased to see that inclusion of the absorbing state does seem to help with this -- at least the difference between reward function choice in Figure 72 becomes a lot less significant with an absorbing state. But how confident can we be the absorbing state has really de-biased this? I would have favored removing the termination condition from these environments, as discussed in Appendix A of [Christiano et al (2017)](https://arxiv.org/pdf/1706.03741.pdf) and followed in much of the other work in this area. I understand re-running the experiments may not be practical -- perhaps consider a break-down of environments with fixed-length episodes vs variable-horizon episodes given your test suite contains a mix of both?\n  3. There are often correlations between the hyperparameters. I think the paper mostly does a good job highlighting these limitations in context, but these caveats have tended to be dropped in summaries. In particular, I think the conclusion that synthetic demos may not predict human demos is significantly undermined by the human demos in question being quite strange, in the sense that the agent *cannot* possibly replicate the human demos. I think you either need to argue that this is a normal or unavoidable situation when working with human data, or modify the environment to allow the agent to transition to an absorbing state early, or at least add some significant hedging to the conclusion.\n\nI also have one question: you write that “Following prior work [14, 31, 46], we subsample expert demonstrations by only using every 20th state-action pair to make the tasks harder.” However, I do not see any mention of this in the GAIL paper [14]. Can you refer me to where in [14] this is discussed?\n\nAs said, the paper was generally well written, but I noticed a few areas that were confusing:\n- Line 29: “and never these changes were all compared together” – ungrammatical. Perhaps “and these changes have never been compared simultaneously”?\n- Line 30: “the low-level”->”low-level”. “might be not even” -> “might not even be”.\n- Line 220: “As noticed in prior work” – this made me expect you were going to confirm that result but you actually find it doesn't deplicate, consider rephrasing? E.g. “Prior work has claimed the initialization ...”\n- Line 223: “An overfitting” -> “An overfit”?\n- Some capitalization issues in references, e.g. \"Openai gym\"->\"OpenAI Gym\".\n\n# Update after rebuttal\n\nI thank the authors for their response. It is nice to know the results mostly do not change between final performance rather than AUC, but I agree that including a brief analysis of this in the paper would be worthwhile. \n\nI continue to have some concerns about the results being confounded by variable-length episodes and the quirks in the human data, but I do sympathize that given the scale of the experiments and the already significant length of the paper, these issues are difficult to correct. I'd suggest the authors consider adding a more explicit discussion of these limitations to the manuscript, however.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations discussed in context but not adequately pushed through to the summary. No real discussion of societal impact of work which is a missed opportunity but I expect it to be robustly positive.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "UowR7JNecpG", "reviewer": "Reviewer_UNYV", "summary": "This paper presents a large-scale empirical study for implementation choices on adversarial imitation learning. The contributions of this work can be summarized as follows:\n\na. implement a high-configurable adversarial imitation learning algorithm and conduct a large-scale empirical study\n\nb. from the empirical study, the authors argue that they find two important results:\n\n1. a choice of discriminator regularizers is not critical \n\n2. artificial demonstrations are not a good proxy for human demonstrations", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents interesting results about algorithmic choices on adversarial imitation learning by conducting rigorous large-scale empirical studies. The paper is clearly written and findings, such as discriminator regularization techniques (e.g. gradient penalty, .. ) are not as crucial as we considered in literature, are interesting. Yet, I have following concerns and comments about this paper:\n\n1. The authors argue that synthetic demonstrations are not a good proxy for human data. \nBut I concern this argument is too obvious to be one of main findings of this empirical study. \n\n2. The authors may argue that unexpected human behaviors make significant differences in optimal hyperparameter settings in Figure 4. But it is not clear to me where those differences come from. Isn’t it simply because of the difference in performance? According to Table 2 in Appendix F, performances of door-expert-v0 and door-human-v0 are significantly different (2882 and 796 respectively), hammer-expert-v0 and hammer-human-v0, too (12794 and 3071 respectively).\n\n3. Hyperparameter selections should be more carefully considered especially for human data. The agent getting higher performance on the evaluation environment is not always better, because the goal of imitation learning is imitating the given expert demonstration.\n\n4. Since expert demonstrations define each problem, varying the quality of expert datasets could be a valuable extension of this empirical study. For example, D4RL dataset contains different versions of each task such as hopper-expert-v0 and hopper-medium-v0, etc. \n. \n\nAlthough I have some concerns on this paper, lessons from this work can be helpful to future imitation learning researchers and practitioners. Hence, I vote for acceptance to this paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "AelDBWNxU_A", "reviewer": "Reviewer_mnBs", "summary": "The paper presents an extensive study on hyperparameters and design choices for adversarial imitation learning, including the choice of divergence (reward function), RL algorithm and whether marginals $p(s,a)$, $p(s)$, $p(s,s')$ or $p(s,a,s')$ are to be matched. Evaluations are conducted on MuJoCo and Adroit environments --- the latter experiments also made use of human demonstrations.\nThe main findings are that demonstrations collected by an RL agent are not good proxies for human demonstrations and that more standard discriminator regularizers like dropout perform similarly well as gradient penalty.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nSignificance\n----------------\nStudies like these can be very helpful for researchers and practitioners. The paper was interesting to read, but afterall I did not get any major new insights. The fact that the differences between human demonstrations and demonstrations from an RL agent affect the performance of the algorithm and the choice of parameters was to be exected. But of course, learning about which parameters might be preferable in a human-demonstration setting is still useful, even though I think that the number of experiments is too small to conclude that JS divergence performs better than KL for example.\n\nOriginality\n--------------\nI don't think that a similar study on adversarial imitation learning exists, so in that sense the work is original. However, it does not present any intriguing ideas and also the interpretations of the results seem a bit shallow. \n\nQuality\n-----------\nOverall, the study was soundly conducted. However, I do have a few criticisms:\n\n* All experiments seem to use a fixed number of demonstrations. That would be a major oversight, given that some algorithms that work well for many demonstrations (for example behavioral cloning) might not work very well for few demonstrations. Evaluating how design choices relate to the number of demonstrations would have been very interesting. On a minor note, also evaluations w.r.t. to noisy / mixed data might have produced more meaningful insights.\n\n* The performance is evaluated based on the agent's return. I think it would be important to also evaluate the performance w.r.t. the different divergences, since these are the actual objectives of the algorithms. It is not at all clear, whether lower divergences always achieve higher rewards.\n\n* The authors say that they plan to release the code, however, I am not overly optimistic since not even the reviewers have been given access to the code. Also, I would expect a study to publish the underlying data, which would be very useful, for example, to conduct the missing evaluations w.r.t. the different divergences (maybe even MMD).\n\nClarity\n--------\nThe presentation of the results is good and the paper is neatly structured.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative societal impact were adequately addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "no ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an extensive experimental study on implementation choices in adversarial imitation learning. All reviewers were leaning towards acceptance of the paper since many observations made in the paper could be useful to practitioners. I am also recommending acceptance of the paper. 

Decision: Accept (Poster)