Task-Relevant Adversarial Imitation Learning
Authors: Konrad Zolna
Scott Reed
Alexander Novikov
Ziyu Wang
Sergio Gómez
David Budden
Serkan Cabi
Misha Denil
Nando de Freitas
Venue:  (2020)
Keywords: adversarial
imitation
robot
manipulation

Abstract: We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent. We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL). Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features. Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such as behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly proposed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to find solutions, even when the other agents have access to task reward. 

Under review as a conference paper at ICLR 2020
TASK-RELEVANT ADVERSARIAL IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We show that a critical problem in adversarial imitation from high-dimensional sensory data the tendency of discriminator networks distinguish agent and expert behaviour using task irreeittellvant features beyond the control of the agent. We analyze this problem in detail and propose solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL). Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features. Compre- hensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly pro- posed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to find solutions, even when the other agents have access to task reward.
INTRODUCTION
Generative Adversarial Networks (GANs) have produced breath-taking conditional image synthesis results (Goodíellow et al., 2014 Brock et al. 2019), and have inspired adversarial learning approaches to imitating behavior. Generative Adversarial Imitation Learning (GAIL) (Ho Ermon 2016), discriminator network is trained to distinguish agent and expert behaviour through its observations, and is then used as reward function. GAIL agents can overcome the exploration challenge by taking advantage of expert demonstrations, while also achieving high asymptotic performance by learning from agent experience.
Despite the huge promise of GAIL,, it has not yet had the same impact GANs; in particular, robust GAIL from pixels for control applications remains a challenge. Here, we study a key shortcoming of GAIL: the tendency of the discriminator mainly exploit task-irrelevant features. For example, by focusing slight background differences, a discriminator can achieve perfect generalization, assigning zero reward all held-out agent observations. However, this discriminator does not yield an informative reward function because it ignores behavior.
GAIL
TRAIL
(a) Block lifting
(b) Block lifting with distractors
GAIL
TRAIL.
(c)
slacking
(d) Block distractors
Figure 1: GAIL and TRAIL succeed at lifting (a). but when distractor objects are added, GAIL fails while TRAIL succeeds (b). Due robustness initial conditions, TRAIL can stack from pixels while standard GAIL fails (c). We witness this difference again in insertion with distractors (d). A video showing agents performing these tasks can be seen at https:: wty.ttt..e/aaG1..... be/

---

Under review as a conference paper at ICLR 2020
Changing distractor prope
Changing agent sppeerance
Changing object sppearsnce
Expert
Demo
Agent Episodo
Figure 2: Illustration of several task-irrelevant changes between the expert demonstrations and the distribution of agent observations, for the lift (red cube) task. The naively-traine discriminator network will use these differences rather than task performance to distinguish agent and expert.
Assuming there is expert policy me that is optimal for unknown reward function, here refer to feature as task-irrelevant if does not affect that reward. For example, if the task is to lift a red block, the positions of other blocks would be task-irrelevant; Figures anddd
This paper makes the following contributions:
1. It reveals fundamental limitation of GAIL by showing that discriminators do in practice exploit task-irreevvant information, thereby resulting poor task performance.
2. It introduces powerful GAIL baselines. In particular, shows that standard regularization and data augmentation are generally useful and improve upon standard GAIL.
3. It shows that these improvements to GAIL, as well as other improvements proposed by Reed et al.] (2018), do not completely solve the problem, allowing GAIL agents to fail catastrophically with the addition of iesk-irrelevant distractors.
4. It introduces Task Relevant Adversarial Imitation Learning (TRAIL), using constrained optimization to force the discriminator to focus on the relevant aspects of the task, which improves performance dramatically on manipulation tasks from pixels (see Figure I].
RELATED WORK
The use of demonstrations to help agent training has been studied extensively in robotics (Bakker Kuniyoshi 1996. Kawato et al. 1994 Miyamoto et al.) 1996) with approaches ranging from Qelarnnng (chhall 1997) to behavioral cloning (BC) (Pomerleau 1989).
BC: BC is effective solving many control problems (Pomerleau) 1989 Finn et al. 2017 Duan
al. 2017 Rahmatizadeh et al. 2018). It has also been successfully applied to initialize RL training (Rajeswaran et 2017). It, however, suffers from compounding errors as initial small deviations from the expert behaviors tend cause bigger differences (Ross et al. 2011). This often necessitates a large number of demonstrations for satisfactory performance. Furthermore, BC typically does not lead to agents that superior to their demonstrators.
Inverse RL: Ziebart et al. (2008) Ng et al. (bbeell & Ng (0044) propose inverse reinforcement learning (IRL) as a way of learning reward functions from demonstrations. Reinforcement learning can then be used to optimize that learned reward. Recently, Finn et al. (201bb) approached continuous robotic control problems with success by applying Maximum Entropy IRL algorithms which are very closely related to GAIL (Finn et al. 201 and have similar drawbacks.
Learning from Demonstrations: Hester al. (2018) developed deep Q-Learning from demonstra- tion (DQfD), in which expert trajectories are added to experience replay and jointly used to train agents along with their own experiences. This was later extended by Vecerik et al.) (2017) anhlen al. (2018) to better handle sparse-reward problems in control and Atari games respectively. Despite their efficiency, this class of methods still requires access to rewards in order to learn.
GAIL: Following the success of Generative Adversarial Networks (Goodfellow et al. 2014 in image generation, GAIL (Ho & Ermon 2016) applies adversarial learning to the problem of imitation. Although many variants are introduced in the literature (Li al. 2017 Fu al. 2018 Merel et al.) 2017 Zhu et al. 2018) Baram et al.) 2017). making GAIL work for high-dimensionll input spaces, particularly pixels, remains challenge.
2

---

Under review as a conference paper at ICLR 2020
A few papers (Peng al.) 2018 Reed et al. 2018 Blondé & Kalousis 2018 seek to address the problem of overfitting the discriminator, Peng et al. (2018) introduces the Variational Bottleneck to regularize the discriminator. Reed al. (2018) proposes not train the vision module of the discriminator (e.g. use the vision module of the critic network instead) and only train a tiny network on top of the vision module to discriminate. [Blondé (Blondd & Kalousis (2088) follow similar approach. Unstructured regularization, however, cannot stop the discriminator from fitting to features that are systematically different between the agents' and the demonstration's 's behavior, like those illustrated in Figure 2 TRAIL, on the other hand, is much less prone to overfitting to these features.
Stadie et al. (017) extend GAIL to the setting of third person imitation, in which the demonstrator and agent observations come from different views. To prevent the discriminator from discriminating based on vietpoint domain, they use gradient flipping from an auxiliary classifier to learn domain-invariant features. Our approach is not to learn domain-invarian features, but instead learn discriminators that only focus on behavior.
Several recent works have focused on improving the sample efficiency of GAIL (Blondé & Kalousis 2018; Sasaki et al., 2018). Common to these approaches and to this work, is the use of off-policy actor critic agents and experience replay, to improve the utilization of available experience.
REINFORCEMENT LEARNING AND ADVERSARIAL IMITATION
Following the notation of Sutton & Barto 2018). a Markov Decision Process (MDP) is a tuple (8,A, A. R. 7,))) with states S, actions A, reward function R(s, a), transition distribution P(s'|s, a), and discount y. An agent in state 8 E S takes action a E A according to its policy л and moves to state s' E S according to the transition distribution. The goal of RL algorithms to find a policy that maximizes the expected sum of discounted rewards, represented by the action value function (T a) E" === 7² R(s, st. a.). where EE is an expectation over trajectories starting from Su = and taking action au = a and thereafter rumning the policy x.
To apply RL, is essential that we have access to the reward function which is often hard to design and evaluate (Singh et al. 2019). In addition, sparse rewards can cause exploration difficulties that pose great challenges to RL algorithms. We therefore look to imitation learning and particularly GAIL to derive reward function from expert demonstrations. In GAIL, reward function is learned by training a discriminator network (ss, a) to distinguish between agent and expert state-action pairs. The GAIL objective is thus formulated as follows:
min max B(x,a)~< (log ((s, a)] + (lo.a)) - D(s, a))] - (x
(I)
where π is the agent policy, mæ the expert policy, and (xx) (optional) entropy regularizer. The reward function is defined simply: R(s, a) - log(1 1 - ((s, a)).
GAIL is theoretically appealing and practically simple. The discriminator, however, can focus on any features to discriminate, whether these features are task-relevant or not. In the next subsection we describe way to constrain the discriminator network in order to prevent it from using task iireeitrele details to distinguish agent and expert data.
TASK-RELEVANT ADVERSARIAL IMITATION LEARNING (TRAIL)
We want the discriminator to focus on task-relevant features. Our proposed solution, TRAIL,, prevents the discriminator from being able to distinguish expert and agent behaviour based on selected aspects of the data. For instance, the discriminator should distinguish agent and expert frames only when meaningful behavior is present in those frames. In the absence of behavior useful to solve the task, e.g. in initial frames prior to the execution of the behavior, the discriminator should be agnostic.
To this end, we propose constraint for the discriminator, such that its accuracy must not be greater than chance on a particular set of observations I that we will call the invariant set. I will include only observations that can be distinguished as agent expert in task-irreeeaant ways. Precisely,
3

---

Under review as a conference paper at ICLR 2020
formulate TRAIL in terms of the following constrained optimization problem:
max E EENTE [log Du(s)] - Bewno [log(1 - Dw(ss)]
(2)
s.t. B [DD66)24 E I Banto [D 8 I 1
To apply the above constraint in practice, That is, given a batch N examples Se ~ TE, $8 ~ Te from the expert and agent, and &e ~ Tg and $8 To both in the set I, maximize the following augmented objective function:
optimize the reverse of the objective function on I.
(se s0, Se. (e)
i= log Dv
(e
+ log
(1-11)
(1 D (s) (s8)
(3)
~ Ein 1 log Dep
(5(1) log (1 DD 88 ))]
Dv
ucourocy(Ss.50)23
where accuracy(:s)) is defined the average of discriminator accuracies:
N
accuracy(6..8)) 2N
(s5) +
1 (b.))
The scalar is tunable hyperparameter.
We used single frames as states but sequences can also be used, and we do not require expert actions. Not requiring actions enables learning in very off-policy settings, where the action dimensions and distributions of the demonstrator (another robot or human) are different from those of the agent.
4.1 THE SELECTION OF THE INVARIANT SET
The selection the invariant set design choice. In general, we could always contrive non- stationary and adversarial ways of making this choice difficult. However, we argue that in many situations of great interest, including our robotic manipulation setup. is easy to propose effective and very general invariant sets.
straightforward way to collect robot data is to execute a random policy. We can then use the resulting random episodes, for both expert and agent, construct the invariant set I. Another way construct I is to use early frames from both expert and agent episodes. Since in early frames little or no task behavior is apparent, this strategy collected. This strategy also improves robustness with respect to variation in the initial conditions of the task; see for example block insertion Figure[:c)
out be effective and no extra data has to be
Importantly, if the set I captures some forms of irrelevance but not all forms. it will nonetheless always help in improving performance. In this regard, TRAIL will dominate its GAIL predecessor whenever the designer has some prior on what aspects of the data might be task irrelevant.
5 EXPERIMENTS
We focus on solving robot manipulation tasks. The environment implements two work-spaces: one with a Kinova Jaco arm (Jaco), and the other with a Sawyer arm (Sawyer). See supplementary material A. methods, are sparse and equal to +I each step when given task is solved and otherwise. The maximum reward for the episode is 200, since this is the length of single evaluation episode.
J ] for a detailed description. Environment rewards, which are not used by GAlL-based
Our agent is based on the off-policy D4PG algorithm (Barth-Maron et al., 2018) because of its stability and data-efficiency (see supplementary material A.6). Following Vecerik al. (2017), add expert demonstrations into the agents' experience replay, and refer to the resulting RL algorithm D4PG from Demonstrations (D4PGfD). For each task we collect 100 human demonstrations.
Data augmentation Traditional data augmentation has proved beneficial in imitation learning (Berseth & Pal 2019). Surprisingly, to the best of knowledge, this has not been explicitly studied in prior publications on GAIL. However, we find that data augmentation is generally useful component prevent discriminator overfitting. It drastically improves the baseline GAIL. agent, and
4

---

Under review as a conference paper at ICLR 2020
is necessary to solve any of the harder manipulation tasks. We distort images by randomly changing brightness, contrast and saturation; random cropping and rotation; adding Gaussian noise. When multiple sensor inputs available (e.g. multiple cameras), we also randomly drop out these inputs, but leaving at least one active.
All discriminator-basse methods in this work (including all baselines) use data augmentation except as indicated in the relevant ablations (see Section[5.2] We also considered regularizing the GAIL discriminator with spectral normalization (Miyato et al. 2018). It performed slightly better than GAIL, but still failed in the presence of distractor objects, and we thus omit spectral normalization in the main experiments for simplicity.
Actor early stopping When the agent has learned the desired behavior, and the resulting data is used for training, the discriminator will become unable to distinguish expert and agent observations based only on behavior. This forces the discriminator to rely on task-irrelevant information.
To avoid this scenario, we propose restart each actor episode after a certain number of steps such that successful behavior is rarely represented in agent data. This enables the discriminator to recognize the goal condition, which appears frequently at the end of demonstration episodes, as representative of expert behavior. To avoid hand-tuning the stopping step number, we found that the discriminator score can used to derive an adaptive stopping criterion. Concretely, we restart an episode if the discriminator score at the current step exceeds the median score of the episode so far for T patience consecutive steps (in practice we set patsence = 10).
We set invariant set hyperparameter À = 1 and use adaptive early stopping for TRAIL. The ablation with À = O, f.e. adaptive early stopping only, is referred to as TRAIL-O. Hence, the only difference between TRAIL-O and TRAIL is that the later uses invariant set constraints.
5.1 BLOCK LIFTING WITH DISTRACTORS
In this section, consider two variants of the lift task in the Sawyer work space: a) lift alone, where only one red cube is present, b) lift distracted, with two extra blocks (blue and green, see Figure 9 We show how adding these additional distractors affects the training procedure.
We first compare our method to baselines. In doing so, the invariant set constructed using the first 10 frames from every episode, as in the rest of the work. This choice does not require us collect any extra data, and hence the comparison with baselines is fair. In the next section, we elaborate on the choice of the invariant and provide additional experimental results.
As baselines, we run BC and GAIL (with data augmentation). Baseline GAIL is the strongest we able implement. The only difference between this baseline and TRAIL. is the use of invariant set constrains and actor early stopping (code, agent configurations like number of actors, and network architecture are the same). We additionally consider the approaches proposed by Reed et al. (2018) as GAll-based baselines; using either a randomly initialized convolutional network, or convolutional critic network, to provide fixed vision features on top of which a tiny discriminator network trained. We call these two baselines random and critic respectively. Finally, to show the importance of actor early stopping, we run TRAIL-O (Fig.
Lift alone
Lift distracted
Lift distracted scoded
legend
Dowwards
Gull. TRAIL D
100-
pc
mttic
50.
11.5m0
Ordal
3
160
staps
slaps
Actor alogo
Figure 3: Results for lift alone, lift distracted, and lift distracted seeded. Only TRAIL excels.
All methods perform satisfactorily on lift alone, but the proposed methods TRAIL-0 and TRAIL do best. As expected, the performance of BC on lift distracted is similar to its performance on lift alone, despite the two additional blocks. The two additional blocks in lift distracted affect the GAlL-based baselines, despite being irrelevant to the task.
5

---

Under review as a conference paper at ICLR 2020
To understand this effect, we conducted additional experiment (lift distracted seeded). Here the initial block positions are randomly drawn from the expert demonstrations. Therefore, is impossible to discriminate between expert and actor episodes using the first few frames of an episode. Note this initialization procedure is not applied to the evaluation actor, keeping the evaluation scores comparable between lift distracted and lift distracted seeded.
This experiment exposes one major culprit behind the performance degradation of GAIL: memoriza- tion. The discriminator can achieve perfect accuracy by memorizing all 100 initial positions from the demonstration set, making the reward function uninformative. By constraining the discriminator, TRAIL squeezes out this irrelevant information and succeeds in solving the task in the presence of distractions. TRAIL is the only method that is able to handle the variety of initial cube positions during training, achieving better than expert performance on lift distracted.
Interestingly, random performs reasonably on lift distracted. Given random's 's strong performance, we conducted additional experiments to evaluate its effectiveness when trained with adaptive early stopping and present the results in Figure 12 of the supplementary material.
CONSTRUCTING THE INVARIANT SET I
In the previous subsection, early frames were used construct the invariant set (TRAIL-early). Here, we evaluate another previously mentioned approach for constructing I; random policy (TRAIL- random). The lift distracted task caused all baselines fail, but was solved by TRAIL. We introduce harder version of the task, where the expert appearance is different, to tease out the differences between TRAIL-early and TRAIL-random. The difference in appearance between the expert and imitator allows the GAIL discriminator to trivially distinguish them. The results and the differences in the expert appearance are presented in Figure
Agont appearance
Expert appearance
T.fft distrmeted,, different hody appon
399
ligend
PC
TMAH esrty
Acter steps
Figure 4: Lift red block, where expert has a different body appearance, and with distractor blocks. TRAIL-random outperforms GAIL, and performs on par with TRAIL-early.
The new task is indeed harder and it takes longer for TRAIL methods to achieve performance better than BC baseline, which is not affected by the different body appearance. GAIL is clearly outperformed and does not take off.
The difference between TRAIL methods is negligible. We also tried mix them but the differences remain imperceptible. Hence, in the following experiments we simply use early frames. This choice pragmatic as it does not require that we collect any extra data, and hence the comparison with GAIL and other baselines is fair. It is also very simple to apply in practice, even if one does not have access the expert setup anymore. Finally, is general and powerful enough to be successfully used across all robotic manipulation tasks considered in this work.
To decide how many initial frames should be used construct I, we conducted an ablation study and found out that the method is not very sensitive to this choice (see supplementary material A.2). Hence, we chose 10 initial frames, and intentionally used the same number for all tasks to further emphasize generality of this choice.
5.2 ABLATION STUDIES
MEASURING DISCRIMINATOR MEMORIZATION OF TASK-IRRELEVANT FEATURES
In this section, we experimentally confirm that memorization a limiting factor GAIL methods. We equipped the discriminator with extra heads whose inputs the final spatial layer of the

---

Under review as a conference paper at ICLR 2020
ResNet for the lift distracted task. The first head is trained on the first frames only, and has the same target as the main head (i.e. discriminating between agent and expert). To train the second head, we randomly divide the expert demonstrations into two equi- -qui-numerous subsets, and the task is to predict to which of these randomly chosen sets the demonstration was assigned to. Both heads are trained via backpropagation but their gradient is not propagated to the ResNet so they do not influence the training procedure directly.
If our claim correct, we expect the extra heads to have higher accuracy for TRAIL-O compared to TRAIL, since TRAIL representations are penalized for having features triggering memorization, i.e. the features should not aid discriminating based on the first frames or predicting a random label for each expert demonstration. No reasonably performing method is able to force 50% accuracy for extra heads since some features are important to solve the task (e.g. the position of the red cube).
We also collected 25 extra holdout demonstrations and visualize the average discriminator prediction on them, and compare with predictions on the training demonstrations.
Overall
Pirst steps (m)
Training expert demos
0.75
0.8
Discriminator P(expert)
0.50
0.6
0.25
Accuracy
0.4
0-
legend
C.
D.
F.
TRAIL
First steps (e)
Demonstration class (c)
Holdout expert demos
TRAIL
1-
1-
0.8
0.50
0.6
0.25
0.4
0
1e5
265
1e5
265
1o5
2a5
Learner steps
Learner steps
Figure 5: Demonstrating the memorization problem on the lift distracted task (here higher accuracy is worse). Accuracy of different discriminator heads is presented (A-D). In A, the overall accuracy for all timesteps. Then main (m) and extra (e) heads accuracy for the first steps are presented in B and C, respectively. Accuracy of the head predicting randomly assigned demonstration class is shown D. Average discriminator predictions for training and holdout demonstration are shown and F.
In Fig. 5 we see the overall accuracy of the main head for TRAIL-O is significantly higher, and the difference is larger for early steps (when TRAIL achieves only 50%, as expected). TRAIL representations are less helpful for extra heads (see Figure CCD)). Finally, TRAIL-0 elearly overfits on training demonstrations, predicting almost the maximum score, while only 0.25 predicted for holdout demonstrations. The TRAIL average predictions for both datasets are almost identical.
ACTOR EARLY STOPPING (TRAIL-0)
In this section, we analyze the importance of adaptive early stopping on 3 tasks in the Jaco work- space: lift red cube (liff), put red cube in box (box), and stack red cube on blue cube (stack). We consider D4PGfD and three GAIL-based models with varying termination policies: a) fixed step (50), b) based on ground truth task rewards, and c) based on adaptive early stopping (TRAIL-O). Using ground truth task rewards, an episode is terminated if the reward at the current step exceeds the median reward of the episode so far for 10 consecutive steps. Results are presented in Figure
Termination based on task reward is clearly superior; although unrealistic in practice, it defines the performance upper-bound and clearly shows that early stopping is beneficial. TRAIL-0 is robust and reaches human performance all tasks. A fixed termination policy, when tuned, can be very effective. The same fixed termination step, however, does not work for all tasks. See Figure [3] in supplementary material for the effects of varying termination steps. Finally, as can be inferred from stack results, the dense rewards provided by TRAIL-O are helpful in solving this challenging problem which is unsolved with D4PGfD even though D4PGfD uses ground truth task rewards.
7

---

Under review as a conference paper at ICLR 2020
Lift
Put ir. box
Stack
300-
1.0
-1.4111
lagend
11romm only
100
Floed step (20)
3.307
Actor steps
Actor steps
Actoor stops
Figure 6: Results for lift, box, and stack on Jaco environment.
DATA AUGMENTATION
Table 5.2, we report best reward obtained in the first 12 hours of training (averaged for seeds; see Figure 14 for full curves). The results show that data augmentation is needed in lift distracted. For the lift alone task, TRAIL-O with data augmentation performs on par with TRAIL.
Task
Regularization
Data augmentation
No data augmentation
TRAIL-0 TRAIL
~16
115
lift alone
~~55
1665
TRAIL-O TRAIL
~0
5
lift distracted
~180
10
Table Influence of data augmentation (evaluated on rewards) for lift alone and lift distracted.
The performance of TRAIL is not affected by the lack of data augmentation on the lift alone task,
whereas the performance of TRAIL-O is.
LEARNING WITH A FIXED, PERFECT DISCRIMINATOR
To assess whether learned discriminators are necessary, we compare TRAIL against agents using fixed reward function corresponding Respert distracted tasks. This baseline simulates an oracle discriminator with perfect generalization, but which is agnostic to behavior. On the lift alone task. agents using this fixed reward achieve roughly half the reward of TRAIL asymptotically. and on lift distracted they do not solve the task (average rewards are less than 5). See supplementary Figure 16 for learning curves.
and Ragent = 0 for the lift alone and lift
5.3 LEARNING FROM OTHER EMBODIMENTS AND PROPS
Since the TRAIL discriminator is trained to ignore lask-irrelevant features, it can learn from demon- strations with different embodiments and props. Figure 7 shows that GAIL even with augmentation fails to learn block lifting from a different embodiment, and performs worse when the expert uses a different prop color. TRAIL solves the task and achieves better performance in both cases.
Expert
Difforent body aрpearance
Expert
Different; prop appearance
250
2E0
Kujert
Rawards
legend
Dowards
100-
Agent
Agent
TRAIL
03..7
8.677
1x7
Aclic stags
slyps
Figure 7: When the expert differs in body or prop appearance, TRAIL outperforms GAIL.
8

---

Under review as a conference paper at ICLR 2020
5.4 EVALUATION ON DIVERSE MANIPULATION TASKS
To further demonstrate benefits of using our proposed method, we present results for TRAIL, TRAIL- the baseline GAIL, and BC on more challenging tasks. Specifically, we consider stack with the Sawyer robot; and insertion and stack banana in the Jaco sprke. The results are shown in Figure The tasks we consider here are much harder evidenced by the performance of BC agents. These experiments suggest that TRAIL is generally useful as an improvement over GAIL,, even when the tasks are not designed include task-irrelevant information.
Stack
Insertion
Stack banana
150
Nowards
Aggand GNIl
100-
100
TAAI
TBALL
00-
30
50
0-50
307
0.288
3.508
140
0.6e8
0.568
Actor
Actor steps
Figure 8: Results comparing TRAIL, TRAIL-0 and GAIL for diverse manipulation tasks.
CONCLUSIONS
To make adversarial imitation work on nontrivial tasks from pixels, crucial prevent the dis- criminator from exploiting task-irrelevant information. Data augmentation is necessary component that prevents the discriminator from overfitting, but alone is not sufficient to solve more challeng- ing problems. Our proposed method TRAIL proved to be remedy which effectively focuses the discriminator on the task even when task-irrelevant features are present.
We showed through ablations that TRAIL benefits from data augmentation, actor early stopping, and the use of invariant set constraints. The last enables our agent to solve the full task suite in our experiments showing its decisive importance.
REFERENCES
Pieter Abhecl and Andrew Ng. Apprenticeship learning via inverse reinforeement learning. Proceedings of the twenty-first international conference on Machine learning, pp. 1. ACM, 2004.
Paul Bakker and Yasuo Kuniyoshi. Robot see, robot do: An overview of robot imitation. In AISB96 Workshop on Learning in Robots and Animals, 1996.
Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-end differentiable adversarial imitation learning. In ICML, 2017.
Gabriel Martn, Matthew W Hoffmann David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804,0m 08617, 2018
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. arXiv preprint arXiv: 1rXivv:707.0000 06887, 2017.
Glen Berseth and Christopher Pal. Visual imitation learning with recurrent siamese networks. arXiv preprint 1eXiv:1901.07 08..077866.. 2019.
Lionel Blondé and Alexandros Kalousis. Sample-efficient imitation learning via generative adversarial nets. arXiv preprint arXiv:1809, 020.44 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: /51 15/1.1229, 2015.

---

Under review as a conference paper at ICLR 2020
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho. Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Advances neural information processing systems, pp. 1087-1098, 2017.
Chelsea Finn, Paul Christiano. Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-hased models. arXiv preprint 16X 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016b.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. arXiv preprini 1XXx:1100.00905, 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. In ICLR, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672 2602222660222 2014.
Todd Hester, Matej Vecerik, Olivier Pietquin. Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan. Andrew Sendonaris, Ian Osband, Gabriel Dulac- Aulac-Arnodd, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565 (663, 2016.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:/803.00090 2018.
Mitsuo Kawato, Francesca Gandolfo, Himaki Gomi, and Yasuhiro Wada. Teaching by showing in kendama based optimization principle. In ICANN. 1994.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arX:v: 14X.v:1412.66980 2014.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. In NIPS, 2017.
Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint 1707.0:707.020300 2017.
Hiroyuki Miyamoto, Stefan Schaal, Francesca Gandolfo, Hiroaki Gomi, Yasuharu Koike, Ricko Osu, Eri Nakano, Yasuhiro Wada, and Mitsuo Kawato. A kendama learning robot based bi-directional theory. Neural networks, 1996.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In leml, pp. 663-670, 2000.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. arXiv preprint arXiv: 181000081,, 2018
Tobias Pohlen, Bilal Piot, Todd Hester. Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Sarth-Maron Hado van Hasselt, John Quan, Mel Večerík, et al, Observe and look further: Achieving consistent performance on atari. arXiv preprint 1805. 1593, 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi- task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2018.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint arXiv: 170../0087 2017.
10

---

Under review as a conference paper at ICLR 2020
Scott Reed, Yusuf Aytar, Ziyu Wang. Tom Paine, Aäron den Oord, Tobias Pfaff, Sergio Gomez, Alexander Novikov, David Budden, and Oriol Vinyals. Visual imitation with minimal adversary. 2018
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell, A reduction of imitation learning and structured prediction no-regret
online online learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 2011.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. 2018.
Stefan Schaal. Learning from demonstration. In NIPS, 1997.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In JCML, 2014.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint urXiv:190.007855 2019.
Bradly C Stadie. Pieter Abbeel, and Ilya Sutskever. arXiv:1703.01700 2017.
Third-person imitation learning. arXiv preprint
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint 167.02260.. 2016.
Matej Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pictquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. CoRR, 2017.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Frez, Serkan Cabi, Saran Tunnaauvunakkol,, János Kramár, Raia Hadsell, Nando Freitas, et al, Reinforcement and imitation learning for diverse visuomotor skills. In RSS, 2018.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1438. Chicago, II, USA, 2008.

---

Under review as a conference paper at ICLR 2020
A SUPPLEMENTARY MATERIAL
A.1 DETAILED DESCRIPTION OF ENVIRONMENT
All our simulations are conducted using MuJoCd Todorov et al. 2012 We test our proposed
algorithms variety of different envrionments using simulated Kinova Jace and Sawyer robot arms see Figure We use the Robotiq 2F85 gripper in conjuction with the Sawyer arm.
Figure 9: Two work spaces, Jaco (left) which uses the Jaco arm and is 20 20 cm, and Sawyer (right) which uses the Sawyer arm and more closely resembles real robot cage and is 35 x 35 cm.
To provide demonstrations, we use the SpaceNavigator 3D motion controlle to set Cartesion velocities of the robot arm. The gripper actions are implemented via the buttons on the controller. All demonstrations in experiments are provided via human telcoperation and we collected 100 demonstrations for each experiment.
Jaco:
When using the Jaco arm, arm and all joints of the hand. The simulation is run with a numerical time step of 10 milliseconds, integrating steps, to get control frequency of 20HZ. The agent uses frontal camera of size 64 x 64 (see Figure a)). For full list of observations the agent sees, please refer to Table a).
joint velocity control (9DOF) where control all joints of
(a) FRONTAL CAMERA
(b) FRONT LEFT CAMERA
(c) FRONT RIGHT CAMERA
Figure 10: Illustration of the pixels inputs to the agent.
Sawyer: When using the Sawyer arm, we Cartesian velocity control (6DOF) for the robot arm and add one additional action for the gripper resulting in degrees of freedom. The simulation is run with a numerical time step of 10 milliseconds, integrating 10 steps, to get control frequency of 10HZ. The agent uses two frontal cameras of size 64 64 situated on the left and right side of the robot cage respectively (see Figure [0 c)). For full list of observations the agent sees, please refer to Table 2(b).
For all environments considered in this paper, we provide sparse rewards (i.e. if task is accomplished, the reward is 1 and () otherwise). In experiments regards our proposed methods, rewards are only used for evaluation purposes and not for training the agent.
Wone mi foco .ar.o....... are
nova 1a0008851attt wa- wwborttrrooboooooo 1p-arll
12

---

Under review as a conference paper at ICLR 2020
Table 2: Observation and dimensions.
a) Jaco
b) Sawyer
Feature Name
Dimensions
Feature Name
Dimensions
Trontal camera
64 64
3
front left camera
64 x 64 x
base force and torque sensors
front right camera arm joint position arm joint velocity wrist force sensor wrist torque sensor hand grasp sensor hand joint position tool center point cartesian orientation tool center point cartesian position hand joint velocity
64 x
arm joints position
6
arm joints velocity
6
7
wrist force and torque sensors
6
3
hand finger joints position
3
hand finger joints velocity
3
hand fingertip sensors
3
grip site position pinch site position
3
9
3
A.2 EARLY FRAMES ABLATION STUDY
Here, we vary the number of early frames of each episode used to form I. We report that a range of values from 1 up 20 works well across both tasks (Put in box and Stack banana), with performance gradually degrading as the value increased beyond 20 (Fig. TTb.
Stack
first frames
Rowords
1+7
2+7
167
260
3+7
4x7
Actor
Actor steps
Figure 11: TRAIL performance, varying the number of first frames each episode used form I.
A.3 TRAIL-O) WITH random
We found out and mentioned the subsection [.] that random benefits from TRAIL-O, Unfortunately. it is still prone to overfitting and hence, worse than our full method - TRAIL. We present random + TRAIL-O accompanied with our methods in Figure 12 Our TRAIL and random TRAIL-O are the only methods exceeding BC performance lift distracted. However, TRAIL performance is clearly better (obtains higher rewards and never gets overfitted).
Lift slone
Lift distracted
Lif cistrac seeded
150
132
legend Ball
1BC 8:
03298
167
3,207
157
2
Acter steps
Acter
Figure 12: Results for lift alone, lift distracted, and lift distracted seeded.
A.4 FIXED TERMINATION POLICY
As mentioned in the subsection 5.2 the most basic early termination policy fixed step termination may be very effective if tuned. Since the tuning may be expensive in practice, recommend using adaptive early stopping (TRAIL-O). However, for the sake of completeness we provide results for fixed step termination policy depending on the hyperparameter tuned. The results for stack task
13

---

Under review as a conference paper at ICLR 2020
presented in Figure 13
The Jaco work space is considered here because Sawyer requires TRAIL to obtain high rewards. As can be inferred from the figure, the performance is very sensitive to the fixed step hyperparameter. We refer to subsection 5.2 for more comments all methods.
Stack
200
legend
D4PDID only
Rewards
rewards based
100-
stop (20)
Fixed step (30)
30
Fixed step (50)
TRAIL-0
Actor steps
Figure 13: Results for stack in Jaco work space. Fixed step termination policy can be very effective but the final performance is very sensitive the hyperparameter. TRAIL-O does need tuning nor access to the environment reward.
A.5 DATA AUGMENTATION
An extra set of experiments on lift alone and lift distracted tasks (described in subsection 5.1 has been performed to show importance of data augmentation.
Because in the subsection 5.2 only the peak performance is presented (Table 5.2), we present here the full curves in Figure [4| The results shows that data augmentation necessary obtain high rewards in lift distracted. For easier lift alone, TRAIL-0 with data augmentation perform at par with TRAIL..
Lift alone
Lift distracted
150
150
legend
Rewards
TRAIL
BC
TRAIL tmo augmentation)
TRAIL-0
0-
165
2w5
3e5
4e5
565
166
2e6
3e6
deB
5e6
Actor steps
Actor steps
Figure 14: Results for lift alone and lift distracted in Sawyer work space. TRAIL and TRAIL-O by default with data augmentation. Additional results for these methods without data augmentation are presented show its importance.
A.6 D4PG
We use D4PG (Barth-Maron et al. 2018) as our main training algorithm. Briefly, D4PG is distributed off-policy reinforcement learning algorithm for continuous control problems. In nutshell, D4PG uses Q-learning for policy evaluation and Deterministic Policy Gradients (DPG) (Silver et al.) 2014) for policy optimization.
important characteristic D4PG that it maintains a replay memory M (possibility prioritized (Horgan et al. 22018) that stores SARS tuples which allows for off-policy learning. D4PG also adopts target networks for increased training stability. In addition to these principles, D4PG utilized distributed training, distributional value functions, and multi-ster returns to further increase efficiency and stability. In this section, we explain the different ingredients of D4PG.
D4PG maintains online value network Q(s, a)0) and online policy network (T)))). The target networks are the same structures as the value and policy network, but are parameterized by different parameters 0' and o' which are periodically updated the current parameters of the online networks.
14

---

Under review as a conference paper at ICLR 2020
Given the Q function, update the policy using DPG:
= =
(5)
Instead of using scalar Q function, D4PG adopts a distributional value function such that Q(s,, E (8e, a [ where 7 is random variable such that 2; w.p. Pi o exp(w(Se, a ()). The z,'s take on Voins discrete values that ranges uniformly between V Vini and Vmaz such that 2; = Min + //max- -Viin for i E {0. (0..... Voins - 1}.
Voins
To construct a bootstrap target, D4PG uses N-step returns. Given sampled tuple from the replay St+N, we construct new random variable Z' such that Z' 24 + En= T w.p. Pt exp(w( (t+NN ((s++Nll')) (0))) Notice, Z' no longer has the same support. We therefore adopt the same projection d employed by Bellemare et al. (2017). The training loss for the value function
memory; St, at, (Te, Te+1s"- -
[()) = E....a tra. -eN hh (1tp://a._....oo//0//2002/000000
(6)
where is the cross entropy.
D4PG is also distributed following Horgan et al. (2018). Since all learning processes only rely on the replay memory. we can easily decouple the 'actors' from the 'learners'. D4PG therefore uses a large number of independent actor processes which act in the environment and write data central replay memory process. The learners could then draw samples from the replay memory for learning. The learner also serves parameter server to the actors which periodically update their policy parameters from the learner.
In our experiments, always have access expert demonstrations. We, therefore adopt the practice from DQID and DDPGID and put the demonstrations into our replay buffers. For more details see Algorithms
2
Algorithm Actor
Given:
an experience replay memory M
for n Repisodes do
for 1 to 1' do
Sample action from task policy: +- T(s))
Execute action a, and observe new state 8t-1, and reward rs. Store transition (St, at, Pt,, St-1) in memory M
end for
end for
Algorithm 2 Learner
Given:
an off-policy RL algorithm A
a replay buffer M
a replay buffer of expert demonstrations. M.
Initialize A.
for Nupdates do
Sample transitions (St, dee 8t41) from M to make minibatch B.
Sample transitions (st, de, Pe, 8141) from Me enlarge the minibatch
Perform actor update step with Eqn. (5)
Perform a critic update step with Eqn.
Update the target actor/critic networks every l steps.
end for
15

---

Under review as a conference paper at ICLR 2020
A.7 NETWORK ARCHITECTURE AND HYPERPARAMETERS
Actor and critic share residual pixel encoder network with eight convolutional layers (3x3 convolu- tions, three 2-layer blocks with 16, 32, 32 channels), instance normalization (Ulannov et al. 2016) and exponential linear units (Clevert et al. 2015) between layers.
The policy is a 3-layer MLP with ReLU activations with hidden layer sizes (300, 200). The critic is 3-layer MLP with ReLU activations with hidden layer sizes (400, 300). For a illustration of the network. Please see Figure 15
Aztione and proppic
Head
3x5 Comv 16
Instraos Norm
Propria
Now Pood Iistance Nor.
Pritcy Head
12M
RD layers
Layer Norm
Figure 15: Network architecture for the policy and critic.
The discriminator network uses pixel encoder of the architecture the actor critic, followed by a 3-layer MLP with ReLU activations and hidden layer sizes (32, 32).
Parameters
Values
Actor/Critic Input Width
64
Actor/Critic Input Height
64
D4PG Parameters
Vmnn
-50 -50
Vmaa
150
Voins
21
N step
1
Actor learning rate
10-4
Critic learning rate
10-4
Optimizer
Adam
Kingma & Ba (014))
Batch size
256
Target update period
100
Discount factor (r)
0.99
Replay capacity
106
Number of actors
32 or 128
Imitation Parameters
Discriminator learning rate
10-4
Discriminator Input Width
48
Discriminator Input Height
48
Table 3: Hyper parameters used in robot manipulation experiments.
A.8 COMPARING AGAINST LEARNING WITH FIXED REWARDS
To check whether learned discriminators are needed, we compare TRAIL against agents using a fixed reward function corresponding to Rexpert 1 and Regent 0 for the lift alone and lift distracted tasks. This baseline simulates an oracle discriminator with perfect generalization, but which is agnostic behavior. On the lift alone task, agents using this fixed reward achieve roughly half the
16

---

Under review as a conference paper at ICLR 2020
reward of TRAIL asymptotically. and on lift distracted they do not solve the task (average rewards are less than 5). The learning curves are presented in Figure [6
Lift
150
legend
Rewards
alone TRAIL, distracted
100
35C
0-
0.267
0.de7
0.6e7
0.8w7
1s?
Actor steps
Figure 16: With fixed rewards, the agent is able learn lift alone somewhat, but performs worse than TRAIL. When distractor blocks are added, the fixed reward agent fails to learn completely.
17


Reviews:
[{"review_id": "r1e0ftvNqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the acute problem of irrelevant discrimination in adversarial imitation learning methods. Ideally, the discriminator in such methods should extract task-dependent features and base its discrimination rule upon them only. However, in practice, the opposite usually occurs. For example, if the state distributions of the expert and agent come from different modalities, e.g., different background light, different POV camera, etc., then almost surely the discriminator will base its classification rule on such differences. In turn, the reward signal will be (constant and) irrelevant. This is a well-known problem, that indeed received little attention in previous work. \nAs a solution, the authors propose to maintain an invariance set representing task-irrelevant states. For example, states of a random policy or states from the beginning of an episode. The authors suggest using the invariance set as a regularization mechanism to the discriminator by punishing it for correctly classifying examples from the invariance set. The idea behind this regularization is to distinguish between expert examples that actually represent the task and expert examples that do not. This is achieved by punishing for the discriminator's accuracy over the invariance set.\nIn addition, the authors present two additional improvements: data augmentation and early actor stopping.\nThe paper.\n\n1. I very much like the problem the authors choose to address. I believe that this is a dominant problem that received little attention so far. The reason is that so far, GAIL style methods used experts and agents that come from the same domains (same simulation), thus \"hiding\" such problems. As experiments scale up to real-world setups, this is no longer the case and such problems arise.\n\n2. Overall, I find the experiments section very confusing. The authors propose a formal improvement (the invariance set) which they apply to their agent, as well as two informal improvements (actor early stopping and data augmentation) which they apply selectively for their agent and the baseline methods. As a result, there exist a large number of variations, according to what improvements are applied to what method. Overall, I find it hard to follow the results and understand the contribution of the different improvements.\n\n3. It is not clear if TRAIL and GAIL are using the same architecture and algorithm. Apparently, the authors could have used GAIL's original policy gradient algorithm with their invariance set modification. If an improvement was evident in such a comparison, then it would have been easier for me to understand the significance of the proposed method. Currently, there are too many differences between the baseline and TRAIL to clearly say that the invariance set made the difference.\n\n4. It is not clear why the constraint in Eq. 2 is placed over the sum of the two terms and not on each term separately since each term is defined over a different state distribution. What would you expect to see different if the constraint was placed on the two terms separately?\n\n5. In the same matter, why do you need the 0.5 factor?\n\n6. The invariance set is referred to extensively before it is first defined. Therefore, the reader is forced to toggle back and forth multiple times in section 4 before understanding equations 2 and 3. Consider giving at least an intuition about the invariance set beforehand to make the reading more fluent.\n\n7. The conclusions section is minimal to non-existing while there are many conclusions to draw from the experiments.\n\n8. Data augmentation was previously studied in the context of adversarial imitation. See for example \"Visual imitation with reinforcement learning using recurrent Siamese networks\" by Berseth et.al.\n\n9. Overall assessment: among the three modifications the authors propose, I would be glad to see if the invariance set modification alone can make a difference. The other two modifications are either not novel (see Berseth et.al.), or highly heuristic (\"...we restart an episode if the discriminator score at the current step exceeds the median score of the episode so far for T_patience consecutive steps...\"). However, it seems that the informal improvements (augmentation and early stopping) play a crucial role in the success of TRAIL (... It drastically improves the baseline GAIL agent, and is necessary to solve any of the harder tasks...\"). It is absolutely fine that the authors stress the importance of the informal improvements but it seems that they are dominating the contribution of the main ingredient.\n\nI also find it problematic that the construction of the invariance set is not straightforward (\"...the selection of the invariance set is a design choice...\"). When added to the way that the data augmentation is carried (sensor dependent), and to the fact the the early stopping mechanism seems dubious for some tasks (I can think of several tasks where early stopping will not work), this method is far from being a \"plug-and-play\" solution. I.e., the paper offers general guidelines on how to solve imitation tasks but not a concise algorithm.\nIn order for me to vote for acceptance, I need the paper to be more concise. In its current state, I see it more as general guidelines for training adversarial imitation algorithms.\n\nHowever, as said in the beginning, I very much believe in the problem that the authors present and encourage them to provide an experiment where the only modification between the baseline and the proposed method is the usage of an invariance set. Success in such an experiment will clearly change my mind.\n\nThere are also some presentation issues, e.g cluttered graphs, that can be addressed.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Sye2efgAtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an extension of the adversarial imitation learning framework where the discriminator is additionally incentivized not to distinguish frames that are different between the expert and the agent in irrelevant ways. The method relies on manually identifying a set of irrelevant frames.\n\nThe paper correctly identifies an important shortcoming of GAIL and proposes a sensible and generic way to overcome it. The experiments are well designed and corroborate the claims made in the paper.\n\nEDIT: I acknowledge reading the other reviews and the author response and stand by my initial assessment.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gMweDDcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE:\nI acknowledge that I‘ve read the author responses as well as other reviews. \nI would keep my score unchanged, i.e. at 6 Weak Accept. \n\n\n#####\n\nThis work considers imitation learning, i.e. the problem of learning an unknown reward function from expert demonstrations. The paper proposes Task-Relevant Adversarial Imitation Learning (TRAIL), a modification of Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) which introduces an additional constraint to the discriminator optimization, namely that the discriminator must not be able to successfully discriminate between expert and agent samples on some predetermined invariant set $\\mathcal{I}$. Technically this constraint is realized with an additional term added to the standard GAIL objective that minimizes the negative (i.e. reversed) discriminator objective whenever discrimination accuracy exceeds 50% (i.e. randomness) on $\\mathcal{I}$. Given that the invariant set $\\mathcal{I}$ includes environment samples not relevant for the task, TRAIL thus aims to prevent discriminator overfitting to task-irrelevant features (e.g. object or agent appearance) and regularizes the discriminator to extract more salient, task-relevant features of expert behavior.\nIn experiments on various robot manipulation tasks (Block lifting/stacking/insertion with and without distraction), the paper shows that TRAIL outperforms Behavioral Cloning (BC), standard GAIL, Deterministic Policy Gradients from Demonstrations (DPGfD) as well as additionally introduced improved GAIL baselines that employ common regularization techniques (early stopping and data augmentation).\n\nI think this work could be accepted since it clearly demonstrates a key issue of GAIL (tendency to overfit to task-irrelevant features) and proposes a simple solution for this issue that empirically proves to be effective. I especially appreciate the comprehensive ablation studies that isolate effects from data augmentation and actor early stopping. The paper overall is well written and easy to follow. \nI must admit, however, that I am not too familiar with this line of research and might be missing some context, e.g. recent competitors.\n\nSome questions that I have:\n(1) Could you elaborate more on how to select the invariant set $\\mathcal{I}$ in practice? The paper proposes random policies or initial frames, but as I understand experiments are only reported for the latter design choice. How would performance differ for different choices? What if the invariant set $\\mathcal{I}$ is misspecified, i.e. includes task-relevant samples? Would TRAIL be robust?\n(2) How is the invariant set $\\mathcal{I}$ chosen for the stack, insertion, and stack banana experiments in Figure 8?\n(3) What if actions would be included in objective (2) again?\n\n\n#####\nMinor comments\n- Is the legend in Figure 14 wrong? Table 1 results and the plots in Figure 14 are inconsistent (e.g. on Lift distracted \"TRAIL\" is best in Figure 14 which should be \"TRAIL - augmentation\" according to Table 1.)\n- Section 5.1: \"Finally, to disentangle the importance of actor early stopping, ...\" >> \"disentangle\" somewhat is a signal word in ML that readers directly associate with representation learning. Maybe just \"show\"/\"infer\"/\"understand\".\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to improve adversarial imitation learning (GAIL) by encouraging the discriminator to focus on task-dependent features.

An advantage of this paper is that it not only improves upon GAIL, but it is doing so after first demonstrating and analyzing an existing issue. 

On the other hand, the presentation of the paper and breadth of experiments could be significantly improved further than the updated version. It would also be necessary to clarify whether the baseline is vanilla PG or D4PG.

A major point for discussion was the selection of the invariance set. The ablation studies and explanation provided during the rebuttal period towards this point are helpful, but somehow we still do not have the full picture to understand well how this method compares to existing literature.

Decision: Reject