Title: TASK-RELEVANT ADVERSARIAL IMITATION LEARNING

Abstract: We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent. We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL). Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features. Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such as behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly proposed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to find solutions, even when the other agents have access to task reward.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) have produced breath-taking conditional image synthesis results (Goodfellow et al., 2014; Brock et al., 2019) , and have inspired adversarial learning approaches to imitating behavior. In Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) , a discriminator network is trained to distinguish agent and expert behaviour through its observations, and is then used as a reward function. GAIL agents can overcome the exploration challenge by taking advantage of expert demonstrations, while also achieving high asymptotic performance by learning from agent experience.

Despite the huge promise of GAIL, it has not yet had the same impact as GANs; in particular, robust GAIL from pixels for control applications remains a challenge. Here, we study a key shortcoming of GAIL: the tendency of the discriminator to mainly exploit task-irrelevant features. For example, by focusing on slight background differences, a discriminator can achieve perfect generalization, assigning zero reward to all held-out agent observations. However, this discriminator does not yield an informative reward function because it ignores behavior. Due to robustness to initial conditions, TRAIL can stack from pixels while standard GAIL fails (c). We witness this difference again in insertion with distractors (d). A video showing agents performing these tasks can be seen at https://youtu.be/Rz5G15rDKcg .

Expert Demo Agent Episode:

Changing distractor props Changing agent appearance Changing object appearance

Figure 2 : Illustration of several task-irrelevant changes between the expert demonstrations and the distribution of agent observations, for the lift (red cube) task. The naively-trained discriminator network will use these differences rather than task performance to distinguish agent and expert.

Assuming there is an expert policy π E that is optimal for an unknown reward function, here we refer to a feature as task-irrelevant if it does not affect that reward. For example, if the task is to lift a red block, the positions of other blocks would be task-irrelevant; see Figures 1 and 2 .

This paper makes the following contributions:

1. It reveals a fundamental limitation of GAIL by showing that discriminators do in practice exploit task-irrelevant information, thereby resulting in poor task performance. 2. It introduces powerful GAIL baselines. In particular, it shows that standard regularization and data augmentation are generally useful and improve upon standard GAIL. 3. It shows that these improvements to GAIL, as well as other improvements proposed by Reed et al. (2018) , do not completely solve the problem, allowing GAIL agents to fail catastrophically with the addition of task-irrelevant distractors. 4. It introduces Task Relevant Adversarial Imitation Learning (TRAIL), using constrained optimization to force the discriminator to focus on the relevant aspects of the task, which improves performance dramatically on manipulation tasks from pixels (see Figure 1 ).

2. RELATED WORK:

The use of demonstrations to help agent training has been studied extensively in robotics (Bakker & Kuniyoshi, 1996; Kawato et al., 1994; Miyamoto et al., 1996) with approaches ranging from Q-learning (Schaal, 1997) to behavioral cloning (BC) (Pomerleau, 1989) .

BC: BC is effective in solving many control problems (Pomerleau, 1989; Finn et al., 2017; Duan et al., 2017; Rahmatizadeh et al., 2018) . It has also been successfully applied to initialize RL training (Rajeswaran et al., 2017) . It, however, suffers from compounding errors as initial small deviations from the expert behaviors tend to cause bigger differences (Ross et al., 2011) . This often necessitates a large number of demonstrations for satisfactory performance. Furthermore, BC typically does not lead to agents that are superior to their demonstrators.

Inverse RL: Ziebart et al. (2008); Ng et al. (2000) ; Abbeel & Ng (2004) propose inverse reinforcement learning (IRL) as a way of learning reward functions from demonstrations. Reinforcement learning can then be used to optimize that learned reward. Recently, Finn et al. (2016b) approached continuous robotic control problems with success by applying Maximum Entropy IRL algorithms which are very closely related to GAIL (Finn et al., 2016a) and have similar drawbacks.

Learning from Demonstrations: Hester et al. (2018) developed deep Q-Learning from demonstration (DQfD), in which expert trajectories are added to experience replay and jointly used to train agents along with their own experiences. This was later extended by Vecerik et al. (2017) and Pohlen et al. (2018) to better handle sparse-reward problems in control and Atari games respectively. Despite their efficiency, this class of methods still requires access to rewards in order to learn.

GAIL: Following the success of Generative Adversarial Networks (Goodfellow et al., 2014) in image generation, GAIL (Ho & Ermon, 2016) applies adversarial learning to the problem of imitation. Although many variants are introduced in the literature (Li et al., 2017; Fu et al., 2018; Merel et al., 2017; Zhu et al., 2018; Baram et al., 2017) , making GAIL work for high-dimensional input spaces, particularly raw pixels, remains a challenge.

A few papers (Peng et al., 2018; Reed et al., 2018; Blondé & Kalousis, 2018) seek to address the problem of overfitting the discriminator. Peng et al. (2018) introduces the Variational Bottleneck to regularize the discriminator. Reed et al. (2018) proposes to not train the vision module of the discriminator (e.g. use the vision module of the critic network instead) and only train a tiny network on top of the vision module to discriminate. Blondé & Kalousis (2018) follow a similar approach. Unstructured regularization, however, cannot stop the discriminator from fitting to features that are systematically different between the agents' and the demonstration's behavior, like those illustrated in Figure 2 . TRAIL, on the other hand, is much less prone to overfitting to these features. Stadie et al. (2017) extend GAIL to the setting of third person imitation, in which the demonstrator and agent observations come from different views. To prevent the discriminator from discriminating based on viewpoint domain, they use gradient flipping from an auxiliary classifier to learn domain-invariant features. Our approach is not to learn domain-invariant features, but instead learn domain-agnostic discriminators that only focus on behavior.

Several recent works have focused on improving the sample efficiency of GAIL (Blondé & Kalousis, 2018; Sasaki et al., 2018) . Common to these approaches and to this work, is the use of off-policy actor critic agents and experience replay, to improve the utilization of available experience.

3. REINFORCEMENT LEARNING AND ADVERSARIAL IMITATION:

Following the notation of Sutton & Barto (2018), a Markov Decision Process (MDP) is a tuple (S, A, R, P, γ) with states S, actions A, reward function R(s, a), transition distribution P (s |s, a), and discount γ. An agent in state s ∈ S takes action a ∈ A according to its policy π and moves to state s ∈ S according to the transition distribution. The goal of RL algorithms is to find a policy that maximizes the expected sum of discounted rewards, represented by the action value function

Q π (s, a) = E π [ ∞ t=0 γ t R(s t , a t )]

, where E π is an expectation over trajectories starting from s 0 = s and taking action a 0 = a and thereafter running the policy π.

To apply RL, it is essential that we have access to the reward function which is often hard to design and evaluate (Singh et al., 2019) . In addition, sparse rewards can cause exploration difficulties that pose great challenges to RL algorithms. We therefore look to imitation learning and particularly GAIL to derive a reward function from expert demonstrations. In GAIL, a reward function is learned by training a discriminator network D(s, a) to distinguish between agent and expert state-action pairs. The GAIL objective is thus formulated as follows:

EQUATION

where π is the agent policy, π E the expert policy, and H(π) an (optional) entropy regularizer. The reward function is defined simply: R(s, a) = -log(1 -D(s, a)).

GAIL is theoretically appealing and practically simple. The discriminator, however, can focus on any features to discriminate, whether these features are task-relevant or not. In the next subsection we describe a way to constrain the discriminator network in order to prevent it from using task-irrelevant details to distinguish agent and expert data.

4. TASK-RELEVANT ADVERSARIAL IMITATION LEARNING (TRAIL):

We want the discriminator to focus on task-relevant features. Our proposed solution, TRAIL, prevents the discriminator from being able to distinguish expert and agent behaviour based on selected aspects of the data. For instance, the discriminator should distinguish agent and expert frames only when meaningful behavior is present in those frames. In the absence of behavior useful to solve the task, e.g. in initial frames prior to the execution of the behavior, the discriminator should be agnostic.

To this end, we propose a constraint for the discriminator, such that its accuracy must not be greater than chance on a particular set of observations I that we will call the invariant set. I will include only observations that can be distinguished as agent or expert in task-irrelevant ways. Precisely, we formulate TRAIL in terms of the following constrained optimization problem:

max ψ E s∼π E [log D ψ (s)] + E s∼π θ [log(1 -D ψ (s))] (2) s.t. 1 2 E s∼π E 1 D ψ (s)≥ 1 2 | s ∈ I + 1 2 E s∼π θ 1 D ψ (s)< 1 2 | s ∈ I ≤ 1 2 .

To apply the above constraint in practice, we can optimize the reverse of the objective function on I.

That is, given a batch of N examples s e ∼ π E , s θ ∼ π θ from the expert and agent, and ŝe ∼ π E and ŝθ ∼ π θ both in the set I, we maximize the following augmented objective function:

L ψ (s e , s θ , ŝe , ŝθ ) = N i=1 log D ψ s (i) e + log 1 -D ψ s (i) θ (3) -λ N i=1 log D ψ ŝ(i) e + log 1 -D ψ ŝ(i) θ 1 accuracy(ŝe,ŝ θ )≥ 1 2

, where accuracy(•, •) is defined as the average of discriminator accuracies:

EQUATION

)

The scalar λ ≥ 0 is a tunable hyperparameter.

We used single frames as states but sequences can also be used, and we do not require expert actions. Not requiring actions enables learning in very off-policy settings, where the action dimensions and distributions of the demonstrator (another robot or human) are different from those of the agent.

4.1. THE SELECTION OF THE INVARIANT SET:

The selection of the invariant set I is a design choice. In general, we could always contrive nonstationary and adversarial ways of making this choice difficult. However, we argue that in many situations of great interest, including our robotic manipulation setup, it is easy to propose effective and very general invariant sets.

A straightforward way to collect robot data is to execute a random policy. We can then use the resulting random episodes, for both expert and agent, to construct the invariant set I. Another way to construct I is to use early frames from both expert and agent episodes. Since in early frames little or no task behavior is apparent, this strategy turns out to be effective and no extra data has to be collected. This strategy also improves robustness with respect to variation in the initial conditions of the task; see for example block insertion in Figure 1(d) .

Importantly, if the set I captures some forms of irrelevance but not all forms, it will nonetheless always help in improving performance. In this regard, TRAIL will dominate its GAIL predecessor whenever the designer has some prior on what aspects of the data might be task irrelevant.

5. EXPERIMENTS:

We focus on solving robot manipulation tasks. The environment implements two work-spaces: one with a Kinova Jaco arm (Jaco), and the other with a Sawyer arm (Sawyer). See supplementary material A.1 for a detailed description. Environment rewards, which are not used by GAIL-based methods, are sparse and equal to +1 for each step when a given task is solved and 0 otherwise. The maximum reward for the episode is 200, since this is the length of a single evaluation episode.

Our agent is based on the off-policy D4PG algorithm (Barth-Maron et al., 2018) because of its stability and data-efficiency (see supplementary material A.6). Following Vecerik et al. (2017) , we add expert demonstrations into the agents' experience replay, and refer to the resulting RL algorithm as D4PG from Demonstrations (D4PGfD). For each task we collect 100 human demonstrations.

Data augmentation:

Traditional data augmentation has proved beneficial in imitation learning (Berseth & Pal, 2019) . Surprisingly, to the best of our knowledge, this has not been explicitly studied in prior publications on GAIL. However, we find that data augmentation is a generally useful component to prevent discriminator overfitting. It drastically improves the baseline GAIL agent, and is necessary to solve any of the harder manipulation tasks. We distort images by randomly changing brightness, contrast and saturation; random cropping and rotation; adding Gaussian noise. When multiple sensor inputs are available (e.g. multiple cameras), we also randomly drop out these inputs, but leaving at least one active.

All discriminator-based methods in this work (including all baselines) use data augmentation except as indicated in the relevant ablations (see Section 5.2). We also considered regularizing the GAIL discriminator with spectral normalization (Miyato et al., 2018) . It performed slightly better than GAIL, but still failed in the presence of distractor objects, and we thus omit spectral normalization in the main experiments for simplicity.

Actor early stopping When the agent has learned the desired behavior, and the resulting data is used for training, the discriminator will become unable to distinguish expert and agent observations based only on behavior. This forces the discriminator to rely on task-irrelevant information.

To avoid this scenario, we propose to restart each actor episode after a certain number of steps such that successful behavior is rarely represented in agent data. This enables the discriminator to recognize the goal condition, which appears frequently at the end of demonstration episodes, as representative of expert behavior. To avoid hand-tuning the stopping step number, we found that the discriminator score can be used to derive an adaptive stopping criterion. Concretely, we restart an episode if the discriminator score at the current step exceeds the median score of the episode so far for T patience consecutive steps (in practice we set T patience = 10).

We set invariant set hyperparameter λ = 1 and use adaptive early stopping for TRAIL. The ablation with λ = 0, i.e. adaptive early stopping only, is referred to as TRAIL-0. Hence, the only difference between TRAIL-0 and TRAIL is that the later uses invariant set constraints.

5.1. BLOCK LIFTING WITH DISTRACTORS:

In this section, we consider two variants of the lift task in the Sawyer work space: a) lift alone, where only one red cube is present, b) lift distracted, with two extra blocks (blue and green, see Figure 9 ). We show how adding these additional distractors affects the training procedure.

We first compare our method to baselines. In doing so, the invariant set is constructed using the first 10 frames from every episode, as in the rest of the work. This choice does not require us to collect any extra data, and hence the comparison with baselines is fair. In the next section, we elaborate on the choice of the invariant set and provide additional experimental results.

As baselines, we run BC and GAIL (with data augmentation). Baseline GAIL is the strongest we were able to implement. The only difference between this baseline and TRAIL is the use of invariant set constrains and actor early stopping (code, agent configurations like number of actors, and network architecture are the same). We additionally consider the approaches proposed by Reed et al. (2018) as GAIL-based baselines; using either a randomly initialized convolutional network, or a convolutional critic network, to provide fixed vision features on top of which a tiny discriminator network is trained.

We call these two baselines random and critic respectively. Finally, to show the importance of actor early stopping, we run TRAIL-0 (Fig. 3 ).

Figure 3 : Results for lift alone, lift distracted, and lift distracted seeded. Only TRAIL excels.

All methods perform satisfactorily on lift alone, but the proposed methods TRAIL-0 and TRAIL do best. As expected, the performance of BC on lift distracted is similar to its performance on lift alone, despite the two additional blocks. The two additional blocks in lift distracted affect the GAIL-based baselines, despite being irrelevant to the task.

To understand this effect, we conducted an additional experiment (lift distracted seeded). Here the initial block positions are randomly drawn from the expert demonstrations. Therefore, it is impossible to discriminate between expert and actor episodes using the first few frames of an episode. Note this initialization procedure is not applied to the evaluation actor, keeping the evaluation scores comparable between lift distracted and lift distracted seeded.

This experiment exposes one major culprit behind the performance degradation of GAIL: memorization. The discriminator can achieve perfect accuracy by memorizing all 100 initial positions from the demonstration set, making the reward function uninformative. By constraining the discriminator, TRAIL squeezes out this irrelevant information and succeeds in solving the task in the presence of distractions. TRAIL is the only method that is able to handle the variety of initial cube positions during training, achieving better than expert performance on lift distracted.

Interestingly, random performs reasonably on lift distracted. Given random's strong performance, we conducted additional experiments to evaluate its effectiveness when trained with adaptive early stopping and present the results in Figure 12 of the supplementary material.

CONSTRUCTING THE INVARIANT SET I:

In the previous subsection, early frames were used to construct the invariant set (TRAIL-early). Here, we evaluate another previously mentioned approach for constructing I; random policy (TRAILrandom). The lift distracted task caused all baselines to fail, but was solved by TRAIL. We introduce a harder version of the task, where the expert appearance is different, to tease out the differences between TRAIL-early and TRAIL-random. The difference in appearance between the expert and imitator allows the GAIL discriminator to trivially distinguish them. The results and the differences in the expert appearance are presented in Figure 4 . The new task is indeed harder and it takes longer for TRAIL methods to achieve performance better than BC baseline, which is not affected by the different body appearance. GAIL is clearly outperformed and does not take off.

The difference between TRAIL methods is negligible. We also tried to mix them but the differences remain imperceptible. Hence, in the following experiments we simply use early frames. This choice is pragmatic as it does not require that we collect any extra data, and hence the comparison with GAIL and other baselines is fair. It is also very simple to apply in practice, even if one does not have access to the expert setup anymore. Finally, it is general and powerful enough to be successfully used across all robotic manipulation tasks considered in this work.

To decide how many initial frames should be used to construct I, we conducted an ablation study and found out that the method is not very sensitive to this choice (see supplementary material A.2). Hence, we chose 10 initial frames, and intentionally used the same number for all tasks to further emphasize generality of this choice.

5.2. ABLATION STUDIES MEASURING DISCRIMINATOR MEMORIZATION OF TASK-IRRELEVANT FEATURES:

In this section, we experimentally confirm that memorization is a limiting factor of GAIL methods. We equipped the discriminator with two extra heads whose inputs are the final spatial layer of the ResNet for the lift distracted task. The first head is trained on the first frames only, and has the same target as the main head (i.e. discriminating between agent and expert). To train the second head, we randomly divide the expert demonstrations into two equi-numerous subsets, and the task is to predict to which of these randomly chosen sets the demonstration was assigned to. Both heads are trained via backpropagation but their gradient is not propagated to the ResNet so they do not influence the training procedure directly.

If our claim is correct, we expect the extra heads to have higher accuracy for TRAIL-0 as compared to TRAIL, since TRAIL representations are penalized for having features triggering memorization, i.e. the features should not aid discriminating based on the first frames or predicting a random label for each expert demonstration. No reasonably performing method is able to force 50% accuracy for extra heads since some features are important to solve the task (e.g. the position of the red cube).

We also collected 25 extra holdout demonstrations and visualize the average discriminator prediction on them, and compare with predictions on the training demonstrations. In Fig. 5 , we see the overall accuracy of the main head for TRAIL-0 is significantly higher, and the difference is larger for early steps (when TRAIL achieves only 50%, as expected). TRAIL representations are less helpful for extra heads (see Figure 5 C-D ). Finally, TRAIL-0 clearly overfits on training demonstrations, predicting almost the maximum score, while only ∼ 0.25 is predicted for holdout demonstrations. The TRAIL average predictions for both datasets are almost identical.

ACTOR EARLY STOPPING (TRAIL-0)

In this section, we analyze the importance of adaptive early stopping on 3 tasks in the Jaco workspace: lift red cube (lift), put red cube in box (box), and stack red cube on blue cube (stack). We consider D4PGfD and three GAIL-based models with varying termination policies: a) fixed step (50), b) based on ground truth task rewards, and c) based on adaptive early stopping (TRAIL-0). Using ground truth task rewards, an episode is terminated if the reward at the current step exceeds the median reward of the episode so far for 10 consecutive steps. Results are presented in Figure 6 .

Termination based on task reward is clearly superior; although unrealistic in practice, it defines the performance upper-bound and clearly shows that early stopping is beneficial. TRAIL-0 is robust and reaches human performance on all tasks. A fixed termination policy, when tuned, can be very effective. The same fixed termination step, however, does not work for all tasks. See Figure 13 in supplementary material for the effects of varying termination steps. Finally, as can be inferred from stack results, the dense rewards provided by TRAIL-0 are helpful in solving this challenging problem which is unsolved with D4PGfD even though D4PGfD uses ground truth task rewards. In Table 5 .2, we report the best reward obtained in the first 12 hours of training (averaged for all seeds; see Figure 14 for full curves). The results show that data augmentation is needed in lift distracted.

For the lift alone task, TRAIL-0 with data augmentation performs on par with TRAIL.

Task Regularization Data augmentation No data augmentation

lift alone TRAIL-0 ∼165 ∼115 TRAIL ∼155 ∼165 lift distracted TRAIL-0 ∼30 ∼5 TRAIL ∼180 ∼10

Table 1 : Influence of data augmentation (evaluated on rewards) for lift alone and lift distracted.

The performance of TRAIL is not affected by the lack of data augmentation on the lift alone task, whereas the performance of TRAIL-0 is.

LEARNING WITH A FIXED, PERFECT DISCRIMINATOR:

To assess whether learned discriminators are necessary, we compare TRAIL against agents using a fixed reward function corresponding to R expert = 1 and R agent = 0 for the lift alone and lift distracted tasks. This baseline simulates an oracle discriminator with perfect generalization, but which is agnostic to behavior. On the lift alone task, agents using this fixed reward achieve roughly half the reward of TRAIL asymptotically, and on lift distracted they do not solve the task (average rewards are less than 5). See supplementary Figure 16 for learning curves.

5.3. LEARNING FROM OTHER EMBODIMENTS AND PROPS:

Since the TRAIL discriminator is trained to ignore task-irrelevant features, it can learn from demonstrations with different embodiments and props. Figure 7 shows that GAIL even with augmentation fails to learn block lifting from a different embodiment, and performs worse when the expert uses a different prop color. TRAIL solves the task and achieves better performance in both cases. 

5.4. EVALUATION ON DIVERSE MANIPULATION TASKS:

To further demonstrate benefits of using our proposed method, we present results for TRAIL, TRAIL-0, the baseline GAIL, and BC on more challenging tasks. Specifically, we consider stack with the Sawyer robot; and insertion and stack banana in the Jaco work-space. The results are shown in Figure 8 . The tasks we consider here are much harder as evidenced by the performance of BC agents. These experiments suggest that TRAIL is generally useful as an improvement over GAIL, even when the tasks are not designed to include task-irrelevant information.

Figure 8 : Results comparing TRAIL, TRAIL-0 and GAIL for diverse manipulation tasks.

6. CONCLUSIONS:

To make adversarial imitation work on nontrivial tasks from pixels, it is crucial to prevent the discriminator from exploiting task-irrelevant information. Data augmentation is a necessary component that prevents the discriminator from overfitting, but alone is not sufficient to solve more challenging problems. Our proposed method TRAIL proved to be a remedy which effectively focuses the discriminator on the task even when task-irrelevant features are present. Here, we vary the number of early frames of each episode used to form I. We report that a range of values from 1 up to 20 works well across both tasks (Put in box and Stack banana), with performance gradually degrading as the value increased beyond 20 (Fig. 11 ). A.3 TRAIL-0 WITH random

We found out and mentioned in the subsection 5.1 that random benefits from TRAIL-0. Unfortunately, it is still prone to overfitting and hence, worse than our full method -TRAIL. We present random + TRAIL-0 accompanied with our methods in Figure 12 . Our TRAIL and random + TRAIL-0 are the only methods exceeding BC performance on lift distracted. However, TRAIL performance is clearly better (obtains higher rewards and never gets overfitted).

Figure 12 : Results for lift alone, lift distracted, and lift distracted seeded.

A.4 FIXED TERMINATION POLICY:

As mentioned in the subsection 5.2, the most basic early termination policy -fixed step terminationmay be very effective if tuned. Since the tuning may be expensive in practice, we recommend using adaptive early stopping (TRAIL-0). However, for the sake of completeness we provide results for fixed step termination policy depending on the hyperparameter tuned. The results for stack task are presented in Figure 13 . The Jaco work space is considered here because Sawyer requires TRAIL to obtain high rewards. As can be inferred from the figure, the performance is very sensitive to the fixed step hyperparameter. We refer to subsection 5.2 for more comments on all methods. Because in the subsection 5.2 only the peak performance is presented (Table 5 .2), we present here the full curves in Figure 14 . The results shows that data augmentation is necessary to obtain high rewards in lift distracted. For easier lift alone, TRAIL-0 with data augmentation perform at par with TRAIL. A.6 D4PG

We use D4PG (Barth-Maron et al., 2018) as our main training algorithm. Briefly, D4PG is a distributed off-policy reinforcement learning algorithm for continuous control problems. In a nutshell, D4PG uses Q-learning for policy evaluation and Deterministic Policy Gradients (DPG) (Silver et al., 2014) for policy optimization. An important characteristic of D4PG is that it maintains a replay memory M (possibility prioritized (Horgan et al., 2018) ) that stores SARS tuples which allows for off-policy learning. D4PG also adopts target networks for increased training stability. In addition to these principles, D4PG utilized distributed training, distributional value functions, and multi-step returns to further increase efficiency and stability. In this section, we explain the different ingredients of D4PG.

D4PG maintains an online value network Q(s, a|θ) and an online policy network π(s|φ). The target networks are of the same structures as the value and policy network, but are parameterized by different parameters θ and φ which are periodically updated to the current parameters of the online networks.

:

www.mujoco.org

https://www.kinovarobotics.com/en/products/assistive-technologies/ kinova-jaco-assistive-robotic-arm

https://www.rethinkrobotics.com/sawyer/

https://robotiq.com/products/2f85-140-adaptive-robot-gripper

https://www.3dconnexion.com/spacemouse_compact/en/

Figures and Tables:

Figure1: GAIL and TRAIL succeed at lifting (a), but when distractor objects are added, GAIL fails while TRAIL succeeds (b). Due to robustness to initial conditions, TRAIL can stack from pixels while standard GAIL fails (c). We witness this difference again in insertion with distractors (d). A video showing agents performing these tasks can be seen at https://youtu.be/Rz5G15rDKcg .

Figure 4: Lift red block, where expert has a different body appearance, and with distractor blocks. TRAIL-random outperforms GAIL, and performs on par with TRAIL-early.

Figure 5: Demonstrating the memorization problem on the lift distracted task (here higher accuracy is worse). Accuracy of different discriminator heads is presented (A-D). In A, the overall accuracy for all timesteps. Then main (m) and extra (e) heads accuracy for the first steps are presented in B and C, respectively. Accuracy of the head predicting randomly assigned demonstration class is shown in D. Average discriminator predictions for training and holdout demonstration are shown in E and F.

Figure 6: Results for lift, box, and stack on Jaco environment.

Figure 7: When the expert differs in body or prop appearance, TRAIL outperforms GAIL.

Figure 11: TRAIL performance, varying the number of first frames in each episode used to form I.

Figure13: Results for stack in Jaco work space. Fixed step termination policy can be very effective but the final performance is very sensitive to the hyperparameter. TRAIL-0 does not need tuning nor access to the environment reward.

Figure 14: Results for lift alone and lift distracted in Sawyer work space. TRAIL and TRAIL-0 are by default with data augmentation. Additional results for these methods without data augmentation are presented to show its importance.

Observation and dimensions.

Table content: <table><tr><td>a) Jaco</td><td/><td>b) Sawyer</td><td/></tr><tr><td>Feature Name</td><td>Dimensions</td><td>Feature Name</td><td>Dimensions</td></tr><tr><td>frontal camera</td><td>64 × 64 × 3</td><td>front left camera</td><td>64 × 64 × 3</td></tr><tr><td>base force and torque sensors</td><td>6</td><td>front right camera</td><td>64 × 64 × 3</td></tr><tr><td>arm joints position</td><td>6</td><td>arm joint position</td><td>7</td></tr><tr><td>arm joints velocity</td><td>6</td><td>arm joint velocity</td><td>7</td></tr><tr><td>wrist force and torque sensors</td><td>6</td><td>wrist force sensor</td><td>3</td></tr><tr><td>hand finger joints position</td><td>3</td><td>wrist torque sensor</td><td>3</td></tr><tr><td>hand finger joints velocity</td><td>3</td><td>hand grasp sensor</td><td>1</td></tr><tr><td>hand fingertip sensors</td><td>3</td><td>hand joint position</td><td>1</td></tr><tr><td>grip site position</td><td>3</td><td>tool center point cartesian orientation</td><td>9</td></tr><tr><td>pinch site position</td><td>3</td><td>tool center point cartesian position</td><td>3</td></tr><tr><td/><td/><td>hand joint velocity</td><td>1</td></tr><tr><td colspan="2">A.2 EARLY FRAMES ABLATION STUDY</td><td/><td/></tr></table>

Peer review (iclr2020):
The paper addresses the acute problem of irrelevant discrimination in adversarial imitation learning methods. Ideally, the discriminator in such methods should extract task-dependent features and base its discrimination rule upon them only. However, in practice, the opposite usually occurs. For example, if the state distributions of the expert and agent come from different modalities, e.g., different background light, different POV camera, etc., then almost surely the discriminator will base its classification rule on such differences. In turn, the reward signal will be (constant and) irrelevant. This is a well-known problem, that indeed received little attention in previous work. As a solution, the authors propose to maintain an invariance set representing task-irrelevant states. For example, states of a random policy or states from the beginning of an episode. The authors suggest using the invariance set as a regularization mechanism to the discriminator by punishing it for correctly classifying examples from the invariance set. The idea behind this regularization is to distinguish between expert examples that actually represent the task and expert examples that do not. This is achieved by punishing for the discriminator*s accuracy over the invariance set. In addition, the authors present two additional improvements: data augmentation and early actor stopping. The paper. 1. I very much like the problem the authors choose to address. I believe that this is a dominant problem that received little attention so far. The reason is that so far, GAIL style methods used experts and agents that come from the same domains (same simulation), thus *hiding* such problems. As experiments scale up to real-world setups, this is no longer the case and such problems arise. 2. Overall, I find the experiments section very confusing. The authors propose a formal improvement (the invariance set) which they apply to their agent, as well as two informal improvements (actor early stopping and data augmentation) which they apply selectively for their agent and the baseline methods. As a result, there exist a large number of variations, according to what improvements are applied to what method. Overall, I find it hard to follow the results and understand the contribution of the different improvements. 3. It is not clear if TRAIL and GAIL are using the same architecture and algorithm. Apparently, the authors could have used GAIL*s original policy gradient algorithm with their invariance set modification. If an improvement was evident in such a comparison, then it would have been easier for me to understand the significance of the proposed method. Currently, there are too many differences between the baseline and TRAIL to clearly say that the invariance set made the difference. 4. It is not clear why the constraint in Eq. 2 is placed over the sum of the two terms and not on each term separately since each term is defined over a different state distribution. What would you expect to see different if the constraint was placed on the two terms separately? 5. In the same matter, why do you need the 0.5 factor? 6. The invariance set is referred to extensively before it is first defined. Therefore, the reader is forced to toggle back and forth multiple times in section 4 before understanding equations 2 and 3. Consider giving at least an intuition about the invariance set beforehand to make the reading more fluent. 7. The conclusions section is minimal to non-existing while there are many conclusions to draw from the experiments. 8. Data augmentation was previously studied in the context of adversarial imitation. See for example *Visual imitation with reinforcement learning using recurrent Siamese networks* by Berseth et.al. 9. Overall assessment: among the three modifications the authors propose, I would be glad to see if the invariance set modification alone can make a difference. The other two modifications are either not novel (see Berseth et.al.), or highly heuristic (*...we restart an episode if the discriminator score at the current step exceeds the median score of the episode so far for T_patience consecutive steps...*). However, it seems that the informal improvements (augmentation and early stopping) play a crucial role in the success of TRAIL (... It drastically improves the baseline GAIL agent, and is necessary to solve any of the harder tasks...*). It is absolutely fine that the authors stress the importance of the informal improvements but it seems that they are dominating the contribution of the main ingredient. I also find it problematic that the construction of the invariance set is not straightforward (*...the selection of the invariance set is a design choice...*). When added to the way that the data augmentation is carried (sensor dependent), and to the fact the the early stopping mechanism seems dubious for some tasks (I can think of several tasks where early stopping will not work), this method is far from being a *plug-and-play* solution. I.e., the paper offers general guidelines on how to solve imitation tasks but not a concise algorithm. In order for me to vote for acceptance, I need the paper to be more concise. In its current state, I see it more as general guidelines for training adversarial imitation algorithms. However, as said in the beginning, I very much believe in the problem that the authors present and encourage them to provide an experiment where the only modification between the baseline and the proposed method is the usage of an invariance set. Success in such an experiment will clearly change my mind. There are also some presentation issues, e.g cluttered graphs, that can be addressed.