Year: 2018

Abstract:
We present an adversarial exploration strategy, a simple yet effective imitation learning scheme that incentivizes exploration of an environment without any extrinsic reward or human demonstration. Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other. The former collects training samples for the latter, and its objective is to maximize the error of the latter. The latter is trained with samples collected by the former, and generates rewards for the former when it fails to predict the actual action taken by the former. In such a competitive setting, the DRL agent learns to generate samples that the inverse dynamics model fails to predict correctly, and the inverse dynamics model learns to adapt to the challenging samples. We further propose a reward structure that ensures the DRL agent collects only moderately hard samples and not overly hard ones that prevent the inverse model from imitating effectively. We evaluate the effectiveness of our method on several OpenAI gym robotic arm and hand manipulation tasks against a number of baseline models. Experimental results show that our method is comparable to that directly trained with expert demonstrations, and superior to the other baselines even without any human priors.

heading: 1 INTRODUCTION
text: Over the past decade, imitation learning (IL) has been successfully applied to a wide range of domains, including robot learning (Englert et al., 2013; Schulman et al., 2013), autonomous navigation (Choudhury et al., 2017; Ross et al., 2013), manipulation tasks (Nair et al., 2017; Prieur et al., 2012), and self-driving cars (Codevilla et al., 2018). Traditionally, IL aims to train an imitator to learn a control policy π only from expert demonstrations. The imitator is typically presented with multiple demonstrations during the training phase, with an aim to distill them into π. To learn π effectively and efficiently, a large set of high-quality demonstrations are necessary. This is especially prevalent in current state-of-the-art IL algorithms, such as dataset aggregation (DAgger) (Ross et al., 2011) and generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016). Although these approaches have been the dominant algorithms in IL, a major bottleneck for them is their reliance on high-quality demonstrations, which often require extensive supervision from human experts. In addition, a serious flaw in the learned policy π is its tendency to overfit to demonstration data, preventing it from generalizing to new ones. To overcome the aforementioned challenges in IL, a number of methods have been investigated to enhance the generalizability and data efficiency, or reduce the degree of human supervision. Initial efforts in this direction were based on the idea of meta learning (Duan et al., 2017; Finn et al., 2017; Yu et al., 2018), in which the imitator is trained from a meta learner that is able to quickly learn a new task with only a few set of demonstrations. However, such schemes still require training the meta-learner with tremendous amount of time and demonstration data, leaving much room for improvement. Thus, a rapidly-growing body of literature based on the concept of using forward/inverse dynamics models to learn π within an environment in a self-supervised fashion (Agrawal et al., 2016; Nair et al., 2017; Pathak et al., 2018) has emerged in the past few years. One key advantage of the concept is that it provides an autonomous way for preparing training data, removing the need of human intervention. In this paper, we call it self-supervised IL.
Self-supervised IL allows an imitator to collect training data by itself instead of using predefined extrinsic reward functions or expert supervision during training. It only needs demonstration during inference, drastically decreasing the time and effort required from human experts. Although the core principles of self-supervised IL are straightforward and have been exploited in many fields (Agrawal et al., 2016; Nair et al., 2017; Pathak et al., 2017; 2018), recent research efforts have been dedicated
to addressing the challenges of multi-modality and multi-step planning. For example, the use of forward consistency loss and forward regularizer have been extensively investigated to enhance the task performance of the imitator (Agrawal et al., 2016; Pathak et al., 2018). This becomes especially essential when the lengths of trajectories grow and demonstration samples are sparse, as multiple paths may co-exist to lead the imitator from its initial observation to the goal observation. The issue of multi-step planning has also drawn a lot of attention from researchers, and is usually tackled by recurrent neural networks (RNNs) and step-by-step demonstrations (Nair et al., 2017; Pathak et al., 2018). The above self-supervised IL approaches report promising results, however, most of them are limited in applicability due to several drawbacks. First, traditional methods of data collection are usually inefficient and time-consuming. Inefficient data collection results in poor exploration, giving rise to a degradation in robustness to varying environmental conditions (e.g., noise in motor control) and generalizability to difficult tasks. Second, human bias in data sampling range tailored to specific interesting configurations is often employed (Agrawal et al., 2016; Nair et al., 2017). Although a more general exploration strategy called curiosity-driven exploration was later proposed in Pathak et al. (2017), it focuses only on exploration in states novel to the forward dynamics model, rather than those directly influential to the inverse dynamics model. Furthermore, it does not discuss the applicability to continuous control domains, and fails in high dimensional action spaces according to our experiments in Section 4. Unlike the approaches discussed above, we do not propose to deal with multi-modality or multi-step planning. Instead, we focus our attention on improving the overall quality of the collected samples in the context of self-supervised IL. This motivates us to equip the model with the necessary knowledge to explore the environment in an efficient and effective fashion.
In this paper, we propose a straightforward and efficient self-supervised IL scheme, called adversarial exploration strategy, which motivates exploration of an environment in a self-supervised manner (i.e., without any extrinsic reward or human demonstration). Inspired by Pinto et al. (2017); Shioya et al. (2018); Sukhbaatar et al. (2018), we implement the proposed strategy by jointly training a deep reinforcement learning (DRL) agent and an inverse dynamics model competing with each other. The former explores the environment to collect training data for the latter, and receives rewards from the latter if the data samples are considered difficult. The latter is trained with the training data collected by the former, and only generates rewards when it fails to predict the true actions performed by the former. In such an adversarial setting, the DRL agent is rewarded only for the failure of the inverse dynamics model. Therefore, the DRL agent learns to sample hard examples to maximize the chances to fail the inverse dynamics model. On the other hand, the inverse dynamics model learns to be robust to the hard examples collected by the DRL agent by minimizing the probability of failures. As a result, as the inverse dynamics model becomes stronger, the DRL agent is also incentivized to search for harder examples to obtain rewards. Overly hard examples, however, may lead to biased exploration and cause instability of the learning process. In order to stabilize the learning curve of the inverse dynamics model, we further propose a reward structure such that the DRL agent is encouraged to explore moderately hard examples for the inverse dynamics model, but refraining from too difficult ones for the latter to learn. The self-regulating feedback structure between the DRL agent and the inverse dynamics model enables them to automatically construct a curriculum for exploration.
We perform extensive experiments to validate adversarial exploration strategy on multiple OpenAI gym (Brockman et al., 2016) robotic arm and hand manipulation task environments simulated by the MuJoCo physics engine (Todorov et al., 2012), including FetchReach, FetchPush, FetchPickAndPlace, FetchSlide, and HandReach. These environments are intentionally selected by us for evaluating the performance of inverse dynamics model, as each of them allows only a very limited set of chained actions to transition the robotic arms and hands to target observations. We examine the effectiveness of our method by comparing it against a number of self-supervised IL schemes. The experimental results show that our method is more effective and data-efficient than the other self-supervised IL schemes for both low- and high-dimensional observation spaces, as well as in environments with high-dimensional action spaces. We also demonstrate that in most of the cases the performance of the inverse dynamics model trained by our method is comparable to that directly trained with expert demonstrations. The above observations suggest that our method is superior to the other self-supervised IL schemes even in the absence of human priors. We further evaluate our method on environments with action space perturbations, and show that our method is able to achieve satisfactory success rates. To justify each of our design decisions, we provide a comprehensive set of ablative analysis and discuss their implications. The contributions of this work are summarized as follows:
• We introduce an adversarial exploration strategy for self-supervised IL. It consists of a DRL agent and an inverse dynamics model developed for efficient exploration and data collection.
• We employ a competitive scheme for the DRL agent and the inverse dynamics model, enabling them to automatically construct a curriculum for exploration of observation space.
• We introduce a reward structure for the proposed scheme to stabilize the training process. • We demonstrate the proposed method and compare it with a number of baselines for multiple
robotic arm and hand manipulation tasks in both low- and high-dimensional state spaces. • We validate that our method is generalizable to tasks with high-dimensional action spaces.
The remainder of this paper is organized as follows. Section 2 introduces background material. Section 3 describes the proposed adversarial exploration strategy in detail. Section 4 reports the experimental results, and provides an in-depth ablative analysis of our method. Section 5 concludes.

heading: 2 BACKGROUND
text: In this section, we briefly review DRL, policy gradient methods, as well as inverse dynamics model.

heading: 2.1 DEEP REINFORCEMENT LEARNING AND POLICY GRADIENT METHODS
text: DRL trains an agent to interact with an environment E . At each timestep t, the agent receives an observation xt ∈ X , where X is the observation space of E . It then takes an action at from the action space A based on its current policy π, receives a reward r, and transitions to the next observation x′. The policy π is represented by a deep neural network with parameters θ, and is expressed as π(a|x, θ). The goal of the agent is to learn a policy to maximize the discounted sum of rewards Gt:
Gt = T∑ τ=t γτ−tr(xτ , aτ ), (1)
where t is the current timestep, γ ∈ (0, 1] the discount factor, and T the horizon. Policy gradient methods (Mnih et al., 2016; Sutton et al., 2000; Williams, 1992) are a class of RL techniques that directly optimize the parameters of a stochastic policy approximator using policy gradients. Although these methods have achieved remarkable success in a variety of domains, the high variance of gradient estimates has been a major challenge. Trust region policy optimization (TRPO) (Schulman et al., 2015) circumvented this problem by applying a trust-region constraint to the scale of policy updates. However, TRPO is a second-order algorithm, which is relatively complicated and not compatible with architectures that embrace noise or parameter sharing (Schulman et al., 2017). In this paper, we employ a more recent family of policy gradient methods, called proximal policy optimization (PPO) (Schulman et al., 2017). PPO is an approximation to TRPO, which similarly prevents large changes to the policy between updates, but requires only first-order optimization. PPO is superior in its generalizability and sample complexity while retaining the stability and reliability of TRPO 1.

heading: 2.2 INVERSE DYNAMICS MODEL
text: An inverse dynamics model I takes as input a pair of observations (x, x′), and predicts the action â required to reach the next observation x′ from the current observation x. It is formally expressed as:
â = I(x, x′|θI), (2)
where (x, x′) are sampled from the collected data, and θI represents the trainable parameters of I . During the training phase, θI is iteratively updated to minimize the loss function LI , expressed as:
LI(a, â|θI) = d(a, â), (3)
where d is a distance metric, and a the ground truth action. During the testing phase, a sequence of observations {x̂0, x̂1, · · · , x̂T } is first captured from an expert demonstration. A pair of observations (x̂t, x̂t+1) is then fed into I at each timestep t. Starting from x̂0, the objective of I is to predict a sequence of actions {â0, â1, · · · , âT−1} and transition the final observation x̂T as close as possible.

heading: 3 METHODOLOGY
text: In this section, we first describe the proposed adversarial exploration strategy. We then explain the training methodology in detail. Finally, we discuss a technique for stabilizing the training process.

heading: 3.1 ADVERSARIAL EXPLORATION STRATEGY
text: Fig. 1 shows a framework that illustrates the proposed adversarial exploration strategy, which includes a DRL agent P and an inverse dynamics model I . Assume that Φπ : {x0, a0, x1, a1 · · · , xT } is the
1For more details on PPO, please refer to supplementary material S.2.
sequence of observations and actions generated by P as it explores E using a policy π. At each timestep t, P collects a 3-tuple training sample (xt, at, xt+1) for I , while I predicts an action ât and generates a reward rt for P . In this work, I is modified from Eq. (2) to include an additional hidden vector ht, which recurrently encodes the information of the past observations. I is thus expressed as:
ât = I(xt, xt+1|ht, θI) ht = f(ht−1, xt),
(4)
where f(·) denotes the recurrent function. θI is iteratively updated to minimize LI , formulated as:
min θI LI(at, ât|θI) = min θI β||at − ât||2, (5)
where β is a scaling constant. We employ mean squared error β||at − ât||2 as the distance metric d(at, ât), since we only consider continuous control domains in this paper. It can be replaced with a cross-entropy loss for discrete control tasks. We directly use LI as the reward rt for P , expressed as:
rt(xt, at, xt+1) = LI(at, ât|θI) = β||at − I(xt, xt+1|ht, θI)||2. (6)
Our method targets at improving both the quality and efficiency of the data collection process performed by P , as well as the performance of I . Therefore, the goal of the proposed framework is twofold. First, P has to learn an adversarial policy πadv(at|xt) such that its cumulated discounted rewards Gt|πadv = ∑T τ=t γ
τ−trt(xτ , aτ , xτ+1) is maximized. Second, I requires to learn an optimal θI such that Eq. (6) is minimized. Minimizing LI (i.e., rt) leads to decreased Gt|πadv , forcing P to enhance πadv to explore more difficult samples to increase Gt|πadv . This implies that P is motivated to focus on I’s weak points, instead of randomly collecting ineffective training samples. Training I with hard samples not only accelerates its learning progress, but also helps to boost its performance.

heading: 3.2 TRAINING METHODOLOGY
text: We describe the training methodology of our adversarial exploration strategy by a pseudocode presented in Algorithm 1. Assume that P ’s policy πadv is parameterized by a set of trainable parameters θP , and is represented as πadv(at|xt, θP ). We create two buffers ZP and ZI for storing the training samples of P and I , respectively. In the beginning, ZP , ZI , E , θP , θI , πadv , as well as a timestep cumulative counter c are initialized. A number of hyperparameters are set to appropriate values, including the number of iterations Niter, the number of episodes Nepisode, the horizon T , as well as the update period TP of θP . At each timestep t, P perceives the current observation xt from E , takes an action at according to πadv(at|xt, θP ), and receives the next observation xt+1 and a termination indicator ξ (lines 9-11). ξ is set to 1 only when t equals T , otherwise it is set to 0. We then store (xt, at, xt+1, ξ) and (xt, at, xt+1) in ZP and ZI , respectively. We update θP every TP timesteps using the samples stored in ZP , as shown in (lines 13-21). At the end of each episode, we update θI with samples drawn from ZI according to the loss function LI defined in Eq. (5) (line 23).

heading: 3.3 STABILIZATION TECHNIQUE
text: Although adversarial exploration strategy is effective in collecting hard samples, it requires additional adjustments if P becomes too strong such that the collected samples are too difficult for I to learn. Overly difficult samples lead to a large variance in gradients derived from LI , which in turn cause a performance drop in I and instability in its learning process. We analyze this phenomenon in greater detail in Section 4.5. To tackle the issue, we propose a training technique that reshapes rt as follows:
rt := −|rt − δ|, (7)
Algorithm 1 Adversarial exploration strategy 1: Initialize ZP , ZI , E , and model parameters θP & θI 2: Initialize πadv(at|xt, θP ) 3: Initialize the timestep cumulative counter c = 0 4: SetNiter ,Nepisode, T , and TP 5: for iteration i = 1 toNiter do 6: for episode e = 1 toNepisode do 7: for timestep t = 0 to T do 8: P perceives xt from E , and predicts an action at according to πadv(at|xt, θP ) 9: xt+1 = E(xt, at) 10: ξ = 1[t == T ] 11: Store (xt, at, xt+1, ξ) in ZP 12: Store (xt, at, xt+1) in ZI 13: if (c% TP ) == 0 then 14: Initialize an empty batchB 15: Initialize a recurrent state ht 16: for (xt, at, xt+1, ξ) in ZP do 17: Evaluate ât = I(xt, xt+1|ht, θI) (calculated from Eq. (4)) 18: Evaluate rt(xt, at, xt+1) = LI(at, ât|θI) (calculated from Eq. (6)) 19: Store (xt, at, xt+1, rt) inB 20: Update θP with the gradient calculated from the samples ofB 21: Reset ZP 22: c = c+ 1 23: Update θI with the gradient calculated from the samples of ZI (according to Eq. (5)) 24: end
where δ is a pre-defined threshold value. This technique poses a restriction on the range of rt, driving P to gather moderate samples instead of overly hard ones. Note that the value of δ affects the learning speed and the final performance. We plot the impact of δ on the learning curve of I in Section 4.5. We further provide an example in our supplementary material to visualize the effect of this technique.

heading: 4 EXPERIMENTAL RESULTS
text: In this section, we present experimental results for a series of robotic tasks, and validate that (i) our method is effective in both low- and high-dimensional observation spaces; (ii) our method is effective in environments with high-dimensional action spaces; (iii) our method is more data efficient than the baseline methods; and (iv) our method is robust against action space perturbations. We first introduce our experimental setup. Then, we report experimental results of robotic arm and hand manipulation tasks. Finally, we present a comprehensive set of ablative analysis to validate our design decisions.

heading: 4.1 EXPERIMENTAL SETUP
text: We first describe the environments and tasks. Next, we explain the evaluation procedure and the method for collecting expert demonstrations. We then walk through the baselines used for comparison.

heading: 4.1.1 ENVIRONMENTS AND TASKS
text: We evaluate our method on a number of robotic arm and hand manipulation tasks via OpenAI gym (Brockman et al., 2016) environments simulated by the MuJoCo (Todorov et al., 2012) physics engine. We use the Fetch and Shadow Dexterous Hand (Plappert et al., 2018b) for the arm and hand manipulation tasks, respectively. For the arm manipulation tasks, which include FetchReach, FetchPush, FetchPickAndPlace, and FetchSlide, the imitator (i.e., the inverse dynamic model I) takes as inputs the positions and velocities of a gripper and a target object. It then infers the gripper’s action in 3-dimensional space to manipulate it. For the hand manipulation task HandReach, the imitator takes as inputs the positions and velocities of the fingers of a robotic hand, and determines the velocities of the joints to achieve the goal. In addition to low-dimensional observations (i.e., position, velocity, and gripper state), we further perform experiments for the above tasks using visual observations (i.e., high-dimensional observations) in the form of camera images taken from a third-person perspective. The detailed description of the above tasks is specified in Plappert et al. (2018b). For the detailed configurations of these tasks, please refer to our supplementary material.

heading: 4.1.2 EVALUATION PROCEDURE
text: The primary objective of our experiments is to demonstrate the efficiency of the proposed adversarial exploration strategy in collecting training data (in a self-supervised manner) for the imitator. We compare our strategy against a number of self-supervised data collection methods (referred to as ”baselines” or ”baseline methods”) described in Section 4.1.4. As different baseline methods employ different data collection strategies, the learning curve of the imitator also varies for different cases. For a fair comparison, the model architecture of the imitator and the amount of training data are fixed
for all cases. All of the experimental results are evaluated and averaged over 20 trials, corresponding to 20 different random initial seeds. In each trial, we train an imitator by the training data collected by a single self-supervised data collection method. At the beginning of each episode, the imitator receives a sequence of observations {x̂0, x̂1, · · · , x̂T } from a successful expert demonstration. At each timestep t, the imitator infers an action ât from an expert observation x̂t+1 and its current observation xt by Eq. (4). We periodically evaluate the imitator every 10K timesteps. The evaluation is performed by averaging the success rates of reaching x̂T over 500 episodes. The configuration of the imitator and the hyperparameters of the baselines are summarized in the supplementary material.

heading: 4.1.3 COLLECTION OF EXPERT DEMONSTRATIONS
text: For each task mentioned in Section 4.1.1, we first randomly configure task-relevant settings (e.g., goal position, initial state, etc.). We then collect demonstrations from non-trivial and successful episodes performed by a pre-trained expert agent (Andrychowicz et al., 2017). Please note that the collected demonstrations only contain sequences of observations. The implementation details of the expert agent and the method for filtering out trivial episodes are presented in our supplementary material.

heading: 4.1.4 BASELINE METHODS
text: We compare our proposed methodology with the following four baseline methods in our experiments.
• Random: This method collects training samples by random exploration. We consider it to be an important baseline because of its simplicity and prevalence in a number of research works on self-supervised IL (Agrawal et al., 2016; Nair et al., 2017; Pathak et al., 2018).
• Demo: This method trains the imitator directly with expert demonstrations. It serves as the performance upper bound, as the training data is the same as the testing data for this method.
• Curiosity: This method trains a DRL agent via curiosity (Pathak et al., 2017; 2018) to collect training samples. Unlike the original implementation, we replace its DRL algorithm with PPO, as training should be done on a single thread for a fair comparison with the other baselines. This is alo an important baseline due to its effectiveness in Pathak et al. (2018).
• Noise (Plappert et al., 2018a): In this method, noise is injected to the parameter space of a DRL agent to encourage exploration (Plappert et al., 2018a). Please note that its exploratory behavior relies entirely on parameter space noise, instead of using any extrinsic reward. We include this method due to its superior performance and data efficiency in many DRL tasks.

heading: 4.2 PERFORMANCE COMPARISON IN ROBOTIC ARM MANIPULATION TASKS
text: We compare the performance of the proposed method and the baselines on the robotic arm manipulation tasks described in Section 4.1.1. As opposed to discrete control domains, these tasks are especially challenging, as the sample complexity grows in continuous control domains. Furthermore, the imitator may not have the complete picture of the environment dynamics, increasing its difficulty to learn an inverse dynamics model. In FetchSlide, for instance, the movement of the object on the slippery surface is affected by both friction and the force exerted by the gripper. It thus motivates us to investigate whether the proposed method can help overcome the challenge. In the subsequent paragraphs, we discuss the experimental results in both low- and high-dimensional observation spaces, and plot them in Figs. 2 and 3, respectively. All of the results are obtained by following the procedure described in Section 4.1.2. The shaded regions in Figs. 2 and 3 represent the confidence intervals.
Low-dimensional observation spaces. Fig. 2 plots the learning curves for all of the methods in low-dimensional observation spaces. In all of the tasks, our method yields superior or comparable performance to the baselines except for Demo, which is trained directly with expert demonstrations. In FetchReach, it can be seen that every method achieves a success rate of 1.0. This implies that it does not require a sophisticated exploration strategy to learn an inverse dynamics model in an environment where the dynamics is relatively simple. It should be noted that although all methods reach the same final success rate, ours learns significantly faster than Demo. In contrast, in FetchPush, our method is comparable to Demo, and demonstrates superior performance to the other baselines. Our method also learns drastically faster than all the other baselines, which confirms that the proposed strategy does improve the performance and efficiency of self-supervised IL. Our method is particularly effective in tasks that require an accurate inverse dynamics model. In FetchPickAndPlace, for example, our method surpasses all the other baselines. However, all methods including Demo fail to learn a successful inverse dynamics model in FetchSlide, which suggests that it is difficult to train an imitator when the outcome of an action is not completely dependent on the action itself. It is worth noting that Curiosity loses to Random in FetchPush and FetchSlide, and Noise performs even worse than these
two methods in all of the tasks. We therefore conclude that Curiosity is not suitable for continuous control tasks, and the parameter space noise strategy cannot be directly applied to self-supervised IL. In addition to the quantitative results presented above, we further discuss the empirical results qualitatively. Please refer our supplementary material for a description of the qualitative results.
High-dimensional observation spaces. Fig. 3 plots the learning curves of all methods in highdimensional observation spaces. It can be seen that our method performs significantly better than the other baseline methods in most of the tasks, and is comparable to Demo. In FetchPickAndPlace, our method is the only one that learns a successful inverse dynamics model. Similar to the results in Fig. 2, Curiosity is no better than Random in high-dimensional observation spaces. Please note that we do not include Noise in Fig. 3 as it performs worse enough already in low-dimensional settings.

heading: 4.3 PERFORMANCE COMPARISON IN ROBOTIC HAND MANIPULATION TASK
text: Fig. 2 plots the learning curves for each of the methods considered. Please note that Curiosity, Noise and our method are pre-trained with 30K samples collected by random exploration, as we observe that these methods on their own suffer from large errors in an early stage during training, which prevents them from learning at all. After the first 30K samples, they are trained with data collected by their exploration strategy instead. From the results in Fig. 2, it can be seen that Demo easily stands out from the other methods as the best-performing model, surpassing them all by a considerable extent. Although our method is not as impressive as Demo, it significantly outperforms all of the other baseline methods, achieving a success rate of 0.4 while the others are still stuck at around 0.2.
The reason that the inverse dynamics models trained by the self-supervised data-collection strategies discussed in this paper (including ours and the other baselines) are not comparable to the Demo baseline in the HandReach task is primarily due to the high-dimensional action space. It is observed that the data collected by the self-supervised data-collection strategies only cover a very limited range of the state space in the HandReach environment. Therefore, the inverse dynamics models trained with these data only learn to imitate trivial poses, leading to the poor success rates presented in Fig. 2.

heading: 4.4 ROBUSTNESS TO ACTION SPACE PERTURBATION
text: We evaluate the performance of the imitator trained in an environment with action space perturbations to validate the robustness of our adversarial exploration strategy. In such an environment, every action taken by the DRL agent is perturbed by a Gaussian random noise, such that the training samples collected by the DRL agent are not inline with its actual intentions. Please note that we only inject noise during the training phase, as we aim to validate the robustness of the proposed data collection strategy. The scale of the injected noise is specified in the supplementary material. We report the performance change rates of various methods for different tasks in Table. 1. The performance change rate is defined as: Prperturb−ProrigProrig , where Prperturb and Prorig represent the highest success rates with and without action space perturbations, respectively. From Table. 1, it can be seen that our method retains the performance for most of the tasks, indicating that our method is robust to action space perturbations during the training phase. Please note that although Curiosity and Noise also achieve a change rate of 0% in HandReach and FetchSlide, they are not considered robust due to their poor performance in the original environment (Fig. 2). Another interesting observation is that our
method even gains some performance from action space perturbations in FetchPush and HandReach, which we leave as one of our future directions. We thus conclude that our method is robust to action space perturbations during the training phase, making it a practical option in real-world settings.

heading: 4.5 ABLATIVE ANALYSIS
text: In this section, we provide a set of ablative analysis. We examine the effectiveness of our method by an investigation of the training loss distribution, the stabilization technique, and the influence of δ. Please note that the value of δ is set to 1.5 by default, as described in our supplementary material.
Training loss distribution. Fig. 4 plots the probability density function (PDF) of LI (derived from Eq. (5)) by kernel density estimation (KDE) for the first 2K training batches during the training phase. The vertical axis corresponds to the probability density, while the horizontal axis represents the scale of LI . The curves Ours (w stab) and Ours (w/o stab) represent the cases where the stabilization technique described in Section 3.3 is employed or not, respectively. We additionally plot the curve Random in Fig. 4 to highlight the effectiveness of our method. It can be observed that both Ours (w stab) and Ours (w/o stab) concentrate on notably higher loss values than Random. This observation implies that adversarial exploration strategy does explore hard samples for inverse dynamics model.
Validation of the stabilization technique. We validate the proposed stabilization technique in terms of the PDF of LI and the learning curve of the imitator, and plot the results in Figs. 4 and 5, respectively. From Fig. 4, it can be observed that the modes of Ours (w stab) are lower than those of Ours (w/o stab) in most cases, implying that the stabilization technique indeed motivates the DRL agents to favor those moderately hard samples. We also observe that for each of the five cases, the mode of Ours (w stab) is close to the value of δ (plotted in a dotted line), indicating that our reward structure presented in Eq. (7) does help to regulate LI (and thus rt) to be around δ. To further demonstrate the effectiveness of the stabilization technique, we compare the learning curves of Ours (w stab) and Ours (w/o stab) in Fig. 5. It is observed that for the initial 10K samples of the five cases, the success rates of Ours (w/o stab) are comparable to those of Ours (w stab). However, their performance degrade drastically during the rest of the training phase. This observation confirms that the stabilization technique does contribute significantly to our adversarial exploration strategy.
Although most of the DRL works suggest that the rewards should be re-scaled or clipped within a range (e.g., from -1 to 1), the unbounded rewards do not introduce any issues during the training process of our experiments. The empirical rationale is that the rewards received by the DRL agent are regulated by Eq. (7) to be around δ, as described in Section 4.5 and depicted in Fig. 4. Without the stabilization technique, however, the learning curves of the inverse dynamics model degrade drastically (as illustrated in Fig. 2), even if the reward clipping technique is applied.
Influence of δ. Fig. 6 compares the learning curves of the imitator for different values of δ. For instance, Ours(0.1) corresponds to δ = 0.1. It is observed that for most of the tasks, the success rates drop when δ is set to an overly high or low value (e.g., 100.0 or 0.0), suggesting that a moderate value of δ is necessary for the stabilization technique. The value of δ can be adjusted dynamically by the adaptive scaling technique presented in Plappert et al. (2018a), which is left as our future direction.
From the analysis presented above, we conclude that the proposed adversarial exploration strategy is effective in collecting difficult training data for the imitator. The analysis also validates that our
stabilization technique indeed leads to superior performance, and is capable of guiding the DRL agent to collect moderately hard samples. This enables the imitator to pursue a stable learning curve.

heading: 5 CONCLUSION
text: In this paper, we presented an adversarial exploration strategy, which consists of a DRL agent and an inverse dynamics model competing with each other for self-supervised IL. The former is encouraged to adversarially collect difficult training data for the latter, such that the training efficiency of the latter is significantly enhanced. Experimental results demonstrated that our method substantially improved the data collection efficiency in multiple robotic arm and hand manipulation tasks, and boosted the performance of the inverse dynamics model in both low- and high-dimensional observation spaces. In addition, we validated that our method is generalizable to environments with high-dimensional action spaces. Moreover, we showed that our method is robust to action space perturbations. Finally, we provided a set of ablative analysis to validate the effectiveness for each of our design decisions.

Review summaries:
The paper proposes a novel exploration strategy for self-supervised imitation learning. An inverse dynamics model is trained on the trajectories collected from a RL-trained policy. The policy is rewarded for generating trajectories on which the inverse dynamics model (IDM) currently works poorly, i.e. on which IDM predicts actions that are far (in terms of mean square error) from the actions performed by the policy. This adversarial training is performed in purely self-supervised way. The evaluation is performed by one-shot imitation of an expert trajectory using the IDM: the action is predicted from the current state of the environment and the next state in the expert’s trajectory. Experimental evaluation shows that the proposed method is superior to baseline exploration strategies for self-supervised imitation learning, including random and curiosity-based exploration.