Auto-Encoding Adversarial Imitation Learning
Authors: Kaifeng Zhang
Rui Zhao
Ziming Zhang
Yang Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: imitation learning
reinforcement learning
auto-encoders

Abstract: Reinforcement learning (RL) provides a powerful framework for decision-making, but its application in practice often requires a carefully designed reward function. Adversarial Imitation Learning (AIL) sheds light on automatic policy acquisition without access to the reward signal from the environment. In this work, we propose Auto-Encoding Adversarial Imitation Learning (AEAIL), a robust and scalable AIL framework. To induce expert policies from demonstrations, AEAIL utilizes the reconstruction error of an auto-encoder as a reward signal, which provides more information for optimizing policies than the prior discriminator-based ones. Subsequently, we use the derived objective functions to train the auto-encoder and the agent policy. Experiments show that our AEAIL performs superior compared to state-of-the-art methods in the MuJoCo environments. More importantly, AEAIL shows much better robustness when the expert demonstrations are noisy. Specifically, our method achieves $11\%$ and $50.7\%$ relative improvement overall compared to the best baseline GAIL and PWIL on clean and noisy expert data, respectively. Video results, open-source code and dataset are available in supplementary materials. 

Under review as a conference paper at ICLR 2023
AUTO-ENCODING
ADVERSARIAL IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) provides powerful framework for decision-making but its application in practice often requires a carefully designed reward function. Adversarial Imitation Learning (AIL) sheds light automatic policy acquisition without access to the reward signal from the environment. In this work, we propose Auto-Encoding Adversarial Imitation Learning (AEAIL); robust and scalable All. framework. To induce expert policies from demonstrations, AEAIL utilizes the reconstruction error of an auto-encoder as reward signal, which provides more information for optimizing policies than the prior discriminator- based ones. Subsequently, we use the derived objective functioon to train the auto-encoder and the agent policy. Experiments show that our AEAIL performs superior compared to stateoof-hhe-at methods in the MuJoCo environments. More importantly, AEAIL shows much better robustness when the expert demonstrations are noisy. Specifically, our method achieves 11% and 50.7% relative improvement overall compared to the best baseline GAIL and PWIL clean and noisy expert data, respectively. Video results, open-source code and dataset are available in supplementary materials.
INTRODUCTION
Reinforcement learning (RL) provides powerful framework for automated decision-making, How- ever, RL still requires significantly engineered reward functions for good practical performance. Imitation learning offers the instruments to learn policies directly from the demonstrations, without an explicit reward function. It enables the agents to learn to solve tasks from expert demonstrations, such helicopter control (Abbeel et al. 2006 2007 Ng al. 2004 Coates al. 2008 Abbeel iet al. 2008a 2010). robot navigation (Ratliff et al. | Abbbel et al. 2008b 2 Ziebart et al. 2008
2010 and building controls (Barrett & Linder 2015).
The goal of imitation learning is to induce the expert policies from expert demonstrations without access the reward signal from the environment. We divide these methods into two broad categories: Behavioral Cloning (BC) and Inverse Reinforcement Learning (IRL). Among IRL, adversarial imitation learning (AIL) induces expert policies by minimizing the distribution distance between expert samples and agent policy rollouts. Prior AIL methods model the reward function as discriminator to learn the mapping from the state-action pair scalar value, i.e., reward (Ho Ermon| 2016 Zhang et al. 22000 Fu et al., 2017). However, the discriminator in the AlL framework would easily find the differences between expert samples and agent- gennt-eeeeeeedd ones, even though some differences are minor. Therefore, the discriminator-base reward function would yield a sparse reward signal to the agent. Consequently, how to make AIL robust and efficient to use is still subject to research.
Our AEAIL is an instance of AIL by formulating the reward function as an auto-encoder. Since auto-encoder reconstruct the full state, unlike traditional discriminator based AlL, our method will not overfit to the minor differences between expert samples and generated samples. In many cases, reward signal provides richer feedback to the policy training process. Thus, our new method achieves better performance on a wide range of tasks.
Our contributions are three-fold:

---

Under review as a conference paper at ICLR 2023
We propose the Auto-Encoding Adversarial Imitation Learning (AEAIL) architecture. It models the reward function as the reconstruction error of an auto-encoder. It focuses on the full-scale differences between the expert and generated samples, and less susceptible to the discriminator attending to minor differences.
Experiments show that our proposed AEAIL achieves the best overall performance on many environments compared to other state-of-the-art baselines.
Empirically, we justify that our AEAIL works under a wide range of distribution divergences and auto encoders. And the major contributing factor of our method is the encoding- decoding process rather than the specific divergence or auto-encoder.
2 RELATED WORK
2.1 IMITATION LEARNING
Adversarial Imitation Learning (AIT) AlL. methods such as GAIL, DAC, f-GAIL., EAIRL, FAIRL
(Ho & Ermon 2016 Kostrikov et al.] 2019 Zhang al. 2020 Qureshi et al.) 2019 Ghasemipour ctt al. (020) formulates the learned reward function as a discriminator that learns to differentiate expert transitions from non-expert ones. Among these methods, GAIL (Ho & Ermon 2016 considers the Jensen- -hasennShanonn divergence. DAC (Kostrikov et al. 2019) extends GAIL to the off-policy setting and significantly improves the sample efficiency of adversarial imitation learning. Furthermore, /-divergence is utilized in f-GAIL (Zhang et al., (020), which is considered improving sample- efficiency. Recently, FAIRL utilizes the forward KL divergence (Ghasemipour et al.] 2020) and achieves great performance, but it is still not robust and efficient enough. These methods rely heavily on carefully tuned iiseriminator-baseee reward function. It might focus on the minor differences between the expert and the generated samples and thus gives sparse reward signal to the agent. In comparison, our auto-encoder-base reward function learns the full-scale differences between the expert and generated samples.
Inverse Reinforcement Learning (IRL) IRL learns a reward function from the expert demonstrations, which assumes the expert performs optimal behaviors. One category of IRL uses a hand-crafted
similarity function to estimate the rewards (Boularias et al.] 201T Klein et al.) 2013 Piot et al.) 2016).
The idea of these methods is still inducing the expert policy by minimizing the distance between
the state action distributions of the expert and sampled trajectories. Primal Wasserstein Imitation Learning (PWIL) (Dadashi et al., 2021) is one such algorithm, which utilizes the upper bound of Wasserstein distance's 's primal form as the optimization objective. The advantage of these methods is that they are relatively more robust and less demonstration dependent than AIL methods. However, the performance of these methods heavily depends on the similarity measurement, and therefore, it varies greatly on different tasks. Compared to PWIL, our method achieves superior performance via automatically acquiring a similarity measurement. Random Expert Distillation (RED) (Wang al. 2019) uses random network distillation compute the rewards for imitation learning. The error between the predictor and the random target network could guide the policy to mimic the expert behaviors. Meanwhile, auto-encoder could he used place of the random network distillation. In comparison, RED uses fixed auto-encoder while our AEAIL utilizes the AE in adversarial manner. ValueDice (Kostrikov et al. 2020) utilizes a critic based reward function which is optimized during training the policy and critic. However, the performance of ValueDice is sensitive to the expert data used. IQ-Learn (Garg et al.] (021) is a dynamics- aware IL method which avoids adversarial training by learning single Q-function, implicitly representing both the reward and the policy.
2.2 IMITATION LEARNING WITH IMPERFECTION
The robustness of adversarial imitation learning is questionable with imperfection in observations (Stadie et al.) 2017 Berseth & Pal 2020), actions, transition models (Gangwani & Peng 2020 Christiano et al. 2016), expert demonstrations (Brown al., 2019 Shiarlis et al. 2016 Jing et al. 2020> and their combination (Kim et al. 2020). Prior robust imitation learning methods require the demonstrations to be annotated with confidence scores (Wu et al.) 2019 Brown et al. 2019
Grollman & Billard 2012). However, these annotations are rather expensive. Compared to them, our auto-encoder-basee reward function won't 't easily overfit to the noisy feature. Therefore, our AEAIL
2

---

Under review as a conference paper at ICLR 2023
min (E (s.a) (8) (s.a)) D (6)))
Expert Samples
8
MSE
(6)
Generated Samples
-
1
5
||x 111
max E (s.a)- (6)]
Figure 1: Training framework of AEAIL. The auto-encoder computes the reconstruction error for these two mini-batches of data examples and optimizes the derived objectives. Therefore, a surrogate reward function (d) provides the signal the agent. The agent can induce the expert policies by iteratively training the auto-encoder and the agent policy.
relatively straightforward and robust to the noisy expert data and does not require any annotated data.
2.3 AUTO-ENCODING BASED GANS
Auto-encoders have been successfully applied improve the training stability and modes capturing in GANs. We classify auto- auto-encoding based GANs into three categories: (1) utilizing an auto-encoder as the discriminator such as energy-basee based GANs (Zhao et al., (2016) and boundary-equilibrium GANs (Berthelot et al. (2017); (2) using denoising auto-encoder to derive auxiliary loss for the generator Warde-Farley & Bengio 2017;; (3) combining variational auto-encoder and GANs to generate both vivid and diverse samples by balancing the objective of reconstructing the training data and confusing the discriminator (Larsen et al. 2016). Our method AEAIL takes inspirations from EBGAN (Zhao et al. 2016), utilizing the auto-encoder in the reward function.
3 BACKGROUND
Inverse Reinforcement Learning (IRL) IRL extracts the policy towards mimicking expert behaviors with the help of learned reward function. On the other hand, AlL directly induces the expert policies by minimizing the distribution distance between expert demonstrations and ggent-generated samples.
Adversarial Imitation Learning (AIL) Assume that we are given set of expert demonstrations, AIL involves discriminator which tries distinguish the expert and policy generated samples and policy that is optimized to confuse the discriminator accordingly. Ideally, the rollouts from the agent will indistinguishable from the expert demonstrations in the end. The discriminator cannot classify the expert samples and policy-generated ones any more at the end of the training process. Consequently, this kind of discriminator based reward function can be seen as pseudo-reward signal, which is only used for optimizing policies during training.
Wasserstein Divergence One popular distribution divergence for measuring the distance between policy samples and expert samples is the Wasserstein divergence (Arjovsky al. 2017
d(p,q) sup Ex-pld(x)] Eyyoy( 161:51
(1)
where p and g two probability distributions, is some function that is 1-Lipschitz continuous. Equation [ measures the distance between these two distributions. Intuitively, we can view it as the minimum cost of turning the piles from distribution into the holes in distribution q-

---

Under review as a conference paper at ICLR 2023
4 METHODOLOGY
4.1 OVERVIEW
The reward function in AIL methods is discriminator, which attempts to assign high values to the regions near the expert demonstrations and assign low values to the other areas. However, the discriminator would easily overfit the minor differences between expert data and policy-generated samples. Consider the CartPole balancing task, the goal of this task is to keep the pole balancing via moving the cart left or right. The state of environment is feature consisting of position, angle, angle's rate, and cart velocity while the action is moving left right. Here, we assume that all the expert states' velocity is 2, for example. When the generated states' velocity is 1.9, and other dimensions of state-action pair are the same as the expert's, s, the discriminator of GAIL would still give a low reward on these generated state-action samples. However, the policy rollouts may perform very well towards mimicking the expert's is behaviors. In other words, the reward function in GAIL. on this example overfits to the minor differences between the expert and the sampled data and leads the policy miss the underlying goal of the expert.
In this paper, we propose an auto-encoder-base based reward function for adversarial imitation learning. It utilizes the reconstruction error of the state action pairs to compute the reward function. The reconstruction eπror based reward signal force the discriminator to consider all information in the state-action tuple, rather than focusing on the minor differences. Therefore, this reward signal wouldn't lead to overconfidence in distinguishing the expert and the generated samples. Recall the CartPole example, the mean square error between states' velocity 1.9 and is very small. And could still feed good reward to the agent under this situation. Therefore, intuitively proposed reward signal can better focus on the full-scale differences between the expert and generated state action pairs, yielding more informative rewards and thus improving the policy learning.
4.2 OUR METHOD
Our approach is to minimize the distance between the state action distribution of the policy va and that of the expert demonstrations DE.
The objective formulation we used in our method is Ho et al. (016);
d(Tg, TT) sup
L (s, a)] E (s,
IrulesK
where the reward function network's parameters are denoted as w and the policy network's parameters represented 6. Minimizing this distance will induce the expert policy from expert demonstra- tions. Therefore, the optimization of the policy Ty and the reward function (w, a) forms a bi-level optimization problem, which can be formally defined
min max (E(s.a)>Dy [n a)] (s, (s,
(3)
Optimizing Equation 3 leads an adversarial formulation for imitation learning. The outer level minimization with respect to the policy leads to learned policy that close to the expert. The inner level maximization recovers reward function that attributes higher values to regions close to the expert data and penalizes all other areas.
In our method, defined as:
auto-encoder based surrogate pscudo-reward function instead, which is
Ton (s) 1/(1 + as,
where AE is the reconstruction eтror of an auto-encoder:
AB(x:) = |[ee o Ene(x: - 3r x²13
Here, represents the state-action pairs. Equation is the mean square error between the sampled state-action pairs and the reconstructed samples. This form of the reward signal uses the reconstruction error of an auto-encoder to score the state-action pairs in the trajectories. Equation monotonically decreasing function over the reconstruction error of the auto-encoder. We give high rewards to the state-action pairs with low reconstruction emoos of the auto-encoding process and vice versa. Section
4

---

Under review as a conference paper at ICLR 2023
4.1 motivates that this form of reward signal focuses more the full-scale differences between the expert and generated samples, and won't I easily overfit to the noise of the expert data.
Training the auto-encoder is adversarial process considering the objective 3 which is minimizing the reconstruction error for the expert samples while maximizing this error for generated samples.
When combining Equation and Equation as:
obtain the training objective for the auto-encoder
L = (s.a) (w a)] (Muu))
(6)
(s.))) [1/(1 + (s, - (L,,)) [1/(1 (E,
With this adversarial objective. the auto-encoder learns to maximize the full-scale differences be- tween the expert and the generated samples, As a result, it gives the agent denser reward signal. Furthermore, the agent can also be more robust when facing noisy expert data due to the robust auto-encoding objective.
Figure I depicts the architecture for AEAIL. The auto-encoder- based reward function takes either expert or the generated state-action pairs and estimates the reconstruction error based rewards accordingly. We iteratively optimize the auto-encoder and the agent policy under this adversarial training paradigm. Algorithm Tin in Appendix A. shows the pseudo-code for training AFAIL.
4.3 THEORETICAL PROPERTIES
Our AEAIL also shares a similar theoretical framework with GAIL, where the auto-encoder based reward function can be theoretical properties GAIL. We make the same set of assumptions for our auto-encoder based reward function as these for the discriminator in GAIL (Guan al. 202I):
a generalized discriminator. Guan al. (202T) investigates the
(i) the auto-encoder based reward function has bounded gradient, i.c., 3C, E IR, such that
a wwlloo
w
Σ
Orw I& S CCr
Ow;
(ii) the auto-encoder based reward function has L-Lipschitz continuous gradient, i.e., 3L, R. such that Vs E S, Va E A. Vun, w2 E W, then:
VV (s. Vurws a)ll S L, w2l12
(iii) the distribution divergence
(y Po) = sup E [r())] Ez (r(x)) <(r(x))
rER
is properly regularized to be strongly concave with respect to the reward function. Here, <(r(x)) is the regularizer. We prove that the reward function in our AEAIL K-Lipschitz with clipping the weights of the auto-encoder. (Proof see in Appendix A.2).
These conditions are assumptions for theoretical convenience since they don't always hold for multi-layer neural network- based discriminator in GAIL proposed auto-encoder. When these assumptions hold, our AEAIL shares the same theoretical properties as GAIL: converges globally.
EXPERIMENTS
Environments: we conduct the experiments on six locomotion tasks with varying dynamics and
difficulties: Swimmer-v2 (Schulman et al. 20115), Hopper-v2 (Levine & Koltun| 2013), Walker2d-v2
(Schulman et al. 2013). HalfChectah-v2 (Heess et al. 2015). Ant-v2 (Schulman al. 2016). and
Humanoid-v2 Tassa al. 2012).
Baselines: the baselines we choose for comparison includes: GAIL (Ho & Ermon 2016). FAIRL
(Ghasemipour et al. 2020), PWIL (Dadashi et al. 2021] and DAC (Kostrikov et al. 2019
Implementation: experimental details, including the implementation and more results, are shown in Appendix A.3 A.5
5

---

Under review as a conference paper at ICLR 2023
Walker2d
Swimmer
Hopper
15C
3000
56
sogust
rendom
GAA
FA.IL
DOC
-50
PVLL
100
HalfCheetah
Humanoid
6000
6000
5000
6000
4000
4000
4000
5000
2000
1000
40 50
80
100
Figure 2: Mean and standard deviation return of the deterministic evaluation policy over 10 rollouts and seeds, reported every 100k timesteps, which are learning from non-noisy expert demonstrations.
Table I: Learned policy performance for different imitation learning algorithms on clean expert data.
Task
Walker2d
Hopper
Swimmer
HalfCheetah
Ant
Humanoid
Expert
4865.1
3194.9
142.2
8193.6
5730.9
5187.9
Random
-0.6 + 0.3
26 +
-5 + 14
-944 + 108
-401 + 233
110 +
GAIL
4259 + 369
3152 + 102
106 + 27
6964 + 807
4998 + 686
3257 + 1393
FAIRL
2912 825
3117 + 75
105 21
6917 : 267
5247 + 259
1695 + 1098
PWIL
4226 + 501
3051 51
6376 + 229
4808 + 645
3466 + 736
DAC
4533 248
3208
4
8420 + 29
1610 + 2328
602 = 232
Ours
4810 111
3125 159
111 + 22
7458 + 191
5394 451
4717 346
5.1 LEARNING FROM EXPERT DEMONSTRATIONS
Imitation with Clean Expert To compare the performance our proposed AEAIL and other baselines, we run the experiments on the clean expert dataset. We used the ground truth average return for evaluating different learned policies. Higher rewards indicate better mimicking expert behaviors. Figure (atrre22eppn depicts the training curves of ground truth rewards for different imitation learning algorithms, and Table shows the final learned policy performance on MuloCo tasks with ground truth episode return.
Results show that PWIL performs well on Walker2d, Hopper, HalfCheetah, and Humanoid but still good enough. GAIL and FAIRL are comparable on Hopper, Swimmer, HalfCheetah, while GAIL somehow better than FAIRL on other tasks. DAC achieves great performance on Walker2d, Hopper and HalfCheetah while it performs poorly on other tasks.
improvement (%))
60
20
19
Our method's overall scaled reward is about 0.921, whereas the best baseline 0.83 for GAIL. There an about 11% relative improvement. Our method outperforms other state-of oftte-of-the-an the-art baselines on all locomotion tasks except for Hopper and HalfCheetah. where our approach doesn't 't outperform DAC. Here would like to point out that our AEAIL has already
Swimmer
Hopper
Cheetah
Walker
Ant
Humanoid
Figure 3: Relative improvement of AEAIL compared to GAIL. Environments are sorted by the number state-action dimensions.

---

Under review as a conference paper at ICLR 2023
Walker2d
Swimmer
Hopper
150
2300
3000
4000
2500
3000
56
1500
2000
GAL
1000
FARL
DAC
PHLL
Durn
100
HalfCheetah
Humanoid
6000
8000
8000
5000
6000
2000
2000
2000
1000
50
Figure 4: Mean and standard deviation return of the deterministic evaluation policy over 10 rollouts and seeds learning from noisy expert demonstrations, reported every 100k timesteps.
Table 2: Learned policy performance for different imitation learning algorithms on noisy expert data.
Task
Walker2d
Hopper
Swimmer
HalfCheetah
Ant
Humanoid
GAIL
3358 + 623
1361 + 725
93 +
3148 + 1096
3353 + 1370
1110 + 296
FAIRL
3686 + 772
2483 + 899
97 +
3451 + 588
-1987 + 1633
544 + 241
PWIL
4043 + 559
2854 = 48
4933 = 343
729 = 1274
2321 + 1038
DAC
529 + 138
357 171
36
2019 = 900
-1586 922
374 + 66
Ours
4149 + 423
2935 + 110
85 15:00 17
7017 + 844
5204 + 421
3721 + 1831
achieved 97.8% of the expert performance on Hopper while 91.7% on HalfChectah, which is very close to completely solving the tasks.
Relative Improvements of AEAIL Figure shows the relative improvement of overall scaled rewards for our AEAIL compared to GAIL on clean expert data. The abscissa axis represents different tasks with the growth of task dimension, from Swimmer (dim=10) to Humanoid (dim=393). With the increasing task dimension, the relative improvement of our AEAIL shows an increasing trend. It indicates that method has much more benefits when applied into high dimensional environments. We hypothesize that the discriminator in GAIL would focus much more on the minor differences between the expert and agent samples with the growth of task dimension. It is harder for GAIL to scale in higher dimensional tasks. On the other hand, our provides more information to the policy learning. Consequently, AEAIL scales relatively better than GAIL in higher dimensional tasks.
process
5.2 LEARNING FROM NOISY EXPERT DEMONSTRATIONS
Imitation with Noisy Expert To show the robustness of our proposed AEAIL, we further conduct experiments on noisy expert data. We add a Gaussian noise distribution 0.3) the expert data for Walker2d, Hopper, Swimmer and HalfCheetah. Since Ant and Humanoid are much more sensitive to noise, we add (0, 0.03) Gaussian noise these two tasks.
Table 2 shows the final learned policy performances on benchmarks. Figure shows the training curves on noisy expert setting. These results show that our method outperforms other state-of-the-ar algorithms all tasks except for Swimmer,
which FAIRL wins. Our AEAIL offers excellent capability in learning from noisy expert data on these tasks. The overall scaled rewards for our AEAIL is 0.813, whereas the best baseline is 0.539 for PWIL on the noisy expert setting. There is an about 50.7% relative improvement. Other baselines including GAIL, FAIRL and especially DAC, are very sensitive to the noisy expert.
7

---

Under review as a conference paper at ICLR 2023
Video Performance We also record vided for the performance of different methods on both clean and noisy data. It shows that only our AEAIL can move forward as quickly as the expert from noisy expert data, while other baselines suffer from various types of failures.
5.3 EMPIRICAL ANALYSIS
A denser reward signal is beneficial for policy search (Ng et al., 1999 Hu et al. 2020). Section 4.] motivates that our cetto-encoder-basee based reward function focused on the full-scale differences between the expert and generated samples. Our method intuitively provides the agent with denser reward signal than discriminator based ones. In this subsection, we will verify this empirically.
Question 1. Is the auto-encoderrbasel reward signal denser than the discriminator-base one?
To answer Question 1, we define the denseness as the amount of reward information for the same batch from agent rollouts. Higher amounts of reward in- formation for the same batch would be beneficial for policy training since it contains much more learning signals. And we define as denser reward signal. Specifically, we compute the differential entropy for each reward batch as:
1.0
0.9
Entropy
0.8
0.7
H(z)= - = lo)) log f())
FAIRL
(7)
0.6
Ours
60
80
100
where the probability density function whose support is set X. For computation convenience, we discrete the rewards into 100 sections to compute the probabilities. Therefore, can get the entropy of each reward batch from agent rollouts.
Timesteps (1e5)
Figure 5: Entropy of learned rewards for state- action pairs in each batch for training the agent-policy overall.
Figure 5 shows the averaged differential entropy curve of the learned pseudo-rewards. This shows the generated samples' entropy in each batch during policy optimization, reported every 100 steps. Our AEAIL gets a much higher entropy during the training process. This means the reward batch contains much more information than the baseline. It also shows that our encoder-based reward signal is indeed denser than the discriminator based one.
5.4 ABLATION STUDY
Question 2. What is the major contributing factor of our AEAIL? Could it be the specific W-distance metric?
To analyze the major contributing factor of our AEAIL, we conduct an ablation study that replaces the distance other distribution divergences. Comparable performances indicate that the major contributing factor is the micoding-decoding process rather than the specific distribution divergence. To justify this hypothesis, we formulate another form of surrogate reward function based on Jensen- Shannon divergence (Goodfellow et al. 2014). Details see in Appendix A.4
Jensen-Shannon divergence (Goodfellow et al. 2014) is defined as:
P₂(x) (Pr + (x)) Ez.p, [log( (P,(x) + PP (x)))
P (a)
IS Pa) EE [log(
(8)
where P, and Py represent the expert data distribution and the agent-generated sample distribution accordingly.
Our JS-based variant induces the expert policy by minimizing the distribution distance between expert and generated samples. Specifically, we define that:
p, (x))/PP,(()) Py(()) exp(-AE(z))
Video results, code and data attached in supplementary materials.
8

---

Under review as a conference paper at ICLR 2023
where AE(x:) is the same with Equation 5
original AEAIL. Minimizing Jensen-Shanne Shannon divergence assigns low mean square errors to the expert samples and high mean square errors to other regions.
shows that our JS-based vari- achieves 10.5% and 44.9% rel- ative improvement compared to the best baseline GAIL and PWIL on clean and noisy expert data, respec- tively. Similar the original AEAIL, our JS iS-based variant also greatly im- proves the imitation performance on these benchmarks. The relative im- provements are comparable between the two distance metrics. It indicates that the major contributing factor our AEAIL is the encoding-decoding process. This also shows that AEAIL works not limited to specific distance metric.
Table 3: Relative improvements for different variants of our AEAIL compared the best baseline GAIL and PWIL on clean and noisy data, respectively.
Improvements
Ours
Ours-JS
Our
Clean Data
11.0%
10.5%
7.6%
Noisy Data
50.7%
44.9%
42.1%
Question 3. Is AEAIL limited to the specific rype of nuto-encoders? How about utilizing variational auto-encoders?
To prove that our AEAIL is not sensitive to a specific type of auto-encoders, we conduct experiments by replacing the vanilla auto-encoder with the variational auto-encoder. If variational auto-encoder leads to similar performance on the benchmarks, illustrates that AEAIL works under a wide range of different auto-encoders. We run the VAB-based variant MuJoCo tasks under the setting as the AE-based one. Details see in Appendix A.5
Table 3 shows VAE-based variant gets 7.6% and 42. 1% relative improvement compared to the best baseline GAIL and PWIL on clean and noisy expert data, respectively. This means that our VAE-based variant improves the imitation performance consider- ably compared to other baselines. It justifies that our AEAIL is flexible with different auto-encoders.
Averaged Return
0.8
0.6
M
0.4
size 12e-(1100,10]
<ix==550,50]
0.2
size = [200.200]
Question 4. How our AEAII. performs with varying hidden size of the AE?
220=f200,50.0000
0.0
slze-(200,100.200
20
40
60
80
100
Timesteps (1e5)
To justify that our AEAIL is not sensitive to different sizes of the auto-encoder, we conduct ablations with different hidden sizes of the auto-encoder. If different sizes of the AE lead to similar imitation performance, it proofs that our AEAIL is not sensitive the hidden size of the auto-encoder.
Figure 6: Training curves with averaged scaled rewards overall for varying hidden sizes of the AE.
Figure 6 shows the averaged overall scaled rewards of AEAIL with different hidden sizes of the auto-encoder. They achieve comparable results for most the choices. However, when the size of AE is too large, the imitation performance of our AEAIL would decline. It depicts that the hidden size of auto-encoder cannot be too large. Otherwise, the AEAIL is not sensitive to the size of the auto-encoder.
CONCLUSIONS
This paper presents a straightforward and robust adversarial imitation learning method based on auto-encoding (AEAIL). We utilize the reconstruction eтror of an auto-encoder as the surrogate pseudo-reward function for reinforcement learning. The advantage is that the auto-encoder based reward function focused on the full-scale differences between the expert and generated samples, which provides a denser reward signal to the agent. As a result, it enables the agents to learn better. Experimental results show that our methods achieve strong competitive performances both clean and noisy expert data. In the future, we want to further investigate our approach in more realistic scenarios, such as autonomous driving and robotics.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pieter Abheel, Varun Ganapathi, and Andrew Y. Ng. A learning vehicular dynamics, with application to modeling helicopters. In Advances Neural Information Processing Systems (NeurIPS), 2006.
Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y. Ng. An application of reinforcement learning to aerobatic helicopter flight. In Advances in Neural Information Processing Systems (NeurIPS), 2007.
Pieter Abbeel, Adam Coates, Timothy Hunter, and Andrew Ng. Autonomous autorotation of re helicopter. In International Symposium on Robolics, 2008a.
Pieter Abbeel, Dmitri Dolov, Andrew Y. Ng, and Sebastian Thrun. Apprenticeship learning for motion planning with application to parking lot navigation. In JEEE/RSj International Conference on Intelligent Robots and Systems, 2008b.
Picter Abbeel, Adam Coates, and Andrew Y. Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 2010.
Martin Arjovsky, Soumith Chintala, and Leon Bottou. Wasserstein gan. In International Conference Machine Learning (ICMI.), 2017.
Enda Barrett and Stephen Linder. Autonomous hva control, a reinforcement learning approach.
Machine Learning and Knowledge Discovery in Databases, 2015.
Glen Berseth and Christopher Pal. Visual imitation with reinforcement learning using recurrent siamese networks. 2020. URL /tt///ttss///peeeeeee.... http://wwweee//tommmdddddd--e-ed
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning, In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 182-189. JMLR Workshop and Conference Proceedings, 2011.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference Machine Learning (ICML), pp. 783-792. PMLR, 2019.
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. 2016.
Adam Coates, Pieter Abbeel, and Andrew Y. Ng. Learning for control from multiple demonstrations. In International Conference on Machine Learning (ICML), 2008.
Robert Dadashi, Leonard Hussenot, Mathieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. International Conference on Learning Representations (ICLR), 2021.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. International Conference on Learning Representations (ICLR), 2017.
Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. In International Conference Learning Representations (ICLR), 2020.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Mathieu Geist, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Seyed Ghasemipour, Kamyar Seyed, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. Conference on Robot Learning (CoRL), pp. 1259- 1277. PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Daniel H Grollman and Aude G Billard. Robot learning from failed demonstrations. International Journal of Social Robotics, 4(4):331-34,, 2012.
Ziwei Guan, Tengyu Xu, and Yingbin Liang. When will generative adversarial imitation learning algorithms attain global convergence. In International Conference Artificial Intelligence and Statistics (AISTATS), pp. 1117-1125. PMLR, 2021.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2944-2952, 2015.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Jonathan Ho, Jayesh Gupta, and Stefano Ermon. Model-free imitation learning with policy optimiza- tion. In International Conference on Machine Learning, pp. 2760-2769. PMLR, 2016.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5109-5116, 2020.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning (ICML), pp. 5286-5295. PMLR, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Edouard Klein, Bilal Piot, Matthieu Geist, and Olivier Pietquin. A cascaded supervised learning approach to inverse reinforcement learning. In Joint European conference on machine learning and knowledge discovery in databases, pp. 1-16. Springer, 2013.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations (ICLR), 2019.
Diseriminator-actorr
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off- off-policy distribution matching. In International Conference on Learning Representations, 2020. URL. https:// open net. www.t/eeno/trree www.ooounnniiiiigaaaaaaa
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. In International Conference Machine Learning (ICML), 2016.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning (ICMI., pp. 1-9, 2013.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In International Conference on Machine Learning (ICML), 1999.
Andrew Y. Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang. Inverted autonomous helicopter flight via reinforcement learning. In International Symposium Experimental Robotics, 2004.
11

---

Under review as a conference paper at ICLR 2023
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Bridging the gap between imitation learning and inverse reinforcement learning. IEEE transactions on neural networks and learning systems, 28 (8):1814-1826, 2016.
Ahmed H. Qureshi, Byron Boots, and Michael C. Yip. Adversarial imitation via variational inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. Maximum margin planning. In International Conference on Machine Learning (ICML), 2006.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning (ICML), pp. 1889-1897, PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensiona continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR), 2016.
Kyriacos Shiarlis, Joao Messias, and Shimon Whiteson. Inverse reinforcement learning from failure. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems (AAMAS), pp. 1060-1068, 2016.
Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third perrd-person imitation learning. In International Conference on Learning Representations (ICLR), 2017.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 190664913. IEEE, 2012.
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. International Conference PMLR, 2019.
Machine Learning, pp.
David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. In International Conference on Learning Representations (ICLR), 2017.
Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imita- tion learning from imperfect demonstration. Proceedings of the 36th International Conference 2019.
on Machine Learning (ICML), pp. 681
Xin Zhang, Yanhua Li. Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Junbo Zhao, Machael Mathieu, and Yann LeCun. Energy-based generative adversarial network. International Conference on Learning Representations (ICLR), 2016.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. AAAI Conference on Artificial Intelligence (AAAI), 2008
Brian D. Ziebart, Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference Machine Learning (ICML), 2010.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1
PSEUDO CODE
Algorithm I Auto-Encoding Adversarial Imitation Learning (AEAIL)
Input: Initial parameters of policy, auto-encoder Uu. we: Expert trajectories Dg. repeat
Sample state-action pairs (8;; 0z) ne, and (sg, ag) ~ DE with same batch size.
Update W, to W4+1 by decreasing with the gradient:
E (s) V my 1/(1 + AEw, (8i, az))] - B(en 1/ + ABm, a))]
Take policy step from 0, 0.+1, using the TRPO update rule with the reward function 1/(1 + AEw, (s, a)), and the objective function for TRPO is:
F(s.a)--1// AE (s, a))].
until Ty converges
Algorithm I depicts the pseudo-code for training AEAIL. The first step is to sample the state-action pairs from expert demonstrations Dg and the trajectories sampled by the current policy with the same batch size. Then we update the auto-encoder by decreasing with the gradient with loss function Eq. 6 Finally, we update the policy assuming that the reward function Eq. which leads to an adversarial training paradigm. We repeat these steps until the approach converges.
A.2 PROOF ON THAT AEAIL IS MINIMIZING THE W-DISTANCE
In this section, we provide proof that clipping the weights of the auto-encoder can make the reward function K-Lipschitz. Therefore, can obtain that our AEAIL actually minimize W-distance.
Theorem 1. (AE Error is 2m(K K + 1)-Lipschitz) Provided that f() is K-Lipschitz, then the recon- struction error of our auto-encoder (x - f()))² is 2 everywhere.
Proof. If fz) is K-Lipschitz, then according to the definition of K-Lipschitz. There exists constant K, for all real Iı and 22, such that:
(f()) ((22) S K| x2l
where fax is realevalued function.
Then, we can obtain:
( [(x1) -x2-- f(x2)) = (2 12) - (f(m)) [(x2)) & (K 1))m 22l
So there exist K1 = K + 1, for all real #i and .2, such that
[(x1 f(x1) (x2 - f(x2)) < K1|z1 x22
We can obtain that (x - fx)) K2 Lipschitz
Let g(z) f(x), we can obtain that g(z) is K2 Lipschit, then there exists constant K, for all real V1 and Ø2, such that:
Ig(x1) - g(x2) S K1 |my - .22
(10)
In our AEAIL, & is the input feature which is bounded while f(x is also bounded since the weights of the auto-encoder is bounded via weight clipping. So g(z) is bounded.
We obtain:
(21) (xz)) 19(x1) g(x2)) (0(x1) + g(x2)) Ki|21 x2l. (g(2)) (x2))
13

---

Under review as a conference paper at ICLR 2023
We assume that g(x) is bounded by constant then:
gg (21) g (x2)) s 2mK ø2l
So there exist K2 2m such that for all real æ1 and m2:
(x)) - (x2)) < K2/æi x2| 2m(K 12 x2l
We can obtain that the reconstruction error (x) = (x - f(x))) is also 2m(K + Lipschitt
Corollary 1. (Our AEAIL minimizing Wasserstein distance) the divergence for AEAIL: d(Tg, To)
sup
E E (s,
(11)
InulesK
is Wasserstein distance.
Proof. Arjovsky et (2017) shows that given F is set of functions from X' RR we can define
dr Po) sup Er p. (x)] - [x
JEF
as an integral probability metric with respect to the function class F. When F is K-Lipschitz functions, we can obtain dF(Pr, Pe) the Wasserstein distance.
For our AEAIL, our reward function is:
r(s, a) =
((8,a)=1//) 1/(1 + | - Dec Enc(x)]12)
So the reward function is K-Lipschitz since we clip the weight of the auto-encoder. This will yield the same behaviour. Therefore, AEAIL. actually optimizes the W-distance.
A.3 IMPLEMENTATION
Data, code, and video see supplementary materials. This section will introduce the implemen- tation details, including tasks, basic settings for all algorithms, baseline implementations, network architecture for our approach.
A.3.1 TASKS
(a)
(b)
(c)
(e)
Figure 7: Illustrations for locomotion tasks we used in our experiments: (a) Swimmer; (b) Hopper; (c) Walker; (d) HalfCheetah; (e) Ant: Humanoid.
The goal for all these tasks is to move forward as quickly as possible. These tasks are more challenging than basic ones due to high degrees of freedom. In addition, great amount of exploration is needed to learn to move forward without getting stuck at local optima. Since we penalize for excessive controls and falling over, during the initial stage of learning, when the robot cannot move forward for a sufficient distance without falling, apparent local optima exist, including staying at the origin or moving forward slowly. Figure depiets locomotion tasks' environments.
14

---

Under review as a conference paper at ICLR 2023
A.3.2 RESOURCES AND LICENSE
We GTX 3080 GPU and 40 CPU to run the experiments. Our algorithm needs 120 hours for training on tasks with 5 random seeds. We use an open source MuJoCo license to run the locomotion tasks. The license can found here: https:/mmujoco.org
A.3.3 BASIC SETTINGS
We sample the expert trajectories with SAC on Walker2d, Hopper, Swimmer, HalfCheetah, and Ant with trajectories while 320 trajectories on Humanoid.
We normalize the state features for each task and use the normalized features as the input for the auto-encoder. We also use the raw data input to compute the final output of the mean square error.
We train these algorithms directly on Walker2d, Hopper, and Swimmer, while on HalfCheetah, Ant, and Humanoid, we use BC to pre-train the policy with 10k iterations. test all the algorithms on these six locomotion tasks with five random seeds and using ten rollouts to estimate the ground truth rewards to plot the training curves.
The maximum length for sampled trajectories is 1024. The discounted factor is 0.995. Each iteration, we update three times of policies and one time of the reward function (discriminator or auto-encoder). While updating TRPO, update the critic five times with learning rate of 2e-4 every iteration.
We clip the auto-encoder's 's parameters into [-0.1, 0.1] to make the AE L-Lipschitz continuous.
A.3.4 BASELINES
We open-source code, OpenAl baselines (Dhariwal et al. 2017), to get the performance of BC, and the training iterations is 100k for all tasks.
GAIL (Ho & Ermon 2016) borrows the GAN (Goodfellow al. 2014) architecture to realize the IRL process, and it is applicable in high dimensional dynamic environments. We implement GAIL with open-source code, OpenAl baselines (Dhariwal al.) (017), with best tunned hyperparameters. We choose TRPO as the reinforcement learning algorithm for a fair comparison.
FAIRL (Ghasemipour al. 2020) utilizes the forward KL divergence the objective, which shows competitive performances on MuloCo tasks. FAIRL is implemented based on GAIL with corresponding objective function as in (Ghasemipour et al. 2020). We also choose TRPO as the reinforcement learning algorithm for a fair comparison.
PWIL (Dadashi et al. 2021) is imitation learning method based offline similarity reward function. It is pretty robust and easy to fine-tune the two pyyee--arametter).. PWIL implemented based on open source code (Dadashi et 2021) with TRPO as the reinforcement learning method for fair comparison.
of its reward function (only
DAC Kostrikov al., 2019 is discriminator-actor- states' rewards to improve the imitation performance. DAC is implemented via its open source code.
algorithm which introduces absorbing
A.3.5 ARCHITECTURE OF OUR METHIOSS
Policy Net: two hidden layers with 64 units, with tanh nonlincarities in between, which is the same with basclines. The learning rate we used for updating the policy is 0.01.
Auto-Encoder Net: (both vanilla auto-encoder and variational auto-encoder) the number layers is the same with the architecture of discriminator in GAIL, i.e., four layers. The two hidden layers are with 100 units, with tanh nonlinearities in between, the final output layer is an identity layer. We normalize the state feature after the input layer, and we use the state action input features compute the mean square error. The learning rate for the auto-encoder is 3e-4.
A.4 ABLATION STUDY: DIFFERENT DISTRIBUTION DIVERGENCES
We conduct ablation study with Jensen- divergence. We run the experiments with new form reward function based on -hasennn- divergence (Goodfellow et al.) 2014) for comparison.
15

---

Under review as a conference paper at ICLR 2023
divergence is defined as:
(x JS Pg) = Ex [oog( 2 (x) + P2) +1 Teerrpooge
p (x) (P, + (x))
(12) where P. and Pg represents the expert data distribution and the generated data distribution. The adversarial training procedure of AEAIL will match the state-action distribution of expert and generated samples.
)
In this form of reward function, we consider the exponential of negative reconstruction error as: (xr) (xx) P,(x)) The derived Jensen-Shannon divergence for updating the auto-encoder based reward function is:
C = - (sss))) (loaa)=blloggee((AAAEE (s, a)))] - (., [log(1 exp(- (s)
(13)
=aa,,))D AE (s, a)] ()) [log(1 exp( exp(-AEwwl (s, a)))]
(14)
Here, represents the parameters of the auto-encoder network. Its formulation of surrogate reward function is:
Tw (s. 0) = log(1 cxp(-AEE a)))
(15)
Minimizing the Censen-Shannor divergence between state-action distribution of the expert and generated trajectories assigns low mean square errors to the expert samples and high mean square eroos to other regions. And intuitively, give high rewards to the regions with low reconstruction errors and vice versa.
Table 4: Learned final policy performance for our AEAIL with W-distance, AE based variant (Ours), JS-distance (Ours-JS), and VAE based variant (Ours- VAE). Top three rows: non-noisy expert; Bottom three rows: noisy expert.
Task
Walker2d
Hopper
Swimmer
HalfCheetah
Ant
Humanoid
Ours
4810 + 111
3125 + 159
111 + 22
7458 = 191
5394 + 451
4717 + 346
Ours-JS
4589 + 498*
3245 65t
123 3
7901 226
5873 1: 512t
3564 432
Ours-VAE
4813 = 3
3198 + 142
106 6
8042 = 208
4901 = 687
3981 + 494
Ours
4149 1 423
2935
17
7017 844
5204 1 421
3721 1831
Ours-JS
4298 + 338
2698 + 128
81 4
6768 = 687*
5381 + 512
3121 + 1254
Ours-VAE
4354 + 3971
3018 + 105
7
6782 + 781
4292 + 418
3082 + 1291
The results in Table show that W-distance based method is comparable with JS-distance based method on almost all tasks. Our IS based variant achieves 10.5% and 44.9% relative improvement compared the best baseline GAIL and PWIL on clean and noisy expert data, respectively. It is similar to the W-distance based variant. This indicates that the -eeoodingdeooinnggg process is the major contributing factor in our AEAIL which improves the imitation performance rather than specific distribution divergence.
A.5 ABLATION STUDY: UTILIZING OTHER AUTO-ENCODERS
We conduct experiments with variational auto-encoder based reward function as an ablation. The variational auto-encoder (Kingma & Welling 2014) directed model that uses learned approximate inference. The critical insight behind training variational auto-encoder is to maximize the variational
lower bound L(q) of the log- likeliioldood for the training examples (Goodfellow et al. 2016):
L(g) = EEzgg(2)a) log Pmodel (modell,xx) H((g)11=))
(16)
= - log Pmodel (x|/² DRL (ec.gg(e D
(17)
< log (z)
(18)
where z is the latent representation for data points æ. And Pmodel is chosen to be a fixed Gaussian distribution (o0, 7). Training the variational auto-encoder is still adversarial process. We notice
16

---

Under review as a conference paper at ICLR 2023
Walker2d
Swimmer
Hopper
6300
600
4000
5000
3000
4000
400
1000
300
2000
2900
100
1000
40
60
100
0
40
50
100
20
40
80
100
HalfCheetah
Ant
Reacher
8000
0000
0.00
4000
Wan
Mr
2000
-40
2000
200
PWIL
20
20
40
50
20
100
Figure 8: Mean and standard deviation return of the deterministic evaluation policy over 10 rollouts and seeds learning from expert demonstrations, reported every 100k timesteps.
that the first term in Eq. 17 i7ss the reconstruction log- loglliielhhoo found in traditional auto-encoders. The objective function for training the variational auto-encoder is:
C = (y + Dx (p(2(ss, a)), (2))de
(19)
(( a) + DEL (p(2 ()), (2))]
where the choice for (s, a) the same with our AE based one, which is Eq. in Section 4.2 When minimizing this objective, the variational auto-encoder based reward function can still assign high values near the expert demonstrations and low values to other regions. the other hand, optimizing this objective is restricting the latent distribution for the expert fixed Gaussian distribution Pmodet (z) and maximizing the KL distance between the generated samples and the same fixed Gaussian distribution.
Table 4 shows the results for our AE based variant and VAE based variant on both clean and noisy expert data. Our VAE based variant gets 7.6% and 42.1% relative improvement compared the best baseline GAIL and PWIL, respectively. Comprehensively, our AEAIL works well with the VAE formulation.
A.6 MORE EXPERIMENTS BUILT UPON DAC
AIL methods formulate the reward function as a discriminator. can easily overfit the minor difference between expert and agent generated samples. On the other hand, our AEAIL is general modification for AIL methods. views the reward function as an auto-encoder. Under this formu- lation, our AEAIL can learn the full-scale difference while providing a denser reward function for agent training. So our AEAIL can built on different variants of AIL methods.
DAC Kostrikov et al. 2019) is good variant of GAIL. it uses absorbing states to solve the reward bias problem and also off-pyy RL agent training improve the sample efficiency. In this section, provide more experiment results built upon DAC with off- off-policy RL agent. The bascline choose PWIL with D4PG. Our AEAIL code is adapted from DAC (Kostrkov et al. 2019). The source code for PWIL is from original paper (Dadashi et al. 2021)
Figure show the training curve of our AEAIL and PWIL. The imitation performance on Reacher shows that our AEAIL also works for negative survival bias tasks. Our AEAIL is better than PWIL on all tasks except for Swimmer. It shows that our AEAIL is general to other variant AIL methods and it also achieves strong performance.
17

---

Under review as a conference paper at ICLR 2023
Policy Sample
Expert Sample
GAIL
DAC
FAIRL
Ours
Figure 10: I-SNE visualization for the latent space of discriminator our auto-encoder on noisy expert data on Walker2d. t-SNE's parame-rr perplexity is 30.0.
A.7 OUR AUTO-ENCODER IS EXPRESSIVE FOR DIFFERENT ROLLOUT SAMPLES
Section motivates that our auto-encoder based reward function focuses on the full scale differences between the expert and generated samples. Therefore, it is expressive for different rollout samples. To justify this hypothesis, we illustrate the generated rewards for different samples. Distinct rewards for different trajectories means the reward signal is expressive.
We collect 50 levels of trajectories with performance from random to expert. This can be realized by saving trajectories every 100 iterations when training PPO. Since the trajectories with different per- formances have different lengths, we normalize the generated rewards with the length of trajectories. We also scale the generated rewards considering the expert samples are 1 and random policy samples are O.
Figure 9 shows that the final learned scaled and normalized rewards for different samples with our AEAIL's reward function on Walker2d. Note that learned rewards under our formulation are readily distinct at different levels. Our steep curve indicates that the auto- entoden derived reward function is ex- pressive indeed.
Mean Learnt Rewards
0.06
0:04
A.8 T-SNE
VISUALIZATION FOR THE REWARD FUNCTION
0.0
AIL can be seen as inducing the expert policy through confusing the discriminator. Ideally, the final policy generated samples and expert samples would demon- strate identical distribution. We visualize the latent representations of the discriminator (in AIL meth- ods) and the auto-encoder (in our method) on both the expert and the sampled from the same expert policy in order to illustrate the relation between discriminator-ban based and auto-encoder based reward function.
10
20
50
Different Samples
Figure 9: Scaled normalized rewards for dif- ferent rollout samples on Walker2d.
samples with t-SNE. Here, the policy-generane generated samples are
samples will be indistinguishable (greatly overlapping) in this latent space. Specifically, we visualize the output of the discriminator's first hidden (middle) layer for the AIL methods. For our AEAIL, we visualize the output of the auto-cncoder's first hidden (middle) layer.
Ideally, the expert and fmal-policy-genennee
However, since the discriminator focus on the minor differences between the expert and generated samples, the embeddings won't fully overlapped. Figure 10 shows that our AEAIL greatly overlaps the two sets of embeddings compared to other discriminator based formulations. reflects that our auto-encoder can't distinguish the expert and generated samples any in the latent space. means our auto-encoder can learn the full-scale difference between expert and generated samples.
18


Reviews:
[{"review_id": "y7GRw19nyvk", "reviewer": "Reviewer_9Nok", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Auto-Encoding Adversarial Imitation Learning (AEAIL) for adversarial imitation learning (AIL). The framework uses the reconstruction error of an auto-encoder as the reward rather than classifier-based reward. The authors suggest that reconstruction errors provide denser rewards. Experimental results show that AEAIL outperforms baseline methods, especially when using noisy demonstrations.", "summary_of_the_review": "The paper proposes an AE-based approach to AIL. While the method shows some promise for noisy demonstrations, it has limited novelty, issue with its theoretical interpretation and discrepancies with previous empirical results. Substantial revision may thus be needed for the current manuscript.", "strength_and_weaknesses": "Strength:\n1. The proposed method is clean and straightforward to implement.\n2. The use of AE to overcome noisy demonstration appears robust and outperforms baselines empirically.\n\nWeakness:\n1. I can't agree with the theoretical claim that the proposed AE discriminator minimizes the Wasserstein distance.  The Wasserstein distance formulation from [1] and used in this paper requires the space of the reward function to be Lipschitz-K. However, this assumption may not hold true for AE in practical applications, as noted by the authors. For the practical attempt to clip the weights, note that [1] clips the weights of the discriminator $f$ directly. However, for AE, the function that needs to be Lipschitz-K is the actual reward $|||x - f(x)|^2$. It remains to be shown that weight clipping is effective for AE reward.\n\nIn addition, AE-based GAN is more closely associated with energy-based GAN (see the theoretical derivation in [1]), which shows that AE minimizes total variation distance, not Wasserstein distance.\n\n2. The definition of reward denseness is unclear to me. I can't find any formal definition of it and the differential entropy in the experiment is only a \"proxy\" to the concept of denseness. The entropy measurement is also problematic for 2 reasons: 1) it appears that simply scaling the reward with a large constant would improve entropy for any reward function. 2) Entropy requires $p(r(s, a))$ to compute. It is unclear to me how the probability of a reward $r(s, a)$ is obtained.\n\n3. I have several concerns over the empirical evaluation. a) Why TRPO is the only RL algorithm used for all methods, instead of more recent one like D4PG. In particular, PWIL originally used D4PG. It is well understood empirically that different RL algorithms have significant impacts on the method performance. Thus it would be preferable to select the best RL algorithm for each individual method. b) Some empirical results are hard to reconcile with previous works. For instance, the humanoid expert performance is quite low compared to previously used (5700 vs ~10000 in PWIL). PWIL was able to achieve a performance of 7000+ using 11 demonstrations in original PWIL, while this work only achieves ~3500 with 320 demonstrations. Could the authors please discuss how to reconcile the discrepancies?\n\n[1] Wasserstein generative adversarial networks, Arjovsky et al., ICML 2017", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iusN5iCgZT", "reviewer": "Reviewer_vULj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel method for adversarial imitation learning that restricts the family of reward functions to a specific function that is monotonically decreasing in the reconstruction error of an auto-encoder (I think that \"reconstruction-error\" and \"auto-encoder\" are misnomers, as the encoder-decoder network is not trained to minimize the reconstruction error).\nThe method is compared to prior AIL methods (GAIL, DAC, PWIL, FAIRL) on Mujoco tasks, where achieves better performance, in particular when the expert data is perturbed by Gaussian noise. Furthermore, several ablations are conducted, for example to test a VAE-style and a GAIL-inspired objective for training the reward function.", "summary_of_the_review": "The paper proposes a novel method for imitation learning that does not seem fully sound, but achieves good results in practice. I think the paper could be accepted in principle, however, currently the evaluation seems to suffer from reward bias, and thus, the value of the empirical results seems questionable. How would the algorithm perform on environments where the agent needs to reach an absorbing state (e.g. MountainCar?), or when learning the reward also for the absorbing states (as proposed by Kostrikov et al. 2019)?", "strength_and_weaknesses": "Strength:\n- The paper is well written \n- The approach is novel\n\nWeaknesses:\n- The algorithm doesn't seem fully sound\n- The experiments may suffer from reward bias ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fqPeg6zyJ_", "reviewer": "Reviewer_ymvo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nIn this work, a version of the GAIL method is proposed where the reward in the form of the discriminator on the state-action distribution is replaced by a reward in the form of the reconstruction error of an autoencoder on state-actions. The method is evaluated in extensive simulated experiments.\n", "summary_of_the_review": "The motivation in terms of unstable GAIL, as well as the high-level idea of the autoencoder-based version make sense. But I think this work is too premature, in terms of writing as well as methodological contribution.", "strength_and_weaknesses": "\n**Strengths:**\n\nI see the motivation.\nClearly, GAIL has its weaknesses in terms of overall instabilities. And one specific problem can be the discriminator becoming \"too good\" (though maybe a bit more detail would be appropriate on what this means, given that, on some level of abstraction, the better the discriminator the better for the imitator; and, as far as I understand, this is more a problem of the particular training dynamics/diverging \"speeds\" of imitator and discriminator).\n\nOn a high level, I see that the idea of learning representations/autoencoders can help here. I see that an autoencoder can be a way to detect the \"anomaly\" of having a non-expert compared to an expert.\n\nExtensive simulation experiments are conducted with mostly good performance of the proposed method.\n\n\n**Weaknesses and improvement points:**\n\n\nHowever, I think this work is too premature, in terms of writing as well as methodological contribution:\n\n\nThe structure, ordering and writing, in particular description of the method, need to be improved. Here are some more and some less problematic parts I see:\n* I'm surprised that Wasserstein divergence/WGAN are introduced quite lengthily in sec 3 and 4, but then not really used? There is no further mentionin of it after eq (2)? That's confusing. I think if the Wasserstein GAN framework is actually not used, and the relation is merely that both have a bi-level/adversarial formulation, then Wasserstein does not have to be introduced so prominently. Btw., I recommend to also include [Xiao at al: Wasserstein adversarial imitation learning] as reference.\n* Related to this: Is the Lipschitz-continuity constraint in the problem in eq. (3) dropped compared to (2)?\n* I think the presentation of IL, IRL,etc., needs to be improved. I wouldn't subsume all those variants of GAIL under IRL! Because IRL is about recovering the expert's underlying reward function - assuming it \"exists\" - while GAIL etc. practically do not recover the expert reward (this can be seen from the fact that in equilibrium, the discriminator is always flat, and this is hardly a sensible expert reward). Overall, I would say, IL is the overarching task where demonstrator state-actions are given and have to be imitated. And then there are all these versions like BC, IRL, GAIL, … I think this is the proper conceptual background, while currently the structure and writing of intro and background (sec 3) leaves room for improvement.\n* I feel that the whole intro, background and method sections jump back and forth between motivation and actual method. This is not fundamentally wrong, but a bit confusing in terms of structure.\n* The CartPole balancing task is referred to without proper introduction. It needs to be introduced properly before discussing details like velocities (sec 4.1).\n\n\n\n\n\nMethod and its novelty/contribuion:\n* I think an autoenconder just learns a lower-dimensional data *manifold* - and not a *distribution* (in contrast to a variational autoencoder, GAN, etc.). But, at least in some cases, this means that *all elements that in fact lie on the manifold will get the same error/reward*. But this may not be expressive enough as a reward function. In contrast, with distributional approaches, even within the manifold there is a further \"weighting/ordering\" between elements of the manifod.\n* As a minor point, intuitively, I feel that an autoencoder should be something that learns a very general, robust representation. I feel the autoencoder/representation in this work - i.e., reconstruction error for expert states - is not robust to distributional changes etc. I think it often happens that in one instance (say, for one initialization), a state-action pair is good (taken by the expert), while in another instance it is bad (taken by non-expert). I.e., whether a state is good or bad can heavily depend on initial state, random fluctuations, etc. E.g., from one starting position in a maze, the expert may never want to visit a certain state, while from another one, the state would lie on the shortest path. It's hard for me to make this intuitive counterargument specific, and maybe GAIL suffers from similar problems, but it does seem to be a limitation.\n* More positively speaking, I think it would be relevant to also conceptually better understand the relation between an autoencoder based appraoch to detect the expert versus non-expert, compared to the more classic distribution-divergence one.\n* I feel sec 4.3's title makes quite a strong claim, but then there is little details on the precise conditions for the convergence guarantees. I don't say the full theory from the mentioned Guan et al is needed, but the treatment just feels a bit incomplete in terms of necessary assumptions.\n* I'm wondering: if - as sec 4.3 seems to indicate - the proposed autoencoder-based method is just a special case of what has already been treated by Guan et al, this may indicate that the contribution of the present work is rather limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pz4qY-5PCGc", "reviewer": "Reviewer_EB8r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use reconstruction loss as a reward function in adversarial imitation learning for solving Mujoco locomotion tasks. The idea is built on top of WGAIL/GAIL method, where an autoencoder replaces the discriminator. \n", "summary_of_the_review": "The Paper demonstrates that reconstruction loss can be used as a discriminator in a GAIL-like framework and present performance in various Mujoco Tasks. However, the proposed approach lacks significant novelty and is also not well justified theoretically. Moreover, some of the statements are given without reference or mathematical support. ", "strength_and_weaknesses": "**Strengths** \nPaper demonstrates that using reconstruction loss as a reward can lead to better performance than some prior methods. \n\n**weaknesses** \n- Overall, the idea of using reconstruction loss in RL already exists*. Therefore, the novelty of the proposed approach is limited.\n-- *Loss is its own Reward: Self-Supervision for Reinforcement Learning\n-- *Improving Sample Efficiency in Model-Free Reinforcement Learning from Images\n- The experimental evaluation is limited as it does not quite convince that the proposed approach results in significant improvement.\n- Theoretical justification to motivate the design choice are not convincing either to support the idea of auto-encoder based surrogate reward function. The statement \"The higher entropy of learned rewards means the reward signal can offer more feedback for each state-action pair, thus providing a denser reward signal to the agent policy.\" is given without any justification or theoretical support. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject