Auto-Encoding Inverse Reinforcement Learning
Authors: Kaifeng Zhang
Rui Zhao
Ziming Zhang
Yang Gao
Venue: ICLR 2022 Submitted (2022)
Keywords: Adversarial Imitation Learning
Inverse Reinforcement Learning
Auto-Encoding

Abstract: Reinforcement learning (RL) provides a powerful framework for decision-making, but its application in practice often requires a carefully designed reward function. Inverse Reinforcement Learning (IRL) has shed light on automatic reward acquisition, but it is still difficult to apply IRL to solve real-world tasks. In this work, we propose Auto-Encoding Inverse Reinforcement Learning (AEIRL), a robust and scalable IRL framework, which belongs to the adversarial imitation learning class. To recover reward functions from expert demonstrations, AEIRL utilizes the reconstruction error of an auto-encoder as the learning signal, which provides more information for optimizing policies, compared to the binary logistic loss. Subsequently, we use the derived objective functions to train the reward function and the RL agent. Experiments show that AEIRL performs superior in comparison with state-of-the-art methods in the MuJoCo environments. More importantly, in more realistic settings, AEIRL shows much better robustness when the expert demonstrations are noisy. Specifically, our method achieves $16\%$ relative improvement compared to the best baseline FAIRL on clean expert data and $38\%$ relative improvement compared to the best baseline PWIL on noisy expert data both with the metric overall averaged scaled rewards. 

Under review as a conference paper at ICLR 2022
AUTO-ENCODING
INVERSE REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) provides a powerful framework for decision- making, but its application in practice often requires carefully designed reward function. Inverse Reinforcement I. Learning (IRL.) has shed light on automatic re- ward acquisition, but it is still difficult to apply IRI. solve real-world tasks. this work, propose Auto-Encoding Inverse Reinforcement Learning (AEIRL). a robust and scalable IRL framework, which belongs to the adversarial imitation learning class. To recover reward functions from expert demonstrations, AEIRL utilizes the reconstruction error of an auto-encoder as the learning signal, which provides more information for optimizing policies, compared to the binary logis- tie loss. Subsequently, we use the derived objective functions to train the reward function and the RL agent. Experiments show that AEIRL performs superior in comparison with statc-of-the-ar methods in the MuJoCo environments. More im- portantly, more realistic settings, AEIRL shows much better robustness when the expert demonstrations are noisy. Specifically, our method achieves 16.1% rel- ative improvement compared the best baseline FAIRL on clean expert data and 46.5% relative improvement compared to the best baseline PWIL noisy expert data both with the metric overall averaged scaled rewards.
INTRODUCTION
Reinforcement learning (RL) provides a powerful framework for automating decision making. How- ever, RI still requires significantly engineered reward functions for good practical performance. To make RL more applicable the real-world, it is important learn a reward function from expert demonstrations. Imitation learning offers the instruments to learn policies directly from the data, without an explicit reward function. Imitation learning enables the agents to learn to solve tasks from expert demonstrations, such as helicopter control (Abbeel et al., 2006; 2007; Ng et al., 2004; Coates et al., 2008; Abbeel et al., 2008a; 2010), robot navigation (Ratliff et al., 2006; Abbeel et al., 2008b; Ziebart al., 2008; 2010), and building controls (Barrett & Linder, 2015).
The goal of imitation learning is to extract the expert policies from the expert demonstrations with- out the access to the reward signal from the environment. The algorithms in this field can be divided into two broad categories: behavioral cloning (BC) and Inverse Reinforcement Learning (IRL.). BC formulates the learning task as a supervised learning problem, which learns a mapping from the states to the actions, using the expert trajectories. However, BC methods suffer from the problem of compounding errors, i.e., covariate shift, which only learn the actions of the expert but not rea- son about what the expert attempts to achieve. In the contrast, IRL directly infers the underlying reward function from the data, and then teaches the RL agent learn policy to achieve the highest accumulated reward. Empirical results show that IRL. methods are more efficient than BC methods in multi-step decision making tasks. However, how to make IRL. stable and efficient to use still subject to research.
Current IRL methods, such as adversarial imitation learning approaches, model the reward function as discriminator to learn the mapping from the state-action pair to scalar value, i.e., the reward (Ho & Ermon, 2016; Fu et al., 2017; Ghasemipour et al., 2020). However, this formulation may be easily over-confident to focus the minor differences between the state-action features of the expert and generated samples. To the best our knowledge, this is the first instance of formulating the reward function auto-encoder, which has the ability to learn full scale differences between

---

Under review as a conference paper at ICLR 2022
expert and alternative policies. The derived formulation ensures the reward signal is informative, which is efficient for optimization. Additionally, the noising capability and makes the agent robust to the noisy expert, in more realistic settings. In the experiments, we show that the proposed method AEIRI, achieves the best overall performance on both clean and noisy expert demonstrations.
process empowers the de-
Our contributions are three-fold:
We propose the Auto-Encoding Inverse Reinforcement Learning (AEIRL) architecture, which models reward function as a surrogate function using the reconstruction error of an auto-encoder. The reconstruction error provides more informative learning signals, compared to the binary logistic loss.
To show the contributing factors of our method, we conduct ablation studies based on dif- ferent distribution divergences and alternative auto-encoders. Experiments show that they achieve comparable results which indicates the encoding-decoding process the major contributing factor for our method.
The experimental results on the MoJoCo tasks show that our method outperforms state-of- the-art imitation learning methods both clean and noisy expert demonstrations. Empir- ical analysis show that our learned reward function can be more informative and robust. Furthermore, the learning processes of our methods are also more stable in general.
RELATED WORK
2.1 INVERSE REINFORCEMENT LEARNING
Adversarial imitation learning such as GAIL, AIRL, DAC, f-GAIL, EAIRL, FAIRL (Ho & Ermon, 2016; Fu et al., 2017; Kostrikov et al., 2019; Zhang et al., 2020; Qureshi et al., 2019; Ghasemipour et al., 2020) formulates the learned reward function expert transitions from non-expert ones. Among these methods, GAIL (Ho & Ermon, 2016) con- siders the Jensen-Shannon divergence, while AIRL (Fu al., 2017) considers the Kullback-Leibler (KL) divergence. DAC (Kostrikov et al., 2019) extends GAIL to the off-policy setting and signifi- cantly improves the sample- -emupleefficiencyy of adversarial imitation learning. Furthermore, f-divergence is utilized in f-GAIL (Zhang et al., 2020), which is considered more sample-efficient. Recently, FAIRL utilizes the forward KI. divergence (Ghasemipour et al., 2020) and achieves better perfor- mance than AIRL (Fu et al., 2017) comprehensively. but it is still not robust enough. However, these methods rely heavily on a carefully tuned discriminator, which might easily overfit to the minor differences between the expert and the generated samples. In comparison, our auto-encoder based reward function helps to learn the full scale differences between the expert and generated samples, which provides more informative reward signals.
discriminator that learns differentiate
The robustness of adversarial imitation learning is also questionable with imperfection in observa- tions (Stadie al., 2017; Berseth & Pal, 2020), actions, transition models (Gangwani & Peng, 2020; Christiano et al., 2016). expert demonstrations (Brown et al., 2019; Shiarlis et al., 2016; Jing et al., 2020) and their combinations (Kim al., 2020). Previous robust IRL methods require the demon- strations to be annotated with confidence scores (Wu et al., 2019; Brown et al., 2019; Grollman & Billard, 2012), when the expert data is noisy. However, these annotations Compared to this, our auto-encoder based reward function helps to denoise the expert data through the -ecooddngg process. Our method AEIRL relatively succinet and robust noisy expert demonstrations and does not require any annotated data.
rather expensive.
Another category of IRL. uses an offline similarity function to estimate the rewards (Boularias al., 2011; Klein et al., 2013; Piot et al., 2016). The idea of these methods is still inducing the expert policy by minimizing the distance between the state action distributions of the expert and sampled trajectories. To the best of our knowledge, the most powerful method in this category is Primal Wasserstein Imitation Learning (PWIL) (Dadashi et al., 2021), which utilizes the upper bound of its primal form as the optimization objective. The advantage of these methods is that they relatively more robust compared to adversarial imitation learning methods, when the expert data noisy. However, the performance of these methods heavily depends the similarity measurement, and
2

---

Under review as a conference paper at ICLR 2022
therefore, it varies greatly different tasks. Compared to PWIL, method achieves superior performance.
2.2 AUTO-ENCODING BASED GANS
Auto-encoders have been successfully applied to improve the training stability and modes captur- ing in GANs. Auto-encoding based GANs can be classified into three categories: (I) utilizing an auto-encoder as the discriminator such as energy-based GANs (Zhao et al., 2016) and boundary- equilibrium GANs (Berthelot et al., 2017); (2) using a denoising auto-encoder derive auxiliary loss for the generator (Warde-Farke & Bengio, 2017); (3) combining variational auto-encoder and GANs to generate both vivid and diverse samples by balancing the objective of reconstructing the training data and confusing the discriminator (Larsen et al., 2016). Our method AEIRL takes in- spirations from EBGAN (Zhao et al., 2016) and utilizes an auto-encoder as the reward function and derives the efficient reconstruction enror based surrogate reward signal and its corresponding objective functions.
BACKGROUND
A Markov decision process (MDP) is a tuple (S,A,T,7,P,r)) In this tuple, is a state space; A is an action space; T is probability matrix for state transitions; y E (0, 1] is discount factor; P is initial-state transition distribution; and r S x A - I is a reward function. Additionally, we also define stochastic policy æ, which is a mapping from states probability distributions over actions.
IRL infers the reward function using the expert demonstrations, which are assumed to be the ob- servations of optimal behaviors (Ng & Russell, 2000). In general, IRL is formulated as a bi-level optimization process meaning that iteratively training the reward function and optimizing the policy. Assume that we are given an expert policy gg. IRL (Ziebart al., 2008; 2010) fits a reward function from a family of functions R with the optimization problem:
min rER
max FF, (rx(a())))) - FF [r(
well
Moreover, the expert policy will only be provided by a set of expert demonstrations, so the expected reward of mh estimated by these trajectories.
IRL. looks for reward function r(s, a) that assigns high values to the expert policy and low values other policies. Therefore, it allows the expert policy to be found with certain reinforcement learning procedure:
RL(r) arg max M, (r(s(s,))) TEll
(2)
The RL process will induce the expert policy via maximizing the expected cumulative rewards. Meanwhile the entropy term can be optionally added to the reinforcement learning objective to encourage the exploration in policy searching. Typically, IRL models the reward funetion as discriminator which can be easily overfit to the expert data. Also its training stability is questionable once confronted with even little noise in the expert data. Therefore, we propose auto-encoding IRL. to utilize auto-encoder based reward function which achieve strong performance both on clean and noisy expert demonstrations.
4 AUTO-ENCODING INVERSE REINFORCEMENT LEARNING
4.1 OVERVIEW
The reward function in GAIL is a discriminator, which attempts to assign high values to the regions near the expert demonstrations and assign low values to the other regions. However, this form of reward function could be easily overfitting to the expert data. Consider the CartPole balancing task, the state of the environment is feature consisting of position, angle, angle's rate, and cart velocity, while the action is moving left or right. Here, we assume that all the expert states" velocity is 2 for example. When the generated states' velocity is 1.9 and other dimensions of state action pairs
3

---

Under review as a conference paper at ICLR 2022
min (B (B( [no( (s (
Expert Samples
g
Auto-Encoder
MSE
Tw (d)
S
Generated Samples -
| -
0
max E (8,a)~~a (6)
Figure 1: Adversarial training framework of auto-encoding inverse reinforcement learning. The auto-encoder computes the reconstruction error for these two mini-batcher data examples and optimize the objectives. The surrogate reward function provides the signal to the agent.
the same as the expert's, the discriminator of GAIL would still give a low reward on these generated expert's
state action pairs. However it may actually perform very well on the goal of mimicking behaviors. In other words, the reward function in GAIL this example could easily overfit to the minor differences between the expert and the sampled data, while missing the underlying goal the expert.
In our paper, we propose an auto-encoder based reward function for inverse reinforcement learning. It utilizes the reconstruction error of the state action pairs to yield informative reward signal. The reconstruction error based reward signal significantly retains the information of state action pairs, rather than focusing the minor differences. Such reward signal wouldn't lead to overconfidence in distinguishing the expert and the generated samples. Recall the CartPole balancing example, the mean square error between states' velocity 1.9 and 2 very small. And it could still feed good reward to the agent under this situation. Thus, the reconstruction error based reward signal focuses on the full scale differences between the expert and generated state action pairs rather than the minor parts.
This yields much more informative reward signal. Figure (See in Section 5.4), shows a more informative reward signal recovered with our method. Furthermore, the training process of the policy becomes smoother in general, which is shown Figure 11 (See Appendix A.6).
The expert demonstrations usually contains noise bacause we sample these human trajectories with sensors and other devices in the real world. There- fore, adversarial imitation learning, such as GAIL, might be easily affected to learn the noisy expert be- haviors, which are not the real intentions of the ex- pert. The reward function under this situation could overfit to the noisy features. Recall the CartPole bal- ancing example, we assume that the expert states' velocity is 2 but it tends to be 2 + due to the noisy sampling process. When the learned policy is good enough as the sampled states' velocity, the discrimi- nator in GAIL still takes it as bad policy, and there- fore, it loses its efficacy, when learning from noisy demonstrations.
Clean Data
Noisy Data
The clean data points near low dimensional manifold illustrated with the bold black line. The gray circle shows the noisy sampling process. And the auto- encoder learns to denoise the noisy data to clean points.
Figure 2:
Figure 2 depicts the denoising process of an auto- encoder. When the auto-encoder is trained to mini- mize the averaged squared errors, the vector points approximately towards the nearest points on the manifold, since the auto-encoder estimates the ter of mass of the clean points (Goodfellow et al., 2016). Therefore, the reconstruction error of the auto-encoder eliminate some effects when learning from noisy expert demonstrations and
4

---

Under review as a conference paper at ICLR 2022
make the reward signal more robust. Recall the CartPole example, the reconstruction error of the auto-encoder is which doesn't 't play an important role in the rewards for the agent.
4.2 MЕТHOО
Our approach is minimize the distance between the state action distribution of the policy vo and the expert demonstrations Dg- The distance we used in our method is the Wasserstein distance (Arjovsky al., 2017):
d(TE, mo) sup E (s, a)] [rw
Yo
where the reward function network's parameters are denoted as w and the policy network's parame- ters are represented as We choose the metric functions in Wasserstein distance as a class reward functions from neural network functions for computational convenience. Minimizing this distance actually inducing the expert policy from expert demonstrations. Therefore, the optimization of the policy To and recovering period of the reward function (s, a) forms bi-level optimization problem, which can be formally defined as:
min max (((s,))~De(ru (n a)] (r Yhe
This leads to an adversarial formulation for inverse RL. The outer level minimization with respect to the policy leads to a learned policy which is close to the expert. The inner level maximization recovers a reward function which attributes higher values to regions close to the expert data, and penalizes all other regions.
In traditional adversarial imitation learning methods, such as WGAIL (Arjovsky et al., 2017; Ho & Ermon, 2016) (Details See in Appendix A.2.3), the reward function is defined as
Two (s, a) (w a);
(5)
where Da (s, a) is the output of the discriminator.
In our method, we use auto nuto-encoder based surrogate reward function, which is defined as: Tau (s, a) = 1/(1 + (s,
(6)
where AE is the reconstruction error of an auto-encoder:
AB(x)= = Dec Enc(x: - 1r11
Here, x represents the state-action pairs. It a mean square error between the sampled state action pairs and the reconstructed samples. This form of the reward signal uses the reconstruction error of an auto-encoder to score the state action pairs in trajectories. It is a monotonically decreasing function over the reconstruction error of the auto-encoder. Low reconstruction errors ensure high reward values for state action pairs and vice versa. Section 4.1 tells that this form of reward signal focuses more on the full scale differences between the expert and generated samples and can help to denoise the expert demonstrations.
Training the nuto-encoder an adversarial process considering the objective 4, which is minimizing the reconstruction error for the expert samples and meanwhile maximizing this error for generated samples. So the auto-encoder based reward function training objective is to minimize:
L E (s.a)~ma [w (s, a)] ((s.a))DD (s.(s.
(8)
(s.a)~mm [1 + AE,.. a))] (e [1/(1 (E,.
The auto-encoder learns to maximize the full scale differences between the expert and the generated samples with the adversarial objective. It leads to a better feedback signal to the agent. Further- more, the auto-encoder also retains more information and helps to denoise the expert data via the grooding-decoding process. This can further help us to match the distribution between the expert and generated behaviors in latent space.
The variational auto-encoder (Kingma & Welling, 2014) is also an alternative form of reward func- tion. When we use an variational auto-encoder based reward function, the objective function for training the variational auto-encoder in our method
((aa,))DD (r DEL (p(21(s, a)), (2))
Ess (s, DRL (pl 2((,, a)), Pmodel 2))]
5

---

Under review as a conference paper at ICLR 2022
where (modd(()) is fixed Gaussian distribution. When minimizing this objective, the latent distri- bution for the expert samples is widely distributed to induce more diverse expert policies.
Figure depicts the architecture for AEIRL. The auto-encoder based reward function takes either expert or the generated state action pairs, and estimates the reconstruction error based rewards ac- cordingly. The auto-encoder and the policy is iteratively optimized under this adversarial training paradigm. Algorithm in Appendix A.1 depicts the pseudo code for training AEIRL.
Not limited to the utilized Wasserstein distance, other distribution divergences also applicable to form the reward functions without loss of generality. Appendix A.5 shows other two forms of objec- tives and reward functions following Jensen-Shanner divergence. Experiments show that different forms of objectives achieve comparable results on almost all locomotion tasks. It indicates that an auto-encoder based reward function the major contributing factor improve the performance on mimicking the expert rather than the different distribution divergences and the corresponding forms of surrogate reward function.
EXPERIMENTS
We conduct the experiments on six locomotion tasks with varying dynamics and difficulty: Swimmer-v2 (Schulman et al., 2015), Hopper-v2 (Levine & Koltun, 2013), Walker2d-v2 (Schulman et al., 2015), HalfCheetah-v2 (Heess et al., 2015), Ant-v2 (Schulman et al., 2016), and Humanoid-v2 (Tassa al., 2012). The goal for all these tasks is to move forward quickly possible.
The basclines we choose for comparison includes: behavior cloning (BC) (Dhariwal et al., 2017), GAIL (Ho & Ermon, 2016), WGAIL (Arjovsky et al., 2017; Ho & Ermon, 2016), FAIRL (Ghasemipour et al., 2020), and PWIL (Dadashi et al., 2021). For a fair comparison, we use TRPO (Schulman al., 2015) the policy search method for all the algorithms, which is implemented by OpenAl Baselines (Dhariwal et al., 2017). Experimental details, including the implementation and more results, are shown in Appendix A.2 A.7.
5.1
LEARNING FROM EXPERT DEMONSTRATIONS
Task
Walker
Hopper
Swimmer
HalfCheetah
Ant
Humanoid
Random
0.6 + 0.3
25.5 4.2
5.33 = 13.9 694 108 -001.4 233.4
110 4.1
Expert
4904.2
2719.8
142.2
1969.2
2646.1
5187.9
BC
459.7 + 374.8
396.5 435.7
78.3 + 39.6 945.2 = 160.7 807.8 = 1843.9
237.8 = 621.4
GAIL
3023.1 +: 254.4
2487.3 +: 80.3
133.2
10.7 1610.2 + 25.3 2262.6
280.7
2078.9 + 433.8
WGAIL
3308.5 +: 361.4 2570.1
122.1
133.4 5.7 1658.4 + 42.7 2356.5 = 299.2
1669.6 + 296.3
FAIRL
3214.0 1: 290.6 2650.4
101.8
136.7
13.7 1563.1 39.8 2563.0
284.6
1513.2 1 462.7
PWIL
3511.4 + 130.3 2809.9 + 47.2
47.9 4.8 1372.3 71.3 1067.9 = 462.3
3157.2 = 507.8
Ours
3897.8 1 177.8 2706
219.7
141.5
3.7 1674.1
33.6 2715.7 1 93.1
3892.3 1 522.9
Ours- VAF 4013.2 + 248.7 2850.5 + 150.7 135.9 + 5.5 1643.9 + 10.8 2332.5 = 938.3 2486.7 = 1418.9
Table 1: Learned policy performance for different imitation learning algorithms on non-noisy expert data, evaluated by using the mode of sampling from the policy distributions. The best results are marked as red.
To evaluate the performance of our proposed methods and other baselines, we run the experiments on the clean expert demonstrations. The metric we used is the ground truth rewards for different learned policies. Higher rewards indicate better mimicking expert behaviors.
Figure depicts the training curves of ground truth rewards for different algorithms. The training curve shows the imitation performance learning from non-noisy expert data. Table shows the final learned policy performance on MuJoCo tasks with ground truth episode return.
GAIL, WGAIL and FAIRL have comparable results, while WGAIL and FAIRL are little better than GAIL comprehensively on these tasks but still not robust enough. PWIL achieves higher perfor- mance compared with GAIL, WGAIL and FAIRL on Walker2d, Hopper and Humanoid tasks but performs worse than BC on Swimmer. Thus, the performance of PWIL varies greatly on different tasks.

---

Under review as a conference paper at ICLR 2022
Walker2d
Swimmer
HalfCheetah
3000
150
125
1500
100
Return
3005
Return
Return
300
-500
-25
20
60
80
10C
Timesteps (1e5)
Timesteps
(1e5)
Timesteps (le5)
Ant
Humanoid
Hopper
3000
2000
4000
2500
Return
Return
Return
7000
export
1500
mendor
2000
-2000
pul
20
20
80
100
30
40
60
80
100
Timesteps (1e5)
Timesteps (1e5)
Timesteps (1e5)
Figure 3: Mean and standard deviation return the evaluation policy over 10 rollouts and 5 seeds, learning from non-noisy expert demonstrations, reported every 100k timesteps. The return is in term the environment's ground truth reward.
Task
Walker2d
Hopper
Swimmer
HalfCheetah
Ant
Humanoid
BC
728.2 + 835.3 195.8
15.7 35.6 + 2.3
898.1 + 827.9
623.5 1364.1
186.7 + 5243.6
GAIL
1254.1
231.8 1478.0 = 405.9
106.4 + 19.7
1538.3
842.9
1085.0 + 622.8
WGAIL
940.9 383.8 2570.1
122.1 95.9 + 15.4
696.411
1457.7
359.8
1479.1 1: 539.8
FAIRL
1302.3 554.1 1732.3 328.2
106.2 + 12.3
940.1
-767.3
518.2
1668.7 = 511.2
PWIL.
3052.4 +: 346.6 2588.9 = 159.7 57.9 + 8.1
1056.3 + 80.1
-818.2
832.9
3218.9 +: 281.4
Ours
3554.6 1: 275.6 1891.8 197.0
105.0 + 2.7
1501.6 92.4
2868.1 + 95.3
3690.2 = 374.8
Ours-VAE VAE 4078.8 295.1 2209.7 = 108.0 88.4 + 9.4 1334.7 + 148.9 2400.5 + 423.1 2159.4 + 1001.0
Table 2: Learned policy performance for different imitation learning algorithms noisy expert data, evaluated by using the mode of sampling from the policy distributions. The best results are marked red.
The overall averaged scaled rewards for our method is about 0.907 where the best baseline is 0.78 for FAIRL. baselines on all locomotion tasks except for Hopper where our method doesn't 't outperform PWIL. Moreover, the variational auto-encoder variant of our method gets the best scores on Walker2d and Hopper. In summary, methods, including vanilla auto-encoder based and variational auto- encoder based variants, get better performance than other baselines comprehensively.
more than 16.1% relative improvement. Our method outperforms state-of-the-art
5.2 LEARNING FROM NOISY DEMONSTRATIONS
To show the robustness of our proposed AEIRL, we run our method and baselines on noisy expert demonstrations. The noise added to the expert data is Gaussian noise (0, 0.3) every dimension of state action pairs on all tasks except for Ant and Humanoid. Since the dimension state action pairs in Ant and Humanoid is much higher than other tasks, we choose (0, 0.03) and (0, 0.01) Gaus- sian noises added to the dataset respectively in order to lower the effects of the noise component. Figure 7 (See in Appendix A.3) depicts training curves for the ground truth rewards for different

---

Under review as a conference paper at ICLR 2022
methods noisy expert data while Table shows the final learned policy performance on MuJoCo tasks with ground truth episode return.
These results show that method outperforms other state- thethh-rrt algorithms all tasks except for Hopper and Swimmer, on which PWIL and GAIL wins respectively. Table and 2 show that the noise component affects the performance of mimicking the expert behaviors. But our method could help to denoise the expert demonstrations and provide more robust reward signals for inducing the expert policies. The overall scaled rewards for our AEIRL is 0.795 where the best baseline is 0.542 for PWIL. Additionally, PWIL also has some capabilities to denoise the expert on part of the tasks. And other adversarial imitation learning methods, includes GAIL, WGAIL and FAIRL, are very sensitive to the noisy expert.
5.3 ANALYSIS ON ROBUSTNESS OF OUR METHOD
Policy Sample
Expert Sample
Non-Noisy Data
Non-Noisy Data
Non-Noisy Data
Non-Noisy Data
Noisy Data
Noisy Data
Noisy Data
Noisy Data
GAIL
WGAIL
FAIRL
Ours
Figure 4: t-SNE visualization of latent representations of the reward function Walker2d for differ- ent methods with hyper- parameeerretr preplerity = 30.0. Top row: on non-noisy expert data, bottom row: on noisy expert data (Gaussian noise (0, 0.5)).
To compare the robustness of different reward signals of different methods, we use t-SNE visual- ization technique visualize the latent space of the reward function network. For the discriminator based methods, we visualize the output of first hidden (middle) layer for instance. So the latent representations for both discriminator based and our auto-encoder based methods will show its bustmess, especially to the corrupted expert data.
Since PWIL (Dadashi al., 2021) considers offline reward function without neural networks, only compare our methods with other three IRI. baselines. Figure 4 depicts that two sets of samples indistinguishable than other baselines both on clean and noisy expert data. It shows better robustness of our auto-encoder based reward function. More details for using t-SNE and results with varying preplezity are shown Appendix A.4
5.4 ABLATION STUDIES
Different Objectives To analyze the major contributing factors of our method, we conduct the ab- lation studies based on different distribution divergences and different formulation of reward func- tions. Comparable performances would indicate that the major contributing factor is that we utilize the reconstruction error of an auto-encoder rather than the formulation of reward function and its corresponding objective. We formulate some other forms of surrogate reward functions based on Jensen-Shannon divergence (Goodfellow et al., 2014) to justify this hypothesis. Figure 9 (See in Appendix A.5) shows that three different formulations of surrogate reward signal achieve compara- ble results on almost all tasks which all based on the vanilla auto-encoder. This shows the major contributing factor in our method is the anto-encoder based reward function which provides more informative learning signals, compared to the discriminator based reward function.
8

---

Under review as a conference paper at ICLR 2022
Alternative Auto-Encoders Our method is not limited to vanilla auto- entooens and we also take the variational auto-encoder as an alternative. We run variational auto-encoder based variant on MuJoCo tasks under the same setting with vanilla auto-encoder. Figure 10 (See in Appendix A.5) shows the training curves for variational auto-encoder based and vanilla auto-encoder based variants. Overall, the scaled rewards of variational auto-encoder based variant is 0.845 and 0.727 clean and noisy expert respectively, which is comparable to the vanilla auto-encoder based method.
Auto-Encoderr based Reward Signal is Denser denser re- ward signal is beneficial for policy searching (Ng et al., 1999; Hu al., 2020). Section 4.1 tells that our auto-encoder based reward function focuses on the full scale differences between the expert and generated samples. Thus, provides more in- formation to the agent compared to the discriminator based reward function. To justify this hypothesis, we want to il- lustrate the generated rewards for different trajectories in the
Trajectory Space
0.14
WGAIL
Ours AE
Learned Rewards
0.12
Ours-VAL VAS--VAE
whole trajectory space. Meanwhile more distinct rewards for different trajectories means the reward signal is denser and more informative. So we use the final learned reward function of our method and the baselines score different trajectories for analysis.
0.06
0.04
0.02
0.00
We collect 50 levels of trajectories with different perfor- mances from random to the expert performance. This can be realized by saving trajectories every 100 iterations via a PPO. Since the trajectories with different performances have dif- ferent lengths, we normalize the generated rewards with the length of trajectories. We also scale the generated rewards considering the expert samples are and random policy sam- ples are O. So the scaled normalized rewards for different lev- els of trajectories reflect the denseness reward signal. And higher degree of distinction indicates it is denser reward signal.
10
20
30
50
Level
Figure 5: Scaled normalized re- wards for different levels in the tra- jectory space Walker2d.
Figure shows that the final learned scaled and normalized rewards for different levels of the tra- jectories. The higher level in the abscissa means higher performance of its trajectories. Note that learned rewards under our formulation are readily distinct at different levels when compared to learned rewards under WGAIL. Our much steeper curves indicates that the auto-encoder derived reward function provides a denser and more informative learning signal when compared to the dis- criminator based reward functions, which all based on Wasserstein distance (Arjovsky et al., 2017).
Stability of Our Method The variance of gradients is computed via different mini-batches of train- ing examples. Lower variance of gradients means smoother updating of the network (Faghri et al., 2020). Now that our auto-encoder based reward function is more informative and denser than the discriminator based ones, how about the training stability of our method? The relative variance of policy gradients to the weights of policy network shows the training stability for different methods. Figure 11 (See in Appendix A.6) shows that our method gets lower relative variance of gradients which means that the policy network training process can be much more stable.
6 CONCLUSIONS
This paper presents succinct and robust adversarial imitation learning method based on Auto- Encoding (AEIRL). We utilize the reconstruction error of an auto-encoder as the surrogate reward function for reinforcement learning. The advantages of our method can be summarized into two aspects: (I) the auto-encoder based reward function focuses on the full scale differences between the expert and generated samples, which won't it easily overfit to the expert data; (2) the auto-encoder denoises the expert data through the reconstruction process. Thus, the learned reward signal in our method is more robust and informative compared with the existing methods. Experimental results show that our methods achieve strong performances on locomotion tasks both clean and noisy expert demonstrations. In the future, we want to further investigate our method in more realistic scenarios, such as autonomous driving.

---

Under review as a conference paper at ICLR 2022
7
REPRODUCIBILITY STATEMENT
Data and source code are attached in supplementary materials.
Experimental implementation details, including our methods and baselines, see in Appendix A.2.
REFERENCES
Pieter Abbeel, Varun Ganapathi, and Andrew Y. N. learning vehicular dynamics, with application to modeling helicopters. In Advances in Neural Information Processing Systems (NeurIPS), 2006.
Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y. Ng. An application of reinforcement learning aerobatic helicopter flight. In Advances in Neural Information Processing Systems (NeurIPS), 2007.
Pieter Abbeel, Adam Coates, Timothy Hunter, and Andrew Y. Ng. Autonomous autorotation of an re helicopter. In International Symposium on Roboties, 2008a.
Pieter Abbeel, Dmitri Dolov, Andrew Y. Ng, and Sebastian Thrun. Apprenticeship learning for motion planning with application to parking lot navigation. In IEEE/RSj International Conference on Intelligent Robots and Systems, 2008b.
Pieter Abbeel, Adam Coates, and Andrew Y. Ng. Autonomous helicopter aerobatics through ap- prenticeship learning. The International Journal of Robotics Research, 2010.
Martin Arjovsky, Soumith Chintala, and Leon Bottou. Wasserstein gan. In International Conference Machine Learning (ICML), 2017. 5, 6, 9, 14
Enda Barrett and Stephen Linder. Autonomous hva control, a reinforcement learning approach. Machine Learming and Knowledge Discovery in Databases, 2015.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv: 01884.08617, 2018. 20
Glen Berseth and Christopher Pal. Visual imitation with reinforcement learning using recurrent siamese networks. 2020. URL https:tpps.. / http://ppeeeeeiieeee wwt.wvwwwwweeeeeeeeeeeee wertt//ormmiiddeee www.e//ooomiiddddddd.h.. Ywr. 2
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative ad- versarial networks. Advances in Neural Information Processing Systems (NeurIPS), 2017. 3
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 182-189 JMLR Workshop and Conference Proceedings, 2011. 2
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference Machine Learning (ICML), pp. 783-792. PMLR, 2019. 2
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from simulation to real world through learning deep inverse dynamies model. 2016. 2
Adam Coates, Pieter Abbeel, and Andrew Y. Ng. Learning for control from multiple demonstrations. In International Conference on Machine Learning (ICML), 2008.
Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imi- tation learning. In International Conference on Learning Representations (ICLR), 2021. 2, 6, 8, 14, 16
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017. 6, 14
10

---

Under review as a conference paper at ICLR 2022
Fartash Faghri, David Duvenaud, David J. Fleet, and Jimmy Ba. A study of gradient variance in deep learning. 2020. 9, 19
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. International Conference on Learning Representations (ICLR), 2017. 1,
Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. In International Conference on Learning Representations (ICLR), 2020. URL https:// openreview.nete Forum? mett//orruidddtlggllyy...
Seyed Ghasemipour, Kamyar Seyed, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning (CoRL), pp. 1259- 1277. PMLR, 2020. 2, 6, 14
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. 4, 18
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2014. 8, 14, 17
Daniel H Grollman and Aude G Billard. Robot learning from failed demonstrations, International Journal of Social Robotics, 4(4):331-32,, 2012. 2
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2944 2944-2952, 2015. 6
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 1, 2, 5, 6, 14
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: new approach of reward shaping. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 9
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 5109-5116, 2020.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imita- tion learning. In International Conference on Machine Learning (ICML), pp. 5286-5295. PMLR, 2020. 2
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Advances in Neural Information Processing Systems (NeurIPS), 2014. 5, 18
Edouard Klein, Bilal Piot, Matthieu Geist, and Olivier Pietquin. A cascaded supervised learning approach to inverse reinforcement learning. In Joint European conference on machine learning and knowledge discovery in databases, pp. 1-16. Springer, 2013.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-actorrrritt Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations (ICLR), 2019. 2
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. International Conference on Machine Learning (ICML), 2016. 3
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference Machine Learning (ICML), pp. 1-9, 2013.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2000.

---

Under review as a conference paper at ICLR 2022
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML). 1999. 9
Andrew Y. Ng. Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang. Inverted autonomous helicopter flight via reinforcement learning. In International Symposium on Experimental Robotics, 2004.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Bridging the gap between imitation learning and inverse reinforcement learning. IEEE transactions neural networks and learning systems, 28 (8):1814-1826, 2016.
Ahmed H. Qureshi, Byron Boots, and Michael C. Yip. Adversarial imitation via variational inverse reinforcement learning. In International Conference Learning Representations (ICLR), 2019.
Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. Maximum margin planning. In International Conference on Machine Learning (ICML), 2006.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning (ICML), pp. 1889-1897.
PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Con- ference on Learning Representations (ICLR), 2016. 6
Kyriacos Shiarlis, Joao Messias, and Shimon Whiteson. Inverse reinforcement learning from fail- ure. Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems (AAMAS), pp. 1060-1068, 2016. 2
Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. In Interna- tional Conference on Learning Representations (ICLR), 2017.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.
IEEE, 2012. 6
David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. International Conference on Learning Representations (ICLR), 2017. 3
Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Im- itation learning from imperfect demonstration. In Proceedings of the 36th International Confer- ence on Machine Learning (ICML), pp. 6818-6827, 2019.
Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020. 2
Junbo Zhao, Machael Mathieu, and Yann LeCun. Energy-based generative adversarial network. International Conference on Learning Representations (ICLR), 2016. 3
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference Artificial Intelligence (AAAI), 2008. 1,
Brian D. Ziebart, Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning (ICML), 2010. 1,
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 PSEUDO CODE
Algorithm Auto-Encoding Inverse Reinforcement Learning (AEIRL)
Require: Initial parameters of policy, auto-encoder 0b, wo; Expert trajectories Dg.
Ensure:
1: for i= ( to N do
2: Sample state-action pairs (8;; a:) ne, and (sg, ag) DE with same batch size.
3: Update w; to wi+1 by decreasing with the gradient:
(s, Vw, 1/(1 + AEw, (s,, a:))] - ((s,,e)) V. 1/ AEw, (s, as,)
4: Take a policy step from 0; to f;-1, using the TRPO update rule with the reward function 11(1 + AE (s, a)), and the objective function for TRPO is:
F(s,a))-1/(1 ABwr (8, a))]
5: end for
Algorithm depicts the pseudo code for training AEIRL. The first step is to sample the state action pairs from expert demonstrations DE and the trajectories sampled by current policy with the same batch size. Then we update the auto-encoder by decreasing with the gradient with loss function Eq. 8. Finally, we update the policy assuming that the reward function is Eq. 6 which leads to an adversarial training paradigm. We runs these steps with N iterations until the policy converges.
A.2 IMPLEMENTATION
A.2.1 TASKS
(a)
(b)
(c)
(d)
(c)
(I)
Figure 6: Illustrations for locomotion tasks we used our experiments: (a) Swimmer; (b) Hopper; (c) Walker; (d) HalfCheetah; (e) Ant; (f) Humanoid.
The goal for all these tasks is to move forward as quickly as possible. These tasks are more challeng- ing than the basic tasks due to high degrees of freedom. In addition, great amount of exploration needed learn move forward without getting stuck local optima. Since penalize for ex- cessive controls as well as falling over, during the initial stage of learning, when the robot is not able to move forward for sufficient distance without falling, apparent local optima exist including staying at the origin or diving forward slowly. Figure depicts locomotion tasks' environments.
A.2.2 BASIC SETTINGS
The expert trajectories sampled with PPO Walker2d. Hopper, and Swimmer, HalfCheetah and Ant with 25 trajectories. And we sample 320 trajectories on Humanoid with SAC to get higher expert performance since PPO is not good enough Humanoid.
For each task, we normalize the state features and use the normalized features as the input for the auto-encoder.Buu use the raw data input to compute the mean square error.
On Walker2d, Hopper, and Swimmer, we train these algorithms directly, while on HalfCheetah, Ant and Humanoid use BC to pre-train the policy with 10k iterations. We test all the algorithms
13

---

Under review as a conference paper at ICLR 2022
these six locomotion tasks with environment seed 0, 3,2,3,4, 4, and using 10 rollouts to estimates the ground truth reward value to plot the training curves.
The maximum length for sampled trajectories is 1024. The discounted factor is 0.995. Each iteration we update 3 times of policies and time of reward function (discriminator or auto-encoderr. While updating TRPO, update the critic 5 times with learning rate 2e-4 every iteration.
A.2.3 BASELINES
We use open source code, OpenAl baselines (Dhariwal et al., 2017), to get the performance of BC and the training iterations is 100k for all tasks.
GAIL (Ho & Ermon, 2016) borrows the GAN (Goodfellow et al., 2014) architecture to realize the inverse reinforcement learning process and it is applicable high dimensional dynamic environ- ments. We implement GAIL. using open source code, OpenAl baselines (Dhariwal et al., 2017), with best tunned hyperparameters. We choose TRPO as the reinforcement learning algorithm.
WGAIL (Arjovsky et al., 2017; Ho & Ermon, 2016) utilizes the Wasserstein distance to train the policy and the reward function which can more stable. WGAIL is implemented based on GAIL with corresponding objective function and reward signals in (Arjovsky et al., 2017). We also choose TRPO as the reinforcement learning algorithm.
FAIRL (Ghasemipour et al., 2020) utilizes the forward KL divergence the objective which shows competitive performances on MuJoCo tasks. FAIRL is implemented based on GAIL with corre- sponding objective function and reward signals in (Ghasemipour et al., 2020). We also choose TRPO as the reinforcement learning algorithm.
PWIL (Dadashi al., 2021) is an adversarial imitation learning method based offline similarity reward function. It is quite robust and easy to fine-tune the hyper-parametenn of its reward function (only two byper-parameters). PWIL is implemented based open source code (Dadashi et al., 2021) with TRPO as the reinforcement learning step for fair comparison (original PWIL code uses D4PG the RL algorithm).
A.2.4 ARCHITECTURE OF OUR METHODS
Policy Net: the same with baselines, two hidden layers with 64 units, with tanh nonlinearities in between. The learning rate to update the policy is 0.01.
Auto-Encoder Net: both vanilla auto-encoders and variational auto-encoder, the same number of layers with the architecture of discriminator in GAIL and WGAIL, layers. The two hideen layers with 100 units, with tanh nonlincarities in between, the final output layer is an identity layer. And we normalize the state feature after the input layer and we use the raw state action input features to compute the square error. Learning rate for the auto-encoder is 3e-4.
A.3 OMITTED EXPERIMENT RESULTS ON NOISY EXPERT DEMONSTRATIONS
To show the robustness of our AEIRL method, we conduct the experiments on noisy expert demon- strations. The noise component is a Gaussian distribution noise which is (0, 0.3) for Walker2d, Hopper, Swimmer, HalfCheetah. Since the effects of the noise component performs more heav- ily on Ant and Humanoid, we choose (0, 0.03) and (0, 0.01) Gaussian distribution noise for Ant and Humanoid respectively. So under this setting, higher final ground truth rewards indicate better robustness of its corresponding algorithm.
Figure 7 shows the training curves for different imitation learning algorithms the noisy expert demonstrations.
Walker2d, HalfCheetah, Ant, and Humanoid, our AEIRL clearly outperforms other bascline algorithms. On the other hand, Swimmer GAIL achieves the best performance while on Hopper, PWIL achieves the best performance.
We scale the ground truth rewards of final policy performance for different methods which considers the expert trajectory's rewards to be and random policy trajectory's 's rewards to be O. And the averaged scaled ground truth rewards is 0.59 while the best baseline is 0.542 for PWIL. So achieve 46.5% relative improvement to PWIL. In summary, our method more robust than other baselines.
14

---

Under review as a conference paper at ICLR 2022
Walker2d
Swimmer
HalfCheetah
3000
150
1500
100
Return
3000
Return
Return
300
25-
-500
20
60
80
10C
Timesteps (1e5)
Timesteps
(1e5)
Timesteps (le5)
Ant
Humanoid
Hopper
5000
3001
7.500
2000
Return
1000
Return
8000
Return
2000
gnil
1500
2000
pwil
OUTS
20
80
100
30
40
60
80
100
Timesteps (1e5)
Timesteps (1e5)
Timesteps (1e5)
Figure 7: Mean and standard deviation return of the evaluation policy over 10 rollouts and 5 seeds learning from noisy expert demonstrations, reported every 100k timesteps. The return is in term of the environment's ground truth reward.
A.4 DETAILS AND MORE RESULTS ON T-SNE VISUALIZATION
The auto-encoder based reward function learns the mapping from state action space a sealar value, ie., the reward. When the system of our method converges, the generated state action samples would be indistinguishable with the expert state action samples. For our auto-encoder based reward function, the latent representations between final policy samples and expert samples would also be indistinguishable.
To compare the robustness of different reward signals of different methods, ization technique to visualize the latent space of the reward function network. For the discriminator based methods, visualize the output of first hidden (middle) layer for instance. And for our auto- encoder based reward function, we also visualize the output of first hidden (middle) layer in the net. So the latent representations for both discriminator based and our auto-encoder based methods will show its robustness, especially to the corrupted expert data.
t-SNE visual-
We plot the t-SNE embedding results with SciKit-Learn tools (i.e., sklearn. .et.arrnmanoo.d....mm TSNE function) with varying preplexity. The hyper-paramete parameters for t-SNE are (other with seikit-learn):
are default
component 2
early exaggerationfllat 12.0
learning_rate 200.0
n_iterint 1000
n_iter n_iter_withour wwt..tt.tttt..... 1e-7
min proggrrutppogeesstnttt.. 300
metric 'euclidean' init 'pca'
15

---

Under review as a conference paper at ICLR 2022
Policy Sample
Expert Sample
Non-Noisy Data
Non-Noisy Data
Non-Noisy Data
Non-Noisy Data
Noisy Data
Noisy Data
Noisy Data
Noisy Data
GAIL
WGAIL
FAIRL
Ours
Figure 8: t-SNE visualization of latent representations of the reward function Walker2d for dif- ferent methods. Top row: on non-noisy expert data (preplezity = 5.0); Second row: bottom row: on non-noisy expert data (preplexity 50.0); Third row: noisy expert data (preplexity 5.0); Bottom row: on noisy expert data (preplexity = 50.0).
Since PWIL (Dadashi al., 2021) considers an offline reward function without neural networks, we only compare our methods with other three IRL baselines. Figure 8 depicts that two sets of samples are more indistinguishable than other baselines both on clean and noisy expert data. The top row of Figure 8 shows the embeddings with preplezity
5.0 in the non non-noisy setting. The second row shows the embeddings with preplezity 50.0 in the non-noisy setting. The third row of Figure 8 shows the embeddings with preplerity
5.0 in the noisy setting. The bottom row shows the embeddings with preplezity 50.0 in the noisy setting. And the noise component is a Gaussian distribution (0, 0.5). And in the main text, Figure shows the default preplezity = 30.0 results.
With varying hyper-parameter preplexity, we can see the overlapped areas for policy and expert samples greater for our method. In detail, WGAIL hardly appear in the outer ring with preplezity = 30.0. And FAIRL a little better than GAIL and WGAIL. Our AEIRL gets maximum overlap between two sets of samples compared to these baselines. And in the noisy settings, the two sets embeddings of GAIL. and WGAIL. are actually isolated with different "preplexity" while they gobally clustered (greatly overlapped) only with preplezity = 30.0 for FAIRL. And the embeddings are greatly overlapped with varing preplexity for our method. So reflects that our auto-encoder based reward function is more robust to both the clean and noisy data.
observe that the policy samples for GAIL and
A.5 ABLATION STUDIES
To illustrate the major contributing factors of our methods, we conduct the ablation studies with different forms of distribution divergences and other alternative auto-encoders.
16

---

Under review as a conference paper at ICLR 2022
Walker2d
Swimmer
HalfCheetah
150
125
100
1000
Return
Return
Return
500
-20
20
60
80
10c
100
Timesteps (1e5)
Timesteps
(1e5)
Timesteps (le5)
Ant
Humanoid
Hopper
1000
3000
2000
Return
Return
Return
1500
2000
1000
-1000
1000
500
-2000
sighaai
20
65
an
100
Timesteps (1e5)
Timesteps (1e5)
Timesteps (1e5)
Figure 9: Mean and standard deviation return of the evaluation policy over 10 rollouts and 5 seeds learning from non-noisy expert demonstrations, reported every 100k timesteps. The return term of the environment's ground truth reward.
Settings
Walker
Hopper
Swimmer
HallCheetah
Ant
Humanoid
3897.8 + 177.8 2706 + 219.7 141.5 + 3.7 1674.1 + 33.6 2715.7 + 98.1 3892.3 + 522.9
Non-Noisy
4055.9 + 304.0 2580.3 + 186.4 146.1 + 2.6 1660.6 + 26.3 2930.4 + 236.8
122 + 17
3589.1 +: 427.4 2770.5 1 66.8
143.1 1:3 1670.6 1 20.4 3062.9 1: 282
2167 + 1212.1
3554.6 275.6 1891.8 197.0
105.0 2.7 1501.6 + 92.4 2868.1 + 95.4
3690.2 + 374.8
Noisy
3364.5 + 377.3 1638.3
308.5
107.8 + 9.6 1444.3 1 16.8 2440.3 + 184.4 123.2 1 23.1
3561.8 + 345.7 1671.7 + 126.7 105.7 + 4.4 1429.2 + 40.1 2674.9 + 134.3 2521.1 + 860.9
Table 3: Learned policy performance for different imitation learning algorithms non-noisy and noisy expert data, evaluated by using the mode of sampling from the policy distributions. The best results are marked as red respectively. Top: Ours; Middle: Ours-exponential; Bottom: Ours- sigmoid.
We conduct the experiments with another forms of reward functions following the Jensen-Shannon divergence (Goodfellow et al., 2014) for comparison. Typically, JS divergence defined
Pr (x(
P^(x) (P) (x)))
JS(P,, Pa) = FFrP, [log(
)] FZ [log(
(10)
(x) +P)()))
where P, and P. represents the real data distribution and the generated data distribution. The adver- sarial training procedure of auto-encoding inverse reinforcement learning will match the state action distribution of expert and generated samples.
In the first form of alternative reward functions, we consider the exponential of negative reconstruc- tion emror (z) (2)). The corresponding -hasennShannon divergence for updating
17

---

Under review as a conference paper at ICLR 2022
the auto-encoder based reward function is:
- Da [og((ex((-AAe, AA (s, a)))] - (s.))rm [og(1 expm111expppppppp exp(-AA.o(s,u))
(11)
(s,))) [AE (s. a)] ((,a))== [log(1 (
(12)
Here, w represents the parameters of the auto-encoder network. So its formulation of surrogate reward function is:
(s, = exp
(13)
In this formulation, minimizing the -hasnnnShannon divergence between state action distribution of the expert and generated trajectories assigns low mean square errors to the expert samples and high mean square errors to other regions.
In the second formulation of reward functions, we consider the sigmoid output of the negative re- construction error as: (z) So it corresponding objective function for updating the auto-encoder is:
L - -E (s,a)-Dy (sigmooid) AA a))] + (s.a)>> [log(1 sigmoid( (-anoodd(AAEuaee AA, as)))))] (14) where w also represents the parameters of the auto-encoder network. And its reward function for- mulation is:
Tw (s, a) log(1 sigmoid( Ae a)))
(15)
In experimental results, we note the first formulation as "ours-exponential" and the second formu- lation as sigmod""" Specifically, we don't add the entropy term to the reinforcement objective which already achieve competitive results on MuJoCo tasks.
These three forms of reward signals achieve comparable performance on all tasks except for Hu- manoid shown in Figure 9. This indicates that an auto-encoder based reward function is the major contributing factor to improve the performance on mimicking the expert rather than the different distribution divergences or forms of reward signal. However, the performance on Humanoid is quite different between these different forms reward signals. It shows that the Wasserstein distance can be more robust to this task.
Our method not limited to vanilla auto-encoders but also other alternative auto-ens To show the differences between our method and other alternative auto-encoders, conduct the experi- ments with variational auto-encoder based reward function. The variational auto-encoder (Kingma & Welling, 2014) is directed model that uses learned approximate inference. The key insight behind training variational auto-encoder is to maximize the variational lower bound L(g) of the log-likelihood for the training examples (Goodfellow et al., 2016):
L(q)= = Ez=g(z)))) log Pmodel (z, xx) - Hl(((2(z))
(16)
= Ez)g(())) log Pmoda DKL (xL(((2t) (1(=)==))llpmoeee(
(17)
< log (2)
(18)
where z is the latent representation for data points x. And Pmodel is chosen to be fixed Gaussian dis- tribution (0, 7). In Eq. 16, we recognize that the second entropy term encourages the variational posterior to place high probability mass on values many as possible, rather than collapse to single most likely value point. When we use an variational auto-encoder based reward function in our method, the latent representations of the expert samples can be widely distributed on a Gaussian distribution. Therefore, matching the expert and generated state action distributions in this latent space leads to more diverse behaviors inducing. Thus, the reward signal would be more informative.
Training the variational auto-encoder is still adversarial process. We notice that the first term in Eq. 17 is the reconstruction log- likelihood found in traditional auto-encoders, So the objective for training the variational auto-encoder is:
L = = Dxs.saa>aDDrrrles + DRL (pL(((=((s((())) ()) Pm)tet(()))
(19)
(aa)rra (s. + DxL (p(2p((( (0,==(s,a)) (2))
where the choice for r. (s, a) is the same with Eq. in Section 4.2. Therefore, low mean square emorr of the variational auto-encoder ensure high reward values for state action pairs and vice versa.
18

---

Under review as a conference paper at ICLR 2022
Walker2d
Swimmer
HalfCheetah
160
140
Return
Return
Return
00
2000
60
40
1000
-500
40
20
40
60
80
100
Timesteps (1e5)
Timesteps (1e5)
Timesteps (le5)
Anl
Humanoid
Hopper
2000
2006
Return
Return
3000
Return
2000
1006
2500
2000
Oure-FF
01
Ours-VAE
20
20
300
20
100
Timesteps (1e5)
Timesteps (1e5)
Timesteps (1e5)
Figure 10: Mean and standard deviation return of the evaluation policy over 10 rollouts and 5 seeds learning from non-noisy expert demonstrations, reported every 100k timesteps. The return is in term of the environment's ground truth reward.
When minimizing this objective, the variational auto-encoder based reward function can still assign high values near the expert demonstrations and low values to other regions which is the same as in typical auto-encoders. On the other hand, optimizing this objective is restricting the latent dis- tribution for the expert to a fixed Gaussian distribution Pmodee and maximizing the KL distance between the generated samples and the same fixed Gaussian distribution. With the bi-level opti- mization process
can induce the expert policy with more diverse behaviors from the expert demonstrations which addresses the problem of mode collapse in GAIL. And a more informative reward signal would be yield.
Figure 10 shows the training curves for vanilla auto-encoder based variant and varational auto- encoder based variant on non-noisy expert demonstrations. The performances for these two variants are comparable on almost all tasks while vanilla auto-encoder based method is better than varational auto-encoder based variant on Humanoid. It indicates that the variational auto-encoder can be an alternative reward function formulation.
A.6 ANALYSIS ON STABILITY OF OUR METHODS
The variance of gradients is computed via different mini-batches of training examples. Lower vari- ance of gradients means smoother updating of the network (Fughri al., 2020). Now that our auto-encoder based reward function is more informative and denser than the discriminator based ones, how about the training stability our method? To show the training stability of different methods, record the norm of weights of the policy network and the variance of policy gradients. The relative variance of policy gradients shows the training stability for different methods.
While we utilize Wasserstein distance in our method, we compare vanilla auto-encoder based variant and variational auto-encoder based variant with WGAIL. could depict the training stability with newly proposed auto-encoder based reward formulation. Figure 11 shows the training stability of our method compared to the baseline WGAIL, depicts that our method can be more stable and the policy network updating is smoother.
19

---

Under review as a conference paper at ICLR 2022
Non-Noisy Expert
Non-Noisy Expert
Nolsy Expert
Noisy Expert
0.
Norm of Weights
Varlance of Grads
Norm of Weights
Variance Grads.
66
100
30 86
200
20
Timesteps (1e5)
Timesteps (1e5)
Timesteps
(1e5)
Timesteps (1e5)
Non-Noisy Expert
Non-Noisy Expert
Noisy Expert
Noisy Expert
VEAA Oure-2
Norm of Weights
Variance of Grads
31
Norm of Weights
Variance
13
330
20
392
Timesteps (1e5)
limesteps (1eb)
Timesteps (1e5)
limesteps (1eb)
Non-Noisy Expert
Non-Noisy Expert
Noisy Expert
Noisy Expert
Norm of Weights
2037
Variance of Grads
Norm or Weights
Varlance of Grads
Oca
223
203
64-
03-
0.
02
100
d5 Timesteps (1e5)
100
Timesteps (1e5)
Timesteps (1e5)
Timesteps (1e5)
Figure 11:
More stable training procedure: norm of weights for the policy network and the corre-
sponding variance of policy gradients for non-noisy and noisy expert data. Top: Walker2d; Middle: Swimmer; Bottom: Hopper.
A.7 ADDITIONAL EXPERIMENTS ON BETTER SYNTHESIS EXPERTS
Task
Walker
Hopper
HalfCheetah
Expert
6121.7
3548.0
8879.1
4177.0
GAIL
2933.4 392.6
2807.1 + 196.
3107.4 1138.9
1866.9 + 688.8
WGAIL
2692.5 + 746.7
2850.3 + 198.8
3502.2
1411.0
1964.6 + 575.1
FAIRL
2425.0 1 216.2
2630.4 1: 263.8
1655.9 +: 591.6
2107.2 1: 324.7
PWIL
1414.8 + 1033.8
2969.5 + 86.7
5608.8 + 411.0
1661.2 + 397.8
Ours
4036.8 159.8
2939.2 183.4
5775.1 +: 644.8
3104. 95.4
Table 4: Learned policy performance for different imitation learning algorithms on non-noisy expert data, evaluated by using the mode of sampling from the policy distributions. The best results are marked as red.
To get more convincing results, we run the experiments on better synthesis experts. We collect the expert demonstration with 16 trajectories each task via D4PG (Barth-Maron et al., 2018).
Figure 12 depicts the training curves of ground truth rewards for different algorithms. The training curve shows the imitation performance learning from non-noisy expert data. Table 4 shows the final learned policy performance on MuJoCo tasks with ground truth episode return.
The overall results are similar with what we have done in the main text. On Walker2d, HalfCheetah and Ant, our method gets the best performance. PWIL gets the best performance and our AEIRL is on par with PWIL on Hopper.
20

---

Under review as a conference paper at ICLR 2022
Walker2d
HalfCheetah
0.00
6000
5020
30.00
4000
2000
60
102
03
10c
Timesteps (1e5)
Timesteps (1e5)
Ant
Hopper
4000
1000
.000
3070
2000
2000
2000
Expert
GAIL
1000
WGAIL
-2000
3000
500
Ours
0000
10
60 80
300
Timesteps (1e5)
Timesteps (1e5)
Figure 12: Mean and standard deviation return the evaluation policy over 10 rollouts and 5 seeds, learning from non-noisy expert demonstrations, reported every 100k timesteps. The return is in term the environment's ground truth reward.
We take the random and expert trajectory as 0 and respectively to compute the scaled final rewards for different methods. The overall averaged scaled rewards for our method is about 0.729 where the best baseline is 0.546 for PWIL. It is more than 33.7% relative improvement. So our method outperforms the other state-of-the-arr baselines mostly.
21


Reviews:
[{"review_id": "reC5K81W55-", "reviewer": "Reviewer_jqo8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The use of auto-encoder and reconstruction error for formulating reward functions is novel.\n* The auto-encoder is flexible and has many choices in the literature, and I appreciate the authors' effort of trying different variants on that in ablation study.\n* The ablation study on distance function for inverse RL loss function is also impressive, which shows the importance of using auto-encoder.\n* I appreciate the effort of visualizing the latent representation of the reward model, though with some concerns on the plot. Please see details below.\n\nSome questions and concerns:\n* I wonder why using TRPO as the baseline agent, especially when obtaining the expert agent. \n  * It's well-known that the performance of learning a reward model (and also a police) is affected by how good the expert agent is, so I wonder if you have tried some more recent and better agents in the literature. \n  * For example, the PWIL paper shows a great agent with much higher score on Humanoid (nearly 9000) and results in the imitating agent to achieve 7000+ scores, and their agent was trained by D4PG.\n* The description of agent and reward model (and/or auto-encoder) architecture is missing. Please add it for completeness and reproducibility.\n* As it has been widely discussed in the literature, there are too many misuses of t-SNE due to the lack of misunderstanding hyper-parameters and possibly more details. I generally recommend the authors to provide more details when doing the t-SNE plots. For example, the authors can provide the value of `perplexity` that used for the figure (you can do that in appendix), and possibly plot more than one plots by varying `perplexity`.\n  * Also, please provide details on which latent space is plotting, for both auto-encoder and discriminators.\n* *Question*: is it correct to say, a well-trained auto-encoder in the loop (e.g, in Fig. 1) can encode expert samples better with lower reconstruction error, while it doesn't capture the features of generated samples and result in higher error?\n* *Question*: is it possible to provide theoretical analysis on why the proposed method can minimize the distance between the state-action pair distributions of expert and agent samples, or other related similarity / distance metrics?", "summary_of_the_paper": "This paper proposes to use auto-encoder for inverse reinforcement learning. The main goal of using the auto-encoder, is to use it as the reward function, which takes the auto-encoder reconstruction error to provide reward signals for the agent. The authors claim that such approach provides more informative signal than existing works, especially adversarial imitation learning approaches. Their experiments demonstrate that this method has a better performance over other baselines, and more robust to demonstration noise.", "summary_of_the_review": "The authors provide a good insight on how the adversarial imitation learning algorithms fail. The sensitivity to minor differences between expert and agent samples does impede the learning of agent because it may be pushed to learn how to imitate the details instead of other more important features, and thus less robust to the noise of data. Given that, the authors introduce auto-encoder to handle this issue, and the experiments confirms the claim by showing its better performance over other baselines.\n\nHowever, I still have some concerns regarding the writing, mostly the lack of important details. The experiments may need to conduct in another round with a more recent and powerful RL agent, which should be easily available now. I also hope to see if the authors can evaluate their approach in more complex and/or real-world environments. The theoretical understanding of the proposed approach is also worth further investigating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "huCyqHJfFL2", "reviewer": "Reviewer_Ui5q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Auto-encoder based reward function was proposed to denoise the expert data through the encoding-decoding process.\n2. The experiments on the MoJoCo showed that the proposed method outperformed state-of-the-art imitation learning on both clean and noisy expert demonstrations.\n3. Ablation studies showed that different reward functions and loss functions also received good performance.\n\nCons:\n1. Theoretical justification to support the idea of auto-encoder based surrogate reward function was missing.\n2. This method was to minimize the Wasserstein distance between the state action distribution of the policy and the expert demonstrations. But, there was not explanation on Lipschitz constraint or weight clipping.\n3. This paper mentioned WGAIL (Arjovsky et al., 2017) many times, but Wasserstein GAN did not use reinforcement learning or imitation learning.\n4. Need theoretical discussion about using the variational auto-encoder based reward function and the Jensen-Shannon divergence to replace auto-encoder and Wasserstein distance, respectively.\n5. Lack of suitable experimental results to support that auto-encoder based reward function was more informative and denser than the discriminator based one.", "summary_of_the_paper": "This paper presented the auto-encoding inverse reinforcement learning where the reward function is recovered from expert demonstrations in inverse RL problem. The reward was represented as a surrogate function which used the auto-encoder with the metric of reconstruction error.", "summary_of_the_review": "A new auto-encoder based reward function was presented. Theoretical and experimental justifications were insufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "goeghmoygVE", "reviewer": "Reviewer_4gyw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is an interesting approach to IRL, using minimax games and autoencoding to enhance training.  While the approach appears novel (IRL is not my main area), the experimental results, while generally positive, are a little mixed in terms of which approach is preferred (especially VAE vs non-VAE). \n\nThe empirical results on noise-free data are impressive (as are some of the noisy results), but I would have liked to have seen more explanation on why their VAE-based approach outperforms non-VAE on some tasks but not others. I.e., which one should be chosen based on the task? \n\nI don't understand the significance of Figure 4.  In what way are your approach's latent representations \"more indistinguishable\" from those of the others?  In particular, I don't see any real difference on the non-noisy data.  Can you quantify this somehow? \n\nI read the authors' response, but it didn't compel me to change my score. ", "summary_of_the_paper": "The authors present an architecture that utilizes autoencoding as an approach to inverse reinforcement learning.  They present a mathematical formalism, pseudocode, and some empirical analyses on MuJoCo tasks.  ", "summary_of_the_review": "The paper is an interesting approach to IRL, using minimax games and autoencoding to enhance training.  While the approach appears novel (IRL is not my main area), the experimental results, while generally positive, are a little mixed in terms of which approach is preferred (especially VAE vs non-VAE). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eXDTQ0S98GA", "reviewer": "Reviewer_vWWw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper Strengths: \n* AEIRL intuitively provides several benefits. (1) it denoises potentially noisy expert demonstrations, whereas prior inverse RL methods struggle to work with noisy expert demonstrations. (2) the auto-encoder also provides a more informative reward signal whereas the binary classification objective in works like GAIL, can easily result in overfitting. \n* The method seems easy to implement only requiring changing the discriminator to an auto-encoder.\n* The main experiments of the paper are thorough. In both non-noisy and noisy demonstrations they compare on 6 standard benchmark tasks against 5 baselines and two versions of their method, all using 5 seeds. \n* Their method performs convincingly better in the non-noisy setting and mostly better in the noisy setting. \n* There are extensive analyses and ablation studies. I especially found the study of the reward signal in Figure 5 to be informative. This adds qualitative evidence to AEIRL learning a more dense reward function. \n\nPaper Weaknesses: \n* Why does AEIRL perform relatively better on non-noisy expert demonstrations compared to noisy expert demonstrations? The motivation of the method is that it can work well from noisy inputs, yet the method is outperformed in several of the noisy settings and has a large performance drop. This is the opposite of what I would expect. \n* While the reward is intuitive, can it recover a true reward function, or is the reward coupled with the policy like with GAIL? \n* This method is hard to scale to higher dimensional tasks like image-based control. AEIRL must learn a decoder to reconstruct observations. The reconstruction error could be less reliable as a reward signal in these higher dimension tasks since learning the decoder is harder. \n* What happened on the Ant task with noise? Many of the methods went from very positive rewards to negative rewards. \n* The numbers in Tables 1 and 2 are too close together, making some of them hard to read. ", "summary_of_the_paper": "This paper proposes the Auto-Encoding Inverse Reinforcement Learning (AEIRL) method for better imitation learning, especially in the presence of noisy expert demonstrations. The paper’s key insight is that auto-encoders can eliminate the effects of noise from expert demonstrations and provide a more stable reward signal based on the auto-encoder error. The experiments feature extensive analyses across standard benchmarks compared to a large number of baselines. ", "summary_of_the_review": "The method is intuitive and performs strongly across in an extensive imitation learning evaluation. Their analyses are also convincing that their. I, therefore, recommend acceptance.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work addresses the problem of learning representations from noisy expert demonstrations in in adversarial imitation learning. The authors build on top of GAIL, which utilizes a discriminator to model a "pseudo"-reward from demonstrations. In this work, the discriminator is replaced with an auto-encoder. The authors hypothesis is that using an auto-encoder helps in 2 ways: 1) denoising expert trajectories for more "robust" learning; 2) using the reconstruction error (instead of binary classification loss) to distinguis experts from samples provides more informative signal for reward learning. 

**Strengths**
on a global perspective this work is well motivated
a novel algorithmic variant of GAIL is proposed 
thorough experimental evaluation

**weaknesses**
The manuscript doesn't clearly distinguish between adversarial imitation learning algorithms (like GAIL) and "true" inverse reinforcement learning algorithms. This makes it unclear what the real goal of the proposed method is. The ultimate goal of adversarial IL is to learn a policy (by inferring a pseudo-reward at "train" time which is then never used again), while the primary goal of IRL is to learn a reward function at train time, which can then be used at test time. The manuscript motivates the algorithm by saying it will have a more informative signal for learning reward functions, but the algorithm itself is an adversarial IL algorithm which primary goal is to learn a policy from demonstrations. Overall, makes the evaluation and analysis confusing. Ideally, the authors would have focussed on the question "Does the reconstruction error lead to better policies?" (through better pseudo-reward modeling) - or would have extended an IRL method.

Second, the motivation is that the autoencoder helps with more "robust" learning, but it's unclear to me that the evaluation really shows that learning is more robust (also because "robustness" is not clearly defined)

The experimental evaluation is a bit of a mixed bag, and it's a unclear why the new algorithm performs better on non-noisy data (when compared to baselines), but not less so on the noisy data.

**Summary**
Overall, this work provides a promising direction, however in it's current form the manuscript is not yet ready for publication.
Decision: Reject