Unbiased Learning with State-Conditioned Rewards in Adversarial Imitation Learning
Authors: Dong-Sig Han
Hyunseo Kim
Hyundo Lee
Je-Hwan Ryu
Byoung-Tak Zhang
Venue:  (2021)
Keywords: Adversarial Learning
Imitation Learning
Inverse Reinforcement Learning
Reinforcement Learning
Transfer Learning

Abstract: Adversarial imitation learning has emerged as a general and scalable framework for automatic reward acquisition. However, we point out that previous methods commonly exploited occupancy-dependent reward learning formulation—which hinders the reconstruction of optimal decision as an energy-based model. Despite the theoretical justification, the occupancy measures tend to cause issues in practice because of high variance and low vulnerability to domain shifts. Another reported problem is termination biases induced by provided rewarding and regularization schemes around terminal states. In order to deal with these issues, this work presents a novel algorithm called causal adversarial inverse reinforcement learning. Our formulation draws a strong connection between adversarial learning and energy-based reinforcement learning; thus, the architecture is capable of recovering a reward function that induces a multi-modal policy. In experiments, we demonstrate that our approach outperforms prior methods in challenging continuous control tasks, even under significant variation in the environments.

Under review as a conference paper at ICLR 2021
UNBIASED LEARNING WITH STATE-COND REWARDS IN ADVERSARIAL IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial imitation learning has emerged as a general and scalable framework for automatic reward acquisition. However, we point out that previous methods commonly exploited accupancy-dependees reward learning formulation wwhic hinders the reconstruction of optimal decision as an energy-based model. Despite the theoretical justification, the occupancy measures tend to cause issues in prac- tice because of high variance and low vulnerability to domain shifts. Another reported problem is termination biases induced by provided rewarding and regu- larization schemes around terminal states. In order to deal with these issues, this work presents novel algorithm called causal adversarial inverse reinforcement learning. Our formulation draws a strong connection between adversarial learn- ing and energy-based reinforcement learning; thus, the architecture is capable of recovering reward function that induces a multi-modal policy. In experiments, we demonstrate that our approach outperforms prior methods in challenging con- tinuous control tasks, even under significant variation in the environments.
INTRODUCTION
Inverse reinforcement learning (IRL) is an algorithm of recovering the ground truth reward function from observed behavior (Ng & Russell] 2000). IRL. ffollwhmss-ollowwdd by appropriate reinforce- ment learning (RL) algorithms- can optimize policy through farsighted cumulative value measures in the given system (Sutton & Barto 2018); hence it can usually achieve more satisfying results than mere supervision. While a few studies have investigated recovering reward functions to con- tinuous spaces (Babes al. 2011 Levine & Koltun 2012), IRL algorithms often fail to find the ground-truth reward function in high-dimensional complex domains (Finn et al. 20166).
The notion of the ground-truth reward requires elaboration since IRL. is an ill-posed problem: there can be numerous solutions to the reward function inducing the same optimal policy (Ng et al.) 1999. Ng & Russell 2000). Recently, adversarial imitation learning (AIL) as a reward acquisition method has shown promising results (Ho & Ermon, 2016). One of the distinctive strengths of AIL is the scalability through parameterized non-linear functions such as neural networks.
The maximum causal entropy principles are widely regarded as the solution when the optimal con- trol problem is modeled as probabilistic inference (Ziebart et al. 2010 Haarnoja al. 2017). In particular, probabilistic modeling using a continuous energy function forms a representation called an energy-based model (EBM). We highlight the following advantages of the energy- based IRL:
It provides a unified framework for stochastic policies to the learning; most probabilistic models can viewed as special types of EBMs (LeCun et al., (2066).
It rationalizes the stochasticity of behavior; this provides robustness in the face of uncertain dy- namics (Ziebart et al. 2010)) and a natural way of modeling complex multi-modal distribution.
AIL reward functions seem be exceptions to these arguments- the AIL framework produces distinct types of rewards that are ever-changing and are intended for discriminating joint densities. We argue that these characteristics hinder proper information projection to the optimal decision.
This work points out that there remain two kinds of biases in AIL. The established AIL algorithms typically formalized by the cumulative densities called occupancy measure. We claim that the accumulated measures contain biases that are not related to modeling purposeful behavior, and the formulation is vulnerable to distributional shifts of an MDP. Empirically, they work as dominant

---

Under review as a conference paper at ICLR 2021
noises in training because of the formulation's innate high variance. The other bias is implicit survival or early termination bias caused by reward formulation, which lacks consideration for the terminal states in finite episodes. These unnormalized rewards often provokes sub-optimal behaviors where the agent learns to maliciously make use of temporal-aware strategies.
This paper proposes adversarial IRL method called causal adversarial inverse reinforcement learning (CAIRL). We primarily associate the reward acquisition method with approaches for energy-based RL and IRL algorithms: the CAIRL reward function can induce complex probabílistic behaviors with multiple modalities. We then show that learning with a dual discriminator architec- ture provides stepwise, state-condine condeccondiio rewards. For handling biases induced by fmite-horizon, the model postulates the reward function satisfies a Bellman equation, including "self-looping" terminal states. As result, learns the reward function satisfying the property of EBMs.
Noteworthy contributions of this work are 1) model-free, energy- based IRL algorithm that is ef- fective in high-din nigh-dimensional environments, 2) dual discriminator architecture for recovering a ro- bust state-conditioned reward function, 3) effective approach for handling terminal states, and 4) meaningful experiments and comparison studies with state-of-the-art algorithms in various topics.
RELATED WORKS
Imitation learning is fundamental approach for modeling intellectual behavior from an expert at specific tasks (Pomerleau 1991, Zhang et al., 2018). For the standard framework called Behavioral Cloning, learning from demonstrations treated as supervised learning for a trajectory dataset. On the other hand, IRL aims to study the reward function of the underlying system, which characterizes the expert. In this perspective, training policy with an IRI. reward function is branch of imitation learning, specialized in dealing with sequential prcision-making problems by recovering the concise representation of a task (Ng & Russell 2000 Abbeel & Ng 2004).
For modeling stochastic expert policies, Boltzmann distributions appeared in early IRL research, such as Bayesian IRL, natural gradient IRL, and maximum likelihood IRL (Ramachandran & Amir 2007 Neu & Szepesvári, 2012; Babes et al.) (011). Notably, maximum entropy IRL (Ziebart al. 2008 is explicitly formulated based on the principle of maximum entropy. The framework has also been derived from causal entropy the derived algorithm can model the purposeful distribution of optimal policy into a reward function (Ziebart et al.) 2010). Our work draws significant inspirations from these prior works and aims to redeem the perspective of probabilistic causality.
Recently, AIL methods (Ho & Ermon 2016; Fu et al. 2017 Ghasemipour et al. 2020) have shown great success on continuous control benchmarks. Each of the them provides a unique divergence minimization scheme by its architecture. In particular, our work shares major components with AIRL. It has been argued that the algorithm does not recover the energy of the expert policy (Liu) et al. 2020). We stress that our work introduces essential concepts to draw an energy- repre- sentation of the expert policy correctly. The discriminator design based on the rich energy-based interpretation of GANs (Zhao et al., 2016][Azadi et al.) 2018 Che et al. 2020) and numerous studies with multiple discriminators (Chongxuan et al. 2017 Gan et al. 2017 7
Choi et al.) 2018).
The issues of finite-horizon tasks were initially raised in RL during the discussion of time limits in MDP benchmarks (Pardo et al., 2017] Tucker et al.] (018)). turned out that the time limits, or even the existence of terminal states, would significantly affect the value learning procedure of RL com- pared to that generated infinite horizon MDPs. IRL suffers from the identical problem that reward learning of finite episodes is not really stable for tasks outside of appropriate benchmarks, Kostrikov et al. (2018) suggested explicitly adding a self-repeating absorbing state (Sutton & Barto (0088) after the terminal state; consequently, AIL discriminators can evaluate the termination frequencies.
BACKGROUND
Markov Decision Process (MDP). We define an MDP as tuple M = AS, where S and A denote the state and action spaces, and y is the discount factor. The transition distribution P. the deterministic state-action reward function and the initial state distribution Po are unknown. Let Tx and Th be sequences of finite states and actions (Sg, op...., .. obtained by a policy and the expert policy ne, respectively. The term Pr denotes the occupancy measures derived by
2

---

Under review as a conference paper at ICLR 2021
Table 1: The objectives for AlL algorithms in form the minimization of statistical divergences.
Method
Optimized Objective (Minimization)
Behavioral Cloning
Exg (=T(a = -Eng [lg + const
GAIL Ho & Ermon
2016
E pr),s,
H((+(-1s)]
AIRL Fu
2017
E. ( a)) -En a) H(px))
FAIRL Ghasemipour et al.
2020
Er DKL (s) = -Err [ (=(())) H(pe)
CAIRL (this work)
Er (
-Ex + H((T(-1))) const
T, and is defined as (s a) = T(als) Ei=0 Pr(s, = s|x). With a slight abuse of notation, we refer to the occupancy measures of states (F((s and (s(s). The expectation of i for an arbitrary function denotes an expected return for infinite-horizon: E,, [c(s a)] & EE 7 cc(ss a)]=).
Maximum Entropy IRL (MaxEnt IRL). Ziebart (2010) and Haarnoja et al. (2017) defined the optimality of stochastic policy with entropy-regularized RL objective follows:
Σ, (s [r(s), 0z) a(H(ar(s:))
arg max TET
where H. denotes the causal entropy function If mh is the MaxEnt RL policy, the softmax Bellman optimality equations can be defined by the following recursive logic:
Q* (st. 04) r(St; 0e)
+ (( [V (s++1]
(I)
(*t) (. [Q" (se, 0.) - log TE (aels:)]
MaxEnt IRL algorithms (Ziebart et al., 2008 2010) are energy-based interpretations of IRL which aim to find behavior abiding the MaxEnt principle. Such algorithms, however, are difficult to be computed when the given spaces are continuous or dynamics are unknown (Finn et al. 2016a).
Adversarial Imitation Learning. Ho & Ermon 2016) considered adversarial learning as model- free, sampling- based approximation to MaxEnt IRL. Instead of exhaustively solving the problem, GAIL performs imitation learning by minimizing the divergence between the state-action occupancy measures from expert and learner through the following logistic objective:
min max E [log Dss, a)] + E,, [log(1 D(s, a))] - H(T))
(2)
TEll D
where D E (0, 1) indicates binary classifier trained to distinguish between Ta and Tg. The AIRL discriminator tries to disentangle reward function that is invariant to dynamics. takes a particular form: Do a) exp(fe a)) (exp(Je,< a)) (als))) Learning with the AIRL can be considered as the reverse KL divergence between occupancy measures. Ghasemipour et al. (220 proposed the FAIRL algorithm an adversarial method for the forward KL divergence.
UNBIASED REINFORCEMENT SIGNALS FOR ENERGY-BASBR MODELS
Our aim in this section is to investigate unbiased probabilistic modeling of the causality of decisions using the MaxEnt framework. In Sec. 4.1 we discuss the energy-based reward function. Sec. 4.2 and Sec. introduce the modeling method of the particular reward function.
4.1 ENERGY-BASED REWARD REPRESENTATION
To accurately manage the MaxEnt framework, the EBM expert set to (g(a)(s)) o exp(-E(sx } = exp(Q(#t, a.)). such that the likelihood of distribution is proportional to soft Q-function. The MaxEnt RL processes be interpreted minimizing the expected KL- divergence using the information projection (Haarnoja et al. 2017 2018):
J [ = Esypr
DKL (-|ss))
exp Q(x.11
Z(e)
For the rest of the paper, occasionally omit the parameter with 1 for simplicity of derivation.
3

---

Under review as a conference paper at ICLR 2021
Apparently in IRL, since the ground-truthh reference will be the expert policy, the general objective of imitation learning: л² = arg min E, [DKL (T(: s) [)][ is indeed the special type of the
MaxEnt RL objectives. We describe state conticcondiiioned energy mergy-based reward as representation:
If y =z 0, it formulates the "myopic" 1 1-step conditional KL divergence: ( [ (-|ss)
If 1, assuming the dynamics are identical, it leads the "far-sighted" cumulative densities:
E E
= Dxt. [rr(ao $1,... ss = s, r) Pr(ao: $1.... 1s0 = 8, x =))]
By the discount factor 7, can control how much subsequent steps we want to model. Therefore, the energy-basee based rewards generalize learning probabilistic inference of conditional decisions.
Note that AIL reward functions, curated in Table usually do not retain these properties. For exam- ple, the AIRL reward function, namely: f(s a) log(Pn(s, a)/ (s)), recovers the expert policy, only in the trivial case 7 O, as TT(aas)) TE, and also difficult to be analyzed. We also highlight that the standard MaxEnt RL with AlL rewards may not precisely recover the EBM. The more discussions are addressed in Appendix B.2
((..a) For Ty > the projection generally is not Ee I(s.a')-
4.2 STATE-CONDITION
REWARDS ON THE PRINCIPLE OF MAXIMUM ENTROPY
We clarify a valid candidate set of reward functions when cutroy--egularizattin is applied. One trivial solution of such function is the log- ligellkeiooodd of expert r* (s, a) log T (a]s), in the con- dition: (*)))= = (+')" = 0. Then, 1-step expectation of r* draws a conditional KL divergence:
(3) By the property of KL. divergence, the above expression is less than or equal to O where the equality holds if and only if (als) = Ta(a)s), ())s), Va E A. Since the optimal value function outputs zero, the reward shaping (Ng al., 1999) of r* is not required. As a result, the optimality of the action is instantaneously determined without consideration of future rewards. It leaves the following remark.
(a-s())s)) [r* (s) H(([(-1))] = -KL. [T(+ss][[
50
Remark 4.1. For arbitrary discounted rates, r' (s, a) = log (a(())) the optimally shaped reward function for learning efficiency in the Shannon reglaizinggl
From this insight, the ligelikel of expert policy can be considered as a desirable state- conditioned reward function for the MaxEnt RL. However, oftentimes directly modeling the log density (such as BC methods) is not practical due to the limited number of samples. We can alter- natively relaxes likelihood estimation by using a state potential function V (Ng al. 1999):
R [r|r(s, a) + (s')- (s) log g (als)) V : IR, Vs,a, s' E A
(4)
Akin to deterministic case, we formalize point that Eq. (4) does not change the learning objective.
Proposition 4.1. Let be a function satisfying Eq. 4 Then the expected cumulative reward of has following property: En [r(s a) + H(([-1s)) = E. [ (EE(s))
+ E. (s))
Note that the term E (s) is independent of л. Compared to the AIRL, the function r E A can be applied for training arbitrary policies since the overlapping state densities log (s())/p((s) detached. The subsequent learning of r provides an projection, which is the closest estimation of Pr(eo, ... |so = 8, RE) for the current state.
4.3 HANDLING FINITE-HORIZON BIASES VIA KL REGULARIZATION ASSUMPTION
In order to mitigate the terminiation biases depicted in Fig. (a). Kostrikov et al. 2018) highlighted the concept of absorbing states and suggested adding synthetic transitions into the trajectories (see Fig. (b)). which explicitly promotes learning for termination frequencies. In contrast, we focus on another intriguing point that the absorbing state makes all episodes virtually have the properties of infinite horizon. Consider that the expert represents the MaxEnt behavior even when self-looping state is encountered. Since the action selection can change neither reward nor transition state, the "expert" would represent complete random behavior, which is identical to the uniform distribution.
In the sense that they do not require the explicit manipulation of trajectories, Fig. D (c) and (d) demonstrate episodes by assuming self-looping terminal states. The two instances are alike for handling the bi- ases as H, KT, + const, but the KI. regularization has slight advantage because terminal
possible simpler examples to turn lenthh episodes into infinite-horizon
4

---

Under review as a conference paper at ICLR 2021
Hr-2
HT-1
Hr-2
Hy-1
Hmax
r-
ST
Survive
T--2
---1
8T
(a) Entropy bonuses finite horizon MDP.
(c) Entropy bonuses with a looping terminal state.
HT=
HT-1
Hmaax
Hmax
KLT-2
KLT-1 1
P--2
7-
ST
8a
9--9
9--1
ST
(b) Entropy bonuses including an absorbing statc.
(d) KL penalties with looping terminal state.
Visualization of entropy regularization methods and biases around terminal states. The red vertices and edges represent absorbing states and maximum entropy action selection. Each regularization is defined as H4 = (+((++)))) and KI, = DKL (-xs(=(-sss))) (P)uit) respectively.
Figure 1:
state value is always zero instead of Σk=o 0 Hmaax In other words, only switching the regulation scheme from Fig. (a) has eliminated the termination bias. Consequently, the KL penalties can be seamlessly integrated into finite-Horizon MDPs without any pre-processing around terminal states.
With the same reasoning as in Sec. 4.2 the term log */(a)))/u can understood as the optimally shaped reward under the KI. regularization. The constant denotes the likelihood of Punif (i.e. Punit (a), Va E A). To this end, we set the learming objective of the shaped reward function: fs, s') ro a) + th. - (s) = log ==(ale)/u
(5) where and hg denote reward and potential networks. h&g denotes target potential network with identical parameter th, yet the gradient computation is disconnected while training, which is
analogous to constructing target value networks in the deep RL domain (Mnih et al. 2015). For
every terminal state, we fixed the output to the optimal solution: ro(sr, a) = h 0, va E A.
CAUSAL ADVERSARIAL INVERSE REINFORCEMENT LEARNING
5.1 A DUAL DISCRIMINATOR ARCHITECTURE
The methodology is grounded on the findings that an AlL discriminator also contains an EBM, and property that an EBM on joint densities can be decomposed into multiple EBMs (Zhao et al., 2016 Che et al. 2020 Azadi et al., 2018). Suppose that the GAIL discriminator is nearly optimal:
Pæ(s,a)
D(s a) exp(-d(ss,)) a)) PF(s,a) + (s,
where d(s,a) a) denotes the logit of D(8, a). We disentangle the logit function to the following form:
PE (E(s.a) a) Pr (s. a)
log PE(s)
(a(a)s)
d(s,a) > log and then we have two log-ratios of state occupancy measures and policy distributions.
+ log
Pr
(als)
The model substitutes log-ratio of state occupancy measures using state-only discriminator D, with nearly optimal logit score of ds(s)) s log(pe(*)/o=(o) The role of D,, is to nullify the difference between state densities by pre-applying it. We propose an architecture of discriminator:
exp[Jo. (8, a, 8')]
(o,
(s, a, expl - + log [a(a|s)
where (s) and Th (als) are pre-computed. Using D,p and ne as scaffolds, the shaped reward function fo. converges Eq. (5) and becomes specialized in evaluating conditional decisions, where learning with the reward function correctly projects to the optimal policy.
The discriminators Dup and Do. are trained for maximizing the following objectives, respectively:
J(Dz) Fegg [log (s) + E.g [log(1 - Ds(s)))
I(Do.w) = Ere [log Do a, s')] + E.,, [log(1 - (s, s'i)] AB
5

---

Under review as a conference paper at ICLR 2021
where (v(ss (s"p (s) denotes the shaping function. To make the re to be close the case log / regularize the function by minimizing squared 1.2noomm with À RR The regularization on the shaping function eventually makes it converge to zero, but it achieves relatively stable results than regularizing [+)]2, The algorithm provides an IRL reward as rr a); ideally, the performance is invariant to a, if all the processes share the same temperature. In some benchmarks, the reward function is constrained by using the softmax activation function. As the terminal states get the highest entropy bonuses, this constraint does not exploit awareness of terminal states. In some benchmarks has the practical advantage of preventing overly pessimistic rewards when IRI. is not sufficiently trained. We defer additional implementation details to Appendix D
5.2 ANALYSES ON THE CAUSAL AIRL ALGORITHM
reguaaeere IRL. We draw a connection between entropy-regularee policy gradient al- gorithms and our method as adversarial training method for the policy network.
Proposition 5.1. If Eg. 5 is satisfied, the following equality holds.
VO DKL. (T ([)))][[((())))) = -EE 1 (8, a) log (als) + V VoH((-b)]]
(8)
where Q (s. a) a ro re(s. a) + E E (ro at) log To (a) (s)) 0, = a, T|.
The proposition again shows the strong relationship between entropy-regularized RL and AIL. Un- like deterministic RL, the policy of entropy-regulan regularized RL is proved to be converged to a unique fixed point in regularized condition (Geist et al. 2019 Yang et al.) 2019). Thus, deduce that the learning scheme of CAIRL leads the policy to be converged to the fixed point of the expert.
Application to Transfer Learning. Suppose that the expert resides in another MDP: Me (S, A, PE, r, Po, 7). Extending our formulation, we can induce that the function (a, (s, o, 8') would converge to log( Pels' s.o). ==(al=)), ). and the expectation of the function with the KL penalty gives P(s)(s.a)
a
E (a (8, 8') (TT) (-)) -DI. [Pr, (a, s' [>)][ppaa, [s)]
(9)
where Pr, (a, s'ss) and Pa(a, s'(s) denote the agent's and the expert's conditional probability dis- tributions of action and transition state, and learning with ra(s, also promotes the identical effect. If the distributions ño and PE are different, the optimal behavior has to adapt to the gap between domains. The energy-based reward function ra(s,a) is robust dynamics mis- alignment as it learns the conditional joint distribution of actions and transition states, namely DKL [Pr(oo. $1.... |sg s, x) Pr(ao, $1.... Sg s, TE)].
Objective of Potential Networks. The reward shaping can be viewed as a value iteration scheme. We relate Eq. 5 to an entropy-regularized operator called mellowmax defined as mm)(X) log(# ( > exp(aar))) /, and show that re and h2 satisfy the mellowmax Bellman optimality. Proposition 5.2. If fo.s 10,v6 (s, a, s') = log = //r all states, actions, and transition states, then
h (+) = log u / exp(ro(s, + "BBlls. (s'))) da
Thus by corresponds to the mellowmax optimal value function of ra with 1.
According to analyses Asadi & Littman| 2017), the standard softmax operator may lead the value learning to multiple fixed-points when 7 ss I. As the mellowmax is a non-expansion operator, the use of the KL penalties achieves relatively stable potential function optimization.
EXPERIMENTAL RESULTS
Our experiments aim understand CAIRL and verify the effectiveness of the algorithm in the sense of our claims. We evaluate our approach on three topics, whose settings are motivated by the previ- ous works (Haarnoja et al. 2017 2018 Fu et al.] 2017). For the RL algorithms, we implemented an algorithm based on OpenAI PPO2 (Schulman et al. 2017) and use the KL regularization order to eliminate the survival biases in MaxEnt RI. addressed in Sec. 4.

---

Under review as a conference paper at ICLR 2021
CAIRL
CAIRL
AlRL-Abeerringg State
CAIRL
AIRL
Figure 2: IRL trajectories of 2D Multi-goal environments.
Figure 3: Ant robot trajectories,
Hopper-v3, 4 expert trajectories
Ant-v3, expert trajectorics
Wolker2ddv3, 4 expert trajectories
A/RL CAIRL
1.0
CAIRL Blinert
GALL
0.0
0.4
0.2
HollCheetah--33 expert trajectories
Humansid-43, expert trajectories
Humanoid-43, 100 expert saperttrajjece trajectories
0.4
0.5
0.1
0.4
0.4
0.2
0.2
1.4
0.0
2334
3338
30H
20N
30H
Figure 4: Training curves of stochastic policies imitation learning benchmarks.
The first experiment setting is a multi-goal environment. From the initial agent position, four goals are located at the four cardinal directions. While deterministic policy commits to a single goal at the earliest attempt, we hypothesize that the optimal MaxEnt policy distribution represents multi-modal behavior, which is reaching all the four goals the same rate. We evaluated algorithms on two settings. In the 2D setting, the agent is point mass. The ground- truth reward function defined as the difference between Gaussian mixture model values points: GMM(X,+ GMM(X) where x, is 2D representation of state. In the 3D setting, the agent is simulated robot where its state defined by the position and its joint values. The detailed setting and expert of each of the environments are provided in Appendix C.1
Multi-Modal Behavior.
Fig. 2 visualizes trajectories obtained by the IRL al- gorithms. The symmetric multi-goal environment ex- permient (Ieft) demonstrates that CAIRL is capable of recovering multi-mode policy resulting to reach all goals. For evaluating survival bias handling. we imple- mented similar task, called asymmetric multi-goal environment, where the right side goal is located sub- stantially further than others (right). Table [ displays an ablation study that shows episode time steps and ra- tios that IRL agent reach the goal in the right-side, The result shows that CAIRL can induce more uniform multi-modal distribution than AIRL variants and that the algorithm is robust to the survival bias. Especially, the CAIRL algorithh the both activation functions achieved high performance; the difference between two cases not significant. Fig, 3 shows result that
Table 2: Statistics of CAIRL and AIRL in the asymmetric multi-goal environment.
Ep. Timestep
Right
Expert
42,61 18851
0.25
AIRL
70.11 #80.3
0.24
Aabsobb
87.22 108.0
0.33
AIRL+softplus
52.33 1.5..08
0.31
CAIRL+no KI.
40.9 119.3
0.23
CAIRL+linear
41.07
117.47
0.24
CAIRL+sofiplus
42.05 # 19.46
0.25
CAIRL outperforms AIRL in the quality of generated trajectories. CAIRL searched all the four goals with the robot, which prominently shows that algorithm can reconstruct rewards from multi-modal policies in the complex control tasks.
Imitation Learning. The second experiment is the imitation learning tasks of continuous domains from the Gym benchmark suite (Brockman et al., 2016). We evaluated our algorithm on five chal- lenging tasks, including Humanoid benchmark with 21 action dimensions.

---

Under review as a conference paper at ICLR 2021
Source Ern.
Target Erw.
Hopper-v3 Slopettopper
Walker2d-v3 SopeWalker2d
Ant-v3 - CrippledAnt
AIRL FAIRI GAIL
AIRL FAIRL GAIL
AIRL CAIRL FAIL
Derwnee
26
16
Deggee os slop
Degres shopp
Range motion 16)
53
Figure 5: Illustrations of transfer learning and total distance traveled by transfer learning agents.
In Fig. 4 training curves are visualized with the results of five individual runs where the scores are rescaled with respect to the expert score, and the shaded regions represent the minimum and max- imum scores. Since expert policies are uni-modal Gaussian policies, we restricted the number of trajectories to 4 except the Humanoid task. In the figure, CAIRL algorithms (blue lines) shows the fast learning speed in the early phase of training and at least the second-best results for all provided experiments. CAIRL clearly outperforms previous methods in the Hopper-v3, Walker2d-v3, and the Humanoid-v3 tasks. These tasks are considered be challenging due the harsh termination condition. We highlight that CAIRL with linear activation (purple lines) also shows competitive per- formance in the conducted experiments. our experiments, FAIRL showed the lowest performance. We suspect that the FAIRL reward formulation r(s, a) = exp(d(s, a)) (-d(s, a)) sometimes give too much penalty for a single step reward, even when the proposed clipping method is applied.
Transfer Learning. The third experiment is the transfer learning tasks. The setup is inspired by Fu jt al. (2017). but the settings are quite different. We trained each IRL. network in the target (test) environment. To simply put, our experiment aims measure the flexibility of the reward learning process. The formulation was designed to show that knowledge transfer requires adaptation. We wanted to find out whether each algorithm provides helpful rewards without pretraining. Also, it is natural to think that only a chunk of trajectories is available in a realistic problem. We implemented three transfer learning tasks called Sloperopper, Slope Walker2d, and CrippledAnt. In the Slope- Hopper and SlopeWalker tasks, the agent has the same configuration with the original 3D models but the ground is tilted by certain degree ranges to [1, 15). In the CrippledAnt task, the robot has noticeably shorter forelegs as colored red in Fig. 5 We additionally restricted the joint angles of forelegs in the range of 50,,7,5,. compared to the original model.
Given the expert trajectories from the source environments, the results of transfer learning tasks are shown in Fig. For each task, we repeated all runs 5 times and report the results which are averaged over scores from the last 0.5 million training steps. In transfer learning setting, CAIRL outperformed other algorithms considerably that achieved the highest performance for every ex- periment. The results imply that the algorithm extracted informative rewards for new environments with different dynamics, such that our reward acquisition methods are robust with the variation be- tween tasks. Other algorithms failed considerably; it empirically validate one of our hypothesis that the cumulative sum of state occupancy measures hiders robust learning when domain shift happens. The experiments have verified that with the proper consideration of temporal dependencies, the AIL algorithms could be extended towards transfer learning problems.
7 CONCLUSION
In this paper. we have proposed causal AIRL algorithm that recovers a robust reward function. We have provided theoretical analyses, including reward shaping in intropy-regularized MDPs, and the connection between adversarial learning and energy-based RI. We have proposed a novel dual discriminator architecture, which learns reward and a value function of regularized Bellman opti- mality equations. Our model can efficiently disentangle biases originated from state occupancy and terminal states. We have verified that the proposed IRL method has clear advantages over AIRL for learning multi-modal behaviors and handling termination biases. The proposed method recovers cante-conditonee rewards, which has advantages of imitation performance in challenging continuous domains. Furthermore, the proposed method outperformed other methods in domain adaptation in the transfer learning experiments.
AIL algorithms in terms of the robustness
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. ACM, 2004.
Kavosh Asadi and Michael Littman. An alternative softmax operator for reinforcement learning- In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 243- 252. JMLR. org. 2017.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrim- inator rejection sampling. arXiv preprint arXiv:1810.06758 2018.
Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L Littman. Apprenticeship learning about multiple intentions. In Proceedings of the 28th International Conference on Ma- chine Learning (ICML-11), pp. 897-904, 2011.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.06060 2020.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wor Ha. Sunghun Kim. and Jacgul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
LI Chongxuan, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In Advances in neural information processing systems, pp. 4088-4098, 2017.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning; Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016b.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint 1rXiv:1710.11244 2017.
Zhe Gan, Liqun Chen, Weiyao Wang, Yuchen Pu, Yizhe Zhang, Hao Liu, Chunyuan Li, and Lawrence Carin. Triangle generative adversarial networks. Advances in neural information processing systems, pp. 5247-5256. 2017.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 2160-2169, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Scyed Kamyar Scyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pp. 1259-1277. PMLR, 2020.
Tuomas Haarnoja. Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1352-1361. JMLR. org, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1856-1865, 2018
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in NIPS, pp. 4563-4573, 2016.

---

Under review as a conference paper at ICLR 2021
Iya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-aactr--iie Addressing sample inefficiency and reward bias in adversarial imitation learning, arXiv preprint arXiv: 2018.
Gerhard Kramer. Directed information for channels with feedback. PhD thesis, ETH Zurich, 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. 2006.
Sergey Levine and Vladlen Koltun. Continuous inverse optimal control with locally optimal exam- ples. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp. 475-482, Madison, WI, USA, 2012. Omnipress. ISBN 978 97814500 14503 1851.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning- arXiv preprint 2XXv:20044093955, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 518(7540):529-533 2015.
Gergely Neu and Csaba Szepesvári. Apprenticeship learning using inverse reinforcement learning and gradient methods. arXiv preprint arXiv: 1206.52200..266 2012.
Andrew Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learming, pp. 663-670, Morgan Kaufmann Publishers Inc., 2000.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In ICML, volume 99, pp. 278-287, 1999.
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. arXiv preprint 17Xi:: 2017.
Dean A Pomerleau.
Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, 3(1):88-97, 1991.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of IJCAI, pp. 2586-2591 2007.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Wenjie Shi, Shiji Song, and Cheng Wu. Soft policy gradient method for maximum entropy deep reinforcement learning. arXiv preprint wwXiv:1909003990 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear pro- gramming. In Proceedings of the 25th international conference on Machine learning, pp. 1032- 1039. ACM, 2008.
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. arXiv preprint 1000011802.1003,, 2018
Wenhao Yang, Xiang Li, and Zhihua Zhang. A regularized approach to sparse optimal policy in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 5938-5948, 2019.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality telcoperation. In 2018 IEEE International Conference Roboties and Automation (ICRA), pp. 1-8. IEEE, 2018.
10

---

Under review as a conference paper at ICLR 2021
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint
2016.
Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. In International Conference on Machine Learning, pp. 7584-7593, 2019.
Brian D Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010.
Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 3:1433-1438, 2008
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 1255-1262. Omnipress, 2010.

---

Under review as a conference paper at ICLR 2021
A PROOFS
Proposition 411, Let r be a function satisfying Eq. 4.. Then the expected cumulative reward of
has following property: Es ((s, H(a(+15)) = - (T) + Eoswo (s)].
Proof: (s)(s) denotes the state-only occuany measure of state
E, (r(, + H(T(-11)) E
(r((,, log ( (s))
Mlog ((as<1((a))s)) - log ((tl)))))))))))))))))))))))))( - (8((8++1) + ((())))
Ylog ( ((cl(s))
2
0
1 ((1+1)) Σ (s))
xg(0r(s)) (log T(aels))
(s:) + Σ (
E
1=0
Tg(ar(s))
"log T(ae(s))
+ 1 (80))
(3( a) [og nE(a|H}] da ds + / (s))) ds
T(als)
(r(s) / a(als) [og [a(a)))] da ds + Esupo [s)]
T(als)
Pa (O))) ds (Ea-)o(V(()))
s
Pa(()
da ds + E (s)]
E, [KL (T(||s]||TT((())))) + Earpo (s))
Therefore, the reward function r(s, a) also provides the same global objective the likelihood log T
Proposition
5.1 If all critic functions are optimal, the following equality holds.
BT DKI. (To (-))))[]] -E,, (0== (als) + V v.)()-)))
where Q (5 a) e ra(s +E ( a:) - log n (a (s)) 5, a, T]
Proof. By the policy gradient theorem Sutton & Barto 2018). we can derive the gradient of o when initial state is
E DKa. To [T)[[())))]]] = =s = (E [ro(s, a) log (a((ss))s = s] + const.)
V
(re(se) log To ( so
(s)
where (T(()) 2 E (ro log (ar (sr)) |s0 T. By the product rule, we get
V= (s) (0 log [ (Q" a) log (o(ls)) da (((as))) / a) (s)) ds da
12

---

Under review as a conference paper at ICLR 2021
By repeatedly unrolling V V (si)). we can derive the following form:
(3) 7 Pr(s + 2, mo) / (Vors(ala))" 3 da dr
where Pr(s- + x. k. æ) the probability of transitioning from state s to state x steps under policy Thus, using the log derivative trick can derive the rest of equations as follows
V Br DKL Te (+s)||[=((1s)]] = Po(s) Vo(s)V E. (
|so ds
1
Prr(s: = s/mr / (VoTo (8. (a(a)) da ds =0
/ Pr)(s) / (s
da ds
/ PFo PT /
voH((-1s) da ds ET 10 (s, log VoH((ss)
Proposition 5.2
fe (s, a, s') = log uulals)/uufr all states, actions, and transition states, then
(s) = log u / exp(ro(s,a) a) + BB (s']] (Mp da
Thus by corresponds to the optimal value function of the mellowmax regularizer with 1.
Proof.
ro(s,a) + - = log ((als log AS' E S
a)) + +1 (s) = log (als) - log
= uexp(re(s, a) + 7 hhy)) = (als)
exp(ro(s, a) + 7T Mhp (8')) (a)sss
(x)(
(10)
exp(ro(s, a) + t (he da / TE(a|s) da (x)(h
u
(s) log u. / expfro(s, + Ee (he [ da
B DISCUSSIONS WITH MAXENT IRL METHODS
In this section, we address similarities between frameworks the maximum causal entropy IRL and the causal AIRL.
B. MAXIMUM CAUSAL ENTROPY FRAMEWORK
Following the notation of Kramer 1998), the causal entropy can be defined as
H(A T ) A EA.sl- log P(A P(ATISS)) = Σ H(A,S Sta, A1a-1)
13

---

Under review as a conference paper at ICLR 2021
The objective of maximum causal entropy IRL can be formulated by the following optimization Ziebart et al. (2010):
arg max H(A (1S") P(A.1S))
such that: (S) [F(S, A)] = B)(S A)]
Vss
where the overall dynamics P(S
1 T- I P( P are given.
According Ziebart al. 2010)) the distribution satisfying the constrained optimization problem can be defined as a recurrence relation:
po(ae)s))= = (())
3(st)
log (5t, 01) = 00 Ferna + Σ p(s ar) log 3(s++1)
91-1
log (st) = logg Σ B(s,, = softmax log (ss,, as)
With the perspective of energy-based RL, we draw the following implications from the solution:
Pa (a()()))) can be understood as the unique fixed point of MaxEnt framework, which can be achieved by the optimization via soft RL algorithms.
ro(st, a,) = 0 Fe..o. can be understood as a reward function which is subject to the linear constraints on feature function.
log ((t,, az) and log (((e)) can be understood the (optimal) soft values satisfying the Bellman equation.
B.2 VANILLA AIRL REWARD FUNCTION DOES NOT FORMULATE AN EBM
Adversarial Inverse Reinforcement Learning (AIRL) (uu et 2017 is a well-known IRL method that applies an adversarial architecture to solve the IRL problem. Formally, AIRL constructs the discriminator as
exp(s(s,a))
D(s, expif(s,a) a)) + (als)
(11)
This is highly motivated by the former GAN-GCL work (Finn al. 2016b), which proposes that one can apply GAN to train the discriminator as
D(T) = 1 exp(c(T)) z exp(c(r)) + m(7)"
(12)
where c(7) denotes the function for trajectories. From the AIL formulation, AIRL uses surrogate reward as
r(s, a) = log D(s, log(1 D(s (s, log (a|s),
(13)
where f(s, a) log(p (s)) and the overall reward can be seen
entropy-regularic
reward function. 7 = 0, evident that AIRL is identical to the standard adversarial learning without temporal sequences. As result, it makes sense to directly optimize the policy by taking the energy model as the target policy instead of the reward function, which leads to the optimal solution as:
exp(Q" x" (a(ss) E.. exp(("(ss, E. exp(f(s.a')) En PTo a8,
exp(f(s,a))
Pro (sn(s,a) a)
7
By the fundamental property of occupancy measure (Theorem 2 of Syed al. 2008). However, for the general case, the projection of Q-value function cannot be close to my like the standard probabilitic inference using energy-based models.
14

---

Under review as a conference paper at ICLR 2021
B.3 CAUSAL ADVERSARIAL INVERSE REINFORCEMENT LEARNING FORMULATION
As a direct interpretation of the maximum causal entropy framework, the goal of CAIRL can be seen training a parameterized distribution over the expert trajectories as the following maximum- likelihood objective:
max [(0) max E pa(p)))
(12)
0
where the conditional distribution po (p,)sss is parameterized as po(aels;) x exp (Qo(s,, a.)). We can compute the gradient with respect as follows:
oy [0) = Erg
log pa (als))
.00
a
= Exg
Q0(8, a)
log Zo(s)]
(13)
00
00
a Q0 a)] - EER En En (s)) 00 2o d']]]
Ex
00
where (s) is the partition function normalizes the distribution pa- Since sampling with wg and Pe is difficult, think of substituting the formulation with the following adversarial learning form:
O (s a) Ex 100 (s.0)) 30
E ExS
(14)
Nevertheless the formulation in Eq. (14) is similar with standard adversarial framework with ii.d data, the approximation is not a safe choice because of unbounded divergences among Po, and Tæ- Therefore, appropriate adversarial learning in sequential decision problems is essentially different
to GANs for joint distribution matching.
In CAIRL, we replace the reward learning objective with training a logistic discriminator:
explfo(s.a)
Dø(s,a) explfe(s. + x(s,a)
where k<ss a) Shannon divergence between of the generated samples:
Ps (x)- The objective of the discriminator maximize the generalized Jensen-
J(Do) Erg [log Do(s, a)] + Do(s,)))))))
For the discriminators, the objective can written as follows:
J(Do) = Erg log Do(s, a)] Fr (log(1 Do(s, a))]
= Ens [ogg explfo(s,.) a)]
explfo a + k(ss , E. [Log explfo a)) x(s,a))
k(8,a)
= Eze [fo(s, a)] + E. [m(s, - 2 Ex [log { explfo(s, a)) + m(s.a))]
where the operator E. denote the expectation using an occupancy measure p=(8,a) = (s..)) Taking the derivative with respect to 6,
0 J(Db) Eng
② fo(s, a)
explfe(s,a) a
2 E.
exp[]f(s, k(s, 20 fo(s,a))
20
00
i fo(s a)
pe(s)po(als)
Enr
Er
00 fo(s.a)
100
1 {P (
(s
Emg
fo a)
- EE
(a(a'ss) 6(s,a') 30 fo(s,aa))
a'EAA
where
(o(as) exp(fo(s,a) and 8(s,a) Pr(s, a) + (x,
( (s.
The optimal can be easily found by considering @ J(2)) = 0, in condition of po (a]s)) (als) with 0(s, a) = 1 for all states and actions. Moreover, for the right hand side of the expression, we can draw the following properties of å:
de
15

---

Under review as a conference paper at ICLR 2021
If (s) > pE(s, a), meaning the supports disjoint, d(s,a) 1.
If (s) = p a), where Po (a]s) is normally closer to mæ than π,, 8(8, a) x
Therefore, the expression matches the Eq. 13 CAIRL approximates the maximum causal entropy framework without exhaustively computing the recursive equation, and also recovers a plausible reward function on both supports of Px and Pg-
C EXPERIMENTAL DETAILS
C.I MUTL-GOAL ENVIRONMENTS
2D Point Environment: Let the 2D coordinate denote the position of point mass on the environment. The agent is generated according to the normal distribution (o,, (0.1) The four goals are located (6, 0), (-6, 0). (0, 6), and (0, -)), where the agent can move maximum 1 unit per timestep for each coordinate. The ground-truth reward is given by the difference between successive values of a Gaussian mixture depicted as Fig. 6 The assymetric multi-goal environment has similar settings, except the scale is five times bigger goal of the east side further located at (60,0) 0).
3D Ant Environment: Let the 2D coordinate representation denotes the orthogonal projec- tion of the position of Ant robot torso. The simulated robot is spawned near the origin. The four goals are located at (30, 0), (-30, 0). (0, 30), and (0, (30), where the agent has to control four legs to reach one of the goals. It requires approximately 1550 timesteps for an expert to reach one of the goals from the initial position. and the robot's joint values represent the state. Since it is hard to train single expert model to represent the desired multi-modal behavior precisely, we evenly merged 2,000 trajectory samples from 8 uni-modal policies specialized in moving to one of the fixed positions.
vector of the current position
Figure 6: Illustrations of multi-goal environments. From the initial agent position, four goals are symmetrically located the four cardinal directions. The ground-truth energy function of the 2D environment and the expert trajectory samples for each environment are displayed.
C.2 TRANSFER LEARNING ENVIRONMENTS
The experiment setting was designed to show that imitation learning for realistic tasks regardless of domain shifts. We trained each IRL network in the target (test) environment. To simply put, our experiment aims to measure the flexibility of reward learning process. We used 1,000 trajectories from the source task transfer learning data.
SlopeHopper SlkpeWalker2d: We used the same 3D model from the Hopper-v3 and Walker2d-v3 Mujoco benchmarks. For each task, the ground is tilted with certain de- gree in [1, 15). For computing a state vector, the models' height is adjusted to the vertical distance from the slope.
Compared to the original Ant model, we shortened the length of two forelegs to half. We additionally restricted the joint angles of forward legs in the range (001,255.50.7551100 While the objective the agent is for running to the right side, the reward can be calculated by the velocity of robot moving along the x-axis.
CrippledAnt:
16

---

Under review as a conference paper at ICLR 2021
D IMPLEMENTATION DETAILS
D.1 ALGORITHMS
Algorithm summarizes the overall IRL. procedures. The term "REG" in Line of the algorithm refers to the shaping regularization minimizing squared L2-norm ( - h(s) with the reg- ularization parameter À E R+ in order to make the overall algorithm well-defined. For stochastic action distributions, the temperature parameter a has to multiplied for exact matching between conditional distributions. For ensuring Lipschitz continuity, every critic network is regularized by gradient penalty (Zhou et al. 2019).
Algorithm Causal adversarial inverse reinforcement learning.
1: Input: Expert trajectory dataset {Tg i=1' temperature a.
2: Initialize policy n6 and critic functions Dip, and D8
3: for step in {1...., N] do
4: Collect Tx by executing T
5: (())) = Ta, (5 āz. 3,, = (1)
6: Update Dup using +
+
- [lo(1 D2((s)) + log (5r) with GP.
T
7: Update Do,s using + Ee
[log(1-14) Da at, s()) + log (3, 3() with REG+GP.
1=1
8: Update with a T0 a) using maximum entropy policy optimization method.
9:
return Te. 00
For policy optimization in the experiments we used PPO implementation train policy by clipped surrogate objective function as follows:
(a(a))) Agal (s, (als(als)
TT(als) 1 c.1 A (s, (8, (ala)als))
CLIP =
ET..
min
(s, a), clip
(15)
where is the clipping range of PPO algorithm, ensuring that the updated policy does not diverge too far from the previous distribution. One thing to be careful the implementation is that the KI. penalty terms are dependent the policy distribution. Therefore for a PPO algorithm with the KI. regularization, the term A a) can be computed as follows:
Thlld (s) = E
(ro(st,a:) a aloog* ko((a:b.///))000 Tota]:
(16)
Quold (s, = ro(0, a) + E
(roo(se, an) - a log (o (o)/))))//)) = 8, ag = a, Total
(17)
Fold (8,a) a) = Tuld (s, V (s),
(Advantage estimate)
(18)
Aoll (8, a) A Told (8,u)
(19)
We refer the work of (Sii al. (2019) for the detailed derivation of entropy-regularized policy gra- dient algorithms.
D.2 NETWORK ARCHITECTURES AND HYPERPARAMETERS
For all policy and discriminator networks, we use networks with 2-layer MLP with 256-dim layers, and the activation function is ReLU. For input layer, we normalize inputs (state and action vectors) by calculating exponential moving average and variance.
Details for Policy Networks For multi-goal environments, the policy function is represented by Gaussian mixture with four modes. In contrast, a single diagonal Gaussian function is used for other unimodal tasks. For enforcing action bounds, we apply invertible squashing function (lanh) to the raw action samples and compute the likelihoods of the bounded actions such as tanh (Normal r)) (Harrooj al., 2018). In our implementation, network and standard deviation network. The gradient penalty regularization is applied to the value network.
separated the distribution network into mean
17

---

Under review as a conference paper at ICLR 2021
Details for CAIRL Networks For networks, the reward network consists softplus activation at the final output layer while the potential network has linear activation. Gradient penalty is also used in the both networks. We do not explicitly construct the target potential networks, instead we use the same potential network, but the target potential network does not involved any gradient computation. Also, to discard the learning of terminal state values, hg is outputs follows:
0
if is terminal state not caused by time limits, otherwise
(+)()) no grad (hp(s)))
Table shows the hyperparameters of conducted experiments.
Table 3: Hyperparameters.
(a) Shared parameters
(b) Environment specific parameters
Parameter
Value
Environment
# of threads
Temp. (a)
Optimizer
Adam(o, 0.99)
Multi-goal (Point)
1
learning (policy)
1 1-104
Multi-goal (Ant)
16
10-2
Learning rate (diseriminator)
4-10
Humanoid-v3
8
10
Discount factor (n)
0.99
Walker2d-v3
3
10
Clipping Range (=)
0.2
Others
1
10-2
Gradient penalty (1)
10
Shaping regularization (A)
10-4
Batch size per rollout
2048
Policy rollouts Discriminator rollouts
Gradient steps iteration
15
E ADDITIONAL EXPERIMENTAL RESULTS
E. ADDITIONAL BENCHMARK RESULTS
Table 4: The evaluation imitation learning benchmark control tasks presented over different random seeds with 4 and 100 expert trajectory data.
Environment
Method
Hopper-v3
HalfCheetah-v3
Walker2d-v3
Ant-v3
Humanoid-v3
Expert
3688.9
3580.3
5297.0
4030.8 2
9049.0+3114
GAIL
3227.6 6
3431. 94:251
2369.8 +507
4090.8.c
88:225
8816.5 55044
AIRL 3229.8
3127.6 296
3326.7 7 11331
3956.51.00
538.8 538.8
FAIRL
374.1 3300
1147.4.13a0
281 4 116
1501 1371
351.9 L30
CAIRL
3565.4 133
3163.5122
5089.6 619.6.3339
3990. .9 254
9644.6 6 1385
GAIL
3513.1 1:273
3493. 947279
4392.4
4093. 2
9199. 4443
AIRL
3349.04311
3482.6. 3482.6.1202
5165. 7
4027.0 .0 2:82
$532.54720
FAIRL
236. 2
744.2+249
081.441116
2106.0488
375.2422
CAIRL
3571.6+309
3246.44211
$268.14100
2999.64229
9469. 5
E.2 VISUALIZATION OF TRAJECTORIES GENERATED BY IMITATION LEARNING AGENTS
Fig. 7
shows the trajectories from the trained models in the 3D environment with different seeds. Firstly, the result suggest that CAIRL recovers multi-modal policies which is equivalent to the prop- erty MaxEnt IRL. At the same time, CAIRL is fundamentally advanced algorithm compared to MaxEnt IRL algorithms because it can be applied various large domain without knowing dynamics by by enlarging number of parameters in neural networks.
18

---

Under review as a conference paper at ICLR 2021
CAIRL
CAIRL #2
CAIRL #2
Figure 7: Multi-goal Ant trajectories with different random seeds
E.3 ABLATION STUDY ON HYPERPARAMETERS
We provide experiments ablation study three controllable hyperparameters in Fig.
A3t.v3
Hopper v3
Hopper v3
mp1
005
J-10.0
n-10
u-2.11
Fpm 10.
0.2
0.2
0.2
2.391
1.010
2.5M
3.31
5.3M
1.00
1.5M
0.5N
1.3M
1.5%
2531
(a)
(b)
(c)
Figure 8: Experiments hyperparameters of CAIRL. (a) If the shaping regularization À is ex- cessively high, learning of the potential-based shaping disabled, and the reward function shows unstable behavior. (b) The gradient penalty is applied for ensuring convergence of discriminators. The results suggest that sufficiently low gradient penalty is preferred for achieving desired perfor- mance. (c) Compared to the other hyperparameters, CAIRL is robust with the temperature a.
E.4 VISUALIZATION OF CRIPPLEDANT LOCOMOTION
provide the visualization of generated samples from a trained agent trained by CAIRL algorithm in Fig- [ Even though dynamics of the environment is considerably misaligned, the algorithm still can teach agent appropriate algorithm to teach an agent to go desired direction. We believe that this work also provided effective transfer learning algorithm for sequential decision problems.
Figure 9: Visualization of trained agent CrippledAnt environment. The model maximizes the movement speed toward the right side, without showing biased locomotion such as moving to the forward or the backward. Also, the orientation of the torso is preserved throughout the episode, implying that the algorithm is capable of imitating experts even under variation in the dynamics.
19

---

Under review as a conference paper at ICLR 2021
E.5 VISUALIZATION OF AIL REWARD FUNCTIONS
We provide the visualization of CAIRL and AIRL rewards in the 2D multi-goal environment. To effectively represent the functions that takes the argument (s,a) S A into the space S, calculated a two-dimensional vector representation of reward for each state by averaging all possible action in A, i... V, [ EeeAlr(s, a). E {0, 1}. As results, we plotted contour maps by the relative value of rewards and corresponding vector fields of reward function for a state grid, which is shown in Figs. 10 and TT] Apparently, it can be observed that the CAIRL reward function is much more analogous to the MaxEnt likelihood depicted by Haarnoja et al. 2017). More importantly, CAIRL is prominently showing that reward modeling provides more informative reward that approximately advise the best direction to reach one of the goals for each state.
3, -3)
(0.0)
8537
(3.0)
0250
Figure 10: Illustration of the CAIRL reward function in the 2D multi-goal environment. Left: recovered reward function which is averaged over action. The goals located in the four cardinal direction (6,0), (-6,0), (0.-6), (0,6). Right: visualization of cach vector field of reward function and the corresponding local contour map.
(-3, -3)
1.333
302
0353
(0.0)
(3, 0)
Figure 11: Illustration of the AIRL reward function.
20


Reviews:
[{"review_id": "MvjbbHCujnE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The line of reasoning and analysis followed in the paper is mostly sound. The paper claims that the use of (state-action) occupancy measure make IL and IRL methods brittle due to the high variance of these measures and their inability to transfer to other domains. These two claims are neither properly defined and grounded in the literature, nor are they isolated experimentally. It is important to show clearly that a) these are problems, b) hitherto unaddressed in the research landscape, and that the proposed methods and techniques address these problems specifically. \n\nThe paper also claims to address  the reward bias in imitation learning due to mis-specified termination cases. Since this claim is not tied to the previous claim, there should be clear experiments showing the advantage of proposed methods addressing each claim in isolation. This would show a) whether either of the proposed solutions can be used alone with benefits, and b) what is the individual impact of each contribution. Besides the solution claiming to address the reward bias needs to be directly compared against the method introduced in Kostrikov2019 (cited in paper). On another note, the method addressing the bias seem to suffer from a logic flaw. The paper states that in a “self-looping” terminal state, the expert policy is equivalent to a the uniform action distribution, and designs the reward (unrelated yet to reward shaping strictly-speaking) accordingly (0 reward when the expert coincides in action distribution with the uniform one over actions). Since, according to Figure 1, such a KL penalty seem to be applied at every transition, it is not difficult to imagine scenarios (highly multi-modal expert, expert in exploratory phase, etc.) where the designed reward will be zero in non-terminal state due to a close-to-uniform behavior displayed by the expert. The designed reward will then fatally fail to allow the policy to imitate such expert properly.\n\nAt the end of page 4, the paper states: “Compared to the AIRL, […] a reward function r [proposed in the paper] can be universally applied for training arbitrary policies since the agent-specific occupancy measure […] is detached.”. In the absence of empirical or theoretical results showcasing the flaws of learning a metric such as the one in AIRL, the claim remains unclear and unjustified. It must be made clear why the reward formulated by AIRL is flawed. Using results from the “state-only” adversarial imitation learning literature might contribute positively to such a ustification and will echo the first claim, however such results are not mentioned nor cited in the paper. Besides, since AIRL uses state-action occupancy measures in its reward design while the proposed reward involves the policy’s action probability directly, the proposed approach is close in essence to behavioral cloning. It differs from BC as it also involves reward shaping, as introduced in Ng1999 (cited in paper). The similarities in reward formulation appears clearly in the table from Ghasemipour2020 (cited in paper), from which Table 1 is heavily inspired. The BC line should be kept to highlight how similar the formulations are. Moreover, while the introduction of potential-based shaping is justified in AIRL, it is not motivated here, and it remains unclear why it was introduced.\n\nIt should be made clearer in the text that the method builds on AIRL, considering it involves of the functional elements introduced in AIRL (shaping, EBM formulation, MaxEnt IRL, derivations, etc.).\n\nAbout the experiments that are presented: the results of CAIRL are encouraging. As said earlier, it is difficult to disentangle the contribution of each component without ablation of the introduced techniques. It would also be interesting to evaluate the proposed IRL method on IRL tasks (reward recovery, as in AIRL), not only on IL tasks (perform well in environment). \nThe transfer learning experiments show high robustness, which (probably, no ablation) show how also using state occupancy measures on top of the rest of the usual AIL machinery makes the agent transfer well to environment sharing the same state spaces, and differing only by dynamics. This observation directly echoes the main result of AIRL, which is that rewards robust to dynamics changes are only function of the state (not action). Discussing this result is a piece that is missing from the submission unfortunately.\n\nThe paper could use a good editing to address language errors.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ERAiWU5FQC_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds on a recent inverse-RL method, AIRL. The authors argue that the rewards learned by AIRL are potentially inefficient since they depend on the ratio of state-action visitation distributions of the expert and the policy. To resolve this, CAIRL derives rewards that excludes these visitation distributions; this is realized in practice by employing another discriminator to approximate the state-visitation distribution ratio. The paper further proposes a mechanism to handle the reward-bias issue in IRL. Experiments with the MuJoCo locomotion tasks show that CAIRL is a competitive algorithm for imitation learning and can also handle dynamics mismatch between the expert and the learner.\n\nI would like the authors to address the following concerns:\n\n1.\tBetter motivation: It is argued that since AIRL rewards depend on the ratio of state-action visitations, it suffers due to the “high variance” of this ratio. Is there an analytic argument for this, or prior work to support the claim? Note that the density ratio is implicitly obtained with adversarial training, which has found empirical success across many ML domains. Moreover, CAIRL requires estimation of a similar ratio (of state-visitation densities), and this estimation feeds into the IRL reward computation. So why would CAIRL not be plagued with the same “high variance” issue?\n\n2.\tReward-bias handling: The paper needs to provide more empirical evidence that the proposed reward-bias handling method is useful. I find the episode timesteps in Table 2 to be insufficient (especially with the variance in those numbers). The matter is made quite confusing by using a softplus on the IRL rewards – this manually enforces a positive bias on the IRL rewards. These are coupled with the negative rewards from the KL-penalty, and “alpha” now becomes a tunable knob to manage the overall reward-bias (as evident by “alpha” values in Table 3.). Note that DAC, which appears to be a much more principled way of handling the reward bias, does not add a modifier function (e.g. softplus) on the IRL rewards. I would encourage the authors to add more tasks where survival bias is detrimental (e.g. lunar-lander, etc.), and also comment on the role of softplus on their IRL rewards.\n\n3.\tTransfer-learning experiments: how was the algorithm (Algo 1.) changed for these experiments? The main section mentions the need for the transition-dynamics ratio in the IRL rewards. How do you obtain this ratio for these experiments? \n\n4.\tThere is an L2-norm penalty added to the loss function (termed as REG). The paper says it is required to “make the algorithm well-defined”. Could you expand on this statement? Also, it would be interesting to do an ablation on this to measure its contribution to the overall performance of CAIRL. Connected question – are the baselines regularized with the REG and GP losses?", "extra_scores": {}, "extra_text": {}}, {"review_id": "4IWr-zl98X", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n========\nThe paper proposes several modifications to adversarial inverse reinforcement learning (AIRL) that aim to improve transfer to different environments, to address termination bias, and to learn more multimodal policies. The most notable change is probably the introduction of an additional discriminator, $D_\\phi(s)$, that is trained on the state marginal. By adding the logits of the state-discriminator to the logits of the state-action(-nextState)-discriminator, $D_{\\theta,\\psi}(s,a,s')$, the latter supposedly estimates the expert's advantage function better. Apart from adding a state-discriminator (1), I could identify the following modifications to AIRL: (2) adding a constant offset to the optimization problem given by the log-density of a uniform action distribution, (3) enforcing positive rewards through a soft-plus, (4) disregarding the gradient of the next states Value $h_{\\bar{\\psi}}(s')$ when updating the discriminator and (5) penalizing the temporal value differences $||\\gamma h_{\\bar{\\psi}}(s') - h(\\psi)||_{2}^2$ via regularization.\nThe resulting algorithm, CAIRL, is evaluated (a) on a 2D and 3D toy problem for testing the multi-modality of the learned policy (b) on MuJoCo locomotion tasks to evaluate imitation learning performance, and (c) on MuJoCo tasks with modified dynamics to test transferability. The transfer setting is different from the setting used by AIRL because the evaluation learns the policy and reward function directly in the test domain and it is not assumed that the original domain can be simulated. Hence, the evaluation tests the robustness of the imitation learning rather than the robustness of the reward function. Especially at the transfer experiments, CAIRL performed better than AIRL.\n\nStrong points\n===========\n- The paper is well-written, tables and figures are helpful\n- Evaluations on different environments with respect to different properties (multimodality, imitation learning, transfer learning)\n- The paper comes with source code\n\nWeak Points\n===========\n- (I) I don't see the motivation for most of the modifications to AIRL. I do not see how they are supposed to improve multimodality or transferability\n- (II) The evaluations seem to be termination-biased by using positive rewards for CAIRL. Hence, the results provide little value\n- (III) the changes seem to be incremental at best (and quite possibly detrimental, esp. (3-5) )\n- (IV) Provided source code only contains demonstrations for the \"Hopper\" imitation learning experiment. There is no straightforward way to reproduce the other evaluations\n\nRecommendation\n===========\nI recommend rejection because there seems to be little logic behind the proposed modification. I don't see the theoretical basis for the claimed benefits and the empirical analysis is possibly dominated by reward bias that was introduced by restricting CAIRL to positive rewards. \n\nSupporting Arguments\n==================\nMy recommendation for rejecting is primarily due to the weak point (I) and secondarily due to the weak point (II). Hence, I will now focus on these points.\n\nI\n---\n\nThe motivations for most of the motivations are either missing or dubious. \n- __Adding a state discriminator (1):__ The paper claims that the additional discriminator improves transferability (based on Eq. 9). AIRL minimizes the reverse KL of the state-action distributions using a reward function that for an optimal discriminator is given by $r^\\star(s,a) \\approx \\log \\frac{\\rho_{E}(s)}{\\rho_{\\pi}(s)} \\log \\pi_{E}(a|s)$. CAIRL minimizes the KL between the policies using a reward function that for an optimal discriminator is given by $r^\\star(s,a) \\approx \\log \\pi_{E}(a|s)$. Intuitively, the term $\\log \\frac{\\rho_{E}(s)}{\\rho_{\\pi}(s)}$ that was dropped by CAIRL, seems to be very useful by pushing the policy towards the demonstrated states and away from those states that are not encountered by the expert. This additional reward to bring the policy close to demonstrated states seems to be a major advantage of AIRL compared to direct behavioral cloning that is known to suffer from covariate shift due to compounding policy errors [1]. It is not clear to me why removing this term improves imitation learning when learning under different dynamics. Based on Eq. 9 the expected unshaped cost corresponds to the KL $D_{\\text{KL}}\\left[ p_{\\pi_\\phi}(a,s'|s) || p_{E}(a,s'|s) \\right]$ and thus minimizes the divergence between these distributions that are obtained under different dynamics. However, wouldn't AIRL analogously be penalized based on the KL $D_{\\text{KL}}\\left[ p_{\\pi_\\phi}(a,s',s) || p_{E}(a,s',s) \\right]$? Why is a policy that puts less emphasis on matching the demonstrated states expected to perform better in a different environment? I would argue that it is more likely that good states remain good under different dynamics than that good actions remain good, but of course, it always depends on the concrete setting. \n\n- __Adding a constant term to the objective (2)__: I am fine with modification (2) since I see how the constant offset of $u = \\log \\pi_{\\text{uniform}}$ can tackle termination bias for CAIRL. However, the paper motivates this modification as a general, simpler way of tackling termination bias compared to the approach by Kostrikov et al. (2018), which is highly misleading. I think Section 4.2. needs to be rewritten.\n\n- __Enforcing positive rewards (3)__: This modification is in my opinion a crude hack that invalidates much of the evaluation based on a dubious motivation by arguing that the expected reward is positive. First of all, the fact that the expectation of a function is positive does not imply that the function itself is positive. Furthermore, the $\\ge$ sign on page 6 is wrong; it should be $E_{\\pi} \\left[ \\log \\pi_E \\right] \\le E_{\\pi} \\left[ \\log \\pi \\right]$ and not the other way around. On a side note, there is also a typo in that equation, because I suppose the log should be in front of the fraction.\n\n- __Disregarding the gradient of the next state's value (4)__: The paper argues that this is analogous to target networks in deep RL. However, (C)AIRL is not training the value net by minimizing the mean squared TD error but minimizes cross-entropy loss based on a special discriminator architecture, so I do not see the connection here.   \n\n- __Penalizing the temporal value differences (5)__: The regularizer would make sense to me if it was penalizing the TD error. However, I don't see the reason for penalizing $||\\gamma h_{\\bar{\\psi}}(s') - h(\\psi)||_{2}^2$. The paper just argues that this regularizer makes \"the overall algorithm well-defined\", without providing any reasoning.\n\nII\n----\nModification (3) adds a reward bias. Enforcing that all state-action pairs produce positive reward introduces survival bias. Entering unhealthy states in the locomotion tasks would enter an absorbing state with a hard-coded reward of 0. Hence, the agent will avoid entering such states. The survival bias might be a major reason, why CAIRL performs better than AIRL when training in a modified environment.\nI also think that evaluation not fair by not addressing reward bias for AIRL. CAIRL makes use of information about environment resets due to unhealthy states by assigning a value of zero to these states. AIRL could also make use of this information by addressing termination bias as proposed by Kostrikov et al. (2018).\n\n\n\nQuestions\n----------\n- At the bottom of page 4 you argue that compared to AIRL, the CAIRL reward can be \"applied for training arbitrary policies since the agent-specific occupancy measure $\\rho_\\pi(s,a)$ is detached\". Why can't the AIRL reward be used for training arbitrary policies?\n\n- Why is CAIRL assumed to be better at learning multi-modal policies? The connection between CAIRL and MaxEnt-IRL is based on the AIRL derivations, so I do not see the reasoning here.\n\nAdditional Feedback\n--------------------\nI think more details should be provided for the multimodal experiment. Can you plot the demonstrations as well? \n\nReferences\n----------\n[1] S. Ross and D. Bagnell.  Efficient reductions for imitation learning.  AISTATS. 2010.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was evaluated by 3 knowledgeable reviewers. All reviewers raised concerns about the motivation of the contribution of the paper. It is unclear why the use of an additional discriminator should reduce the variance of the log density ratio estimate. Also, the derivations were found to be not convincing or intuitive. These concerns have also not been alleviated after a rather extensive discussion of the reviewers with the authors. Moreover, the transfer setting to a new environment was unclear as it does not show how the  reward function transfers to new dynamics, so the transfer experiments rather evaluate how well the algorithm can imitate a policy on a different dynamics, but it does not tell that the extracted reward function is valid. While the experimental results seem promising, the authors are encouraged to improve the motivation of contribution, check which of the "incremental" contributions are very necessary and improve their evaluation on the transfer scenario.
Decision: Reject