Reparameterized Variational Divergence Minimization for Stable Imitation
Authors: Dilip Arumugam
Debadeepta Dey
Alekh Agarwal
Asli Celikyilmaz
Elnaz Nouri
Eric Horvitz
Bill Dolan
Venue:  (2020)
Keywords: Imitation Learning
Reinforcement Learning
Adversarial Learning
Learning from Demonstration

Abstract: State-of-the-art results in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior. Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes. 
Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can result in significant differences in model performance. Choices including Wasserstein distance and various $f$-divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning. Unfortunately, we find that in practice this existing imitation-learning framework for using $f$-divergences suffers from numerical instabilities stemming from the combination of function approximation and policy-gradient reinforcement learning. In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as $f$-divergence minimization before further extending the framework to handle the problem of imitation from observations only. Empirically, we demonstrate that our design choices for coupling imitation learning and $f$-divergences are critical to recovering successful imitation policies. Moreover, we find that with the appropriate choice of $f$-divergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continous-control tasks with low-dimensional observation spaces. With high-dimensional observations, we still observe a significant gap with and without action labels, offering an interesting avenue for future work.

Under review as a conference paper at ICLR 2020
REPARAMETERIZED VARIATIONAL MINIMIZATION FOR STABLE IMITATION
DIVERGENCE
Anonymous authors Paper under double-bling review
ABSTRACT
State-of-the-art results imitation learning are currently held by adversarial meth- ods that iteratively estimate the divergence between student and expert policies and then minimize this divergence bring the imitation policy closer expert behavior. Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes. Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is algorithmic design choice, and that different choices result in significant differences in model performance. Choices including Wasserstein distance and various f f-divergences have already been explored in the adversarial networks lit- erature, while more recently the latter class has been investigated for imitation learning (Ke et al. 2019). Unfortunately, we find that in practice this existing imitation-learning framework for using f-divergences suffers from numerical in- stabilities stemming from the combination of function approximation and policy- gradient reinforcement learning. In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as f-divergence mini- mization before further extending the framework to handle the problem of imita- tion from observations only. Empirically, we demonstrate that our design choices for coupling imitation learning and f-divergences are critical to recovering suc- cessful imitation policies. Moreover, we find that with the appropriate choice of f- divergence,
obtain caaattionfrro-oosseeeee baseline approaches and more closely match expert performance in continous- control tasks with low-dimensional observation spaces. With high-dimensional observations, we still observe a significant gap with and without action labels, offering interesting avenue for future work.
algorithms that outperform
INTRODUCTION
Imitation Learning (IL) (Osa et al. 2018) refers to paradigm of reinforcement learning in which the learning agent has access to an optimal, eeward-maximizing expert for the underlying environment. In most work, this access is provided through dataset of trajectories where each observed state annotated with the action prescribed by the expert policy. This is often an extremely powerful learning paradigm in contrast to standard reinforcement learning, since not all tasks of interest admit casily-specified reward functions. Additionally, not all environments are amenable to the prolonged and potentially unsafe exploration needed for mwwiii-maimiizing. agents to arrive satisfactory policies (Achiam et al. 2017 Chow et al. 2019).
While the traditional formulation of the IL problem assumes access to optimal expert action labels, the provision of such information can often be laborious (in the case of real, human expert) or incur significant financial cost (such as using elaborate instrumentation to record expert actions). Addi- tionally, this restrictive assumption removes a vast number of rich, observation-only data sources from consideration (Zhou et al.) 2018). To bypass these challenges, recent work (Liu et al.) 2018 Torabi et al.) 2018a b: Edwards et al. 2019 Sun al. 2019) has explored what is perhaps more natural problem formulation in which an agent must recover an imitation policy from a dataset con- taining only expert observation sequences. While this Imitation Learning from Observations (ILfO) setting carries tremendous potential, such as enabling an agent to learn complex tasks from watching freely available videos the Internet, it also is fraught with significant additional challenges. In

---

Under review as a conference paper at ICLR 2020
this paper, show how to incorporate recent advances in cenerative-adversatt training of deep neural networks to tackle imitation learming problems and advance the state-of oftte-of-the-nn the-art in ILfO.
With these considerations in mind, the overarching goal of this work is to enable sample-efficien imitation from expert demonstrations, both with and without the provision of expert action labels.
The rich literature Generative Adversarial Networks (Goodfellow
et al.] 2014) has expanded in cent years to include alternative for- mulations of the underlying objec- tive that yield qualitatively differ- ent solutions the saddle-point op- timization problem (Li et al. 2015
Art-12
2300
Retum
Dziugaite al., 2015; Zhao et al. 2016 Nowozin et al. [2016] Ar-
jovsky et al., 2017; Guirajani et al.
2017). Of notable interest are the findings of Nowozin et al. (2016)
who present Variational Divergence Minimization (VDM), a generaliza- tion of the generative-adversarii ap- proach to arbitrary choices of dis- tance measures between probability distributions drawn from the class of f-divergences (Ali & Silvey] 1966] Csiszár et al.) (2004). Applying VDM with varying choices of f- divergence, Nowozin et al. (2016) encounter learned synthetic distribu-
309
200
200
Figure Evaluating the original f-VIM framework (and its ILfO counterpart, f-VIMO) in the Ant (S E "I" and Hopper (S ") domains with the Total Variation dis- tance. E-VIM/VIMO-signaen the frameworks, detailed Sections 4.2 a Note that, in both plots, the lines for TV-VIM and TV-VIMO overlap.
denotes our instantiation of
tions that can exhibit differences from one another while producing equally realistic samples. Trans- lating this idea for imitation is complicated by the fact that the optimization of the generator occurs via policy-gradien sarial IL. and f-divergences (Ke et al.] 2019).. despite being well-motivated, fails to account for this difference; the end results (shown partially in Figure [ where TV-VIM is the method of Ke et al. 2019), and discussed further in later sections) are imitation-learning algorithms that scale poorly to environments with nigher-dimensione observations.
reinforcement learning (Sutton al. 2000). Existing work in combining adver-
In this work, we assess the effect of the VDM principle and consideration of alternative f- divergences in the contexts of IL and ILfO. We begin by reparameterizing the framework of Ke et al. (2019) for the standard IL. problem. Our version transparently exposes the choices practitioners must make when designing adversarial imitation algorithms for arbitrary choices of fddivergence. We then offer single instantiation of our framework that, in practice, allows stable training of good policies across multiple choices of /-divergence. An example is illustrated in Figure where our methods
result in significantly superior policies. We go on to extend our framework to encapsulate the ILfO setting and examine the efficacy of the resulting new algorithms across range of continuous-come control tasks in the MuJoCo (ooorov et al. 2012) do- main. Our empirical results validate our framework as a viable unification adversarial imitation methods under the VDM principle. With the assistance of recent advances in stabilizing regulariza- tion for adversarial training [Mescheder et al. 2018), improvements in performance can be attained under an appropriate choice of divvergeeece. However, there is still significant performance gap between the recovered imitation policies and expert behavior for tasks with high dimensional obser- vations, leaving open directions for future work in developing improved ILfO algorithms.
and TV-VVMO--igmaan
2 RELATED WORK
The algorithms presented in this work fall in with inverse reinforcement learning (IRL) (gg al. Abbeel & Ng 2004 Syed & Schapire 2007] Ziebart et al., 20088] |Finn et al., 2016 Ho Ermon 2016) approaches II.. Early successes in this regime tend to rely hand-engineered feature rep-
2

---

Under review as a conference paper at ICLR 2020
resentations for success (Abbeel & Ng 2004 Ziebart et al. 2008 Levine et al. 2011 Only in recent years, with the aid of deep neural networks, has there been a surge in the number of ap- proaches that capable of scaling to the raw, high-dimensional observations found in woold control problems (Finn et al. 2016: Ho & Ermon, 2016; Duan et al. 2017 Li et al., 2017: Fu al. 2017; Kim & Park] 2018). Our work focuses attention exclusively
adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive ex- perts (Ho & Ermon 2016 Li et al. 2017 Fu al. 2017: Kostrikov et al. [2018); at the heart of these methods is the Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) ap- proach which produces high-fidelity imitation policies and achieves state-of-the-an results across numerous matinuuus-connro) benchmarks by leveraging the expressive power of Generative Ad- versarial Networks (GANs) (Goodfellow et al.] 2014) for modeling complex distributions over a high-dimensional support. From an IRI. perspective, GAIL. can be viewed as iteratively optimiz- ing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient gradient reinforcement learning (Sutton et al. 2000), allows the agent to shift its own behavior closer to that of the expert. From the perspective of GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert.
While a large body of prior work exists for II., recent work has drawn attention to the more chal- lenging problem of imitation learning from observation (Sermanet al. [017 Liu et al. 2018 Goo & Nickum 2018; Kimura et al. [2018) Torabi et al. 201 Edwards et al. 2019 Sunn et al. 2019). To more closely resemble observational learning in humans and leverage the wealth of publicly- available, observation-on -onlyy data sources, the ILfO problem considers learning from expert demon- stration data where no expert action labels are provided. Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via rein- forcement learning (Gupta et al. 2017: Sermanet et al. [017 Dwibedi al. 2018 Liu al. 2018). In contrast. Torabi et al. (2018a) introduce Behavioral Cloning from Observation (BCO) which lever- ages state-action trajectories collected under random policy train inverse dynamics model for inferring the action responsible for a transition between two input states (assuming the two repre- sent a state and next-state pair). With this inverse model in hand, the observation-oni demonstration data can be converted into the more traditional dataset of state-action pairs over which standard BC can be applied. Recognizing the previously discussed limitations of BC approaches, (Torabi et al. (0188) introduce the natural GAIL counterpart for ILfO, Generative Adversarial Imitation from Observation (GAIFO); GAIFO identical to GAIL except the distributions under consideration in the adversarial game are over state transitions (state and next-state pairs), opposed to state- action pairs requiring expert action labels. While Torabi et al. (2018b) offer empirical results for continuous-contró tasks with low-dimensional features as well as raw image observations, GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algo- rithms that achieve expert performance across wide spectrum of tasks. A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms. For more in-depth survey of ILfO approaches, refer readers to Torabi et al. (2019). We refer readers to the Appendix for a broader overview of prior work.
BACKGROUND
We begin by formulating the problems of imitation learning and imitation learning from observation respectively before taking a closer look at f-divergences and connecting them to imitation learning.
3.1 IMITATION LEARNING & IMITATION FROM OBSERVATION
We operate within the Markov Decision Process (MDP) formalism [Bellman 1957 Puterman 2014) defined as five-tuple M AS R, T, 7) where S denotes (potentially infinite) set of states, A denotes a (potentially infinite) set of actions, R
x A x S > R is a reward function, T S A + ((S) is a transition function, and 7 E [0, 1) is discount factor. At each timestep, the agent observes the current state of the world, 8, E S, and randomly samples an action according to its stochastic policy क S - ((A). The environment then transitions to new state according to
3

---

Under review as a conference paper at ICLR 2020
the transition function T and produces reward signal according to the reward function R that is communicative of the agent's progress through the overall task.
Unlike, the traditional reinforcement learning paradigm, the Decision-makingg problem presented in IL lacks a concrete reward function; lieu of R. a learner is provided with a dataset of expert demonstrations D {n,, 72, -.. ... TN] where each r; (811, di1, $22, a.2....) represents the sequence of states and corresponding actions taken by an expert policy, π². Naturally, the goal an IL. algorithm is synthesize policy ı using D, along with access to the MDP M, whose behavior matches that of π.
While the previous section outlines several possible avenues for using D arrive at a satisfactory imitation policy, our work focuses on adversarial methods that build around GAIL (Ho & Ermon 2016). Following from the widespread success of GANs (Goodfellow al.) 2014). GAIL offers highly-performant approach to IL wherein, at each iteration of the algorithm, transitions sampled from the current imitation policy are first used to update discriminator, (s. a), that acts a binary classifier to distinguish between state-action pairs sampled according to the distributions induced by the expert and student. Subsequently, treating the imitation policy as a generator, policy-gradice reinforcement learning is used to shift the current policy towards expert behavior, issuing higher rewards for those generated state-action pairs that are regarded as belonging to the expert according to (s, a). More formally, this minimax optimization follows as
min max (ea))=r+ [oog(D((>..)) a))] + E(aa)=p [log(1 - (s,
(I)
where p² (8, a) and (s, a) denote the undiscounted stationary distributions over state-action pairs for the expert and imitation policies respectively. Here (w, (V, a)) where (s, a)
represents the unconstrained output of a discriminator neural network with parameters w and (v) = (1 + =- denotes the sigmoid activation function. Since the imitation policy only exerts control over the latter term in the above objective, the per-timestep reward function maximized by reinforcement learning is given as r(s, a, = log(1 Dw, a)). In practice, an entropy reg- ularization term is often added to the objective when optimizing the imitation policy so as to avoid
premature convergence to a suboptimal solution (Mnih et al. 2016 Ho Ermon 2016 2017).
Neu et al.
In order to accommodate various observation-only data sources Zhhou et al. 2018) and remove the
burden of requiring expert action labels, the imitation from observation setting adjusts the expert demonstration dataset 'D such that each trajectory 7; = (#:1, ..., consists only of expert obser- vation sequences. Retaining the goal of recovering behavior, Torabi et al. (2015 8b)) introduce GAIFO as the natural extension of GAIL for matching the state transition distribution of the expert policy. Note that an objective for matching the stationary distribution over expert state transitions enables the provision of per-timestep feedback while simul- taneously avoid the issues temporal alignment that arise when trying match trajectories directly. The resulting algorithm iteratively finds a solution to the following minimax optimization:
imitation policy that closely resembles expert
min max (a,''))=p"" (ss, + ((x.'")pp [lg(1 - (s,
(2)
where p² ss) and (s. s') now denote the analogous stationary distributions over successive state pairs while (s, = (s, represents binary classifier over state pairs. Similar to GAIL, the imitation policy is optimized via policy-gradien reinforcement learning with per- per-timestep rewards computed according to r(s, a, s') = log(1 Dw, s')) and using entropy regularization needed.
4 APPROACH
In this section, we begin with an overview of f f-divergences, their connection to GANs, and their impact on IL through the f-VIM framework (Ke et al. (011) (Section[4.1 I.1). We then present an alter- native view of the framework that transparently exposes the fundamental choice practictioners must make in order to circumvent practical issues that arise when applying f f-VIM to high-diig high-dimensional tasks (Section 4.2). We conclude by presenting our approach for Il.fO f-divergence minimization
4

---

Under review as a conference paper at ICLR 2020
Name
Outpur Activation
dom,-
Conjugate (*)
dom,
Conjugate Inverse f f-((()) (t)
Total Variation (TV)
Lanh(o)
S 5
1.
Kullback-I Leibler (KI.)
exp(t
+ log(t)
Reverse KL (RKL)
exp(p)
R--
log(-()
R
exp(
GAN
log(1 exp(-11)
R-
log(1 exp(t))
R-
log(1 exp( -xx((-t) (t))
Table I: Table of various f-divergences studied in this work as well as the specific choices of activation function yf given by Nowozin al. (2016) and utilized in Ke al. (20T9). Also shown the convex conjugates, inverse convex conjugates, and their respective domains.
(Section 4.3 followed by brief discussion of regularization technique used to stabilize discrimi- nator training in our experiments (Section 4.4).
4.1 f- J-DIVERGENCES AND IMITATION LEARNING
The GAIL and GAIFO approaches engage in an adversarial game where the discriminator esti- mates the divergence between state-action or state transition distributions according to the Jensen- Shannon divergence (Goodfellow et al. 2014). this work, our focus is on more general class of divergences, that includes the Sensen-Shanne divergence, known as Ali-Silvey distances or f- divergences (Ali & Silvey, 1966, Csiszár et al. 2004). For two distributions P and Q with support over domain X' and corresponding continuous densities and we have the f-divergence between them according to:
Dy(P))Q) = (xx)s(( p(x) dr ((x)
(3)
x
where R.+ -> I is convex, lower-semicontinnout function such that f(1) 0. As illus- trated in T different choices of function / yield well-known divergences between probability distributions. In order to accommodate the tractable estimation of f-divergences when only pro- vided samples from and Q. Nguyen et al.] (2010) offer an approach for variational estimation of f-divergences. Central to their procedure is the of the convex conjugate function Fenchel con- jugate (Hiriart-Urruty & Lemaréchal 2004). f", which exists for all convex, lower-semicootimuous functions ff and defined as the following supremum:
(*(() = sup {ut - f(u)} agdomy
(4)
Using the duality of the convex conjugate (f** = f). Nguyen et al. (2010) represent f(u) sup {tu (t)} enabling variational bound: (Edomy-
I a(x))
p(x) Mm) } ((x)
Dy(PQQQ=
sup tEdomy+
p(x) / q(z) ((=)F(((())de)
sup TET
x
sup
[T-p(T(T))) - Lf (T((())))
(5)
TET
where T is arbitrary class of functions T this variational lower bound for GANs that utilize arbitrary J fddivergences, or f-GANs. Specifically, the two distributions of interest are the real data distribution /' and synthetic distribution repre- sented by generative model Qo with parameters 0. The variational function is also parameterized as Te acting as the discriminator. This gives rise the VDM principle which defines the f-GAN objective
dom Nowozin al. (2016) extend the of
min max E, E/r -
Eu (Ta
(6)
Nowozin et al. (016)) represent the variational function as (x) = (x)) such that Vz( X IR represents the unconstrained discriminator network while gf - dom y- is activation
5

---

Under review as a conference paper at ICLR 2020
function chosen in accordance with the f-divergence being optimized. Table 1 includes the "some- what arbitrary" but effective choices for 9 suggested by Nyoooozin et al. (2016) and we refer readers to their excellent work for more details and properties of f-dvergeences and G-GNNs.
Recently, Ke ct al. (2119) have formalized the generalization from GAN to f-GAN for the traditional IL problem. They offer the f- f-Variational Imitation (f-VIM) framework for the specific case of estimating and then minimizing the divergence between state-action distributions induced by expert and imitation policies:
min max B (x,a))pp-
- (f
where Vu : S x A - R denotes the discriminator network that will supply per- per-timestep rewards during the outer policy optimization which itself is carried out over policy parameters 0 via policy- gradient reinforcement learning (Sutton et al., 2000). In particular, the per-timestep rewards pro- vided to the agent are given according to r(,, a, (V)r(V(s))))
While Ke et al. (209)) do an excellent job of motivating the use of f-divergences for II. (by for- malizing the relationship between divergences over trajectory distributions vs. state-action distri- butions) and connecting f f-VIM to existing imitation-learning algorithms, their experiments focus on smaller problems to study the and the implications of such behavior depending on the multimodality of the expert trajectory dis- tribution. Meanwhile, in the course of attempting apply f-VIM to large-scale imitation problems, we empirically observe numerical instabilities stemming from function approximation, demanding reformulation of the framework.
aspects of different, ff f-divergences
4.2 REPARAMETERIZING f- VIM
In their presentation of the f-VIM framework, Ke et al. (2019) retain the choices for activation function 9r introduced by Nowozin et al. (2016) for f-GNNs. Recall that these choices of g play a critical role in defining the reward function optimized by the imitation policy on each iteration of f-VIM, r(s, a, 8²) = ("( (v, a))). It is well known in the reinforcement-leaming literature that the nature of the rewards provided to an agent have strong implications on learning success and efficiency (Ng et al., 1999, Singh et al.] (010). While the activation choices made for f-GANs suitable given that both optimization problems are carried out by backpropagation, we assert that special care must be taken when specifying these activations (and implicitly, the reward func- tion) for initation-learinng algorithms. A combination of convex conjugate and activation function could induce a reward function that engenders numerical instability or a simply challenging reward landscape, depending on the underlying policy-gradient algorithm utilized (Henderson al. 2018)) Empirically, we found that the particular activation choices for the KL. and reverse KL divergences shown in Table (linear and exponential, respectively) produced mmitation-learinng algorithms that, in all of our evaluation environments, failed to complete execution due to numerical instabilities caused by exploding policy gradients. In the case of the Total Variation distance, the correspond- ing f-GAN activation for the variational function is a tanh, requiring a learning agent to traverse a reward interval of [-1, 1] by crossing an intermediate region with reward signals centered around O.
To refactor the f-VIM framework so that it more clearly exposes the choice of reward function to practictioners and shifts the issues of reward scale away from the imitation policy, we propose uniformly applying an activation function g (v) (r-)(((v) where ((- denotes the inverse of the convex conjugate (see Table[]. Here r is effectively a free parameter that can be set according to one of the many heuristics used throughout the field of deep reinforcement learning for maintaining a reasonable reward scale (Mnih et al. 2015) 2016 Henderson et al. 2018 so long as it obeys the domain of the inverse conjugate dom d....-.. In selecting 9y accordingly, the reparameterized saddle- point optimization for f-VIM becomes
min max ((ea))-= [f" (r(V #
(( (r(V a))l
(8)
where the per-timestep rewards used during policy optimization are given by r(s, s') rVV,,s, a)). In applying this choice, we shift the undesirable scale of the latter term in VDM to- wards the discriminator, expecting it to be indifferent since training is done by backpropagation. As one potential instantiation, we consider r(u) = (u) where (-) denotes the sigmoid function lead- ing to bounded rewards in the interval [0. 1] that conveniently adhere dom for almost all of

---

Under review as a conference paper at ICLR 2020
evaluate imitation-leaming algorithms with this choice against those using f-VIM with the original f-GAN activations; we find that, with- out regard for the scale of rewards and the underlying reinforeement-eaniing problem being solved, the f f-GAN activation choices either produce degenerate solutions or completely fail to produce an imitation policy altogether.
the f f-divergences examined in this work"] In Section 5
4.3 JDDIVERGENCES AND IMITATION FROM OBSERVATION
Applying the variational lower bound of Ngyyen et al. 2010) and the corresponding f-GAN tension, we can now present our Variational Imitation from Observation (f-VIMO) extension for a general family of ILfO algorithms that leverage the VDM principle in the underlying saddle-poing optimization. Since optimization of the generator will continue to be carried out by policy-gradient reinforcement learning, we adhere our reparameterization of the f-VIM framework and present the V-VIMO objective as:
min max I. (s.8")=--* (r(V -rt
(9)
with the per-timestep rewards given according to r(s, 8') r(V2(1, 8)). We present the full approach as Algorithm| [ Just as in Section 4.2 we again call attention to Line 5 where the dis- criminator outputs (acting as individual reward signals) scale the policy gradient, unlike the more conventional discriminator optimization of Line 4 by backpropagation: this key difference is the pri- mary motivator for our specific reparameterization
the f-VIM framework. Just as the previous section, we take r(u) (u) as particularly convenient choice of activation given its agreement to the inverse conjugate domains dom,--1 for many choices of f-divergence and we employ this instantiation throughout all of our experiments. We leave the examination of alternative choices for r to future work.
Algorithm f-VIMO
1: INPUT: Dataset of expert trajectories D, initial policy and discriminator parameters Oo and wo. number of iterations N, discount factor 7
. 2: for 0, .N do
3: Sample trajectories from current imitation policy Ti ~ Te,
4: Wi+1 = w; V [F1o))) (',)))))) - (er)")) (r(V s')l)
5: Update 0, 0 via policy-gradient update with rewards given by (8,(
0:+1 0; + (o,,,,,))
Vo Volog(To,( (als)) - (V = 8, ==
6: end for
4.4 DISCRIMINATOR REGULARIZATION
The refactored version of f-VIM presented in Section 4.2 is fundamentally addressing instability issues that may occur on the generator side of adversarial training: in our experiments, we also ex- amine the utility of regularizing the discriminator side of the optimization for improved stability. Following from a line of work examining the underlying mathematical properties of GAN opti- mization (Roth al., 2017 2018 Mescheder et al. 2018). we opt for the simple gradient-based regularization of Mescheder et al. (2018) which (for f VIMOO) augments the discriminator loss with the following regularization term:
R(w) (ss ||V- (r(V.
(10)
where v is a hyperparameter controlling the strength of the regularization. The form of this specific penalty follows from the analysis of Roth et (017)); intuitively, its purpose is to disincentivize the
For Total Variation distance, we ru)) ( to remain within dom,-...

---

Under review as a conference paper at ICLR 2020
500
$110011
Figure 2: Comparing our TV-VIM and TV-VIMO frameworks (Equations 9 with sigmoid rewards against the original TV-VIM formulation (Equation 7 and its TV- counterpart with activations prescribed by Nowozin et al. (2016) (Table
discriminator from producing a non-zero gradient that shifts away from the Nash equilibrium of the minimax optimization when presented with a generator that perfectly matches the true data distri- bution. While originally developed for traditional GANs and shown to empirically exhibit stronger convergence properties over Wasserstein GANs (Gulrajani et al., 2017), this effect is still desirable for the adversarial IL setting where the reward function (discriminator) used for optimizing the imi- tation policy should stop changing once the expert taate-transition distribution has been matched. In practice, we compare f-VIM and f-VIMO both with and without the use of this regularization term and find that R(w) can improve the stability and convergence f-VIMO across almost all domains.
5 EXPERIMENTS
We examine four instantiations of the f-VIM and f- V-VIOO frameworks (as presented in Sections 4.2 and 4.3) corresponding to imitation algorithms with the following choices of f-divergence: GAN, Kullback -eiblee, reverse KL, and Total Variation. We conduct our evaluation across four MuJoCo environments (Todorov et al., 2012) of varying difficulty: Ant, Hopper, HalfCheetah, and Walker (see the Appendix for more details on individual environments).
The core questions seek to answer through our empirical results are as follows:
1. What are the implications of the choice of activation for the variational function in f-VIM imitation policy performance?
2. Do f-divergences act meaningful axis of variation for IL and ILfO algorithms?
3. What is the impact of discriminator regularization on the stability and convergence proper- ties of /-VIM/f-VIN
4. How does the impact of different f-divergences vary with the amount of expert demonstra- tion data provided?
To answer the first three questions above, report the average total reward achieved by the imita- tion policy throughout the course of learning with rewards as defined by the corresponding OpenAI Gym environment (Brockman et al. 2016). Shading in all plots denote 95% confidence intervals computed over 10 random trials with TO random seeds. Expert demonstration datasets of 50 trajec- tories were collected from agents trained via Proximal Policy Optimization (PPO) (Schulman al.) 2017): 20 expert demonstrations were randomly subsampled at the start of learning and held fixed for the duration of the algorithm. We also utilize PPO as the underlying cenfforeemen-leermiii gorithm for training the imitation policy with a clipping parameter of 0.2, advantage normalization, entropy regularization coefficient le 1e-3, and the Adam optimizer (Kingma & Ba 2014). Just as in a discount factor of 7 0.995 and apply Generalized Advantage Esti- mation (Schulman et al. 2015) with parameter À 0.97. We run both f-VIM and f-VIMO for a total of 500 iterations, collecting 50000 environment samples per iteration. The policy and discrim- inator architectures are identically two separate multi-layer perceptrons each with two hidden layers of 100 units separated by tanh nonlinearities. A grid search was used for determining the initial
al-
Ho & Ermon (2016)
8

---

Under review as a conference paper at ICLR 2020
9Che8tth-22 walorde--0
Figure 3: Comparing f-VIM and f-VIMO across four MuJoCo environments with f-divergences: GAN, Kullback-Leibler (KL), reverse KL (RKL), and Total Variation distance (TV). We also ex- amine the effect of discriminator regularization (Equation 10 with v 10 per Mescheder et al. 20188
A6-2
Doent Dere shoten
bost Denenessen
Figure 4: Evaluating f-VIM and f-VIMO with and without regularization across four MuJoCo environments with varying amounts of expert demonstration data.
learning rate, number of PPO epochs, and number of epochs used for discriminator training (please the Appendix for more details) and we report results for the best hyperparameter settings.
To address our final question, we take the best hyperparameter settings recovered when given 20 expert demonstrations and re-run all algorithms with {1, 5. 10, 15) expert demonstrations that randomly sampled at the start of each random trial and held fixed for the duration of the algorithm. We then record the average return of the final imitation policy for each level of expert demonstration.

---

Under review as a conference paper at ICLR 2020
6 RESULTS & DISCUSSION
To highlight the importance of carefully selecting the variational function activation Jr and validate our modifications to the f-VIM framework, we present results in
to the original f-VIM framework of Ke et al. (2019) and its natural ILfO counterpart. Activation functions for the original methods are chosen according to the choices outlined in Ke et al. (2019); Nowozin et al. (2016). In our experiments using the KL and reverse KL divergences, we found that none of the trials reached completion due to numerical instabilities caused by exploding policy gradients. Consequently, we only present results for the Total Variation distance. We observe that under the original J-GAN activation selection, we fail to produce meaningful imitation policies with learning stagnating after 100 iterations or less. As previously mentioned, suspect that this stems from the use of tanh with TV leading to a dissipating reward signal.
We present results in Figure 2 to assess the utility of varying the choice of divergence in f-VIM and f-VIMO across each domain. In considering the impact of f-divergence choice, we find that most of the domains must be examined in isolation to observe particular subset of f f-divergences that stand out. In the IL setting, we find that varying the choice of f-divergence can yield different learning curves but, ultimately, produce near-optimal (if not optimal) imitation policies across all domains. In contrast, we find meaningful choices of f-divergence in the ILfO setting including {KL, TV} for Hopper, RKL for HalfCheetah, and {GAN, TV} for Walker. We note that the use of discriminator regularization per Mescheder et al. 2018) is crucial to achieving these performance gains, whereas the regularization generally fails to help performance in the II. setting. This finding supportive of the logical intuition that ILfO poses a fundamentally more-challenging problem than standard II.
As negative result, we find that the Ant domain (the most difficult environment with S R and A C IR*) still poses a challenge for ILfO algorithms across the board. More specifically, we observe that discriminator regularization hurts learning in both the IL and ILfO settings. While the choice of RKL does manage to produce a marginal improvement over GAIFO, the gap between existing state- of-the-art and expert performance remains unchanged. It is an open challenge for future work to either identify the techniques needed to achieve optimal imitation policies from observations only or characterize fundamental performance gap when faced with sufficiently large observation spaces.
In Figure 4 we vary the total number of expert demonstrations available during learning and observe that certain choices of f-divergences can be more robust in the face of less expert data, both in the IL and ILFO settings. We find that KL-VIM and TV-VIM are slightly more performant than GAIL when only provided with a single expert demonstration. Notably, in each domain we see that certain choices of divergence for f-VIMO do a better job of residing close their f-VIM counterparts suggesting that future improvements may come from examining f-divergences in the small-data regime. This idea is further exemplified when accounting for results collected while using discriminator regularization (Mescheder et al., 2018). We refer readers to the Appendix for the associated learning curves.
Our work leaves many open directions for future work to close the performance gap between student and expert policies in the II.OO setting. While we found the sigmoid function to be a suitable instanti- ation of our framework, exploring alternative choices of variational function activations could prove useful in synthesizing performant ILfO algorithms. Alternative choices of f-divergences could lead to more substantial improvements than the choices we examine in this paper. Moreover, while this work has a direct focus on of f-divergences, Integral Probability Metrics (IPMs) (Müller 1997] Gret- ton et al. 2012) represent distinct but well-established family of divergences between probability distributions. The success of Total Variation distance in our experiments, which doubles as both a f- divergence and IPM (Sriperumbudur et al. 2009), is suggestive of future work building IPM-based ILfO algorithms (Sun et al. 2019
7 CONCLUSION
In this work, we present a general framework for imitation learning and imitation learning from observations under arbitrary choices of f-divergence. We empirically validate single instantiation of our framework across multiple f-divergences, demonstrating that we overcome the shortcomings of prior work and offer a wide class of II. and IIfO algorithms capable of scaling larger problems.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In ICML, 2017.
Syed Mumtaz Ali and Samuel Silvey. general class of coefficients of divergence of one distri- bution from another. Journal of the Royal Statistical Society: Series (Methodological), 28(1): 131-142, 1966.
Brenna Argall, Sonia Chernova, Manuela M. Veloso, and Brett Browning. A survey robot learning from demonstration. Robotics and Autonomous Systems, 57:469-488. 2009.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875, 2017.
Wasserstein gan. arXiv preprint
Christopher G. Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, 1997.
Michael Bain and Claude Sommut. A framework for behavioural claning. Machine intelligence, 15 (15):103, 1999.
Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, pp- 679-684, 1957.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End end learning for self-driving cars. arXiv preprint arXiv: 1604.07316, 2016.
Diana Borsa, Nicolas Heess, Bilal Piot, Siqi Liu, Leonard Hasenclever, Remi Munos, and Olivier Pietquin. Observational learning by reinforcement learning. In Proceedings of the 18th Interna- fional Conference Autonomous Agents and MultiAgent Systems, pp. 1117-1124, International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. ArXiv, abs/ abs/1606.01140, 2016.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar A. Duéñez-Guzzmá. Lyapunov-based safe policy optimization for continuous control. ArXiv, abs/1901.10031, 2019. 2019.
Imre Csiszár, Paul C Shields, et al. Information theory and statistics: A tutorial. Foundations and Trends Communications and Information Theory, 1(4):417-528, 2004.
Yan Duan, Marcin Andrychowicz. Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. Advances in neural information processing systems, pp. 1087-1098, 2017.
Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable representations from visual observations. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1577-1584, 2018.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint
2015.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International Conference on Machine Learning, pp. 1755-1763, 2019.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv: 1/248, 2017.

---

Under review as a conference paper at ICLR 2020
Alessandro Giusti, Jerome Guzzi, Dan C. Ciresan, Fang-Lin He, Juan P. Rodriguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni A. Di Caro, Davide Scara- muzza, and Luca Maria Gambardella. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 1:661-667, 2016.
Wonjoon Goo and Scott Niekum. One-shot learning of multi-step tasks from observation via activity localization in auxiliary video. 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-26628 2014.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773, 2012.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky. Vincent Dumoulin, and Aaron C. Courville. Im- proved training of wasserstein gans. In NIPS, 2017.
Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. ArXiv, abs/ fobs//703.02949 2017.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In 7'hirty-Second AAAI Conference Artificial Intel- ligence, 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Oshand, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jean-Bage Saan-Bptistte -rrutt and Claude Lemaréchal. Fundamentals of convex analysis. 2004.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 45665-473,, 2016.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. arXiv preprins arXiv:190S.12880 2019.
Kee-Eung Kim and Hyun Soo Park. Imitation learning via kernel mean embedding. In Thirty-Secont AAAI Conference Artificial Intelligence, 2018.
Daiki Kimura, Subhajit Chaudhury, Ryuki Tachibana, and Sakyasingha Dasgupta. Internal model from observations for reward shaping. ArXiv, abs/1806.01267, 2018.
Diederik Kingma and Jimmy Ba. Adam:
method for stochastic optimization. CORR,
abs/1412.69885 2014.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Diseriminattr-actoo-orittt Addressing sample inefficiency and reward bias in adversarial imitation learning. In ICLR, 2018.
Michael Laskey, Sam Staszak, Wesley Yu-Shu Hsieh, Jeffrey Mahler, Florian T Pokomn,, Anca D Dragan, and Ken Goldberg. Shiv: Reducing supervisor burden dagger using support vectors for efficient learning from demonstrations in high dimensional state spaces. In 2016 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 462-469. IEEE, 2016.
Michael Laskey, Caleb Chuck, Jonathan Lee, Jeffrey Mahler, Sanjay Krishnan, Kevin Jamieson, Anca Dragan, and Ken Goldberg. Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations. In 20/7 IEEE International Conference on Robotics and Automation (ICRA), pp. 358-365. IEEE, 2017a
Michael Laskey, Jonathan Lee, Roy Fox, Anea Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on Robot Learning, pp. 143-156, 2017b.
Hoang M Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hierar- chical imitation and reinforcement learning. arXiv preprint arXiv: 888.0000000 2018.
12

---

Under review as a conference paper at ICLR 2020
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. In Advances in Neural Information Processing Systems, pp. 19-27, 2011.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International 2015.
Conference Machine Learning, pp.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, pp- 3812-3822, 2017.
Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions Information Theory, 52(10):4394 4412, 2006.
Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. 2018 IEEE International Confer- on Roboties and Automation (ICRA), pp. 1118 1118-1125, 2018.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International Conference on Machine Learning, pp. 3478-3487, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioamnis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51 518:529-533 -332, 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICMI., 2016.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429443, 1997. doi: 10.2307/1428011.
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized markov
decision processes. arXiv preprint arXiv: 2017.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning.
Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In ICML, 1999.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-5861. 2010.
Scott Nickum, Sarah Osentoski, George Konidaris, Sachin Chitta, Bhaskara Marthi, and Andrew G Barto. Learning grounded finite-state representations from unstructured demonstrations. The International Journal Robotics Research, 34(2):131-157, 2015.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, al. An algorithmic perspective on imitation learning. Foundations and Trends Robotics, 7(1-2); 1-179, 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, pp. 305-313, 1989.
Martin L Puterman. Markov Decision Processes.: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 10087, 2017.
13

---

Under review as a conference paper at ICLR 2020
Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:140..597, 2014.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, 2011.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in neural information pro- cessing systems, pp. 2018 2028, 2017.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Adversarially robust training through structured gradient regularization. arXiv preprint arXiv: 2018.
Tim Salimans and Richard Chen. Learning montezuma's revenge from single demonstration. arXiv preprint arXiv:18/2.0333 2018.
Stefan Schaal. Learning from demonstration. In Advances neural information processing systems, pp. 1040-1046, 1997.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:/506.022 024388 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/ 77700777777 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Tone-contrastive networks: Self-supervised learning from video. 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 1134 141,, 2017.
Satinder P. Singh, Richard L. Lewis, Andrew G. Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2:70-82, 2010.
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics,) \phi-divergences and binary classification. arXiv preprint arXiv:0901.2699 2009.
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In Proceedings of the 34th International Conference on Machine Ctarning-Volunee 70, pp. 3309-3318, JMLR. org, 2017.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining rein- forcement learning & imitation learning. arXiv preprint /805. 1240, 2018.
Wen Sun, Anirudh Vemula, Byron Boots, and Andrew Bagnell. Provably efficient imitation learn- ing from observation alone. arXiv preprint arXiv: 10948, 2019.
Richard Sutton and Andrew G Barto. Introduction to reinforcement learning. 1998.
Richard Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- tion processing systems. pp. 1057-1063, 2000.
Umar Syed and Robert E. Schapire. A thne-thccreetc approach to apprenticeship learning. In NIPS, 2007.
Dhruva Tirumala, Hyconwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Tch, and Nicolas Heess. Exploiting hicrarchy for learning and transfer in kl-regularized rl. arXiv preprint 100300/990307433 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033
IEEE, 2012.
14

---

Under review as a conference paper at ICLR 2020
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:180S.01954. 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: urXiv:887..066 01011100700..... 2018b.
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. ArXiv, abs/ 2019.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. 2016.
arXiv preprint
Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008
A RELATED WORK
A.1 LEARNING FROM DEMONSTRATION
Our work broadly falls within the category of Learning from Demonstration (LfD) (Schaal 1997]
Atkeson & Schaal 1997; Argall et al. 2009). where agent must leverage demonstration data (typ- ically provided as trajectories, each consisting of expert state-action pairs) to produce an imitation policy that correctly captures the demonstrated behavior. Within the context of LID, finer dis- tinction can be made between behavioral cloning (BC) (Bain & Sommut 1999, Pomerlcau 1989)
and inverse reinforcement learning (IRL) (Ng et al. Abbeel & Ng] 2004 Syed & Schapire 2007
Ziebart et al. 2008 Finn et al.) 2016 Ho & Ermon 2016) approaches; BC approaches view the demonstration data standard dataset of input-output pairs and apply traditional supervised-
learning techniques to recover an imitation policy. Alternatively IRL-based methods synthesize an estimate of the reward function used to train the expert policy before subsequently applying suiffrreementlearniig algorithm (Sutton & Barto| 1998 Abbeel & 2004g to recover the corre- sponding imitation policy. Although not a focus of this work, we also acknowledge the myriad of approaches that operate at the intersection of II. and reinforcement learning or augment reinforce- ment learning with II. (Rajeswaran al., 2017 Hester et al., 2018
Salimans & Chen 2018 Sun
et al.
2018 Borsa et al. 2019: Tirumala et al.) 2019).
While BC approaches have been successful in some settings (Niekum et al.] 2015 Giusti et al. 2016 Bojarskii et 2016)), they are also susceptible to failures stemming from covariate shift where minute errors in the actions of the imitation policy compound and force the agent into regions of the state space not captured in the original demonstration data. While some preventative measures for covariate shift do exist (Laskey et al.] 2017b), a more principled solution can be found in methods like DAgger (Ross et al. 2011) and its descendants (Ross & Bagnell 2014 [Sun et al. 2017 Le et al. 2018) that remedy covariate shift by querying an expert to provide on-policy action labels. It is worth noting, however, that these approaches are only feasible in settings that admit such online interaction with an expert (Laskey et al. 2016) and, even then, failure modes leading to poor imitation policies do exist (Laskey al. 2017a).
The algorithms presented in this work fall in with IRL-based approaches to IL. Early successes in this regime tend to rely on hand-engineered feature representations for success (Abbeel & Ng 2004 Ziebart et al.) 2008 Levine et al., 201T). Only in recent years, with the aid of deep neural networks, has there been surge in the number of approaches that are capable of scaling to the raw, high- dimensional observations found in real-world control problems (Finn et al. 2016; (oo & Ermon, 2016 Duan et al.. 2017 Li et al. 2017 Fu et al. 2017 Kim Park] 2018). Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across range of imitation tasks without requiring interactive experts (Ho & Ermon 2016 Li et al. 2017 Fu et al.) 2017 Kostrikov et al., 2018); at the heart of these methods is the Generative Adversarial Imitation Learning (GAIL) (Ho & Enmon| 2016) approach which produces
imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the
15

---

Under review as a conference paper at ICLR 2020
expressive power of Generative Adversarial Networks (GANs) (Goodfellow al. 2014) for mod- eling complex distributions over a high-dimensionn support. From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradien reinforcement learning (Sutton et al.) 2000), allows the agent to shift its own behavior closer that of the expert. From the perspective of GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce state-action visitation distribution similar to that of the expert.
While a large body of prior work exists for IL, numerous recent works have drawn attention to the more challenging problem of imitation learning from observation (Sermanet et al.] 2017 Liu et al.)
2018
Goo Niekum 2018 Kimura et al. 2018 Torabi et al. 201 bJ Edwards et al. 2019 [unn
et al. 2019). In an effort to more closely resemble observational learning in humans and leverage observation-only data sources, the II.OO problem considers learning from expert demonstration data where no expert action labels are provided. Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies serve cost signal to be minimized via reinforcement learning (Gupta et al. 2017] Sermanet et al. 2017| Dwibedi et al.) 2018 Liu al. 2018). In contrast, [Torabi et al. (201 Sa) introduce Behavioral Cloning from OB- servation (BCO) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states (assuming the two represent a state and next-state pair). With this inverse model in hand, the observation-only demonstration data can be converted into the more traditional dataset of state- action pairs over which standard BC can be applied. Recognizing the previously discussed limi- tations of BC approaches, Torabi et al. (2018b) introduce the natural GAIL counterpart for ILfO, Generative Adversarial Imitation from Observation (GAIFO); GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions (state and next-state pairs), as opposed to state-action pairs requiring expert action labels. While Torabi et al. (2018)) offer empirical results for continuous-comt raw image observations, GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks. A central question of this work is to explore how alternative formulations of the GAN ob- jective that underlies GAIFO might yield superior ILfO algorithms. For more in-depth survey of ILfO approaches, we refer readers Torabi et al. 2019).
the wealth of
tasks with low-dimensional features as well as
A.2 GENERATIVE ADVERSARIAL NETWORKS
With focus on generative imnerativeaadeesariia methods for II, this work leverages several related ideas in the GAN literature for offering alternative formulations as well as improving understanding of their underlying mathematical foundations (Li et al. [203]] Dziugaite et al. 2015 Zhao et 2016 Nowozin et al. 2016 Roth et al., 2017: Arjovsky et al.) 2017: Guirajani et al. 2017 Roth et al,, 2018 Mescheder et al. 2018)) Crttical to the ideas presented in many of these previous works is an understanding that discriminator networks are estimating a divergence between two probability distributions of interest, usually taken to be the real data distribution and the fake synthetic distri- bution represented by the generator. Formal characterizations of this divergence, either by Integral Probability Metrics (IPMs) (Müller 1997: Gretton et al., 2012) f-divergences (Ali & Silvey)
1966 Csiszár et al., 2004 Liese & Vajda, 2006). yield different variations on the classic GAN for- mulation which is itself slight variation on the Jensen-Shannon (JS) divergence (Li et al.) 2015
Dziugaite et al. 20155 Zhao et al. 2016 Nowozin et al. 2016 Arjovsky et al. 2017 Gulrajani et al., 2017]. Following from work by Nowozin et al. (2016) to generalize the GAN objective to arbitrary j-divergences, Ke et al. (2019) offer a generalization of GAIL to an arbitrary choice of f-divergence
for quantifying the gap between the state-action visitation distributions of the imitation and expert policies; moreover, Ke et al.) (2019) propose a unifying framework for II., f- /-rriiational IMitation (f-VIM), in which they highlight a correspondence between particular choices of f-divergences and existing IL algorithms (specifically BC > Kullback-Leibler (KL) divergence, DAgger < Total- Variation distance, and GAIL > JS-divergence P). While Ke et al. (0119) focus on providing empirical results in smaller toy problems to better understand the interplay between f f-divergence
The discriminator loss optimized in the original GAN formulation 2- Das - log(4) where Das denotes the divergence (Goodfellow et al. 2014 Nowozin al. 2016).
16

---

Under review as a conference paper at ICLR 2020
choice and the multimodality of the expert trajectory distribution, we provide empirical evaluation of their f-VIM framework across a range of contimous control tasks in the Mujoco domain (Todorov et al. 2012). Empirically, find that some of the design choices f f-VIM inherits from the original J-GAN work (Nowozin et al., 2016) are problematic when coupled with adversarial II. and training of the generator by volicy-gradient reinforcement learning, instead of via direct backpropagation as in traditional GANs. Consequently, we refactor their framework to expose this point and provide one practical instantiation that works well empirically. We then go extend the f-VIM frame- work to the IFO problem (f-VIMO) and evaluate the resulting algorithms empirically against the state-of-the-arts GAIFO.
B EXPERIMENT DETAILS
Here provide details of the MuJoCo environments (Todorov et al. 2012) used in experi- ments as well as the details of the hyperparameter search conducted for all algorithms (II. and II.fO) presented.
B.1 MuJoCo ENVIRONMENTS
All environments have continuous observation and action spaces of varying dimensionality (as shown below). All algorithms evaluated each environment were trained for a total of 500 iter- ations, collecting 50, 000 environment transitions per iteration.
Task
Observation Space
Action Space
Ant-v2
RTT
R$
Hopper-v2
R 11
I
HalfCheetah-v2
17
R
Walker2d-v2
IR.
R
B.2 HYPERPARAMETERS
Below we outline the full set of hyperparameters examined for all experiments presented in this work. We conducted full grid search over 10 random trials with 10 random seeds and report results for the best hyperparameter setting.
Hyperparameter
Values {le le o} (5. 10)
Discriminator learning rate
PPO epochs
Discriminator epochs
(1,5,10) 10)
Preliminary experiments were conducted to test smaller values for PPO epochs and policy learning rates before settling the grid shown above.
17

---

Under review as a conference paper at ICLR 2020
C ADDITIONAL RESULTS
C.I UNREGULARIZED f /-VIM/VIMO
Figure 5: Comparing f-VIM and f-VMMO across four MuJoCo environments with f-divergences:
GAN, Kullback-Leibler (KL), reverse KL (RKL), and Total Variation distance (TV).
190
Figure 6: Evaluating f-VIM and f-VIMO across four MuJoCo environments with varying amounts of expert demonstration data.
18

---

Under review as a conference paper at ICLR 2020
C.2 SAMPLE COMPLEXITY LEARNING CURVES
Ant-v2
Happern-2
IChratah-122
Walcer28-22
2000
1060
-2090
430
5040
Sivertpe 3221171
6230
3000
6830
Vi00, 10
200
6008
1000
11000
300
390
300
500
189
283
300
5000
189
200
400
304
206
300
5049
3009
2000
2009
Eerocoo
Benacin
Eteratoon
Itertion
Figure 7: Learning curves for f-VIM and f-VIMO across four MuJoCo environments using only expert demonstration.
21t-22
Hepper-22
Ho ICheetah-12
Welker2d-22
3000
3003
0ar5, p-00
3000
3008
Uperrt
230
123
3000
400
2009
AMlk w-12.
Egert
189
400
5000
3009
200
Toree 1000
De 1010001
Berscon
Steration
Figure 8: Learning curves for F-VIM and f-VIMO across four MuJoCo environments using only expert demonstrations.
19

---

Under review as a conference paper at ICLR 2020
Hopper-v2
ICheetahv22
Walker2d
X000
3000
p-14
p-223
300
0875 1-15
Open
6000
2049
1080
180
1008
436
3000
6000
3003
300
0000
3000
Sipert
330
300
100
183
208
206
450
Figure 9: Learning curves for f-VIM and f-VIMO across four MuJoCo environments using only 10 expert demonstrations.
Hepper-22
Ho ICheetah-12
Welter2d-22
3000
3003
yold 6ar5, p-00
2000
2000
3008
Lppert
123
3000
400
2009
Eipert
189
3000
3009
200
Toreee was 1060
D0 1010001
Berscon
Steration
Bter
Figure 10: Learning curves for f-VIM and V-IMO across four MuJoCo environments using only 15 expert demonstrations.
20

---

Under review as a conference paper at ICLR 2020
Hopper-v2
ICheetahu2
Walker2dv2
X000
3000
p-14
p-223
3W0
0875 1-1
Opent
6000
2049
1080
100
100
180
436
206
3000
200
3003
4800
p-0.0
300
3000
6000
3000
Sipert
300
200
100
133
208
304
208
450 130
Figure 11: Learning curves for f-VIM and f- f-VMOO across four MuJoCo environments using only 20 expert demonstrations.
C.3 /-DIVERGENCE VARIATIONAL BOUND SWAP
Throughout this paper, we advocate for the use of the following variational lower bound to the f-divergence for both f-VIM and f-VIMO:
Dy(p"' =))=))= 2 min max E (0.0))===- (s- (rrn-(((VVVolleee (r(V (8,8'))) -
(11)
In particular, we value the above form as it clearly exposes the choice reward function for the imitation policy free parameter that, in practice, has strong implications for the stability and convergence of adversarial IL/ILiO algorithms. Alternatively, one may consider appealing to the original lower bound of Nguyen al. (2010), used in f-GANs (Nowozin al. 2016) unmodified, but swapping the positions of the two distributions:
Dy(p=0l1pT 2 min max (V
(s.'' (f
(12)
Consequently, the term in this lower bound pertaining to the imitation policy now similar that of the bound in Equation 11: namely, almost arbitrary activation function, gy. applied to the output of the variational function (discriminator) Vg. The difference being that the codomain of gf must obey the domain of the convex conjugate, f*, while the codomain of must respect the domain of the inverse convex conjugate, --1
We evaluate these two choices empirically below for the specific choice of the KL-divergence in the Ant and Hopper domains (the two most difficult domains of our evaluation). We find that the original unswapped bound in Equation II used throughout this paper outperforms the variants with the distributions swapper, for both the IL and ILfO settings. Crucially, we find that the KL-VIM in the Ant domain no longer achieves expert performance while optimizing the swapped bound.
21

---

Under review as a conference paper at ICLR 2020
Ant-v2
4000
3000
Average Return
1000
LLVVIM-sigmoiddsaap
TLVVIMo-igmold-wwpp
KL-VIM-sigmoid, =
(L.VINO-sigmodd y
-1000
Expert
100
200
300
400
500
Iteration
Figure 12: Learning curves for KL- VIM and KL-VIME in Ant with 20 expert demonstrations using the regular and swapped variational lower bound.
Hopper-v2
3000
Average Return
2500
2000
1500
LL-VIM--igoodd-swap. 0.0
w.VVIM--iggmoidswapppp y 0.0
KL-VIM-sigmoid, 0.0 0.0
500
KL-VIMO-sigmoid, 0.0
Expert
100
200
300
400
500
Iteration
Figure 13: Learning curves for KL-VIM and KL-VIMO in Hopper with 20 expert demonstrations using the regular and swapped variational lower bound.
22


Reviews:
[{"review_id": "ryg7NuiTtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the application of the f-VIM framework (Ke et. al., 2019) to the problem of imitation learning from observations (no expert actions). The authors first identify a potential source of numerical instability in the application of f-VIM to imitation learning – the rewards for the policy-gradient RL are given by a combination of a convex conjugate and an activation function. To alleviate this, f-VIM is reparameterized by curating the activation using conjugate inverse (Equation 8), yielding a potentially more stable reward for deep-RL. \n\nI have the following concerns about the paper:\n\n1.\tLack of novelty – Although I appreciate the reparameterization applied to f-VIM to make it potentially more stable for imitation learning in large state- and action-spaces, I don’t think that by itself meets the bar for ICLR. Algorithm 1 is basically the GAILFO algorithm (Torabi et al. 2018) written in the f-Vim framework, with the proposed reparameterization. The discriminator regularization (Section 4.4) has been used before.\n\n2.\tExperiments – Figure 2 shows the improvement with TV when using the reparameterization, and the authors mention in text about the difficulty with KL and reverse-KL. What about the JS divergence (GAIL)? Does reparameterization help or affect that?\n\n3.\tIn Figure 3, is GAIL from the original paper, or does it use the sigmoid rewards? Figure 3 does not offer any evidence that the proposed methods in the paper lead to algorithms that should be preferred over the current state-of-the-art in imitation learning with divergence minimization such GAIL and WAIL.\n\nMinor comment:\nIn Table 1: GAN is not a divergence. Please use Jensen-Shannon, with the corresponding tweaks to the columns. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxFuo-vYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Summary:\nThe paper proposes an IL method based on the f-divergence. Specifically, the paper extends f-VIM (Ke et al., 2019), which uses the f-divergence for IL, by using a sigmoid function for discriminator output’s activation function. This choice of activation function yields an alternative objective function, where the reward function for an RL agent does not directly depend on the convex conjugate of the function f; the paper claims that this independency improves stability of policy learning. This proposed method is named f-VIM-sigmoid. The paper extends f-VIM-sigmoid to the setting of IL with observation and proposes f-VIMO-sigmoid. Experiments on Mujoco locomotion tasks show that f-VIM-sigmoid and f-VIMO-sigmoid perform better than existing methods.\n\n* Rating:\nThe paper proposes a simple but interesting approach to improve stability of adversarial IL. However, the paper has issues regarding baseline methods, motivation, supports of the claim, and experiments (see below). These issues should be addressed. At the present, I vote for rejection.\n\n* Major comments:\n- Discussion and comparing against a simple baseline method based on swapping distributions: \nTo make the reward function be independent of the convex conjugate f*, it is possible to simply swapping the distributions P and Q in the definition of the f-divergence. More specifically, instead of minimizing D_f(P||Q), we can minimize D_f(Q||P), where P is a data distribution and Q is a generator. In this case, pi* and pi_theta in Eq. (7) swap, and the RL agent minimizes the cost function g_f(V_w(s,a)). This cost function does not directly depend on f*, similarly to the reward function r(V_w(s,a)) in Eq. (8). This swapping is simpler and more flexible than re-parameterizing, while achieving the same goal as f-VIM-sigmoid. This swapping method should be discussed and compared against the proposed methods.\n\n- Need stronger baseline methods for ILfO: \nThe paper should evaluate f-VIMO-sigmoid against stronger baselines, e.g., forward adversarial IL (Sun et al., 2019) which outperforms GAIL-based methods in the ILfO setting. \n\n[1] Wen Sun, Anirudh Vemula, Byron Boots, and J Andrew Bagnell. Provably efficient imitation learning from observation alone. ICML, 2019.\n\n- Using the f-divergence for ILfO is not well motivated: \nThe paper does not provide good motivations for using f-divergence in ILfO. This makes the paper quite difficult to follow, since there is no connection between f-divergence and ILfO.\n\n- The experiments focus on evaluating existing methods rather than the proposed methods: \nSpecifically, the proposed methods are evaluated with only one choice of the divergence (TV) in Figure 2. Meanwhile, most of Section 6 and results (Figure 3 and 4, and additional results in the appendix) focus on evaluating the existing methods (f-VIM and f-VIMO) with different choices of divergence. \n\n- The experiments in Figure 2 do not support the claim regarding stability: \nThe paper claims to improve stability of IL by using the proposed re-parameterization. However, the experimental results do not support this claim, and the questions asked in Section 5 are not related to this claimed. Instead, it seems that re-parameterization helps avoiding local optima (possibly due to a biased reward function, see below), while stability is improved by regularizing the discriminator. I could not see how the re-parameterization improves the policy stability as claimed. \n\n- The experiments in Figure 2 seem unfair, since TV-VIM-sigmoid incorporates priors about survival bonuses: \nSpecifically, TV-VIM-sigmoid uses sigmoid which yields strictly positive rewards, while TV-VIM uses tanh which yields positive and negative rewards. As discussed by Kostrikov et al., 2019, using strictly positive rewards incorporate strong priors about the survival bonuses, which exist in the locomotion task used in the experiments. Therefore, TV-VIM-sigmoid uses strong priors while TV-VIM does not. In order to make the comparison fairer, I suggest the authors to evaluate TV-VIM with sigmoid reward output, or include environments that do not have survival bonuses.\n\n[2] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. ICLR, 2019\n\n* Minor comments: \n- The abstract is long and could be shorten. \n- Figures are too small and difficult to see, especially the legends.\n- Table 1 should describe the form of f in addition to its conjugate.\n- The title of the Algorithm 1 should be f-VIMO-sigmoid instead of f-VIMO. \n\n** Update after response.\nI read the response. I thank the authors for clarifying the claims as well as the new experiments with the swap formulation. However, improving clarity of the claims is considered a major revision. I still keep the vote of rejection. \n\nRegarding reward bias. As the authors acknowledge, the improvement achieved by using reparameterization+sigmoid can be explained by two equally-plausible reasons: 1) reparameterization+sigmoid improves stability (as claimed) and 2) sigmoid gives biased rewards. The issue here is that we do not know which is the actual reason, given the current experiments in the paper. As I commented, evaluating TV-VIM with sigmoid but without reparameterization will help address this issue.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgezmwptS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The submission performs empirical analysis on f-VIM (Ke, 2019), a method for imitation learning by f-divergence minimization. The paper especially focues on a state-only formulation akin to GAILfO (Torabi et al., 2018b). The main contributions are:\n1) The paper identifies numerical proplems with the output activations of f-VIM and suggest a scheme to choose them such that the resulting rewards are bounded.\n2) A regularizer that was proposed by Mescheder et al. (2018) for GANs is tested in the adversarial imitation learning setting.\n3) In order to handle state-only demonstrations, the technique of GAILfO is applied to f-VIM (then denoted f-VIMO) which inputs state-nextStates instead of state-actions to the discriminator.\n\nContribution / Significance:\nI think that the contributions of the paper are rather marginal. I do think that the choice of output activation may have large impact on the performance and it seems that the activation suggested by Ke et al. (2019) are somewhat arbitrary. However, the activations proposed in the current submission are also seem somewhat arbitrary and are not accompanied by any theoretical analysis. \n2) and 3) are marginal combinations of existing work that are only insufficiently evaluated and do not seem particular effective.\nHence, I think that the current submission is of rather limited interest.\n\nSoundness:\nThe \"reparametrization\" of f-VIM is motivated based on exploding policy gradients when using unbounded reward functions, especially when minimizing the (R)KL. \nI am not convinced by this motivation, given that GAIL and AIRL (which approximatly minimizes the RKL) use unbounded reward functions and do not seem to suffer from such problems.\n\nEvaluation:\nThe effect of the \"reparametrization\" is only evaluated for total variation. The regularization loss is only evaluated with a single fixed coefficient of 10 on all experiment. I think that a sweep over the coefficient would be mandatory, especially given that current experiments do not show a clear benefit of the regularization loss (the regularized version performs worse on roughly half of the experiments). \nWhen learning from observations only, the submission only evaluates the proposed combination of f-VIM and GAILfO. However, it seems like it would be perfectly possible to handle state-only observations by simply making the discriminator independent of the actions, i.e. using D(s,a) = D(s). Such technique matches the marginal distributions over states and is commonly applied to GAIL, e.g. by Peng et al. [1].\nIt is not clear whether the reported problems of learning from observations only is really a general problem of the learning setting (as claimed in the submission) or a problem of the proposed method.\n\nClarity:\nThe paper is well written and easy to follow. Using different linestyles to distinguish the learning with regularization versus without regularization would help a lot.\n\nDecision:\nDue to the marginal contribution and the insufficient evaluation I have to recommend rejection.\n\n\nQuestion:\nI am maily interested in the authors response to my critique, especially regarding\n- the choice not to compare with state-only f-VIM, and\n- the motivation of the proposed output activations.\n\n\n[1] Peng, Xue Bin, et al. \"Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow.\" arXiv preprint arXiv:1810.00821 (2018). ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission performs empirical analysis on f-VIM (Ke, 2019), a method for imitation learning by f-divergence minimization. The paper especially focues on a state-only formulation akin to GAILfO (Torabi et al., 2018b). The main contributions are:
1) The paper identifies numerical proplems with the output activations of f-VIM and suggest a scheme to choose them such that the resulting rewards are bounded.
2) A regularizer that was proposed by Mescheder et al. (2018) for GANs is tested in the adversarial imitation learning setting.
3) In order to handle state-only demonstrations, the technique of GAILfO is applied to f-VIM (then denoted f-VIMO) which inputs state-nextStates instead of state-actions to the discriminator.

The reviewers found the submitted paper hard to follow, which suggests a revision might make more apparent the author's contributions in later submissions of this work. 
Decision: Reject