Adversarial Imitation Learning via Boosting
Authors: Jonathan Daniel Chang
Dhruv Sreenivas
Yingbing Huang
Kianté Brantley
Wen Sun
Venue: ICLR 2024 poster (2024)
Keywords: Adversarial Imitation Learning
Boosting
Reinforcement Learning
Primary area: reinforcement learning

Abstract: Adversarial imitation learning (AIL) has stood out as a dominant framework across various imitation learning (IL) applications, with Discriminator Actor Critic (DAC) demonstrating the effectiveness of off-policy learning algorithms in improving sample efficiency and scalability to higher-dimensional observations. Despite DAC’s empirical success, the original AIL objective is on-policy and DAC’s ad-hoc application of off-policy training does not guarantee successful imitation. Follow-up work such as ValueDICE tackles this issue by deriving a fully off-policy AIL objective. Instead in this work, we develop a novel and principled AIL algorithm via the framework of boosting. Like boosting, our new algorithm, AILBoost, maintains an ensemble of weighted weak learners (i.e., policies) and trains a discriminator that witnesses the maximum discrepancy between the distributions of the ensemble and the expert policy. We maintain a weighted replay buffer to represent the state-action distribution induced by the ensemble, allowing us to train discriminators using the entire data collected so far. Empirically, we evaluate our algorithm on both controller state-based and pixel-based environments from the DeepMind Control Suite. AILBoost outperforms DAC on both types of environments, demonstrating the benefit of properly weighting replay buffer data for off-policy training. On state-based environments, AILBoost outperforms ValueDICE and IQ-Learn, achieving state-of-the-art performance with as little as one expert trajectory.

Published as a conference paper at ICLR 2024
ADVERSARIAL IMITATION LEARNING VIA BOOSTING
Jonathan D. Chang
Dhruy Sreemivas'
Department of Computer Science
Department of Computer Science Cornell University ds84 4 40corneell.....1.cmmm edu
Cornell University jdc39 (133660com 60corelll........ooo edu
Yingbing Huang
Kianté Brantley
Department of Electrical and Computer Engineering University of Illinois Urbana- Chaa-hhap yh213 aenoo.seeee
Department of Computer Science Cornell University kdb82 @correllllll 1.edu
Wen Sun
Department of Computer Science Corelll University ws4550 coonccnne 1. edu
ABSTRACT
Adversarial imitation learning (AIL) has stood out dominant framework across various imitation learning (IL) applications, with Discriminator Actor Critic (DAC) (Kostrikov et al. 2019) demonstrating the effectiveness of off off-policy learning algo- rithms in improving sample efficiency and scalability to higher-dimensional obser- vations. Despite DAC's empirical success, the original AIL objective is on-policy and DAC's ad-hoe application of off-policy training does not guarantee successful imitation (Kostrikov et al., 2019 2020). Follow-up work such as ValueDICH (Kostrikov et al. (2200) tackles this issue by deriving a fully off-policy AIL ob- jective. Instead this work, we develop novel and principled AIL algorithm via the framework of boosting. Like boosting, our new algorithm, A AILBoost, maintains ensemble of properly weighted weak learners (i.e., policies) and trains a discriminator that witnesses the maximum discrepancy between the distributions of the ensemble and the expert policy. We maintain a weighted replay buffer represent the state-action distribution induced by the ensemble, allowing us to train discriminators using the entire data collected so far. In the weighted replay buffer, the contribution of the data from older policies properly discounted with the weight computed based on the boosting framework. Empirically, we evaluate algorithm both controller state-based and pixel-based environments from the DeepMind Control Suite. AILBoost outperforms DAC on both types of environments, demonstrating the benefit of properly weighting replay buffer data for off- off-policy training. On state-based environments, AILBoost outper- forms Val uelDCE and Io-Learn(Garg al. (202)), achieving competitive performance with as little as one expert trajectory.
INTRODUCTION
Imitation learning (IL) is promising paradigm for learning general policies without rewards from
demonstration data, achieving remarkable success autonomous driving (Bronstein et al. 2022
Pomerleau 1988), video games (Baker et al.) [2022] Shah et al. 2022) and graphics (Peng et al. 2021).
Adversarial Imitation Learning (AIL) is an incredibly successful approach for imitation learning
(Ho & Ermon 2016 Fu et al. 2018 Kostrikov et al.) 2019 Ke et al. (020). These methods cast IL as a distribution matching problem whereby the learning agent minimizes the divergence between the expert demonstrator's distribution and the state-acti aate-action distribution induced by the agent. First
"Equal Contribution

---

Published as a conference paper at ICLR 2024
introduced by (Ho & Ermon 2016), this divergence minimization can be achieved in an iterative procedure reminiscent of GAN algorithms (Goodfellow et al. (014) with our learned reward function and policy being the discriminator and generator respectively.
Originally, a limitation of many AlL methods was that they were on-policy. That is, for on-policy AIL methods like GAIL (Ho & Ermon 2016) and AIRL (Fu et al. 2018), the algorithm would draw fresh samples from the current policy in every iteration for the distribution matching process while discarding all old samples, rendering the sample complexity of these algorithms to be prohibitively large in many applications. Follow-up works (Kostrikov et al. 2019) Sasaki et al. 2019) attempt to relax the on-policy requirement by creating off-policy methods that utilize the entire history of observed data during the learning process. This history is often represented by replay buffer and methods such as Discriminator Actor Critic (DAC) show large improvements and sample complexity over their on-policy counterparts. However, these methods modify the distribution matching objective as a divergence minimization between the replay buffer's and the expert's 's distribution, losing the guarantee of matching the expert's behavior.
scalability
Algorithms like ValueDICE (Kostrikov et al. 2020) address this problem by deriving a new formulation of the AIL divergence minimization objective to be entirely off-policy. VaCEeDICE, however, in principle relies on the environments to have deterministic dynamics In this work, we consider a new perspective towards making AIL off--olliy. We present a new principled off- off-policy All. algorithm,, AT ABoos,,, via the gradient boosting framework (Mason et al. 1999). A ATLBoost maintains ensemble of properly weighted weak learners or policies well weighted replay buffer to represent the state-action distribution induced by our ensemble. Our distribution matching objective is then minimize the divergence between the weighted replay buffer's distribution (i.e., the state-action distribution induced by the ensemble) and the expert demonstrator's 's distribution, making the divergence minimization problem an off aff-policy learning problem. Similar to boosting and gradient boosting, at every iteration, we aim to find a weak learner, such that when added to the ensemble, the divergence between the updated ensemble's distribution and the expert's distribution decreases. In other words, our approach can be understood as performing gradient boosting in the state-action occupancy space, where black-box RL optimizer is used a weak learning procedure to train weak learners, i.c., policies.
We evaluate AILBoos: on the DeepMind Control Suite (Tassa et al. 2018) and compare against a range of off-policy AIL algorithms (Behavior cloning, ValueDICE, DAC) as well as state-of-the art IL algorithm, IQ-Learn. We show that our algorithm is comparable to or more sample efficient than tfte-of-thheaa II. algorithms in various continuous control tasks, achieving strong imitation performance with as little as one expert demonstration. We also show that our approach scales to basedn- partially observable domains, where we again outperform DAC.
2 RELATED WORKS
Off-policy and Offline IL There has also been a wide variety of research conducted on off-policy and offline IL, where the goal is to be either more sample efficient safer by utilizing a replay buffer or not collecting any environmental transitions during training, respectively. The most prominent of (DAC) (Kostrikov al.,
said methods, and the closest to our work, is Discriminator-Actor- 2019). which essentially replaces the on-policy RL algorithm in the ad versariaal IL setup with an off-policy one such as DDPG (Lillicrap al. 2019)) or SAC (Haarnoja et al. 2018). However, as mentioned previously, DAC doesn't i necessarily guarantee a distribution match between the expert and the learned policy, prompting further work to be done. Further work has primarily focused on weighting on-policy and off-policy data differently in both the policy update and the discriminator update, Va ValueDICE (Kostrikov et al., 2020) mitigates this problem by deriving an objective from the original distribution matching problem that only requires off-policy samples to compute. More recently, methods such as Ie-Learn (Garg et al. (02T] have been developed to learn soft Q functions over the environment space, which encodes both reward and a policy for inverse reinforcement learning, and model-based methods such as IL (Rafailov et al. 2021 have shown that using expressive world models (Hafner et al. 2020) leads to strong imitation results in
One cannot derive an unbiased estimate of the objective function proposed in ValueDICE unless it has infinite expert samples and the transition is deterministic (Kostrikov et al. 2020). See section for more detailed discussion.
2

---

Published as a conference paper at ICLR 2024
domains with high- high-dimensional observations. Other off off-policy IL works include SoftDICE Sun ct al. 2021 SparseDICE (Camacho et al. 2021). and AdVIL/AdRRILDAAeuulL (Swamy et al. 2021
Orthogonally, on the offline side, where environment interaction is prohibited. works both on the model-based side (Chang et al.] 202T) and the model-free side (Kim et al., 2022 Yu et al. 2023) has shown that distribution matching is still possible in these settings. These approaches generally operate either by learning a transition model of the environment, with which roll out in to do policy optimization (Chang et al. 202T). or optimizing a modified version of the objective introduced in (Kostrikov et 20200) by using samples from the suboptimal offline dataset as opposed on-policy samples for computation.
Boosting style approach in deep learning & RL The idea of using boosting for policy learning is not new in the deep learning or reinforcement learning literature. On the deep learning side, AdaGAN (Tolstikhin et al., 2017) apply standard adaptive boosting to GANs (Goodfellow et al.) (014) to address and fix issues such as mode collapse, while concurrent work (Grover & Ermon 2017) showed benefits of boosting in general Bayesian mixture models. In RL, the conservative policy iteration (CPI) (Kakade & Langford| 2002) can be understood as performing gradient boosting in the policy space (Scherrer & Geist. 2014). The authors in (Hazan et al.] (019)) use gradient boosting style approach to learn maximum entropy policies. In this work, we perform gradient boosting in the space of state-action occupancy measures, which leads to principled off-policy IL approach.
3 PRELIMINARIES
We consider a discounted infinite horizon MDP . = P.A, (o) where S is the state of states, A is the set of actions, r' : A + I is the reward function and r(s, a) is the reward for the given state-action pair, 7 (0, 1) is the discount factor, μl E ASS) is the initial state distribution, and P: P:SxAr< + (S)) is the transition function. A policy π S - ((A) interacts said MDP, creating majectories composed of state-action pairs {(s), 44)TT= We denote d. to represent the state- action visitation distribution induced by π at timestep t and d= = (1 1--7 En Y'd as the average state-action visitation distribution induced by policy и. We define the value function and Q-function of our policy as V (s) = F E== The goal of RL is to find policy that maximizes the expected cumulative reward.
( (r(()))) = s] and (x,(sa)) = r(s, a) + (( [V* (8'))
In imitation learning, instead of having access to the reward function, we assume access to demon- strations 2e (si,a;)) a.)).= from an expert policy n² that our policy can take advantage of while training. Note that n might not necessarily be a Markovian policy. It is possible that x² is an ensemble of weighted Markovian policies, i.e., n" {aa, T with a, 2 0. E, a; = 1, which means that for each episode, n² will first randomly sample a policy т. with probability a, at t = 0, and then execute My for the entire episode (i.e., no switch to other policies during the execution for an episode). It is well known that the space of state action distributions induced by such ensembles larger than the space of state-action distributions induced by Markovian policies (Hazan al., 2019). The goal in IL is then to learn a policy that robustly mimics the expert. The simplest imitation learning algorithm to address this issue is behavior cloning (BC): argmin E( [<(m()), a) where f is classification loss and II is our policy class. Though this objective is simple, it is known to suffer from covariate shift at test time (Pomerleau 1988; Ross et al. 2011). Instead of minimizing action distribution divergence conditioned
expert states, algorithms such inverse RL (Ziebart et al. 2008) and adversarial II. (Ho & Ermon 2016; Finn et al., 2016 Ke et al., 2020; (unn et al. 2019) directly minimize some divergence metrics between state-action distributions, which help address the covariate shift issue (Agarwal et al. 2019)..
3.1 ADVERSARIAL IMITATION LEARNING (AIL)
The goal of AIL is to directly minimize some divergence between some behavior policy state-action visitation d' and an expert policy state-action visitation d= The choice of divergence results in variously different AIL algorithms.
The most popular AIL algorithm is Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) which minimizes the JS-divergence. This algorithm is on-policy adversarial imitation
3

---

Published as a conference paper at ICLR 2024
learning algorithm that connects Generative Adversarial Networks (GANs) (Goodfellow al. 2014) and maximum entropy IRL (Ziebart et al. 2008). GAIL trains a binary classifier called the discriminator (ss to distinguish between samples from the expert distribution and the policy generated distribution. Using the discriminator to define a reward function, GAIL then executes an on-policy RL algorithm such Trust Region Policy Optimization (TRPO) (Schulman et al.] (017a) or Proximal Policy Optimization (PPO) (Schulman et al., 2017b) to maximize the reward. That gives us the following adversarial objective:
min max Egga= Dlog D(s,, a)] + Rs [log(1 ((), (T) D
where (TT) is an entropy regularization term. The first term in eq. (I) can be viewed as a pseudo reward that can be optimized with respect to the the policy on-policy samples. Note that GAIL typically optimizes both policies and discriminators using on-policy samples, making it quite sample inefficient. Using different divergences, there various reward functions that can be optimized with this framework (Orsini et al., 2021). In this work, while our proposed approach in general is capable of optimizing many common divergences, experiments. Reverse KL divergence has been studied in prior works including Fuu et al. (2018): Ke al. (2020). But different from prior works, we propose an off- off-policy method for optimizing reverse KL by leveraging the framework of boosting.
mainly focus on reverse KL divergence in our
3.2
DISCRIMINATOR ACTOR CRITIC (DAC)
One reason GAIL, need lot of interactions with the environment to learn properly is because of the de- pendency on using on-policy approaches to optimize discriminators and policies. In particular, GAIL does not reuse any old samples. Discriminator Actor Critic (DAC) (Kostrikov et al. 2019) extends CAIL algorithms to take advantage of off- off-policy learning to optimize the discriminators and policies.
DAC introduces a replay buffer R to represent the history of transitions observed throughout training in the context of IRL. This replay buffer allows DAC to perform off- off-policy training of the policy and the discriminator (similar to (Sasaki al. 2019h). Formally, DAC optimizes its discriminator with the objective:
max Ea [log D(s, a)] + E, [log(1 - D(s,a)))
(2)
D
where this objective minimize the divergence between the expert distribution and the replay buffer R distribution. Intuitively, this divergence does not strictly capture the divergence of our policy distribution and the expert distribution, but mixture of evenly weighted policies learned up until the current policy. To rigorously recover a divergence between our policy distribution and the expert distribution we need to apply importance weights:
min max MaxxE,,a<R "I
pr(s,)) log a) + Eg [log(1 D(s, (H()). While this objec- (s.a)
tive recovers the on-policy objective of GAIL (Equation D). the authors note that estimating the density ratio is difficult and has high variance in practice. Furthermore, they note that the not using importance weights (Equation (2) works well in practice, but does not guarantee successful imitation, especially when the distribution induced by the replay buffer, R, is far from our current policy's 's state-action distribution. This is fundamental problem of DAC.
3.3 VALUEDICE
ValueDICE (Kostrikov al., 2020) was proposed to address the density estimation issue of off- policy AIL algorithms formalized in DAC (see section 3.2). ValueDICE aims to minimize the reverse KL divergence written in its Donsker- Varadhan (Donsker & Varadhan 1983) dual form:
-KL((=(ld"))=
min log (s,a)aai [c=(s.a) B( (a(s,a)
ValueDICE performs a change of variable Motivated from Due 1 DICE Nachum et al. 2019),
using the Bellman operator 2 with respect to the policy n: xx(s a) () BB (s) resulting
AA bellman operator B= is defined follows: given any function f(x, we have B f(s, a) r(). + Pe fs', (T')), as
4

---

Published as a conference paper at ICLR 2024
the following objective:
max min U=SxA
l,g (exp (v(s,0) B (T, a))] - (1) [u(so, ao)]
(4)
Now the objective function does not contain on- on-policy distribution d' (in fact only the initial state distribution Mo and the expert distribution). Despite being able to only using d and foo the authors have identified two aspects of the objective that will yield biased estimates. First, the first expectation has logarithm outside of it which would make mini-batch estimates of this expectation biased. Moreover, inside the first expectation term, we have v(s, a) vs, a) with 8 being the Bellman operator. This limits ValueDICE's objective to only be unbiased for environments with deterministic transitions. This is related to the famous double sampling issue in TD learning. Although many popular RL benchmarks have deterministic transitions (Bellemare al. 2013 Tassa et al. 2018 Todorov et al. 2012), this was a limitation not present in the GATI.
In this work, we take different perspective than Val ueDICE to derive off- off-policy AlL algorithm. Different from Va LueDICE, our approach is both off-policy and is amenable to mini-batch updates even with stochastic environment transition dynamics.
ALGORITHM
Our algorithm, Adversarial Imitation Learning via Boosting (A (AILBoost)- motivated by classic gra- dient boosting algorithms (Friedman [200]] Mason et al. 1999) attempts to mitigate a fundamental issue related off-policy imitation learning formalized in DAC (see section 3.2) The key idea is to treat learned policies as weak learners, form an ensemble of them (with a proper weighting scheme derived from gradient boosting perspective), and update the ensemble via gradient boosting.
Weighted policy ensemble. Our algorithm will learn a weighted ensemble of policies, denoted as क == {ar, in with a; 2 0, L; a; and म; being some Markovian policy. The way the mixture works is that when executing ग, at the beginning of an episode, a Markovian policy n; is sampled with probability as, and then n. is executed for the entire episode (i.e., no policy switch in an episode). Note that π itself is not a Markovian policy anymore due to the sampling process at the beginning of the episode, and in fact, such mixture policy's induced state-action distribution can be richer than that from Markovian policies (Hazan et al. 2019). This is consistent with the idea of boosting: by combining weak learners, i.c., Markovian policies, we form a more powerful policy. Given the above definition of we immediately have dt ;= E. a,d"i i.e., the weighted mixture of the state-action distributions induced by Markovian policies T..
Notation wise, given a dataset DD we denote ED [xx) as the empirical function average across the dataset, i.e., [x[s(x)] = EnED /(x) (D].
4.1 AILBOOST: ADVERSARIAL IMITATION LEARNING VIA BOOSTING
We would like to minimize the reverse KL divergence between our policy state-action dis- tribution dt and the expert distribution dxL denoted by E(dTT dd KL
Eea d (s,s a) (n(d"(ss a)/d" (s, a)). The reasons that we focus reverse KL is that (1) it has been argued that the mode secking property of reverse KL is more suitable for imitation learning (ee et al. 2020). (2) reverse KL is on-policy in nature, i.c., it focuses on minimizing the divergence of our policy's action distribution and the expert's at the states from our policy, which help address the covariate shift issue, and (3) the baselines we consider in experiments, DAC and Val ICE, all minimize the reverse KL divergence such AIRI in practice At a high level, our approach directly optimizes E(dT. d= via gradient boosting (Mason et al. 1999) in the state-action occupancy space. Our ensemble induces the following mixture state-action occupancy measure:
d=
a.d"i aa 20. 0.
To compute new weak learner T1+1, we will first compute the functional gradient of loss & with respect to dxH i.e., ve(d,d" Ha=d.. The new weak learner u++1 is learned via the following
"See the official repository
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 AILBoOST (Adversarial Imitation Learning via Boosting) Require: number of iterations T' expert data D weighting parameter a
1: Initialize T weight = L, replay buffer B =
2: for fort= l.... do
3: Construct the t-th dataset D, = {(s}. az) f where 8j. 0j ~ d Vj.
4: Compute discriminator 9 using the weighted replay buffer:
P a.. (-ex(((((,))) 9 argmax
Σ Ra [0(s,
(5)
5: Set B < BUD,
6: Compute weak learner m441 via an off-policy RL approach (e.g., SAC) on reward -ğ(s,a)
with replay buffer B
7: Set O; f a - a) for S t, and at+1 a
8: end for
9: Return Ensemble π {(o), T1)
optimization procedure: T = argmax (d VE(d, d" d"d=aa))) Namely, we aim to search for new policy T++1 such that its state-action occupancy measure - is aligned with the negative gradient - V as much as possible. Note that the above optimization problem can be understood as an RL procedure where the reward function is defined - Vidd, dd=) E SRS Once we compute the weak learner Tt-1, we mix it into the policy ensemble with a fixed learning rate a E (0, 1) denoted as d= = (1 - add + ad Note that the above mixing step can be interpreted as gradient boosting in the state-action occupancy space directly: we re-write the update procedure as d= d + (dd d= where the ascent direction d d is approximating the (negative) functional gradient - VE, since argmax, (d" d= V)) = πt+1 by the definition of 4z+1. It has been shown that such procedure guaranteed to minimize the objective function (i.e., reverse KL in this case) as long as the objective is smooth (our loss & will be smooth as long as d is non-zero everywhere) (e.g., see (Hazan et al. 2019) for the claim)
Algorithmically, we first express the reverse KL divergence in its variational form Nowozin et al. Ke et al. 2020):
2016
KL (dd Idd" ) max E. Es |- exp(e(s, a))] Es g(s,a)
where S x A + R discriminator. The benefit of using this variational form is that computing the functional (sub)-gradient of the reverse KL with respect to d is easy, which is I = argmax, [E. [-exp ex a))l + Eggard g(s, i... ie.. we have 9 being a functional sub- gradient of the loss KL(d"[ld")) with respect to dtt The maximum discriminator will serve reward function for learning the next weak learner 1 1. that
M++1 argmax Eg [s, a)] (d",
To compute ğ in practice, we need unbiased estimates of the expectations via sample averaging which can be done easily in our case. The expectation _ay can be easily approximated by the expert dataset D To approximate Es where d is a mixture distribution, we maintain replay buffer D; for each weak learner x; which contains samples s, ~ dd and then weight D; via the weight og associated with т; In summary, we optimize g as shown Eq. 5j (the highlighted red part denotes the empirical expectation induced by weighted replay buffer). The optimization problem in 5g can be solved via stochastic gradient ascent on g With ÿ. we can optimize for me+1 using any off- RL algorithm, making the entire algorithm off tif-policy. In our experiments, we use SAC the RL oracle for argmax,, Eg Ú a)). Once T++1 is computed, we mix 7&+1 into the mixture,
'Note that similar AdaBoost, each weaker not directly optimizing the original objective, but the weighted combination of weaker learners optimizes the original objective function the reverse KI. our case. 'Note that unlike ValueDICF, here we can easily finite number of samples to obtain unbiased estimate of the loss by replacing expectations by their corresponding sample averages.

---

Published as a conference paper at ICLR 2024
and adjust the weights of older policies accordingly, i.e., a++1 a, and a; f- a; a), Vi t. Note that this weighting scheme ensures that older policies get less weighted in the ensemble.
Remark 1. The use of SAC as the weak learning algorithm and the new way of computing discrimi- nator from Eq. make the whole training process completely off--ooliy. Particularly, unlike most adversarial II approaches, which compute discriminators by comparing on-policy samples from the latest policy and the expert samples, we train the discriminator using all the data collected so far (with proper weighting derived based on the boosting framework). The connection to boosting and the proper weighting provides a principled way of leveraging off-policy samples for updating discriminators. As we will show, compared to DAC which also uses off-policy samples for training policies and discriminators, our principled approach leads to better performance.
Alg 4.1 AILBoost, summarizes the above procedure. In Line TO use SAC as the RL oracle for computing the weak learner. In practice, we do not run SAC from scratch every time in Line [O] Instead, SAC maintains its own replay buffer which contains all interactions it has with the environ- ment so far. When computing *t+1, we first update the reward in the replay buffer using the latest learned reward function - 9. and we always warm start from nT. We include the detailed algorithmic description in Appendix
Memory cost. Note that at the end, our algorithm returns a weighted ensemble of Markovian policies. Comparing to prior works such as DAC, the maintenance of weak learners may increase additional memory cost. However, the benefit of the weighted ensemble that it induces richer state-action distributions than that of Markovian policies. In practice, if memory cost really becomes burden (not in our experiments even with image-basee control policies), we may just keep the latest few policies (note that very old policy has exponentially small weight anyway).
EXPERIMENTS
In this section we aim to empirically investigate the following questions: (1) How does AILBoost
perform relative to other off-policy and state-of-the-art IL methods? (2) Does AILBoost enjoy the sample complexity and scalability benefits of modern off-policy IL methods? (3) How robust is AILBoost across various different adversarial training schedules?
We evaluate AILBcost on 5 environments on the al.)
Task
Difficulty
DeepMind Control Suite benchmark<Tassa 2018): Walker Walk, Cheetah Run, Ball in Cup Catch, Quadruped Walk, and Humanoid Stand. For each game, we train an expert RL agent using the environment's reward and collect demon- strations which we use as the expert dataset throughout our experiments. We compare AILBoost against the following baselines: DAC, an empirically succesful off- policy II. algorithm; IQ-Learn, rn, a state-of-the-ae IL algorithm; Val ueD ICE, another off- off-policy II. method; and BC on the expert data used across all algorithms. We emphasize our comparison to IQ-Leard it has been shown outperform many other imitation learning baselines (e.g., SQIL (Reddy et al. 2019) across a variety of control tasks (Garg et al. 202T).
Ball in Cup Catch Wa Walker Walk Cheetah Run Quadruped Walk Medium
Easy
Easy
Medium
Humanoid Stand
Hard
Table 1: Spread of environments evaluated from the DeepMind Control Suite with hard- ness designations from (Yarats et al. 2022).
The base RI. algorithm we used for training the expert, well for A ATLBosst and DAC, was SAC for controller stte-bbsed experiments and DrQ-v2 (Yarats et al. 2022) for image-based experiments. For IQ-Learn and ValueDICE, we used their respective codebases and hyperparameters provided by the authors and both methods use SAC as their base RL algorithm. Please refer to Appendix B for experimental details, training hyperparameters, and expert dataset specifications.
5.1 CONTROLLER STATE-BASED EXPERIMENTS
Figure shows our aggregate results across the five DeepMind Control Suite (DMC) tasks that we tested on. We chose these five tasks by difficulty as shown in Table [] For evaluation, we follow the recommendations of (Agarwal et al. (02T) and report the aggregate inter-quartile mean, mean, and optimiality gap of A A.Boosst and all the baselines on the DMC suite with 95% confidence intervals.
7

---

Published as a conference paper at ICLR 2024
IQM
Mean
Optimal Gap
AlLBoost
DAC
IQ-Learn
10 Demos
ValueDICE
BC
0.60
0.75
0.90
0.60
0.75
0.90
0.15
0.30
0.45
AlLBoost
DAC
IQ-Leam
5 Demos
ValueDICE
BC
0.4
0.6
0.8
1.0
0.45
0.60
0.75
0.90
0.15
0.30
0.45
0.60
AlLBoost
DAC
IQ-Leam ValueDICE
1 Demo
BC
0.25
0.50 0.75 1.00
0.6
0.8
0.2
0.8
Expert Normalized Score
Figure 1: Aggregate metrics on DMC environments with 95% confidence intervals (CIs) based on environments spanning easy, medium, and hard tasks. Higher inter-quartile mean (IQM) and mean scores (right) and lower optimality gap (left) is better. The CIs were calculated with percentile bootstrap with stratified sampling over three random seeds and all metrics are reported on the expert normalized scores. AILBoost outperforms DAC, ValueDICE, IQ-Learn, and BC across all metrics, amount of expert demonstrations, and tasks.
Ball In Cup Catch
Cheetah Run
Humanold Stand
1000
1000
1000
Meen Score
800
600
600
200
0.00
0.25
0.75
1:00
125
0.00
0.25
0.50
1.00
1.25
Samples
1e8
Samples
1e6
Samples es
1e8
Expert
BC
IQ-Leam
ValueDICE
DAC
AlLBocst
Figure 2: Learning curves with 1 expert trajectory across 3 random seeds. Note AILBoost successfully imitates expert on all environments where other baselines fail and achieves better sample complexity than DAC. Note that when the environment difficulty level increases, our method shows a larger performance gap compared to baselines (e.g., humanoid stand).
We find that A.LBoost not only outperforms all baselines but also consistently matches the expert with only expert trajectory.
When we inspect the 1 trajectory case closer, Figure 2 shows the learning curves on three repre- sentative (I easy, I medium, hard task) environments where we see AILBoost maintain high sample efficiency and strong imitation while state-of-the-an baselines like IQ IQ-Learn completely fail on Humanoid stand. Finally, we note that AILBoost greatly outperforms ValueD ICE which aimed to make AIL off- policy from different perspective. We refer readers to Figure in the appendix for the learning curves on all five environments with different numbers of expert demonstrations.
8

---

Published as a conference paper at ICLR 2024
Walker Walk
Cheetah Run
600
600
Mean Score
400
200
50000 100000 150000 200000 250000 Samples
50000 100000 150000 200000 250000 Samples
Expert
AlLBoost
Figure 3: Image based:
performance on image-based DMC environments, Walker Wal and Cheetah Run, comparing A LBoost, DAC, and BC on three random seeds.
5.2 IMAGE-BASED EXPERIMENTS
Figure 3 demonstrates the scalability of AILBoost subset of environments with 10 expert trajectories. For these experiments, we use DrQ-v2 (Yarats et al. 2022) as the underlying off-policy RL algorithm for both DAC and AILBoost. On Walker Walk and Cheetah Run, we see comparable to better performance than DAC demonstrating that our boosting strategy successfully maintains the empirical, scaling properties of DAC. Furthermore, algorithms show the versatility of AILBoost for IL.
of different off off-policy RL
5.3 SENSITIVITY TO GRADIENT-BASED OPTIMIZATION FOR WEAK LEARNERS AND DISCRIMINATORS
Our algorithm relies on solving optimization problems in Eq. 6 and Eq. for weak learners and discriminators, where weak learner is optimized by SAC and discriminators optimized by SGD. While it is hard to guarantee in general that we can exactly solve the optimization problem due to our policies and discriminators are both being non-convex neural networks, we in general found that approximately solving Eq. 6 and Eq. via gradient based update is enough to ensure good AILBoost across a variety optimization schedules. Overall,
performance. this section, we find that AILBoost to be robust to optimization schedules approximately optimizing Eq. 6 and Eq. [5] with sufficient amount of gradient updates ensures successful imitation; however, there exists sample complexity cost when over-optimizing either the discriminator or the policy.
Figure shows our investigation how sensitive AILBoost is to different optimization schedules for both the policy and discriminator on two representative DMC environments. In particular, we test with expert demonstrations, where we vary the number of discriminator and policy updates. We test the following update schemes:
1000 policy updates per 100 discriminator updates
1000 policy updates per 10 discriminator updates
1000 policy updates per discriminator update
100 policy updates per 100 discriminator updates
These ranges, test various optimization schemes around the schedule that we chose for the main results. We find that the more policy updates we do per discriminator update, the algorithm becomes significantly less sample efficient despite asymptotically reaching expert performance. We also found that insufficient amount of updates the discriminator general hurts the performance. This is also expected since insufficient update on the discriminators may result a ğ which does not optimize Eq. 5 well enough.

---

Published as a conference paper at ICLR 2024
Ball in Cup Catch
Walker Walk
1,000
Meal Score
600
Expert 1000
200
1000 P. 10
100 P. 1.0
Samples
Samples
Figure 4: Policy and Discriminator Update Schedules: Learning curves for AILBoost on two representative DMC environments, Walker Walk and Ball in Cup Cat ch, when optimizing with varying policy and discriminator update schemes across seeds.
CONCLUSION
In this work, present fully off-policy adversarial imitation learning algorithm, AILBoost. Different from previous attempts at making AlL off-policy, via the gradient boosting framework, AILBoost provides principled way of re-using old data for learning discriminators and policies. We show that our algorithm achieves state-of-the-art performance on state-based results on the DeepMind Control Suite while being able to scale high-dimensional, pixel observations. We excited to extend this framework to discrete control as well as investigate imitation learning from observations alone under this boosting framework.
ACKNOWLEDGEMENTS
We would like acknowledge the support of NSF under grant IIS-2154711, CAREER 2339395, and Cornell Infosys Collaboration. Jonathan Chang is supported by Linkedln under the Linkedln- Cornell Grant. Kiante Brantley supported by NSF under grant No. 2127309 the Computing Research Association for the CIFellows Project.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Alckh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept. UW Seattle, Seattle, WA, USA, Tech. Rep, 32, 2019.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforeement learning at the edge of the statistical precipice. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 39304-29322 Curran Asso- ciates, Inc., 2021, URL URLhttps://prrceeddnns neurips co/paper files/paper Paper.
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. doi: 10. 10.1613/jair.391.. URL /
3912
Eli Bronstein, Mark Palatucci, Dominik Notz, Brandyn White, Alex Kuefler, Yiren Lu, Supratik Paul, Payam Nikdel, Paul Mougin, Hongge Chen, Justin Fu, Austin Abrams, Punit Shah, Evan Racah, Benjamin Frenkel, Shimon Whiteson, and Dragomir Anguelov. Hierarchical model-based imitation learning for planning in autonomous driving, 2022. URL /tttpp://arxiv.org
abs/2210. 09539
Alberto Camacho, Izzeddin Gur, Marcin Lukasz Moczulski, Ofir Nachum, and Aleksandra Faust. Sparsedice: Imitation learning for temporally sparse data via regularization. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021.
Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mit- igating covariate shift in imitation learning via offline data with partial coverage. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances Neural Information Processing Systems. volume 34, pp. 965 Curran Asso- ciates, Inc., 2021. URL h rwwrips....... cc/paper__p.p.....eepe fi Files/paper/ 3021/File/07d555 22277111177155533330003388888883
M. D. Donsker and S. R.S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-212 March 1983. ISSN 0010-3640. doi: 10.1002/cpa.3310066200
Chelsea Finn, Paul Christiano. Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852. 2016.
Jerome H. Friedman. Greedy function approximation: gradient boosting machine. The Annals of Statistics, 29(5): 1232. 2001. doi: 20.1114/00//0111223355 URL https://doi: /rg/
10.1214/aos/101203342
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning, 2018.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 4028-4039, 2021.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Aditya Grover and Stefano Ermon. Boosted generative models, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
11

---

Published as a conference paper at ICLR 2024
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations, 2020.
URL
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning pp. 2681-2691. PMLR, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, 2016.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In
Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization, 2020.
Geon-Hycong Kim, Scokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang. and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations (ICLR), 2022. URL httts::/ http:///aaaeeeeeeeeeeeoooooaoaaa
net Bttp:/oormmmdddddddaaaaaaaa
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-rrttt Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference Learning Representations, 2019. URL https:// openreview net Hod fpoA5Km) F
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020. URL https://
openreview. fmr.mmoormmmmm ? Hyg-JC4FDr
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.
Llew Mason, Jonathan Baxter, Peter Bartlett, and Marcus Frean. Boosting algorithms gradient descent. In S. Solla, T. Leen, and K. Müller Advances in Neural Information Processing Sys- tems, volume 12. MIT Press, 1999. URL http//ttppee//preeei.gg..eeooorp
pdf
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pictquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? Advances in Neural Information Processing Systems, 34: 14656-14668, 2021.
Xue Bin Peng. Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP. ACM Transactions on Gruphics, 40(4):1-20, jul 2021. doi: 10. 10.114573450626.4556670 URL https::/doit .org/
10.114582F345022 3459670]
Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 1. Morgan- Kaufmann, 1988. httttp:://ppooeecinnsseeee
waapaper__iieeeeeee files/paper/ pdf
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Visual adversarial imitation lear- ing using variational models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. pp. 3016 3028. Curran Associates, Inc., 2021. URL http::/::/.....eee.o....ee/n.o.. http://iinn.n..l....o.ooo/////oa fi ft/p///per///00//11//// 3 http:///000777.o.aaeaaeaeaaaaee.
12

---

Published as a conference paper at ICLR 2024
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint 19Xiv:1905.11108, 2019.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. In International conference on learning representations, 2019.
Bruno Scherrer and Matthieu Geist. Local policy search in convex space and conservative policy iteration as boosted policy search. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Parl III 14, pp. 35-50. Springer, 2014.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b.
Rohin Shah, Steven H. Wang, Cody Wild, Stephanie Milani, Anssi Kanervisto, Vinicius G. Goecks, Nicholas Waytowich, David Watkins-Valle Valls, Bharat Prakash, Edmund Mills, Divyansh Garg, Alexander Fries, Alexandra Souly, Chan Jun Shern, Daniel del Castillo, and Tom Lieberum. Retrospective on the 2021 basalt competition on learning from human feedback, 2022.
Mingfei Sun, Anuj Mahajan, Katja Hofmann, and Shimon Whiteson. Softdice for imitation learning: Rethinking off-policy distribution matching. arXiv preprint urXiv:2106.03/55, 2021.
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International conference machine learning, pp. 2019.
PMLR,
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu, Of moments and matching: A game-theoretic framework for closing the imitation gap. In Proceedings of the 38th International Conference on Machine Learning. 2021.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. 2012. doi: 10.1109/RROS.0111 2012666619.
Ilya Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, and Bernhard Schölkopf. Adagan: Boosting generative models, 2017.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https: :/ sac, 2020.
github. com/ http:////eeeiiiaaatt///rrrroaeee
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control:
Improved data-augmented reinforcement learning. In International Conference Learning Representations, 2022. URL.
Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offline imitation learning with suboptimal demonstrations via relaxed distribution matching, 2023.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL. USA, 2008.
13

---

Published as a conference paper at ICLR 2024
A DETAILED ALGORITHM PSEUDOCODE
Appendix A presents a more detailed pseudocode of LILoos... The main detail here is the 2-step process learning our discriminator using weighted replay buffer of weak learner samples and then learning a weak learner for a certain number of RL steps.
Algorithm AII BooSO (Adversarial Imitation Learning via Boosting)
Require: number of iterations T, expert data D weighting parameter a Initialize и weight 01 1, replay buffer
2: for = 1,...., do
3: Construct the 1-th dataset D. [(8j, a,)}3= where 8,, 0g ~ d Vj.
4: Set B <- 13 UDD
5: for of Discriminator Updates do
6: Sample batch from B with respective sample weights aict
7: Update discriminator 9 via Eq.
8: end for
9:
for # of Weak Learner Updates do
10:
Compute weak learner # via off-policy RL approach (e.g., SAC) reward -ğ(s, a)
with replay buffer B with uniform weights on all samples end for
12:
Set + a;(1 - a) for i S t, and ar+1 = a
end for
14: Return Ensemble π {(a;, m
After learning our ensemble. we evaluate it by randomly sampling a policy, Ti, from our ensemble with probability ag, With this weighted sampling, we then collect trajectory. Appendix A details this process.
Algorithm 3 AILBooST EVALUATION
Require: Ensemble
(a,, TT) ) I: for # of Evaluation Trajectories do
2: Sample n; ~ π with probability o;
3: Collect trajectory using
4: end for
14

---

Published as a conference paper at ICLR 2024
B IMPLEMENTATION AND EXPERIMENT DETAILS
Here we detail all environment specifications and hyperparameters used in the main text.
B. ENVIRONMENT DETAILS
Following the standards used by DrQ-v2 Yarats al. 2022), environments have maximum horizon length of 500 timesteps. This is achieved by setting each environment's action repeat to be 2 frames. For image based tasks, each state is stacked frames that are each x 84 dimensional RGB images (thus 9 84 x 84).
Task
Action Space Dimension
Task Traits
Reward Type
Ball in Cup Catch Walker Walk Cheetah Run
2
swing, catch walk run walk stand dense
sparse dense dense
6
6
Quadruped Walk
12
dense
Humanoid Stand
21
Table 2: Task descriptions, action space dimension, and reward type for each tested environment.
B.2 DATASET DETAILS
Using the publicly released implementation for SAC and DrQ-v2, we trained high quality expert policies for state-based and image-base based environments respectively, We refer the readers to (Yarats)
et al.
2022 and (Haamoja et al. 2018 Yarats & Kostrikov 2020 for exact hyperparameters.
Expert Performance
Random Performance
Task
Ball in Cup Catch
980.8
16.4
Walker Walk
966.6
19.9
Cheetah Run
910.5
0.2
Quadruped Walk
959,2
17.9
Humanoid Stand
927.8
3.9
Walker Walk (Vision)
823.1
9.6
Cheetah Run (Vision)
806.3
0.3
Table 3: Average expert and random performance calculated by averaging 50 trajectories collected from the expert and random policies respectively. Vision experts are denoted (vision)
B.3 HYPERPARAMETERS
For VaLueDICE and Io-Learn, we used the base hyperparameters they reported for the MuloCo benchmark suite. In order ensure good performance, tried different configurations for every environments (i.e. the configuration for Cheetah Run for Wa 1 ker Wa k) since despite using the same physies engine and models, there are minor differences for DeepMind Control Suite. For DAC and A LIBoot,, we used our own implementations. Table that all perraaaaaeeee are shared between DAC and AILBoost except for the update frequency of the disciminrator vs the policy. Note that this is one of the core differences between DAC and AILBoost.
the hyperparameters used. Note
For AILBoost predominanty tested 4 hyperparameters: of discriminator updates, steps to learn weak learners, weighting parameter a, and the TD n-step. For the of discriminator updates we tested 10. 100, 500, 1000. and 5000. For the the steps to learn weak learners, tested 1000, 5000, 10000, 20000, and 100000. For a, we swept 0.95, 0.7, 0.4, 0.2, and 0.05. Finally, we tested either TD n-step 1 or 3.
15

---

Published as a conference paper at ICLR 2024
Setting
Values
Policy Architecture (state)
3 layer MLP with 1024 hidden units each
DAC (state)
total number of steps: 10c6
replay buffer size: 1e6
learning rate: le-4
action repeat: 2
batch size: 256
TD n-step:
discount factor: 0.99
gradient penalty coeff: 10.0
policy update frequency:
AILBoost (state)
Samples per Weak Learner (N): 1000
of Weak Learners (T): 100
Steps to learn Weak Learne: 1000
Discriminator updates: 100
Weighting Parameter (a): 0.05
Policy Architecture (vision)
Model Architecture from Yarats et al. 2022
DAC (vision)
total number of steps: 20e6
replay buffer size: 1c6
learning rate: le-4
action repeat: 2
batch size: 512
TD n-step: 3
discount factor: 0.99
gradient penalty coeff: 10.0
policy update frequency:
AILBoost (vision)
Samples per Weak Learner (N): 10000
of Weak Learners (T): 100
Steps to learn Weak Learner: 20000
Discriminator updates: 500
Weighting Parameter (a): 0.05
Table 4: Hyperparameters used for DAC and AI LBostt. All of DAC's hyperparameters are shared by AILBoost except for the parameters colored in blue. In particular, the update frequency of the disciminrator vs the policy is one of the core differences between DAC and AILBoost.
16

---

Published as a conference paper at ICLR 2024
C ADDITIONAL RESULTS
C.I AGGREGATE PERFORMANCE COMPARISONS
Following the recommendations of (Agarwal 202I), we do an additional diagnostic of measuring the probability of improvement between two algorithms. This metric measures how likely it is for X outperform Y a randomly selected task from the benchmark suite. Specifically, P(X > Y) J Ln P(Xm Ym) where P(Xm >> Ym) is the probability of X outperforming Y on task m. Note that this measurement does not account the size of improvement. F shows comparison. AILBoost shows significant improvement over all other algorithms other than DAC. In conjuction with Figure I
we see that although the chance of AILBoost doing better than DAC z 50%, the size of improvement AILBoost has over DAC denoted by the IQM and Mean significantly larger.
Algorithm Y
Algorithm X
AlLBoost
DAC
AILBoost
IQ-Learn
AlLBoost
ValueDICE
AILBoost
BC
DAC
IQ-Learn
DAC
ValueDICE
DAC
BC
IQ-Learn
ValueDICE
IQ-Learn
BC
ValueDICE
BC
0.4
0.6
0.8
1.0
P(X Y)
Figure 5: Probability of improvement between all tested baselines and AILBoost.
17

---

Published as a conference paper at ICLR 2024
C.2
LEARNING CURVES
Here we present the complete suite of learning curves for all 5 environments.
Walker Walk
10 Demas
5 Demos
Demo
1000
1000
1000
Ww
600
600
0.00
0.25
0.50
0.75
1:00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
125
150
1e6
Ie6
166
Cheetah Run
10 Demos
5 Demos
Demo
1000
1000
1000
800
800
600
600
600
0.00
025
0.50
0.75
1:00
1:25
1.50
0.00
0.5
050
0.75
1.00
125
1.50
0.00
025
0.50
0.75
1:00
125
150
Ie6
Ie6
165
Quadruped Walk
10 Demas
5 Demos
Demo
1000
1000
1000
800
800
800
500
600
600
400
400
0.00
025
650
075
1:00
125
1.50
0.00
025
0.50
0.75
1.00
1.25
1.50
0.00
0.5
0.50
0.75
1.00
125
1.50
1e6
1e6
166
Ball in Cup Catch
10 Demas
5 Demos
Demo
1000
1000
1000
500
800
800
Score
500
600
600
400
400
400
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.50
1.25
1.50
0.75
1.00
1.25
1550
Ie6
Ie6
leb
Humanoid Stand
10 Demas
Demms
Demo
1000
1000
1000
800
800
800
600
600
600
400
400
400
200
200
2
les
l65
Expert
ValueDICE
AILBoost
Figure 6: Learning curves for A Aloost and all baselines on the DMC environments with 10, 5, and expert trajectories across 3 seeds.
18

---

Published as a conference paper at ICLR 2024
C.3 LEARNING CURVES ACROSS DIFFERENT OPTIMIZATION SCHEDULES
Here we present the full suite of learning curves where we vary how often the policy and the discriminator update relative each other. We keep every other hyperparameter constant in this ablation.
Ball Cup Catch
Walker Walk
Ousonuped WaRk
Cheotah Run
360
Samshe
Expart 2200 120
1002
1000
200 200
20 P.
Figure 7: Learning curves for AILBoost on 4 out of the 5 DMC environments with 5 expert trajectories across seeds, where we vary the number of policy updates and discriminator updates the agent takes over time.
19


Reviews:
[{"review_id": "p1VLtLxcHD", "reviewer": "Reviewer_mXvr", "summary": "Authors proposed AILBoost that exploits gradient boosting algorithm with the variational form of reverse KL divergence. Similar to the sample-based discrepancy minimization in AdaGAN (Tolstikhin et al., 2017), AILBoost adapted weighted replay buffer, where each trajectory and its importance are stored together in the replay buffer, which is later used to minimize the discrepancy between the expert's state-action occupancy measure and the mixture of agent's state-action occupancy measures. The empirical studies with controller-state-based and pixed-based environments show that AILBoost outperforms ValueDICE (Kostrikov et al., 2020) and DAC, both of which are well-known baselines in the AIL literature.", "questions": "- More succinct Abstract and Introduction are needed. I think they are quite dense in its current form. One example I could think is adding a small figure to describe AILBoost's contribution, but this is not a mandatory comment to follow.\n- There are two different forms of reverse KLD; (1) Donsker-Varadhan dual form from ValueDICE (2) Variational form from f-divergence GAIL. In AILBoost, the second form was used. Do we have any reason for not using the first form?\n- The idea of using weighted replay buffer and boosting seems applicable to general AIL frameworks, although authors applied this only to the variational form of reverse KLD. Can we apply this idea to DAC, and if possible, can we see the performance for those cases? One reason I'm asking is the training procedure that combines DAC with boosting may be simpler than the one with rev KLD.\n- In page 7, line 1, \"we always warm start from $\\pi_t$\". ---> What does this mean? Also, Appendix A doesn't appear in the manuscript. \n- AILBoost's computational complexity increases as $t$ increases in Algorithm 1. Can you please compare training times among DAC, ValueDICE and AILBoost?\n- In Figure 1, \"AILBoost outperforms DAC, ValueDICE, IQ-Learn, and BC across all metrics, amount of expert demonstrations, and tasks\" ---> This is true from IQM perspective, but not true when 1 demonstration is considered with Mean and Optimality Gap. \n- Figure 2,3,4's file sizes seem to be too large, which I think end up the heavy file size (34MB) of pdf file and makes text loading slow. Can we make those figures' sizes smaller?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Using gradient boosting with weighted replay buffer and applying it to adversarial imitation learning is interesting. \n- Empirical results were evaluated with many metrics (IQM, Mean, Optimality GAP), which makes the results more reliable.\n- Potential issues and limitations with the algorithm's scalability are discussed.\n- Literature review on AIL is clearly done.", "weaknesses": "- Presentation should be improved: Abstract and Introduction (especially, Abstract) include too many details about existing works, which will be mentioned in Related Works and Preliminaries. Making Abstract and Introduction more succinct seems needed. \n- The algorithm's complexity grows due to using all previous histories. However, this can be approximated by ignoring old samples, as mentioned by authors in Section 4. \n- Contributions are focused on empirical sides and not on the theoretical sides; I don't think this is a crucial weakness, though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZJm0NVqdd", "reviewer": "Reviewer_o1rE", "summary": "In this paper, the authors propose a novel adversarial imitation learning algorithm, AILBoost, which follows the framework of boosting. On the policy side, AILBoost maintains a weighted ensemble of policies and performs gradient boosting in the domain of state-action occupancy measures. On the discriminator side, AILBoost employs a weighted replay buffer combined with an expert dataset to train discriminators that maximize the discrepancy between the current mixed policy and the expert in an online manner. Empirically, AILBoost outperforms benchmarks in continuous control tasks across both controller state-based and pixel-based environments.", "questions": "Questions:\n1.Why using weighted mixing instead of using the DAgger[1] style average mixing?\n\n2.Is the smooth loss condition in Section 4.1 hard to satisfy? Is the specific condition that $d^{\\pi}$'s state action distribution covers the expert's state action distribution? Could you point out the exact theorem that makes this claim?\n\n3.Can I get a definition for \"off-policy\"? I believe the proposed algorithm is still \"on-policy\" as it compares the current mixed policy state distribution to the expert's.\n\n4.The noise during the policy rollout each round might compound via the buffer replay and disrupt the analysis on the discriminator's side. If this is the case and we need to gather new samples from the mixed policy every round, does this challenge the claim of being fully off-policy?\n\n5. It would be interesting to see experiments where the learner is truly weaker than the expert, such that the expert is nonrealizable. Is weak learnability in imitation learning sufficient for near-expert performance?\n\n6.In Algorithm 2, why do we calculate $\\pi_{t+1}$ without the expert dataset? Is this practical? Any further justification?\n\n7.Why in Algorithm 2, line 10, is the replay buffer given uniform weight for all samples? Any further justification?\n\n8.Could you explain more about the meaning of 1000 policy updates per 100 discriminator updates in section 5.3?\n\n9. As a follow up question from question 1, I believe the returned mixed policy is following the online reduction similar to DAgger[1], among DAgger style algorithms, MoBIL[2] is also using weighted policies, which may be worth mentioning.\n\nMinor suggestions:\n\nPlease mention \"10 trajectories\" in the caption of Figure 3.\nPlease provide more description on the performance of behavior cloning in Figure 3.\nCan you provide each task's plots corresponding to Figure 1 in the appendix? Also, all plots related to Figure 2 and Figure 3 would be appreciated.\n\n[1]Ross S, Gordon G, Bagnell D. A reduction of imitation learning and structured prediction to no-regret online learning[C]//Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011: 627-635.\n[2] Cheng C A, Yan X, Theodorou E, et al. Accelerating imitation learning with predictive models[C]//The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019: 3187-3196.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The author applies a mixed policy class which naturally connects with the buffer replay under the adversarial imitation learning framework, this is an interesting discovery.\n2.By using a weighted mix of the learned policies, AILBoost performs gradient boosting, making the policy update smoother, which proves to be useful.", "weaknesses": "1.While thorough in its overall idea, this work lacks in-depth theoretical analysis. I suspect that noise during the policy rollout in each round might compound through the buffer replay, potentially disrupting the analysis of the discriminator.\n\n2.The \"weak learner\" presented in this study doesn't conform to the traditional definition of weak learnability, which could be misleading. Contrary to AdaBoost, if the learner's base policy class fails to match the performance of the expert, then even mixing them at the initial state won't achieve expert-level performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03Xh2rqKNG", "reviewer": "Reviewer_Ah7S", "summary": "This paper presents a novel algorithm called AILBoost for adversarial imitation learning (AIL) in the framework of boosting. AIL has been successful in various imitation learning applications, but existing methods have limitations in terms of off-policy training and sample efficiency. AILBoost addresses these limitations by maintaining an ensemble of weak learners (policies) and training a discriminator to maximize the discrepancy between the ensemble and the expert policy. A weighted replay buffer is used to represent the state-action distribution induced by the ensemble, allowing for training using the entire collected data. The algorithm is evaluated on state-based and pixel-based environments from the DeepMind Control Suite and outperforms existing methods in terms of sample efficiency and performance. The paper also discusses the benefits of boosting in AIL and the robustness of AILBoost across different training schedules.", "questions": "1. Could you provide more insights into the computational complexity of AILBoost? Specifically, how does the algorithm scale with the number of weak learners, the size of the replay buffer, and the dimensionality of the state and action spaces? This information would be valuable for understanding the practical feasibility of AILBoost in larger and more complex environments.\n\n2. In Section 4, you mention that AILBoost maintains a weighted replay buffer to represent the state-action distribution induced by the ensemble. Could you provide more details on how the weights are computed and updated in the replay buffer? Additionally, how does the size of the replay buffer affect the performance of AILBoost? It would be helpful to understand the trade-off between memory usage and performance.\n\n3. The paper mentions that AILBoost achieves competitive performance with as little as one expert trajectory. Could you provide more insights into the limitations and trade-offs of using a small number of expert demonstrations? How does the performance of AILBoost change as the number of expert trajectories increases? It would be interesting to see if there is a threshold or diminishing returns in terms of performance improvement with more expert demonstrations.\n\n4. In the experimental evaluation, you compare AILBoost with existing off-policy AIL algorithms. Could you provide more insights into the reasons behind the superior performance of AILBoost compared to these baselines? What are the key factors or design choices in AILBoost that contribute to its improved performance? This information would help in understanding the specific advantages of AILBoost over existing methods.\n\n5. The paper briefly mentions the limitations of AILBoost, such as increased memory cost. Could you elaborate on other potential limitations or failure modes of the algorithm? Are there any specific scenarios or environments where AILBoost may not perform well? Providing a more comprehensive discussion on the limitations of AILBoost would help in understanding its practical applicability and potential areas for future improvement.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths of the Paper:\n\n1. The paper introduces a novel algorithm, AILBoost, for adversarial imitation learning that leverages the framework of boosting. This approach is unique and different from existing methods in the field. The use of boosting in the context of AIL is a creative combination of ideas that brings new insights and potential benefits.\n\n2. The paper provides a thorough and well-structured description of the AILBoost algorithm, including the theoretical foundations, algorithmic details, and empirical evaluation. The authors present a clear motivation for their approach and provide a comprehensive analysis of its performance compared to existing methods. The empirical evaluation is conducted on a benchmark suite of environments, and the results demonstrate the effectiveness and superiority of AILBoost.\n\n3. The paper is well-written and easy to follow. The authors provide clear explanations of the concepts, algorithms, and experimental setup. The organization of the paper is logical, with sections dedicated to preliminaries, algorithm description, and experimental results. The use of figures and tables further enhances the clarity of the presentation.\n\n4. The paper addresses important limitations in existing adversarial imitation learning methods, particularly in terms of off-policy training and sample efficiency. By introducing AILBoost, the authors provide a principled and effective solution that improves the performance and scalability of AIL algorithms. The empirical results demonstrate the significance of AILBoost, as it consistently outperforms existing methods across different environments and tasks.", "weaknesses": "Weaknesses of the Paper:\n\n1. While the paper compares AILBoost with existing off-policy AIL algorithms such as DAC, ValueDICE, and IQ-Learn, it would be beneficial to include a comparison with more state-of-the-art IL methods as well. This would provide a more comprehensive evaluation and demonstrate how AILBoost performs relative to the best-performing IL algorithms in the field.\n\n2. The paper lacks a detailed theoretical analysis of the AILBoost algorithm. While the authors provide some intuition and connections to boosting algorithms, a more rigorous theoretical analysis would strengthen the paper's claims. Specifically, providing formal proofs or guarantees of convergence, optimality, or sample complexity would enhance the theoretical foundation of AILBoost.\n\n3. The paper briefly mentions the hyperparameters used in the experiments but does not provide a thorough discussion on their selection or sensitivity analysis. It would be valuable to explore the impact of different hyperparameter choices on the performance of AILBoost and provide insights into the robustness and generalizability of the algorithm.\n\n4. The paper does not include ablation studies to analyze the individual components or design choices of AILBoost. By systematically removing or modifying specific components of the algorithm and evaluating their impact on performance, the authors could gain a deeper understanding of the contributions of each component and provide insights into their importance.\n\n5. The paper briefly mentions that AILBoost may have increased memory cost due to maintaining weak learners. However, a more comprehensive discussion on the limitations of AILBoost, such as computational complexity, scalability to larger environments, or potential failure modes, would provide a more balanced perspective on the algorithm's practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6y89rqOGup", "reviewer": "Reviewer_xvjX", "summary": "This paper proposes a AIL method so-called AILBoost with policy boosting. In each round, the policy is ensembled considering the weakness of current policy. The ensemble training coefficient selection is simple with $\\alpha (1-\\alpha)$. It indicates that ensembling the policy is just to cover the small difference between expert and agent rollouts. Experiments show that AILBoost outperforms ValueDICE and DAC in state based and image based environments with small performance gains.", "questions": "I have a question about how the policies are ensembled? Is there a random number to choose which weak learner to perform? (this way keeps the multi-modalities for sampling) Or the policies are added with the outputs? Or any other ensemble methods? I am confused about this. However, ensembling the outputs of different policies may be wrong?\n\nI do think it is a good idea to ensemble the weak learner for adversarial imitation learning. If the authors could resolve my confusion, I would like to raise my score.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is easy to follow. Experiments are well conducted. The performance of AILBoost is convincing.", "weaknesses": "I do think this paper has some inspirations for AIL community, especially the experiment results are convincing.\n\nPls see Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)