C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory
Authors: Tianjiao Luo
Tim Pearce
Huayu Chen
Jianfei Chen
Jun Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Control Theory
Stability Analysis
Primary area: reinforcement_learning

Abstract: Generative Adversarial Imitation Learning (GAIL) provides a promising approach to training a generative policy to imitate a demonstrator. It uses on-policy Reinforcement Learning (RL) to optimize a reward signal derived from an adversarial discriminator. However, optimizing GAIL is difficult in practise, with the training loss oscillating during training, slowing convergence. This optimization instability can prevent GAIL from finding a good policy, harming its final performance. In this paper, we study GAIL’s optimization from a control-theoretic perspective. We show that GAIL cannot converge to the desired equilibrium. In response, we analyze the training dynamics of GAIL in function space and design a novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a simplified “one-step” setting. Going from theory to practice, we propose Controlled-GAIL (C-GAIL), which adds a differentiable regularization term on the GAIL objective to stabilize training. Empirically, the C-GAIL regularizer improves the training of various existing GAIL methods, including the popular GAIL-DAC, by speeding up the convergence, reducing the range of oscillation, and matching the expert distribution more closely.

C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory
Tianjiao Luo', Tim Pearce?, Huayu Chen', Jianfei Chen', Jun Zhu
Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing 100084, China Microsoft Research
(luotj21, chenhuay21)@maill tsinghua, edu. cn (jianfeic, dcszj)@tsinghhu edu, en
Abstract
Generative Adversarial Imitation Learning (GAIL) provides a promising approach to training generative policy imitate demonstrator. It uses on-policy Rein- forcement Learning (RL) to optimize reward signal derived from adversarial discriminator. However, optimizing GAIL is difficult in practise, with the training loss oscillating during training, slowing convergence. This optimization instability can prevent GAIL from finding good policy. harming its final performance. In this paper, study GAIL's optimization from control-theoreli perspective. We show that GAIL cannot converge to the desired equilibrium. In response, we analyze the training dynamics of GAIL in function space and design novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in simplified "one-step" setting. Going from theory to practice, propose Controlled-Gar (C-GAIL), which adds a differentiable regularization term on the GAIL objective stabilize training. Empirically, the C-GAIL regularizer improves the training of various existing GAIL methods, including the popular GAIL-DAC, by speeding up the convergence, reducing the range of oscillation, and matching the expert distribution more closely]
Introduction
Generative Adversarial Imitation Learning (GAIL) [ aims to learn a decision-making policy in sequential environment by imitating trajectories collected from expert demonstrator. Inspired by Generative Adversarial Networks (GANs) 22, GAIL consists of learned policy serving as a generator, and discriminator distinguishing expert trajectories from generated ones. The learned policy is optimized through Reinforcement Learning (RL) with a reward signal derived from the discriminator. This paradigm offers distinct advantages over other imitation learning strategies such as Inverse Reinforcement Learning (IRL), which requires an explicit model of the reward function 3 and Behavior Cloning (BC), which suffers from a distribution mismatch during roll-outs (4)
Meanwhile, GAIL does bring certain challenges. One key issue it inherits from GANs instability during training (5) GAIL presents a difficult minimax optimization problem, where the convergence of the discriminator and the policy generator towards their optimal points not guaranteed in general. This problem manifests in practice as oscillating training curves and an inconsistency in matching the expert's performance (Fig. []. However, recent empirical works (6-10 on GAIL mostly focus on improving the sample efficiency and final return of the learned policy, without directly resolving the problem of unstable training. On the other hand, theoretical works [1T]4]] on GAIL's convergence based on strong assumptions and do not yield practical algorithm for stabilizing training.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Half-Cheetah
Walker2d
Reacher
1.2
1.2
0.8
0.4
GAL- GML-DAC
0.2
C-GAIL-DAC
0.2
0.0
Random
-0.2
Expert
0.0
-1.0
-0.2
2M
0.4M
0.6M
1M
0.2M
0.4M
0.5M
1M
0.2M
0.4M
0.6M
0.8M
Ant
Hopper
1.25
1.2
0.75
0.8
0.50
0.6
0.00
0.2
0.0
-0.75
02.24
11:00
01.0
02:00
154
Figure 1: Normalized return curves for controlled GAIL-DAC with four expert demonstrations on five MuJoCo environments averaged over five random seeds. The x-axis represents the number of gradient step updates in millions and the y-axis represents the normalized environment reward, where I stands for the expert policy return and 0 stands for the random policy return
In this paper, we study GAIL's training stability using control theory. We describe how the generative policy and discriminator evolve over training at timestep t with dynamical system of differential equations. We study whether the system can converge to the desired state where the generator perfectly matches with the expert policy and the discriminator cannot distinguish generated from expert trajectories. This surprisingly reveals that the desirable state is not an equilibrium of the system, indicating that existing algorithms do not converge to the expert policy, even with unlimited data and model capacity. In response, we study a "one-step GAIL" setting, and design a controller that does create an equilibrium at the desired state. We theoretically prove that this controller achieves asymptotic stability around this desired state, which means that if initialized within a sufficiently close radius, the generator and discriminator will indeed converge to it.
Motivated by our theoretical analysis, we propose C-GAIL, which incorporates pragmatic controller that can be added as regularization term to the loss function to stabilize training in practice. Empirically we find that our method speeds up the convergence, reduces the range of oscillation the return curves (shown on Fig. [T). and matches the expert's distribution more closely on GAIL-DAC and other imitation learning methods for a suite of MuloCo control tasks.
1.1 Related Work
Adversarial imitation learning. Inspired by GANs and IRL, Adversarial Imitation Learning (AIL) has emerged as popular technique learn from demonstrations. GAIL formulated the problem as matching an occupancy measure under the maximum entropy RL framework, with discriminator providing the policy reward signal, bypassing the need recover the expert's reward function. Several advancements were subsequently proposed to enhance performance and stability, For instance, AIRL (6) replaced the Shannon-Jensen divergence of GAIL by KL. divergence. Baram et al. 15 explored combining GAIL with model-based reinforcement learning. DAC [9] utilized replay buffer to remove the need for importance sampling and address the issue of absorbing states. Other empirical works such as 7110 [6] helped improve the sample efficiency and final return of GAIL. In contrast, our work focuses on the orthogonal direction of training stability.
Meanwhile, the convergence behaviors of AIL have also been investigated theoretically. Chen et al. [ proved that GAIL convergences to a stationary point (not necessarily the desired state). The convergence to the desired state has only been established under strong assumptions such as i.i.d. samples and linear MDP [2 I and strongly concave objective functions [14. However, existing theory has not analyzed the convergence behavior to the desired state in a general setting, and has so far not presented practically useful algorithms to improve GAIL's convergence. Our analysis in Sec. 3.2 show that GAIL actually cannot converge to the desired state under general settings.

---

Additionally, our proposed controller achieves not only a theoretical convergence guarantee, but also empirical improvements in terms of convergence speed and range of oscillation.
Control theory in GANs. Control theory has recently emerged as a promising technique for studying the convergence of GANs. Xu et al. [17) designed linear controller which offers GANs local stability. Luo et al. [18) utilized a Brownian motion controller which was shown to offer GANs global exponential stability. However, for GAIL, the policy generator involves an MDP transition, which results in a much more complicated dynamical system induced by a policy acting in an MDP rather than a static data generating distribution. Prior theoretical analysis and controllers are therefore inapplicable. We adopt different analysis and controlling techniques, to present new stability guarantee, controller, and theoretical results for the different dynamical system of GAIL.
2
Preliminaries
We start by formally introducing our problem setting, well as necessary definitions and theorems relating to the stability of dynamical systems represented by Ordinary Differential Equations (ODEs).
2.1 Problem Setting
Consider a Markov Decision Process (MDP), described by the tuple A, Po, 7) where S is the state space, A is the action space, (ss'' [s, a) the transition probability function, r(s, a) is the reward function, Po is the probability distribution of the initial state So, and y [0. 1] is the discount factor. We work on the y-discounted infinite horizon setting, and define the expectation with respect to policy II as the (discounted) expectation over the trajectory it generates. For some arbitrary function 9 we have Eg [g(s, a)] - E[ 9(sn, a,)) ). where a, T(an (s.). 80 Po. 8n+1 P(sn+1 [8,; an). Note that we use n to represent the environment timestep. reserving t to denote the training timestep of GAIL. For a policy
II, We define its (unnormalized) state occupancy P=(s) = "n=n 7 (sn = s ). We denote (= a) = E,, log (3,, +>o { = s, Og = a] and the advantage function A ax) = Q" a) E, (Q" a)]. We assume the setting where we are given dataset of trajectories TÆ consisting of state-action tuples, collected from an expert policy TE- We assume access to interact in the environment in order to learn policy æ, but do not make use of any external reward signal (except during evaluation).
2.2 Dynamical Systems and Control Theory
In this paper, we consider dynamical systems represented by ODE of the form
dax(t) di f(x(()),
where represents some property of the system, t refers the timestep of the system and f is function. The necessary condition for a solution trajectory (z(t)}120 converging to some steady state value is the existence of 'equilibrium'.
Definition 2.1. (Equilibrium) 19 A point 7 is an equilibrium of system 1 if ((x) 0. Such an equilibrium also called a fixed point, critical point, or steady state.
Note that dynamical system is unable to converge if an equilibrium does not exist. A second important property of dynamical systems is 'stability". The stability of a dynamical system can be described with Lyapunov stability criteria. More formally, suppose (x(t)) is a solution trajectory of the above system (I) with equilibrium F, we define two types of stability.
Definition 2.2. (Lyapunov Stability) 20 System ① is Lyapunov Stable if given any e 0, there exists a 5 > 0 such that whenever (]z((0) z
< 8, we have [|x(t) æ|| < c for 0 < t < co.
Definition 2.3. (Asymptotic Stability) 20 System IT is asymptotic stable if it is Lyapunov stable, (0)
and there exists a ò > 0 such that whenever
F s 6, we have lim; [I=(()) 2 0.
Note that a dynamical system can be Lyapnuov stable but not asymptotic stable. However, every asymptotic stable dynamical system is Lyapnuov stable.
The field of control theory has studied how to drive dynamical systems to desired states. This can be achieved through the addition of a 'controller' to allow influence over the dynamical system's
3

---

evolution, for example creating an equilibrium at some desired state, and making the dynamical system stable around it. Definition 2.4. (Controller) 21 A controller of a dynamical system is a function u(t) such that
dx((t)
f(((()) + u().
dt
The equilibrium and stability criteria introduced for dynamical system (.) equally apply to this controlled dynamical system (2). In order to analyze the stability of a controller u(t) of the controlled dynamical system given an equilibrium Σ. the following result will be useful.
Theorem 2.5. (Principle of Linearized Stability) [22] A controlled dynamical system 2 with equilibrium λ is asymptotically stable if all eigenvalues of [(f
) + u(t)) have negative real parts, where [(f(f) + u(t)) represents the Jacobian of f (x((t) + u(t) evaluated at E.
Corollary 2.6. (() + u(t)) has positive determinant and negative trace, all its eigenvalues have negative real parts and the system is asymptotically stable
3 Analyzing GAIL a Dynamical System
In this section, we study the training stability of GAIL through the lens of control theory. We derive the differential equations governing the training process of GAIL,, framing it dynamical system. Then, we analyze the convergence of GAIL and find that it cannot converge to the desired equilibrium due the entropy term. For simplicity, we limit the theoretical analysis the original GAIL[] among many variants 6410, while the controller proposed in the next section is general.
3.1 GAIL Dynamics
GAIL consists of a learned generative policy Tg S + A and a discriminator Da, : S x A -> (0. 1).
The discriminator estimates the probability that an input state-action pair is from the expert policy, rather than the learned policy. GAIL alternatively updates the policy and discriminator parameters, and w. (The parameter subscripts are subsequently dropped for clarity.) The GAIL objective M is E,, log(o a))l E [log(1 D(s. a}] XH(T) (T), where Te is the expert demonstrator policy, и is the learned policy, and ((T) (T) = Ez [- log ग (a|s)] is its entropy. Respectively, the objective functions for the discriminator and policy (to be maximized and minimized respectively) are,
VD(D, m) E.r D(s, a)] + Ex [log(1 D(s, a))]
(3)
(D) = E, (log D(s, a)] AE, | log π (a]s)].
To describe GAIL as a dynamical system, we express how π and evolve during training. For the analysis to be tractable, we study the training dynamics from variational perspective, by directly considering the optimization of x and D in their respective function spaces. This approach has been used in other theoretical deep learning works 2 23 to avoid complications of the parameter space.
We start by considering optimizing Eq. (3) with functional gradient descent with discrete iterations in- (V and Tm+1(als)) (a]s) = [m((as) - -B"V= -B (Dmmi)) dexed by m: Dm+1(s, s, a) = Dm (s. a) + 3 aD.. (s.a)
8x (als) where 3 is the learning rate, the discrete iteration number, 8VD(Dm (similarly for (Dm(mm)) 8D..(ssa)) 24 defined via OVD(Dm, Rm)
is the functional derivative which implies the total change in VD upon variation of function D,, a linear superposition 25 of the local changes summed over the whole range of (s, a) value pairs.
8V aDm (s. ds da,
(D..,s..))
We then consider the limit / + 0, where discrete dynamics become continuous ('gradient flow') dD.(s.o) (D. and dale dt (s,()
(D..s) Formally, we consider the evolution of the discriminator function D, S x A + and the policy generator ग. : + A over continuous time t rather than discrete time m. We derive the training dynamic of GAIL in the following theorem.
de
(a)())))
Theorem 3.1. The training dynamic of GAIL takes the form (detailed proof in Appendix Lemma C.2
aD)(s,a) Pr.. (s) Di(s,a)
Px (s)
(4)
dl
d dal(s P.re - (s.)) dt
(5)
4

---

3.2 On the Convergence of GAIL
Now, we study the optimization stability of GAIL using the dynamical system Eq. 5 The desirable outcome of the GAIL training process, is for the learned policy to perfectly match the expert policy, and the discriminator to be unable distinguish between the expert and learned policy.
Definition 3.2. (Desired state) We define the desired outcome of the GAIL training process the discriminator and policy reaching DJ (s. a) = 2. .? (a = π {a)s)).
We are interested in understanding whether GAIL converges to the desired state. As discussed in Sec. [.2] the desired state should be the equilibrium of the dynamical system Eq. (5) for such convergence. According to Def. 2.1 the dynamical system should equal to zero at this point, but present the following theorem (proved in Proposition C.3 and
C.5
Theorem 3.3. The training dynamics of GAIL does not converge to the desired state, and we have
(s, P=; (s) (a)s) PaT (s)m=(((s) dt
D- =0, d=j (als)
P=;, (5) (s, # (6)
Dj
dt
Hence, the desired state is not an equilibrium, and GAIL will not converge to it. Since (al()) # O, even if the system is forced to the desired state, it will drift away from it. We find that the non- equilibrium result is due to the entropy term (*), and equilibrium could be achieved by simply setting À = (Corollary (.4). However, the entropy term is essential since it resolves the exploration issue and prevents over-fitting Therefore, aim to design controller that not only keeps the entropy term but also improves the theoretical convergence guarantee.
4 Controlled GAIL
Having shown in Section 3 that GAIL does converge to the desired state, this section considers adding a controller to enable the convergence. We design controllers for both the discriminator and the policy. We show that this controlled system converges to the desired equilibrium and also achieves asymptotic stability in simplified "one-step" setting.
4.1 Controlling the Training Process of GAIL
Establishing the convergence for GAIL is challenging since the occupancy measure Px involves an expectation over the states generated by playing the policy æ for infinite many steps. We simplify the analysis by truncating the trajectory length one: we only consider the evolution from timestep n to n + 1. We refer this simplified setting as "one-step GAIL", and the convergence guarantee of our proposed algorithm will be established in this simplified setting. Let p(s) be the probability of the state at s on timestep n. The objectives for the discriminator and the policy can then be simplified as,
VD(D,T) /./ log + TE(als) log(1 D(s, a)) ds da.
(D, x) = 1. (()))(a(s)) log ((s a) + (p(s) ds da.
The gradient flow dynamical system of these functions is,
(s. a) p(s)T((a(s) p(s)Tg(als)
(7)
Dz(s,a) D,(s,a)-1
(al(s)s) (lo) Dr(s.a) dt
+ A).
(8)
With this "one-step" simplification, the GAIL dynamics now reveal clearer structure. For given (s, a) pair, the change of D(s, a) and x (als) only depends on D(s, a), r(ass) p(s) and (a(s))) for the same (s, a) pair, without the need to access function values of other (s. a) pairs. Therefore, we can decompose Eq. & which are ODEs of functions, into series of ODEs of sealar values. Each ODE only models the dynamics of two scalar values (D(s, a), <(lss) for particular (x, a) pair. We will add controller to the scalar ODEs, to asymptotically stabilize their dynamical system. Proving that each scalar ODE is stable suggests that the functional ODE will also be stable. Note that
5

---

such decomposition is not possible without the "one-step" simplification, since the evolution of D and π for all (s, a) pairs is coupled through (r((s) and A (s, a) in Eq. (5)
Based on the above discussion, we now consider the stability of a system of ODEs for two scalar variables (D(s, a), (a)ss))- With 8, a given, we simplify the notation as z=(() ()(ss,), v(t)= = Te(sla), E == (a)s))) c:= p(s), so each scalar ODE can be rewritten as,
dx(()) cy(t)
eF
dy(t)
dt xtt) r(t) -1' dt = -clo
clooololff x+)) - cA (og - ct.
(9)
We showed earlier that the GAIL dynamic in Eq. (5) does not converge the desired state. Similarly, neither does our simplified 'one-step" dynamic in Eq. (9) converge to the desired state. We now consider the addition of controllers push our dynamical system to the desired stated. Specifically, we consider linear negative feedback control [26), which can be applied to a dynamical system to reduce its oscillation. We specify our controlled GAIL. system as,
dr((t) cy(t)
cE
x(t) r(t)-1* + ut
(10)
dt x(t)
dy(t) -loog xt)
eA log u(t) cd u2(().
(11)
dt
where i1 (t) and ut(t) are the controllers to be designed for the discriminator and policy respectively. Since the derivative of the discriminator with respect to time evaluated at the desired state (Def. 3.2)
already equals zero, the discriminator is already able to reach its desired state. Nevertheless, the discriminator can still benefit from a controller to speed up the rate of convergence we choose a linear negative feedback controller for uy (t) to push the discriminator towards its desired state. On the other hand, the derivative of the policy generator evaluated at its desired state in Eq. (④) does not equal zero. Therefore, u2(() should be set to make Eq. (11) equal to zero evaluated at the desired
state. We have designed it to cancel out all terms in Eq. ( at this desired state, and also provide feasible hyperparameter values for an asymptotically stable system. Hence, we select un (t) and uz to be the following functions,
un kx(((t) 2
(12)
(t) eA cdlog E + clog 2 + cd + a v() E
(13)
where k, a are hyperparameters. Intuitively, as k gets larger, the discriminator will be pushed harder towards the optimal value of 1/2. This means the discriminator would converge at a faster speed but may also have larger radius of oscillation.
4.2 Analyzing the Stability of Controlled GAIL
In this section, we apply Theorem 2.3 to formally prove that the controlled GAIL dynamical system described in Eq. (10) & is asymptotically stable (Def. 2.3 and give bounds with X, a, and k.
For simplicity, let us define (t) = (z))) and a function f such that f(=(t) is the vector (oy(t) cB
- k (z(t)= 1) clog +cA log E log ((t)-cr log x(t)+aaaa v(t)
E a] T Therefore,
(=())
controlled training dynamic of GAIL in Eq. 1O)p and Eq. (11) can be transformed to the following vector form
(14) Theorem 4.1. Let assumption 4.2 hold. The training dynamic of GAlL. in Eq. 14 is asymptotically stable (proof in Appendix().
d(z(()) ((x())d
Assumption 4.2. We assume a R.K > O, 8c22 Sca 4c² + ck ka 0, and k2 42+32c -cxta
0.
32c
Proof sketch. The first step in proving asymptotic stability of the system in Eq. 14 is to verify whether our desired state is an equilibrium (Def. 2.I.. We substitute the desired state, 2 (t) ((= = O. We then find the linearized system about the desired state d(z(()) = (())==)))))))dd Under Assumption 4.2 we show that
(H. T)T into system (4) and verify that d(=
(I((((()=())))) > 0 and (J( (())) < O. Finally we invoke Theorem 2.5 and Corollary
2.6 to conclude that the system in Eq. [4] is asymptotically stable.

---

Algorithm The C-GAIL algorithm
1: Input: Expert trajectory Tи sampled from TE, initial parameters Co, and @o for generator and
discriminator.
2: repeat
3: Sample trajectory from Wg.
4: Update discriminator parameters & with gradient from,
E, logg Dlo, a) (D(s. 2) + Erg log a(s,a) (D(s, a) - -
5: Update policy parameters with (D, (DD, π in Eq.
6: until Stopping criteria reached
5 A Practical Method to Stabilize GAIL
In this section, extend controller from the "one-step" setting back to the general setting and instantiate our controller as a regularization term on the original GAIL loss function. This results in proposed variant C-GAIL; method to stabilize the training process of GAIL.
Since the controllers Eq. 133 are defined in the dynamical system setting, we need to integrate these with respect to time, in order to recover an objective function that can be practically optimized by a GAIL algorithm. Recalling that (D, T and V (D, .T) are the original GAIL loss functions for the discriminator and policy (Eq. (3)) we define (D, x) and Vi (D, x) modified loss functions with the integral of our controller applied, such that
h (D(s,)) 1 }
(D( VD(D, ET,,
2
2
a (als) (clog 2 2 T=(ass)
(i(DT)) (D( BR
+ cAllo + + -c a)-aarsals)
Note that the training dynamics with these loss functions are identical to Eq. stability under the 'one-step' setting.
10 13 with guaranteed
While (D, π) can be computed directly, the inclusion of Tg, the expert policy, in V π) is problematic the very goal of the algorithm is to learn Tg, and we do not have access to it during training. Hence, in our practical implementations, we only use our modified loss (D T) to update the discriminator, but use the original unmodified policy objective V2 (D. T) for the policy. In other words, we only add the controller to the discriminator objective Vn (D, x). This approximation has no convergence guarantee, even in the one-step setting. Nevertheless, the control theory-motivated approach effectively stabilizes GAIL in practice, as we shall see in Sec. 6 Intuitively, C-GAIL pushes the discriminator to its equilibrium
faster speed by introducing penalty controller centered at With proper selection of the hyperparameter & (ablation study provided in Appendix EE) the policy generator is able train the discriminator at the same pace. leading GAIL's training converge faster with smaller range of oscillation, and match the expert distribution more closely.
Our C-GAIL algorithm is listed in Alg. T. It can be implemented by simply adding a regularization term to the discriminator loss. Hence, our method also compatible with other variants of GAIL, by straightforwardly incorporating the regularization into their discriminator objective function.
6 Evaluation
This section evaluates the benefit of integrating the controller developed in Section 4 with popular variants of GAIL. We the algorithms on their ability imitate an expert policy in simulated continuous control problems in MuloCo (27). Specifically, we consider applying our controller to two popular GAIL algorithms both the original 'vanilla' GAIL [ and also GAIL-DAC (9) (DAC) to improve sample efficiency
state-of-the-ar variant which uses iiscriminnttr-actor-crrii and reduce the bias of the reward function. Additionally, we include supplementary experiments compared with other GAIL variants such Jena et al. [ and Xiao et al. (8] in appendix F
7

---

Hall-Chhetah
Walker
Reacher
GML DAC
C-GALL-DA
1.5
0.09 0.1M 0.214 0.3M 0.5M 0:M 0.W. 6.80 1.0M
0,014 3160
0.004
29 dM BM PVPM
Ant
Hopper
3.5
15
3,00 1M 646 0.714 01854 1.0M
0.064 0.754 RT0 7M00 AM
State Wasserstein distance (lower is better) between expert and learned policies, over number of gradient step updates. Our controlled variant matches the expert distribution more closely.
Figure 2:
6.1 Experimental Setup
We incorporate our controller vanilla GAIL and GAIL-DAC, naming our controlled variants C-GAIL and C-GAIL-DAC. We leverage the implementations of Gleave et (28) (vanilla GAIL) and Kostrikov et al. [9] (GAIL-DAC), Gleave et al. (28) also provide other common imitation learning
frameworks BC, AIRL, and dataset aggregation (DAgger) 29
which also compare
For C-GAIL-DAC, we test five MuJuCo environments: Half-Cheetahh Ant, Hopper, Reacher and Walker 2D. Our experiments follow the same settings Kostrikov et al. [9]. The discriminator architecture has a two-layer MLP with 100 hidden units and tanh activations. The networks are optimized using Adam with learning rate of 10-3 decayed by 0.5 every 10 gradient steps. We vary the number of provided expert demonstrations: 74, 11, 15, 18), though unless stated we report results using four demonstrations. We assess the normalized return over training for GAIL-DAC and C-GAIL-DAC to evaluate their speed of convergence and stability, reporting the mean and standard deviation over five random seeds. The normalization is done with 0 random policy's 's return and to the expert policy return.
In addition to recovering the expert's return, generator's s and the expert's 's state distribution are matched, for which we use the state Wasserstein [30]. This requires samples from two distributions, collected by rolling out the expert and learned policy for 100 trajectories each. We then use the POT library's 's 'emd2' function [31] to compute the Wasserstein distance, using the L.2 cost function with uniform weighting across samples.
also interested in how closely policy
To evaluate C-GAIL, we follow the experimental protocol from Gleave et al. (28], both for GAIL and other imitation learning baselines. These evaluated on Ant, Hopper, Swimmer, Half-Cheetah and Walker 2D. For C-GAIL, we change only the loss and other GAIL settings are held constant. We assess performance in terms of the normalized return. We use this set up to ablate the controller strength hyperparameter of C-GAIL (Appendix B varying & E {0.1, 1, 10) (ablation study of a is not included since algorithm only involves controller for the discriminator in practice). Our experiments are conducted on single NVIDIA GeForce GTX TITAN X.
Table 1: Mean and standard deviation for returns of various IL algorithms and environments
Ant
Half Cheetah
Hopper
Swimmer
Walker2d
Random
-349 31
-293 36
-53 1 62
3 8
- 1 75
Expert
2408 + 110
3465 + 162
2631 + 19
298 +
2631 + 112
Controlled GAIL
2411 + 21
3435 = 50
2636 88
298 +
2633 12
GAIL
2087 + 187
3293 239
2579 - 85
295 +
2589 + 121
BC
1937 + 227
3465 = 151
2830 265
298 + 1
2672 +
AIRL
-121 + 28
1837 + 218
2536 + 142
269 +
1329 + 134
DAgger
3027 + 187
1693 + 74
2751 + 11
34
2174 132

---

6.2 Results
We compure GAIL-DAC to C-GAIL-DAC in Figure I (return), 2 (state Wasserstein), and 3 (con- vergence speed). Figure ! shows that C-GAIL-DAC speeds up the rate of convergence and reduces the oscillation in the return training curves across all environments. For instance, on Hopper, C- GAIL-DAC converges 5x faster than GAIL-DAC with less oscillations. On Reacher, the return of GAIL-DAC continues to spike even after matching the expert return, but this does not happen with C-GAIL-DAC. On Walker 2D, the return of GAIL-DAC oscillates throughout training, whereas method achieves a higher return at has reduced the range of oscillation by more than 3 times. For Half-Cheetah, our method converges 2x faster than GAIL-DAC. For Ant environment, C-GAIL-DAC reduces the range of oscillations by around 10x.
In addition to matching the expert's 's return faster and with more stability, Figure 2 shows that C-GAIL- DAC also more closely matches the expert's ' state distribution than GAIL-DAC, with the difference persisting even towards the end of training for various numbers of expert trajectories. Toward the end of training, the state Wasserstein for C-GAIL-DACC is more than two times smaller than the state Wasserstein for GAIL-DAC on all five environments.
Figure 3 shows that these improvements hold for differing numbers of provided demonstrations. plots the number of gradient steps for GAIL-DAC and C-GAIL-DAC to reach 95% of the max-return for vaious numbers of expert demonstrations. Our method is able to converge faster than GAIL-DAC regardless of the number of demonstrations.
Hyperparameter sensitivity. We evaluate the sensitivity to the controller's hyperparameter & using vanilla GAIL. Figure + (Appendix E plots normalized returns. For some environments, minor gains can be found by tuning this hyperparameter, though in general for all values tested, the return curves of C-GAIL approach the expert policy's return earlier and with less oscillations than GAIL. This is an important result as it shows that our regularizer can easily be applied by practitioners without the need for fine- mee-grained hyperparameter sweep.
Other imitation learning methods. Table TableIbenchmarte benchmarks C-GAIL against other imitation learning methods, including BC, AIRL, and DAgger, some of which have quite different requirements to the GAIL framework. The table shows that C-GAIL is competitive with many other paradigms, and in consistently offers the lowest variance between runs of any method. Moreover, we include supplementary experiments compared with Jena al. (7) and Xiao al. [8] in appendix F
7 Discussion & Conclusion
This work helped understand and address the issue of training instability in GAIL using the lens of control theory. This advances recent findings showing its effectiveness other adversarial learning frameworks. We formulated GAIL's training as dynamical system and designed controller that stabilizes at the desired state, encouraging convergence to this point. We showed theoretically that our controlled system achieves asymptotic stability under "one-step" setting. We proposed
Ant
Walker
Reacher
GAL-DAC --GAI--AAC
0.
0.8
0.8
0.7
8.6
0.4
0.5
0.5
0.3
04
5
10
12
14
16
18
10
12
14
16
18
6
10
12
14
16
16
Half-Cheetah
Hopper
3.55
0.54
0.3
0.50
0.2
12
14
12
14
16
18
Figure 3: Number of gradient step updates (in millions) required to reach 95% of the max-return for various numbers of expert trajectories on MuloCo environments averaged over five random seeds.

---

practical realization of this named C-GAIL, which reaches expert returns both faster and with less oscillation than the uncontrolled variants, and also matches their state distribution more closely.
Whilst our controller theoretically converges to the desired state, and empirically stabilizes training, recognize several limitations of our work. In description of GAIL training continuous dynamical system, we do not account for the updating of generator and discriminator being discrete as in practice, In our practical implementation of the controller, we only apply the portion of the loss function acting on the discriminator, since the generator portion requires knowing the likelihood of an action under the expert policy (which precisely what we aim to learn!). We leave future work to explore whether estimating the expert policy and incorporating controller for the policy generator brings benefit.
Acknowledgments and Disclosure of Funding
We thank Professor Yang Gao for support in discussions. This work was supported by the National Science and Technology Major Project (2021ZD0) 10502). NSFC Projects (Nos. 62350080, 62376131, 62061 136001, 62106123, 62076147, U19A2081, 61972224), and the High Performance Computing Center, Tsinghua University. J.Z is also supported by the XPlorer Prize.
References
[1] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[2] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[3] Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, page 2, 2000.
[4] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9329-9338, 2019.
[5] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning. pages 3481-3490. PMLR, 2018.
[6] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11245 2017.
[7] Rohit Jena, Changliu Liu, and Katia Sycara. Augmenting gail with be for sample efficient imitation learning. In Conference on Robot Learning, pages 80-90. PMLR, 2021.
[8] Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning. arXiv preprint 100.:/900001133.. 2019.
[9] llya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-acto actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning, arXiv preprint 18X.:v1809.02925. 2018.
[10] Gokul Swamy, Nived Rajaraman, Matt Peng, Sanjiban Choudhury, J Bagnell, Steven Z Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation. Advances in Neural Information Processing Systems, 35:7077-7088. 2022.
[11] Minshuo Chen, Yizhou Wang, Tianyi Liu, Zhuoran Yang, Xingguo Li, Zhaoran Wang, and Tuo Zhao. On computation and generalization of generative adversarial imitation learning. arXiv preprint arXiv:2001.02792. 2020.
[12] Qi Cai, Mingyi Hong, Yongxin Chen, and Zhaoran Wang. On the global convergence of imitation learning: case for linear quadratic regulator. arXiv preprint arXiv: 1901 2019.
10

---

[13] Yufeng Zhang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Generative adversarial imitation learning with neural network parameterization: Global optimality and convergence rate. In International Conference on Machine Learning, pages 11044-1100 1104.. PMLR, 2020.
[14] Ziwei Guan, Tengyu Xu, and Yingbin Liang. When will generative adversarial imitation learning algorithms attain global convergence. In International Conference Artificial Intelligence and Statistics, pages 1117-1125. PMLR, 2021.
[15] Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-end differentiable adversarial imitation learning. In International Conference on Machine Learning, pages 390-399. PMLR, 2017.
[16] Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:2914-2924, 2020.
[17] Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Understanding and stabilizing gans' training dynamics using control theory. In International Conference Machine Learning, pages 10566-10575. PMLR, 2020.
[18] Tianjiao Luo, Ziyu Zhu, Jianfei Chen, and Jun Zhu. Stabilizing gans' training with brownian motion controller. ICML, 2023.
[19] Edward L Ince. Ordinary differential equations. Courier Corporation, 1956.
[20] Paul Glendinning. Stability, instability and chaos: an introduction to the theory of nonlinear differential equations. Cambridge university press, 1994.
[21] William L. Brogan. Modern control theory. Pearson education india, 1991.
[22] Joseph P La Salle. The stability of dynamical systems. SIAM, 1976.
[23] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[24] Jacques Lafontaine et al. An introduction to differential manifolds. Springer, 2015.
[25] Walter Greiner and Joachim Reinhardt. Field quantization. Springer Science & Business Media, 2013.
[26] Stephen P Boyd and Craig H Barratt. Linear controller design: limits of performance, volume Citeseer, 1991.
[27] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[28] Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H Wang, Sam Toyer, Maximilian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell. imitation: Clean imitation learning implementations. arXiv preprint UrXiv:2211.11977. 2022.
[29] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635 JMLR Workshop and Conference Proceedings, 2011.
[30] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. ICLR, 2023.
[31] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, al. Pot: Python optimal transport. The Journal of Machine Learning Research, 22(1 (21):357133578 2021.
[32] Christopher M Bishop and Nasser M. Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.

---

A
Broader impact
This work has provided an algorithmic advancement in imitation learning. such, we have been cognisant of various issues such as those related to learning from human demonstrators c.g. privacy issues when collecting data. However, this work avoids such matters by using trained agents as the demonstrators. More broadly, methods providing more efficient and stable learning, which we believe general has positive impact.
work as step towards more principled machine learning
B Basics of Functional Derivatives
We provide some background on functional derivatives, which are necessary for the derivations in Appendix For more rigorous and detailed introduction, please refer to the Appendix D of [32]
Definition B.1. (functionals) We define functional F Fly] - R to be operator that takes function (() and returns a scalar value Fly].
Definition B.2. (functionals derivatives) We consider how much functional Fly) changes when make a small change (n](z)l<>) to the function. If we have
f
OF a ())m()) da
Fly hn) F(y)
dF + hn)
lim h-+
Oy
dh
h==
such that
/
OF By (x)(()) dz + O(h3),
Fly + -o))= FFy)
then By
aF is called the functional derivative of with respect to function y.
Theorem B.3. (chain rule) If the function is controlled by some weights 0 can rewritten as yo-
We have
OF Oye Oy 00
(x) (x)
0e
C Detailed Theoretical Analysis of General GAIL
max (D( = E2 [log (s,, a)l + Ex,, [log(1 D(s, D
(15)
min V3 (D, = E,, (s,, a)] AE 1- log (a]s)).
(16)
Lemma C.1. Given that Ty is parameterized policy. Define the training objective for entropy- regularized policy optimization
((0) = Eng [r(s, a)) AE log To (a(s))]
Its gradient satisfies
0 80 J(0) = Ezol
alog no(a|s) (s, = E.ol
3 log (a(s))
A (s,a)],
00
a0
where QTa (8, a) and A (s, a) are defined as
QT* == Eno (r,,, a) Aog log (a(3))) s, 3,00 a],
(s
12

---

Proof.
00 (()
00 Em (s) AE., log mo(as)]]
/ Pro (s) + x δ
Pro (s) log ro(ass)dads
08
00
/ Opxo (s)To(a
(s, a)dads A /
Ppso (s)o((a)
/ Pro (s)mo(als)
Olog rTe (als))
log no (als)(als)datll
dads
00
00
00
/ Opzo (s)mo(als)
(r(s,a) Alog (a + A / Pro (s)
1 (ro(als)
dads
00
To(als)
00
f Opxo (s)
(r(s,a) + Alog ( + A / (5) 0
(a sa(assddads
00
00
Opxo (s) (r(s,a) + A (
00
O (r(s, Alog oo (a|s)]
0==0
00
The above derivation suggests that we can view the entropy term additional fixed reward r'(s, a) = Alog (a)s). Applying the Policy Gradient Theorem, we have
Olog no(als) (=* = -E,, 1 Bog (a)s)) A (s,
08 J(0) = Ezol
00
00
where TT is similar the classic Q-function but with an extra "entropy reward" term.
Lemma C.2. The functional derivatives for the two optimization objectives
VD(D, T) = E. llo D(s, a)] + Erg [log(1 - D(s, a))
VD(D,)) mT E, [log ((s, a)] - AE log loglllo(aa (a|s)]
respectively satisfy
@VD (s)
Pa (s)Te(als)
0D D(s a)
1 as
0V, = (r(s))
where A follows the same definition in Lemmn C.I
(s, [log ((5,a)+Alog (a]s)] 8,40 = a), A (²(aaa) = ( Q"
Proof. Regarding aVp 0D by definition of E2 we have VD(D, - =
Pr (s) log a)dads + f Pre (s) log(1 D(s, dadss
according to the chain rule 25 of functional derivative, have
OVp Pa (s)
Pag ( ())TT(als)
aD D(s,a)
1 ((s,a)
Regarding @v 31 suppose parameterized by The chain rule for functional derivative states /
@v=
TV, dm
3m 00 dads.
00
13

---

According to Lemma
C.1
we have
0v,
i ((as)
Ex
A (s,
00
00
((als
Px (*()))(als)
A a]dads
00
/ px(() (T((s))
A" (s, a)dads.
00
Therefore, we have
0V dπ
Pr -
Proposition C.3.
The constrained optimizaation proble
min π) = Eg log (s,, a)] -AE, log x (a|s)] s.t. / s(all)) =
does not converge when ग = गE and (88, a) = 2
Vss VorVs Namely,
0v
Ōn ((s_a)==((s.)) # O.
When Tg and (ss 1/ we have
(s a) =Frs! (X log (π(a(s) log 2/80 8, 00 = a)
p(sn = =s|s0 s,a0 a) / mo(gi3]Alogm<((33 log dadd
Σ
n=0
7mh I p(sn = -5s00 s.00 = () + log 2|ds
=
n=0
(s -Q" a) - (s,
Σe 7"h p(8 = 8150 = 8) - p(8n = =8|80 = 8,00 = a))AH(TTE(-11)
n n=
According Lemma C.2
ov,
Zn PaT -re(()AAE(S,0)) Pre (TT
PTe (8n (Tg(((())1))) Since ag can by any policy distribution determined by the expert dataset, (TT((- (3)) can be any value for various and Additionally, for different actions + az, we cannot guarantee
Recall that we have A (8, a) = En 7n1 ("[ (8n 5/80
5 00
Png (on 8/80 8, 01) = Pag ( = 3/80
ag
02). Thus 8Va. is not constant and 0t
relies action A (s,
a) = 0 cannot hold, and thus
0V= + O.
Corollary C.4. This is corollary of Proposition C.3 When T ( a) TT(S, a), D(s,a) 2.2 and
the entropy term excluded from the GAIL. objective, we find, Bn
Proof. Exclusion of the entropy term can be achieved by setting X = 0. Then we have,
@v,
=P (A,
(17)
D0
[p(ss, 3|su 8) p(s,. -5/su 0.00 =a)(A((T((-1s)
(18)
=0
(19)
14

---

Proposition C.5.
The optimization problem
max (D(D,T) T) = Ex D(s, a) Eng [(s, a))
D
converges when π = Tg and D(s, o.) = 1 for Vs, a. Namely,
@VD p
3D ((s.))==p(ss)..(sss)>) = O.
Proof. According the chain rule of functional derivative, have
@ 0R, flog DJ + BI [lo((1 D)
0D
0D
E
E Exgl
D
D
EEFR D
1 - D
E [2 2]
D Proof of Theorem 4.1
Theorem D.1. Let assumption 4.2 holds. The training dynamic of GAIL in Eq. (4) is asymptotically stable.
Proof. To analyze the convergence and stability behavior of system 14 first we need to verify definition 2.1 to make sure our goal functions are equilibrium points. Then we apply theorem [25 to prove system 14 is asymptotically stable. Notice that 2" (t) (g. E)T, then we substitute this goal
function system 14
f(2
We the compute the linearized system near the goal function such that
d(z(t)) = J(f(2" (t))))((f()
(20)
where J is the Jacobian of function f. Therefore,
cg(t)
I()(=
x(1)
(w(()-
(
(21)
e +
vit)
(2.E)
which after evaluation becomes
ScFF - k
2c
(22)
-2c
ca+a
E
Then compute the determinate and trace of ([[[("(()))) which
det(I(f(z* (8c2), x 8ca 4c2) (ck) ka)
(23)
E
(race([([((*())))) -8E² kE +a
(24)
E
Since E = (a)s))) has range [0, 1}, therefore we have
> 0, if
ck - ka > 0
(25)
8c² X 8ca 462 + ckd - ka
0
(26)
15

---

The graph of (t)) is also downward hyperbola with middle point
(-k k )=+)))--ctaa)) Therefore, (I if
16c
32c
2 + 32c( --ch + a
<0.
(27)
32c
Note that this implies ckA - ka > 0, since
k + 32c( -c + a)
<0
(28)
32c
c³ + <0
(29)
32c
122
cd
0 > 0
(30)
32c
k2
ed
0
> 0.
(31)
32c
As result, system 14
asymptotically stable assumptions 4.2
hold.
16

---

E
Ablation on
Retur Plot
Return
1.02
1.00
Nyw in My
NM morr Poc.m
20
14
26
3.0
2.0 TimsHouur
2.5
3.0
TimeHHour
3.00
MOO
NMr TMrro (ee Noun
0.59
Mer loo
0.8
0.1
0.52
2.5
Time/oour
TimeHouu
Return for Walker2d
0.8
0.6
0.4
0.2
k 0.
0.0
Figure 4: Normalized returns curves for controlled GAIL with k = 0.1, = 1, and k 10 on MuJoCo environments, where the y-axis, represents expert policy return and represents random policy return
17

---

Comparison with Other IL Methods
BC+GAIL
WAIL
GAIL-DAC
Ours
Expert Trajectories
200
4
4
Half-Cheetah
4558.09 = 89.50
3660.49 + 217.60
5097.51 + 62.93
5102.32 + 5.80
Hopper
3554.35 + 165.7
3573.74 + 12.98
3521.14 + 36.81
3586.24 + 8.78
Reacher
-7.98 = 2.66
-9.81 + 3.23
-10.56 5.61
-8.61 + 1.46
Ant
3941.69 # 944.67
4173.97 213.46
3268. 29 1301.12
4239.40 42.81
Walker2d
6799.93 # 387.85
5274.72 + 981.72
4538.99 # 302.64
5912.83 146.58
Table 2: Return for each environment on various GAIL algorithms.
BC+GAIL
WAIL
GAIL-DAC
Ours
Expert Trajectories
200
4
Half-Cheetah
0.73M
15.24M
0.13M
0.08M
Hopper
30.36M
74.11M
0.51M
0.06M
Reacher
0.68M
10.84M
0.02M
0.01M
Ant
0.52M
98.50M
0.92M
0.34M
Walker2d
1.26M
79.48M
0.43M
0.41M
Table Number of Iterations needed to reach 95% return
Expert
DiffAIL
C-DifIALL
Hopper
3402
3382.03
142.86
3388.28 + 41.23
HalfCheetah
4463
5362.25 + 96.92
4837.34 + 30.58
Ant
4228
5142.60 90.05
4206.64 + 36.52
Walker2d
6717
6292.28 + 97.65
6343.89 89 + 33.67
Table Final reward with trajectory diffusion-basee GAIL, [2], both vanilla and controlled
variant. Mean and standard deviation over five runs
18

---

G
Comparison in Atari tasks
Expert (PPO)
GAIL
C-GAIL
BeamRider
2637.45 + 1378.23
1087.60 + 559.09
1835.27 381.87
Pong
21.32 = 0.0
+ + 18.13
0.34
Q*bert
598.73 + 127.07
-7.27 24.95
428.87
Seaquest
1840.26 = 0.0
1474.04 201.62
1389.47 80.24 23912.73 + 32.69
Hero
27814.14 = 46.01
13942.51 67.13
Table 5: Final reward in five Atari tasks. Mean and standard deviation over ten runs
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state our contribution and scope clearly in the abstract and introduction. Overall, we asymptotically stabilize the training process of GAIL and achieve a faster rate of convergence, smaller range of oscillation and match the expert distribution more closely.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our paper in Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: We provide our assumptions in Section 2 and Assumption 4.2 Our complete
proofs are provided in Section Cand and D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include the implementation of our method in Algorithm Additionally, submit additional zip file reproduce experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We submit an additional zip file to reproduce our experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ptapp..cc/pbblicguuu.ee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify our settings Section 6.1 and we provide an ablation study on hyperparameters in Section F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the error bars Figure T and 3 The methodology for their
calculation is included in their respective captions and Section
6.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on computer resources in Section 6.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that htt.r//p......bbitei/taaaccaaaa.
didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research comforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts of our work in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not think our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all the original papers used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "xdQhERQxNI", "reviewer": "Reviewer_PPsU", "summary": "This paper addresses the problem of unstable training of Generative Adversarial Imitation Learning (GAIL). To this end, the paper studies the convergence of GAIL from a control-theoretic perspective and proposes to employ a regularization term for the discriminator loss function, which can stabilize the training. The experiments in the locomotion domain, e.g., Half-Cheetah and Walker2D, show that the proposed method leads to learned policies that match experts better, achieve better performance, and converge faster. I believe this work provides insightful analyses, presents a promising method, and sufficiently evaluates the proposed method. Hence, I recommend accepting this paper.", "questions": "See above", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Motivation and novelty**\n- The motivation for stabilizing GAIL training is convincing.\n- Studying this from a control-theoretic perspective is novel to the best of my knowledge.\n\n**Clarity**\n- The overall writing is clear.\n\n**Experimental results**\n- The experiments in the locomotion domain, e.g., Half-Cheetah and Walker2D, show that the proposed method leads to learned policies that match experts better, achieve better performance, and converge faster.", "weaknesses": "**Figure 2 standard deviation**\n- Are the results reported in Figure 2 aggregated from five random seeds? Since Figure 2 does not show standard deviation, I am unsure if the gaps are statistically significant.\n\n**Table 1 Hopper results**\n- I am wondering why BC (2830) outperforms the expert (2631) in Hopper.\n\n**Limited domains for evaluation**\n- The evaluation is limited to locomotion, e.g., Half-Cheetah and Walker2D. Experimenting with the proposed method and the baselines in other domains, such as navigation (grid world or point maze in D4RL), robot arm manipulation (OpenAI Fetch or Shadow Dexterous Hand), and games (Atari) would significantly strengthen the results.\n\n**Visualized state distributions**\n- I appreciate the authors showing the state Wasserstein distance between expert and learned policies in Figure 2. I feel it would be informative to visualize the state distributions of expert and learned policies. For example, we can use a grid world navigation task with discrete state and action spaces. Then, we can visualize the state distributions of expert and learned policies as heatmaps and put them side by side for comparison. I believe this would make the claim that C-GAIL-DAC can match the expert state distributions better and more convincing.\n\n**Related work**\n- Including the descriptions of more recent IL methods could make the related work more comprehensive, such as\n    - Diffusion policy (https://arxiv.org/abs/2303.04137v4 https://arxiv.org/abs/2403.03954)\n    - Consistency Policy (https://arxiv.org/abs/2405.07503)\n    - Diffusion BC (https://arxiv.org/abs/2302.13335)\n    - DiffAIL (https://arxiv.org/abs/2312.06348) / DRAIL (https://arxiv.org/abs/2405.16194v1)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We are delighted that we were able to communicate the value of our work. We address your questions below, and have provided several new results in the global rebuttal.\n\n**Q1. Figure 2 standard deviation**\n \nA1. The results in Figure 2 are indeed aggregated from five random seeds. Thank you for pointing out that we did not report error bars on the plot. We present the standard deviation as tables in global rebuttal Q4 and will modify our figure. \n\n**Q2. Table 1 Hopper results exceed expert**\n\nA2. This is a good observation, and we do not have a comprehensive answer for why the agent average slightly surpasses expert performance. This also happened in the new DiffAIL results. It's possible that the agent overfits to one of the more successful demonstration trajectories. However, we notice that the expert return is within the error bars of our method. \n\n\n**Q3. Additional experiment domains**\n\nA3. We agree that our results could be more impactful with further domains. We have added in additional environment on Atari as per the global rebuttal.\n\n\n**Q4. Additional IL algorithms in related work**\n\nA4. Thank you for pointing out this gap in our related work, which we will add to the next version of our paper. Moreover, inspired by the reviewer's comment, we have conducted new experiments on DiffAIL in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jgWmWZ0b1n", "reviewer": "Reviewer_DQNe", "summary": "The paper titled \"C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory\" addresses the challenge of training instability in Generative Adversarial Imitation Learning (GAIL), a method used to train a generative policy to imitate a demonstrator's behavior. The authors analyze GAIL's optimization from a control-theoretic perspective and identify that GAIL does not converge to the desired equilibrium due to the entropy term in its objective. To resolve this, they propose Controlled-GAIL (C-GAIL), which introduces a differentiable regularization term to the GAIL objective to stabilize training. Empirical results demonstrate that C-GAIL improves upon existing GAIL methods, including GAIL-DAC, by accelerating convergence, reducing oscillation, and more closely matching the expert's policy distribution. The paper contributes a novel control-theoretic approach to stabilize adversarial training in imitation learning, offering both theoretical insights and practical algorithmic advancements.", "questions": "1. While the paper demonstrates the effectiveness of C-GAIL in the context of MuJoCo control tasks, how well does the proposed control-theoretic approach generalize to other domains, such as autonomous driving or game playing, where the dynamics of the environment and the complexity of the tasks might be significantly different?\n\n2. Given that the practical updates in GAIL training are discrete whereas the theoretical model assumes a continuous dynamical system, what is the impact of this discrepancy on the long-term stability and performance of the learned policies, especially in tasks with high stochasticity or where the environment reacts non-linearly to actions?\n\n3. The paper mentions that a proper selection of the hyperparameter k is crucial for the effectiveness of the C-GAIL controller. Can the authors provide more insights on how to determine the optimal values for these hyperparameters in a data-driven manner, without relying on extensive hyperparameter tuning, and how sensitive is the performance of C-GAIL to these choices?", "limitations": "1. The paper formulates GAIL as a continuous dynamical system for the purpose of stability analysis, but in practice, updates to the generator and discriminator are discrete. This discrepancy between the theoretical model and practical implementation may impact the applicability of the theoretical results in real-world scenarios.\n\n2. The practical implementation of the controller is applied only to the discriminator's objective function. The paper does not provide a method for incorporating a controller for the policy generator, which would require knowledge of the expert policy that is not available during training.\n\n3. The stability guarantees are based on certain assumptions, such as hyperparameter conditions. The paper does not fully explore how the results might be affected when these assumptions do not hold, which could limit the generalizability of the findings to different settings or problem domains.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a novel application of control theory to stabilize the training process of Generative Adversarial Imitation Learning (GAIL). By analyzing GAIL from a control-theoretic perspective, the authors provide a deeper understanding of the optimization challenges inherent in GAIL and propose a theoretical solution that ensures asymptotic stability.\n\n2. The authors develop Controlled-GAIL (C-GAIL), a practical algorithm that incorporates a regularization term derived from control theory. This algorithm is shown to improve the stability and convergence of GAIL in empirical tests, offering a tangible advancement that can be applied to existing GAIL methods to enhance their performance.\n\n3. The paper provides extensive empirical evidence to support the effectiveness of C-GAIL. Through experiments on MuJoCo control tasks, the authors demonstrate that C-GAIL achieves faster convergence, reduces the range of oscillation in training, and more closely matches the expert's policy distribution compared to both the original GAIL and other variants, showcasing the robustness and practical applicability of their approach.", "weaknesses": "1. While the paper proposes a theoretically sound controller for stabilizing GAIL training, the practical implementation of the controller is only applied to the discriminator's objective function. The paper acknowledges that the generator's portion of the controller, which would require knowledge of the expert policy, is not used during training. This limitation means the full potential of the control-theoretic approach is not realized in practice.\n\n2. The paper formulates GAIL training as a continuous dynamical system for the purpose of stability analysis. However, in actual practice, the updates to the generator and discriminator are discrete. The discrepancy between the theoretical model and the practical implementation could potentially affect the real-world applicability of the proposed controller.\n\n3. The stability guarantees provided by the paper rely on certain assumptions, such as specific conditions on the hyperparameters of the controller. These assumptions might not hold in more general settings or across different problem domains, which could limit the broad applicability of the results. Additionally, the paper does not fully explore how violations of these assumptions might impact the performance of C-GAIL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We are pleased we were able to communicate the strengths of the work effectively. We value your feedback on the gap between theory and practice. We have included a general response to this point in the global rebuttal, and here offer more targeted comments. We hope this, along with further experiments in game playing environments, and clarifying the importance of hyperparameter selection, might warrant an uplift in your score.\n\n**Q1 Theoretical controller to practice**\n\nA1. As noted by the reviewer, we were unable to implement the derived generator regularizer since we don't have access to the expert policy. We would like to emphasize that the *theoretical contribution* made by our work remains -- our theory is non-trivial, offering a new way to study GAIL, and insight into what an ideal generator regularizer would look like, which may inspire future work. We showed empirically that the discriminator alone is still of significant practical benefit by itself.\n\n**Q2. Continuous vs Discrete system**\n\nA2. Gradient flow [1] is a widely used technique across optimization theory to transform discrete gradient step updates to a continuous differential equation. This is critical to allow systems to be studied analytically. We agree with the reviewer that understanding what is lost in this conversion is of interest. However, given the ubiquity of the technique we feel it is beyond the scope of our work to study this in depth in our specific setting. We do note that our empirical results, which use discrete updates (algorithm 1), show the benefit of our continuous study does transfer to the discrete setting.\n\n\n**Q3. Hyperparameter assumption requirements**\n\nA3. Assumption 4.2 specifies an allowed range of hyperparameters in order for convergence to be guaranteed in theory. It is not clear how impactful this is when choosing $k$ at implementation time, since one needs to know $c$ ($p(s)$) for each state. So the allowed bounds are more valuable from a theoretical perspective. \n\n**Q4. Experiments on other tasks**\n\nA4. Thank you for this suggestion. We have added game environments (Atari) under the global rebuttal.\n\n**Q5 Selection of hyperparameters**\n\nA5. Thank you for this comment, which suggests we have not communicated the effect of the hyperparameter $k$ clearly. We provided an ablation study on $k$ in Figure 4. This shows that C-GAIL is effective under a wide range of $k$ values, from 0.1 to 10, with only marginal gains to be found by tuning it. As such, one of the benefits of our method is the insensitivity to $k$ and lack of need for tuning. We will emphasize this point more clearly in the next version.\n\n[1] Conley, C., 1988. The gradient structure of a flow: I. Ergodic Theory Dynam. Systems, 8.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0aAYLjXNC3", "reviewer": "Reviewer_jmhJ", "summary": "The paper formulated training process of GAIL as a dynamic system. From control theory’s view, authors pointed out GAIL would not converge to the desired state where the generator perfectly matches with the expert policy and the discriminator cannot distinguish generated from expert trajectories. Hence, authors proposed a new regularizer to stabilize GAIL.", "questions": "1. Is there a missing parenthesis in the formula at line 181? $\\tilde{V}_D(D, \\pi)=\\iint_0 p(s) \\pi(a \\mid s) \\log D(s, a)+\\pi_E(a \\mid s) \\log (1-D(s, a)) d s d a,$\n\n2. It seems tricky that theoretical analysis is done under a strict condition, where one-time step environment is considered and controllers are applied both on policy and discriminator, but original policy objective is applied for updating in              implementation, although experimental results are good. Could you explain it more, making it convincing?", "limitations": "The authors have discussed the limitations and future work in Section 7.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I have appreciated the logical framework regarding training process of GAIL as a dynamics system, where more control theory tools could be introduced in. \n\n2. Theoretical analyses are solid, making proposed regularized more convincing.\n\n3. The topic selection is meaningful, revealing the issue of training instability in GAIL. Proposed controller is quite necessary for stabilizing GAIL.", "weaknesses": "1. This work showed GAIL cannot converge to the only desired state because of biased entropy term. Is it possible that entropy term introduces subtle bias in but improves stability a lot? The case will make your work meaningless. I think more experiment should be interpreted from this view. \n\n2. The GAIL actually minimizes JS divergence between $\\rho_{\\pi_{E}}$ and $\\rho_{\\pi}$. For simplicity, the proposed 'C-GAIL' substituted $\\rho_{\\pi}$ with $\\rho$. Does it ignore differences on state distribution? Why the performance still be good?\n\n3. Did this work just substitute a biased regularizer with an unbiased one? It seems that authors should refer success of experiments only to strong convexity of applied controller. I think more details should be supplemented for making your analysis meaningful. \n\n4. More advanced benchmarks should be compared in the experimental section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we are pleased to have communicated the value of our control-theoretic approach. We hope our clarification around the biased-ness of GAIL, and addition of further experiments, might encourage an increase your evaluation of the paper.\n\n**Q1. The entropy term and bias**\n\nA1. We don't believe it's correct to say \"the entropy term introduces subtle bias in but improves stability\". The role of the entropy term is to allow the generator to explore the state-action space and discover how to reproduce the expert trajectories. But it does not improve stability directly. We have proved that the entropy term prevents the system from converging to the desired global equilibrium, it is not clear to us that there is some other \"biased\" equilibrium that it is pushed to instead -- empirical evidence comes from the oscillations in the GAIL-DAC training curves (Figure 1) suggesting there is no alternative more stable local equilibrium.\n\n\n**Q2. Ignoring differences on state distribution and performance**\n\nA2. We indeed ignore the effect of policy changes to the state distribution (switching $\\rho_{\\pi_E}$ with $\\rho$) and only consider the influence of $\\pi$ on actions in our one-step setting. We believe this is not uncommon in RL theory to simplify analysis. One example is TRPO [1], which similarly allows the original policy distribution to approximate the updated distribution and provides a theoretical justification for this approximation (Eq2 to Eq6 in its paper). \n\nNotice that the methodology of starting with simplified setting in theory and extend it to general setting in practice is also commonly used in GANs (global rebuttal Q3). \n\nOur intuition for why performance is still good with only a controller on the discriminator is that C-GAIL pushes the discriminator to its equilibrium at a faster speed by introducing a penalty controller centered at $\\frac{1}{2}$. This leads GAIL’s training to converge faster with a smaller range of oscillation, and match the expert distribution more closely.\n\n\n\n\n**Q3. Biased regularizer and strong convexity**\n\nA3. We do not believe \"bias\" is the right term here. In control theory we aim for certain equilibria. As per our above response, it's not clear that GAIL with entropy converges to any equilibrium. Further more, we are not replacing this entropy term, but introducing an additional regularizer that does make the system stable around the desired equilibrium.\n\nWe are unsure what the convexity comment refers to. None of our results rely on this assumption.\n\n**Q4 More benchmarks**\n\nA4. Thank you for this suggestion. We have added additional experiments as per the global rebuttal.\n\n**Q5. Missing parenthesis**\n\nA5. Thank you for pointing it out. We will modify it.\n\n\n**Q6. One-step setting and lack of policy regularizer**\n\nA6. Please see our response A2 and global rebuttal A3.\n\n\n[1] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. ICML, 2015.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rfupfvf21K", "reviewer": "Reviewer_qTQn", "summary": "This paper proposes a stabilized version of the Generative Adversarial Imitation Learning (GAIL) through control theory, addressing limitations related to achieving equilibrium and slow convergence. The authors conduct a theoretical analysis using control theory to investigate the properties that influence equilibrium achievement. Empirical validation demonstrates that the proposed modifications improve performance and training stability.", "questions": "1. How does the simplified one-step GAIL compare to the full GAIL in terms of performance and convergence?\n2. Can the authors provide a quantitative analysis of training oscillations shown in Figure 1 and correlate them with training stability? It would be beneficial to quantify these oscillations, as they are a major contribution to the paper. It is mentioned 3 times for Walker2d at line 296, but how is this number computed? Similarly, 10x is mentioned in line 298. Additionally, how do these oscillations indicate training stability, and what is the correlation?\n3. How is the controller designed and implemented in the system evaluated? Is there a way to automate the design of controllers u_1 and u_2, and what are the empirical hyperparameters involved?\n4. In Figure 2, what is the high distance value on the y-axis for C-GAIL-DAC?  Could the authors explain the initial higher values?\n5. The number of demonstrations (four) mentioned (line 273) and used in the paper seems small. Can the authors elaborate on how this number suffices for learning effective policies?", "limitations": "Yes. The authors have addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an important problem in the stability of GAIL, which is crucial for the practical implementation of imitation learning techniques.\n- The paper provides a theoretical foundation, analyzes the GAIL objective, and demonstrates its limitations and proposed improvements.\n- The experimental results validate the theoretical claims, showing improved performance and stability in training.", "weaknesses": "- It is unclear how the one-step GAIL, a simplified version, generalizes to the full GAIL framework and impacts policy learning.\n- The practical settings used in the experiment do not guarantee convergence, raising questions about the need for evaluating one-step GAIL.\n- The paper mentions using a linear negative feedback controller to reduce oscillations but lacks details on its implementation and usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We're pleased you have seen value in our new control-theoretic approach to understanding and stabilizing GAIL. We appreciate several nuanced observations you raised, and respond to these below.\n\n**Q1. One-step vs full GAIL**\n\nA1. Theoretically, our controllers work under one-step setting. We follow the insights from one-step setting and extend our controller to full GAIL as C-GAIL.  It is difficult to know how to study the difference between the one-step and full setting analytically -- if able to do this, we could directly design controllers for the full setting! It is an interesting direction for future work. However, we have empirically studied the effectiveness of our C-GAIL under the full setting through our experiments. Also see our response A2 to R jmhJ, which notes similar approximations have been used in popular methods such as TRPO, and the global rebuttal A3.\n\n\n**Q2. Quantifying oscillations**\n\nA2. This is a good point. We have used the term \"oscillations\" informally in the text, without defining it rigorously. A direct way to interpret this is as variance in the reward curves across runs -- viewing the size of the shaded standard deviation in Figure 1 would be one way to quantify this (or reading off the final standard deviations in Table 2). Another way is to consider the oscillations between successive training time steps -- we do not have a single metric capturing this, but it can be viewed on Figure 1 also. \n\n**Q3. Design and implementation of the controllers**\n\nA3. Note that our controllers $u_1$ and $u_2$ only need to be derived once for a given dynamical system. Automating their design in an arbitrary new dynamical system is an interesting direction for future work, but beyond the scope of this paper. We have chosen to use additive controllers, which are implemented as an addition term to the objective functions (line 238-239). This is summarized in Algorithm 1.  Notice that C-GAIL only introduces one additional hyperparameter $k$ (this is one of the attractions of our method). We provide an ablation study in appendix E, finding that C-GAIL is not particularly sensitive to the choice of $k$. \n\n\n**Q4. High initial value on C-GAIL-DAC** \n\nA4. Thank you for this interesting observation. To reiterate; Figure 2 presents the state Wassertein distance for GAIL-DAC and C-GAIL-DAC. The state Wassertein distance measures the difference between expert and generator distributions at a given point in training. Higher values indicate a larger difference between expert and generator. As noted by the reviewer, in 4/5 environments C-GAIL-DAC briefly has a higher (worse) metric than GAIL-DAC, though in the long run always ends up lower (better). \nWe are not certain why this occurs, particularly as the reward curves are almost always strictly better for C-GAIL-DAC (Figure 1). Intuitively, since our controller for the discriminator encourages convergence to $ \\frac{1}{2}$, the corresponding generator may have a larger distance to the expert one initially. However, as the discriminator approaches its equilibrium, the generator may become stable too as a response.\n\n**Q5. Number of expert demonstrations**\n\nA5. Thank you for this feedback -- we will add a discussion on this in the next version of the paper. We used four expert demonstration to be consistent with the original GAIL-DAC paper [1]. As we understand, one of the advantages of GAIL is that it requires less expert trajectories than other methods (e.g. BC). It's possible that other environments and GAIL variants may require larger numbers of demonstrations (it also depends on the length of each trajectory). We do provide an ablation in Figure 3 with up to 18 demonstrations.\n\n[1] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lxp8VbbnO3", "reviewer": "Reviewer_9gMV", "summary": "This paper theoretically analyzes the training dynamics of GAIL, a widely discussed issue in GANs, pointing out that the original GAIL cannot converge to the desired equilibrium. From a control theory perspective, the paper propose C-GAIL, which can achieve asymptotic stability. The paper demonstrate that C-GAIL, compared to several baseline methods, accelerates convergence, reduces oscillations, and more closely approximates the expert distribution in 5 typical MoJoCo control tasks.", "questions": "- What are the main differences between C-GAIL and similar methods in GANs? It is recommended to provide a separate discussion.\n- The authors claim that the reason for the original GAIL's failure to converge to equilibrium is due to the entropy regularization term. Removing entropy can ensure convergence to equilibrium at the cost of exploration. However, if $V^{\\prime}_{\\pi}$ is not computable, will it affect the policy's exploration? Compared to directly removing entropy, does C-GAIL show an advantage in exploration?\n- The theoretical analysis is based on the original GAIL, which measures JS divergence. For variants like WGAIL, LS-GAIL, and f-GAIL, can convergence and stability still be guaranteed?\n- How long is each expert demonstration in the experiments? How were the expert demonstrations obtained?\n- In Figure 2, what causes the significant drop in the blue line in the Hopper environment? Were the curves plotted using the averages of multiple experiments?\n- $\\bar{s}$ and $\\bar{a}$ in line 89 are undefined.\n- Table 2 lacks the expert reward baseline as a reference.", "limitations": "The limitations of the method have been outlined in the discussion.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Well-written paper with clear objectives. The authors have the intention to share the code.\n- The idea is novel for GAIL, although it draws on concepts from GAN.\n- Theoretical analysis is complete.\n- The method is simple to implement and can be easily applied to various existing GAIL methods.", "weaknesses": "- The theoretical analysis is based on quite a few assumptions, and the final implementation method is only a rough approximation of the theoretical results.\n- The experimental section is not comprehensive enough. Given that the method is simple and easy to deploy, can its effectiveness be validated in more GAIL variants and environments?\n- There is a lack of comparison with the latest GAIL variant algorithms; only GAIL (2016) and GAIL-DAC (2018) are shown in Figures 1, 2, and 3.\n- There is a divergence between the theoretical motivation of the paper and the final algorithm implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We hope our response might provide sufficient evidence that our method is general enough to consider upgrading your score. In particular please note the additional experiments on a new environment, and the three additional variants (two already included in Appendix F, one new in the global rebuttal).\n\n**Q1. From theory to practical settings**\n\nA1. Please see our global review A3.\n\n\n**Q2. More environments**\n\nA2. We have included a set of Atari results under the global rebuttal. Please let us know any specific additional environments you'd like to see.\n\n\n**Q3. Latest GAIL variants**\n\nA3. While the results in the paper's main body indeed focus on GAIL and GAIL-DAC, Appendix F Table 2 already includes comparisons with two other recent GAIL variants -- BC-GAIL and WAIL. We will make these results more prominent in the next version of the paper. In addition, we have included a diffusion variant in the global rebuttal. Please let us know any specific additional variants you'd like to see.\n\n\n**Q4. Differences between C-GAIL and similar methods in GANs?**\n\nA4. Compared with GANs, the policy generator of GAIL involves an MDP transition, which results in a much more complicated dynamical system induced by a policy acting in an MDP rather than a static data generating distribution. Prior theoretical analysis and controllers are therefore inapplicable. We adopt different analysis (different ODEs) and controlling techniques (local linearization), to present a new stability guarantee, controller, and theoretical results for the different dynamical system of GAIL.\n\n\n**Q5. Exploration for entropy term**\n\nA5. Thank you for this comment. We understand that inclusion of the entropy term is critical to allow the generator policy to explore the state space. C-GAIL maintains this entropy term, while additionally adding a controller to stabilize the system. Removing the entropy term directly as you suggest is an interesting proposal, but we expect this would cause the algorithm to collapse to some local minimum without fully exploring the space. \n\n\n**Q6. Theoretical guarantee for other GAIL variants**\n\nA6. Our work presents a methodology to analyse the training process as a dynamical system and design a controller to stabilize its training. To design a controller for other GAIL variants, we would start with their objective functions and take a derivative to get their dynamical system respectively. The only difference is on the design of the generator's controller to ensure local linearization theorem. The controller for the discriminator will remain the same.\n\nIn this manner, since our implementation of C-GAIL only adds controllers for the discriminator, our C-GAIL is compatible with WAIL, LSGAIL and f-GAIL. We have provided our result compared with WAIL in appendix F.\n\n\n**Q7. Length of expert demonstration and how obtained**\n\nA7. Our main experiments follow the same settings as GAIL-DAC [1], which in turn followed vanilla GAIL [2] which used 50 timesteps per trajectory. The expert policies were trained with TRPO.\n\n\n\n**Q8. Drop in Hopper Figure 2,  number of random seeds**\n\nA8. This is a good observation. We are not certain why there is a sudden dip in Wasserstein distance for Hopper. It's possible that, because the distance is computed between a low number of expert and generated trajectories, there is noise in our measurement, leading to these unexpected spikes. \n\nFigure 2 is computed by averaging five random seeds. We apologize for the confusion as we did not originally include the error bars. We present the standard deviation as table in global rebuttal Q4 and will modify our figure.\n\n\n\n**Q9. Undefined notations**\n\nA9. Thank you for pointing it out. We will properly define them to avoid confusion. They are substitutes notation for $s$ and $a$ since we have the initial condition $s_0 = s$ and $a_0 = a$.\n\n**Q10. Table 2 missing expert return**\n\nA10. Thank you for noticing this. We will add an extra row for expert return. Since we are conducting on the same Mujuco tasks as Figure 1 and Table 1, the expert return is the same. \n\n\n[1] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925, 2018.\n\n[2] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565–4573, 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal for common questions

Thanks to all reviewers for their constructive feedback. We were pleased our work received a favorable assessment. Whilst we address each reviewer's questions individually, this global rebuttal summarizes our response to common points highlighted by several reviewers. 1) Whether C-GAIL's improved stability holds in further environments. 2) How our controlled variant interacts with other SOTA AIL methods. 3) Clarifying the gap between our theory and practical implementation. 4) We computed standard deviations from Figure 2.

**Q1. Additional environments**

A1. 
As noted by R PPsU, DQNe & 9gMV, our experiments focused on continuous control in Mujoco tasks. To explore whether C-GAIL brings benefit in other environments we have run additional experiments using (vanilla) GAIL in several Atari games. C-GAIL achieves higher final reward on 4/5 games, always with lower variance. 


|| BeamRider | Pong | Q*bert | Seaquest| Hero
|---| -------- | ------- |------- |------- |------- |
Expert (PPO)| 2637.45 $\pm$ 1378.23| 21.32 $\pm$ 0.0| 598.73 $\pm$ 127.07 | 1840.26 $\pm$ 0.0| 27814.14 $\pm$ 46.01|
GAIL |1087.60 $\pm$ 559.09	|-1.73 $\pm$ 18.13| -7.27 $\pm$ 24.95| 1474.04 $\pm$ 201.62| 13942.51 $\pm$ 67.13| 	
C-GAIL | 1835.27 $\pm$ 381.84 | 0.34 $\pm$ 8.93 | 428.87 $\pm$ 12.72 | 1389.47 $\pm$ 80.24| 23912.73 $\pm$ 32.69

Table R1. Final reward in five Atari tasks [1]. Mean and standard deviation over ten runs. Based on the implementation here: https://github.com/yunke-wang/gail_atari.


**Q2. Additional GAIL variants**

A2.
R 9gMV requested a comparison with more GAIL variants. R PPsU brought our attention to a new family of diffusion-based GAIL algorithms. We were curious to investigate whether the C-GAIL regularizer would bring benefit in this new class of GAIL methods. Run in Mujoco, the table below presents results. Whilst the final performance for both the controlled and non-controlled variants all roughly match (or exceed) the expert return, the standard deviation of C-DiffAIL is consistently smaller than DiffAIL.

|| Hopper | HalfCheetah | Ant | Walker2d|
|---| -------- | ------- |------- |------- |
Expert| 3402| 4463| 4228| 6717|
DiffAIL | 3382.03 $\pm$ 142.86 | 5362.25 $\pm$ 96.92| 5142.60 $\pm$ 90.05 | 6292.28 $\pm$ 97.65| 
C-DiffAIL| 3388.28 $\pm$ 41.23 | 4837.34 $\pm$ 30.58 | 4206.64 $\pm$ 36.52 | 6343.89 $\pm$ 33.67

Table R2. Final reward with 1 trajectory in diffusion-based GAIL [2], both vanilla and controlled variant. Mean and standard deviation over five runs. Based on the implementation here: https://github.com/ML-Group-SDU/DiffAIL.




**Q3. From theory to practice**

A3. 
Several reviewers correctly noted differences between the assumptions made in our theory, and the practical implementation of our algorithm. These include moving from a continuous flow to discrete updates (R DQNe), studying the one-step setting (R qTQn), only implementing the discriminator regularizer (R DQNe). (R 9gMV also notes this more generally.) 

While we provide a specific responses to each assumption separately in our individual responses, we wanted to make a more general response here. We'd like to emphasize that theoretical studies of deep learning usually make simplifying assumptions in order to make progress analytically. This approach of beginning from simplified theory and applying insights it to a practical setting has often been found to be valuable. For example, approaches to stabilize GANs often follow this approach [3][4][5]. 

**Q4. Figure 2 error bars** 

A4. Figure 2 is computed by averaging five random seeds. We did not include error bars in the original version. We present the standard deviation as table below and will modify our figure in the next version. Note that the standard deviations of C-GAIL-DAC are nearly always lower than for GAIL-DAC.

|              | 0.2M | 0.4M | 0.6M | 0.8M | 1M |
|--------------|------|------|------|------|----|
| Half-Cheetah | 1.32|    0.95  |   0.83 | 0. 86| 0.81 |
| Walker       |  0.65  |   0.42 | 0.46  | 0.41  | 0.43 |
| Reacher      | 0.53   |  0.46   | 0.53|  0.55 | 0.52   |
| Ant          |  0.87   | 0.75  | 0.79 | 0.76 | 0.82   |
| Hopper       |   0.37   | 0.29 | 0.34  | 0.67  | 0.46   |
Table R3. Standard deviation for the state Wasserstein distance of GAIL-DAC 

|              | 0.2M | 0.4M | 0.6M | 0.8M | 1M |
|--------------|------|------|------|------|----|
| Half-Cheetah | 0.77 |  0.49|  0.44 | 0.36   | 0.38 |
| Walker       | 0.52  | 0.37   | 0.36   | 0.35   | 0.36  |
| Reacher      | 0.61 | 0.41   | 0.37  | 0.39   |0.38    |
| Ant          | 0.65 | 0.62    | 0.58    |0.58  | 0.56   |
| Hopper       |  0.23  | 0.20  | 0.16  | 0.14   | 0.15   |
Table R4. Standard deviation for the state Wasserstein distance of C-GAIL-DAC


[1] Wang et al., 2021. Learning to weight imperfect demonstrations. ICML. 

[2] Wang et al., 2024. DiffAIL: Diffusion Adversarial Imitation Learning. AAAI.

[3] Mescheder et al., 2018. Which training methods for GANs do actually converge? ICML.

[4] Xu et al., 2020. Understanding and stabilizing GANs’ training dynamics using control theory. ICML.

[5] Luo et al., 2023. Stabilizing GANs’ training with brownian motion controller. ICML.

Meta-review: This paper makes contributions to stabilize GAIL (generative adversarial imitation learning) via control theory. Four reviewers are all supportive on the paper and two of them are recommending clear acceptance. Through the author-reviewer discussion, several concerns of the reviewers are addressed and the assessment improved.

Strengths of the paper include solid theoretical contributions by connecting control theory with GAIL and novel regularization methods to stabilize GAIL with rich empirical verifications. There are also weaknesses mentioned by the reviewers and many of them are resolved with additional experiments and explanation, including standard deviations in figure 2 and more testing environments.

Therefore, this paper is recommended for acceptance. The authors are also encouraged to discuss (and incorporate the posted discussion on) the simplified assumptions and the generalizability of the theories.
Decision: Accept (poster)