Incompatibility between Deterministic Policy and Generative Adversarial Imitation Learning
Authors: Wanying Wang
Yirui Zhou
Chaomin Shen
Yangchun Zhang
Jian Tang
Zhiyuan Xu
Yaxin Peng
Venue: Submitted to ICLR 2023 (2023)

Abstract: Deterministic policies are widely applied in generative adversarial imitation learning (GAIL). When adopting these policies, some GAIL variants modify the reward function to avoid training instability. However, the mechanism behind this instability is still largely unknown. In this paper, we capture the instability through the underlying exploding gradients theoretically in the updating process. Our novelties lie in the following aspects: 1) By employing multivariate Gaussian policy with small covariance to approximate deterministic policy, we establish and prove the probabilistic lower bound for the exploding gradients, which can describe the degree of instability universally, while the stochastic policy will never suffer from such pathology subsequently. 2) We also prove that the modified reward function of adversarial inverse reinforcement learning (AIRL) can relieve exploding gradients, but at the expense of ``non-confidence''. Experiments and a toy demo support our analysis.

Under review as a conference paper at ICLR 2023
INSTABILITY IN GENERATIVE ADVERSARIAL IMITA- TION LEARNING WITH DETERMINISTIC POLICY
Anonymous authors Paper under double-blind review
ABSTRACT
Deterministic policies are widely applied in generative adversarial imitation learn- ing (GAIL). When adopting these policies, some GAIL variants modify the reward function to avoid training instability. However, the mechanism behind this insta- bility is still largely unknown. In this paper, we capture the instability through the underlying exploding gradients theoretically in the updating process. Our novel- ties lie in: 1) We establish and prove a probabilistic lower bound for exploding gra- dients, which can describe the instability universally, while the stochastic policy will never suffer from such pathology subsequently, by employing the multivari- ate Gaussian policy with small covariance to approximate deterministic policy. 2) We also prove that the modified reward function of adversarial inverse reinforce- ment learning (AIRL) can relieve exploding gradients. Experiments support our analysis.
INTRODUCTION
Imitation learning (IL) trains a policy directly from expert demonstrations without reward signals (Ng et al. 2000 Syed & Schapire 2007 Ho & Ermon] 2016). It has been broadly studied under the
twin umbrellas of behavioral cloning (BC) (Pomerleau 1991) and inverse reinforcement learning
(IRL) (Ziebart et al. 2008). Generative adversarill imitation learning (GAIL) (Ho & Ermon (0166) established by the trust region policy optimization (TRPO) (Schulman al., 2015) policy training, plugs the inspiration of generative adversarial networks (GANs) (Goodfellow et al., 2014) into the maximum entropy IRI. The discriminator GAIL. aims to distinguish whether a state-action pair comes from the expert demonstration or is generated by the agent. Meanwhile, the learned policy generates interaction data for confusing the discriminator. GAIL is promising for many real-world scenarios where designing reward functions to learn the optimal control policies requires significant effort. It has made remarkable achievements in physical-worrs tasks, e.g., robot manipulation (Jabri, 2021]. mobile robot navigating (Tai al. 2018), commodities search (Shi et al. 2019), endovascular catheterization (Chi et al. 2020). etc.
The learned policy in GAIL be effectively accomplished by reinforcement learning (RL) meth- ods (Sutton & Barto 2018 [Puterman] (0014), which are divided into stochastic policy algorithms and deterministic policy algorithms, incorporating the two classes of algorithms into GAIL denoted as ST-GAIL and DE-GAIL respectively in our paper. For ST-GAIL, one can refer to the proximal pol- icy optimization (PPO)-GAIL (Chen al., 2020), the natural policy gradient (NPG)-GAIL (Guan et al. 202T) and the two-stage stochastic gradient (TSSG) (Zhou et al. 2022). These algorithms high- -igh-diimnsional environments against
have shown that GAIL can ensure global convergence
traditional IRL. methods (Ng et al. 20000 Ziebart et al. [0088] Boularias et al. 2011). Unfortunately. ST-GAIL methods have low sample efficiency and cost long time to train the learned policy well
Zuo et al. 2020).
In comparison, some related works (Kostrikov et al. 2019 Zuo et al. 2020) imply that deterministic policies are capable of enhancing sample efficiency when training GAIL variants. Kostrikov et al. 2019)) proposed the discriminator actit--rritttooiiiiiiii (DAC) algorithm, which defines the reward function with the GAIL discriminator for the policy trained by the twin delayed deep deterministic policy gradient (TD3) (Fujimoto et al. 2018). It reduces policy- envirceenvironment interaction sample complexity by average factor of 10. Deterministic generative adversarial imitation learning (DGAIL.) (Zuo)

---

Under review as a conference paper at ICLR 2023
et al. 2020) utilizes the modified deep deterministic policy gradient (DDPG) (Lillicrap et al. 2015)
to update the policy, and achieves a faster learning speed than ST-GAIL.
These DE-GAIL methods not only effectively improve sample efficiency, but also mitigate instabil- ity caused by the reward function - -og((1 - [s,, a)), referred to as the positive logarithmic reward function (PLR). This classical logarithmic reward function is one of the primary shapes and is fre- quently used in GAIL. For PLR. Kostrikoy et al. (2019) discussed the reward bias in a simple toy demo: Zuo al. 22200) exhibited its instability through contrast experiments, and improved the sta- bility of DE-GAIL by introducing the idea of learning from demonstrations (LfD) into the generator.
However, the instability caused by PLR is largely unknown. We investigate the perfor- mance of DDPG-GAIL and TD3-GAIL, i.e.,
DOPG-GAL in Walkerdda2
TD3-GAL Walker2d-2
1500
1000-
3003
2000
replacing TRPO to RL algorithms DDPG and TD3 with unchanged PLR, with 10 expert demonstrations displayed Fig. [ emerges extreme instability under 11 random seeds. The learning effects under some seeds are capable of reaching expert levels (valid), while others hardly learn anything (invalid). At some point, the two corner cases cannot be well captured by the bias analysis introduced through the toy demo in Kostrikov et al. (019)).
2003
1500
1000
1003
Number Samples
Number Samples
Figure 1: Learning curves of DDPG-GAIL and TD3-GAIL. with different random seeds in Walker2d-v2.
In this paper, we incorporate different deterministic policy algorithm, i.e., the softmax deep double deterministic policy gradients (SD3) (Pan et al.) 2020) into GAIL with PLR. SD3-GAIL exhibits the same instability as the experiment results in DDPG-GAIL and TD3-GAIL, This implies that instabil- ity is not special case in DE-GAIL. In addition, the instability of DE-GAIL is mainly down to the invalid case that appeared in the experiment. Then, we prove that there exist exploding phenomena in the absolute gradients of the policy loss, describing the invalidity theoretically and universally. Further, we conclude that the discriminator will possibly degenerate the probabilistic lower bound for exploding gradients, with respect to the mismatching between the learned policy and the expert demonstration, i.e., the significant state-action distribution discrepancy between the learned sattenaction pair and that of the expert. Finally, we disclose that the outliers in- terval under the modified reward function in adversarial inverse reinforcement learning (AIRL) (Fu eet al. 2018) is smaller than that under PLR in GAIL. Such modified reward function shows its superiority of stability. Our contributions can be summarized as follows:
1. Meanwhile, we give
Establish and prove the probability of exploding gradients theorem to theoretically describe the instability DE-GAIL. contrast, our analysis strikes that stochastic policy can avoid exploding gradients.
Compared with the consistent trend in ST-GAIL, we point out the instability is caused by deterministic policies, rather than GANs.
The reward function in AIRL is shown to reduce the probability of exploding gradients.
2 RELATED WORK
In large and high-dimensionall environments, Ho & Ermon 2016) proposed GAIL, which is pro- cessed on TRPO (Schulman et al. 2015). It gains significant performance in imitating complex
expert policies (Ghasemipour al. 2019 Xu et al. 2020 Ke et al. 2020 Chen et al., (02T).
To accelerate the GAIL learning process. natural idea is to use deterministic policy gradients. Sample-efficient adversarial mimic (SAM) (Blondé & Kalousis 2019) method integrates DDPG (Lillicrap et a (015)) into GAIL, and adds a penalty on the gradient of the discriminator mean- while. Zuo et al. (2020) proposed deterministic generative adversarial imitation learning (DGAIL), which combines the modified DDPG with LID to train the generator under the guidance of the discriminator. The reward function in DGAIL is set as (ss. TD3 (Fujimoto et al. 2018 and off- off-policy training of the discriminator are performed in DAC (Kostrikov et al. 2019) to reduce policy-environmee interaction sample complexity by an average factor of 10. The revised reward function in DAC is log( D(s, a)) log(1 D(s,a)) a)).
2

---

Under review as a conference paper at ICLR 2023
Notably, these works achieve gratifying results benefiting from modifications reward functions. When implementing GAIL with PLR directly, Kostrikov et al. (2019) pointed out the reward bias via special toy demo; DGAIL exhibited its instability merely from the perspective of experimental results. Differently, we illustrate this phenomenon through a universal theory.
3 PRELIMINARY
In this section, we introduce the definition of a Markov decision process, reproducing kernel Hilbert space and generative adversarial imitation learning setup.
3.1 MARKOV DECISION PROCESS
A discounted Markov decision process (MDP) is characterized by 5-tuple (S, PM in the standard RL setting. S and A denote the finite state space and action space, respectively. r(s,
A in state S. PM (s' S A x + [0, 1] denotes the transition distribution and y is the discount factor. A policy ss)s) specifies action distribution conditioned state The objective of RL to maximize the expected seward-to-go TTT)) = Ex 1Er= y'r (St, 0z) |su aol-
S A R is the reward function for performing action
Induced by policy и,, we define the discounted stationary state distribution as
dth (8) (1 -
Pr(s: = =;
t=
Here Pr(s; = 8; T) denotes the probability of reaching state at time I, which is given by
II PM (su+1 Sus an) i (au s.) Pr (so) dsda,
where ds dso ... ds:-1 and da -ag imply integration over the previous states and actions. Similarly, the discounted stationary state-action distribution is defined as
8
p² (1 Σ
Pr(s, = = 0))),
which measures the overall "frequency" of visiting state-action pair under the policy relationship between p² (s, a) and d= (s) can be described as
The
(=( = ग (a)
(1)
3.2 REPRODUCING KERNEL HILBERT SPACE
Given a set S, for any e E IP and x E S, linear functional mapping h E H to (c, h(x)) is continuous, then vector-valued reproducing kernel Hilbert space (RKHS) H is a Hilbert space of functions : - Rº (Micchelli & Pontil 2005). For all y E S. (3) a symmetric function that is positive definite matrix, thereby (Kz <<I, y)e E H. It has the reproducing property (h, (z = h((x) c. Here (.,)) and (:, // denote the inner product in RR and in H. respectively. We denote H Hp.
3.3 GENERATIVE ADVERSARIAL IMITATION LEARNING
GAIL combines IRL with GANs, treating RL methods the generator. GAIL takes the advantage of the discriminator D(s, a) to calculate the difference between the distribution of the state-action pair induced by the learned policy and the expert policy Tg, thereby providing the reward for the agent. Moreover, the policy and discriminator can be approximated by RKHS (Ormoneit & Sen
2002). The optimization problem in GAIL is
min D
(( [log D(s, a))l + [p [log(1 D(s, a))],
(2)

---

Under review as a conference paper at ICLR 2023
where the policy mimics the expert policy via the reward function r(s,a) log(1 D(s, a)). When the discriminator reaches its optimal
(s, (s, (p + (s,
the optimization objective the learned policy is formalized as minimizing the state-action distri- bution discrepancy between the imitated policy and the expert policy with the divergence:
(JS)
1 D (0",
p²
p
1
p² p
min
pe
4
THE PRINCIPLE OF INSTABILITY IN GAIL WITH DETERMINISTIC POLICIES
Inspired by the instability of DE-GAIL in MuJoCo environments (Subsect. 4.1 impute the invalidity to exploding phenomenon in the absolute gradients of the policy loss in Sub- sect. 4.2 Additionally, the probabilistic lower bound for exploding gradient is provided. In contrast, ST-GAIL will not suffer from such pathology. Finally, we present that the reward function in AIRL
theoretically
relieves the exploding gradients Subsect. 4.3
Fig. shows the architecture for our analysis.
Valid
MuJoCo Experiments
Invalid
Theorem
Theorem
Exploding Gradients
Exploding Gradients
Normal Training
Normal Training
Pr(4)) Pr(E)
Pr(4)2 PrE))
Degenerate
D a))
D(s
00
Equivalent Relationship
Proposition 2
Prop.
Combination Reward Function Decrease Pr(x)
=1
D(,,9)): :
Discriminater
(s,4)): Optimal Discriminator
D (s,a):: Regular Discriminator
A: Explocing Gradients
E: Hr, Ms.)) Mismulching ((s,)- C|2 for any C
An illustration of our proposed theorem of exploding gradients in DE-GAIL, Due to exploding gradients, the invalid case experiments corresponds to diverging from the optimal dis- criminator or degenerating to 0 or 1.
Figure 2:
4.1 ILLNESS REWARD EMPIRICAL EVALUATIONS IN MuJoCo ENVIRONMENTS
We replicate the experimental setup of Zhou et al. 2002). Expert trajectories are created by the SAC agent in Hopper-v2, v2, HalfCheetahh-// v2 and Walker2d-v2 respectively. The size of expert demonstra- tion data 10th obtained with a 0.01 standard deviation. The mean return of the demonstration data in each environment is 3433, 9890 and 3509 respectively.
When training GAIL, we use two-layer networks to approximate the kernel function (see Arora et al. (2019)). The reward function is set as r(s, a) log(1 D(s, a)) (PLR). First, well- performed TSSG is revisited as the baseline for subsequent comparison. Then we explore DDPG and TD3, which also cause GAIL fail as Kostrikov al. (2019). A brief description of the
4

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfchestah-v2
Wa Wailker2d-v2
4000
12000
5030
10000
Return
Return
2000
1030
2000
1000
1000
-2000-
1350
1350
2.0
3.0
0.0
0.5
10
255
0.0
0.5
Number of Samples
Number Samples
Numbar Samples
Figure 3: Comparison of SD3-GAIL, TD3-GAIL, DDPG-GAIL and TSSG in three different envi- ronments. Solid lines and dashed lines correspond to the average performance of the four algorithms and expert demonstrations, respectively.
training procedure is laid out Appendix A. (Alg. D. The best evaluation results of DDPG-GAIL and TD3-GAIL are shown in Fig. 3 Compared to TSSG, we can observe that DDPG-GAIL GAIL and
TD3-GAIL not only obtain a lower mean return but also suffer from obviously higher variance. We
suspect this is owing to the inaccurate Q Q-value estimations of DDPG and TD3 Pann al. 2020
Waberzd-z
4900
4300
4200
4000
2000
3000
3006
Return
Sctuum
12:0
1009
35
Number of Sampples
Number Samples
Numbbe S.SSmlps
Number Sumples
(a)
(b)
(c)
(d)
Figure The performances the four algorithms under different seeds in Walker2d-v2.
We next experimentally examine whether better value estimation will improve GAIL. SD3 (Pan itt al. 2020), a deterministic policy algorithm that enables a smaller absolute bias of true values and value estimates than TD3, is conducted into the framework of GAIL, as shown in Alg. [ Its best performances are presented in Fig. We conclude the results in two aspects:
SD3- GDBL improves the average return compared DDPG-GAIL and TD3-GAIL, and even achieves higher upper limit than TSSG. This phenomenon is possibly attributed to the accuracy of Q-value estimation in SD3.
Although the enhancement of average return SD3-GAIL compared DDPG-GAIL and TD3-GAIL, the variance of SD3- remains high. In order to thoroughly observe the phenomenon of high variance of SD3-GAIL, DDPG-GAIL and TD3-GAIL, we specify the performance of the four algorithms under 11 random seeds in Walker2d-v2 as exhibited in Fig. The three deterministic policy algorithms reveal extreme instability under random seeds. Specifically,
The performances under some seeds are capable of reaching expert levels, such as SD3-GAIL, TD3-GAIL and DDPG-GAIL with seed 0.
Some seeds almost learn nothing, such as SD3-GAIL, TD3-GAIL and DDPG-GAIL with seed 7.
SD3-GAIL with seed and seed 9 learn only part of the expert policy.
While TSSG maintains similar trend under different random seeds, the instability is not caused by GANs itself depicted by Arjovsky & Bottou (001))
These empirical evaluations manifest that DE-GAIL algorithms suffer from pathological training dynamics. We attempt to explain this phenomenon by employing the statement of reward bias introduced by Kostrikov et al. (2019). In their specific toy demo, repeated trajectories may be trained with the stochastic learned policy, which brings about higher return than expert under PLR
5

---

Under review as a conference paper at ICLR 2023
r(s,a) - log(l--1111 1og(1-D(ss Ds,, a)), the strictly positive reward function. This is known the reward bias. Conversely, the learned policy is deterministic, the trajectories will be trained with no repetition. Furthermore, the expert return will not be exceeded by the return of the deterministic learned policy technically, in other words, the reward bias vanishes. This can not interpret DE-GAIL as inferior to ST-GAIL which appeared in our experiment. An illustrative example of the invalid case is shown in Appendix A.2 In the sequel, we will explain such instability from the perspective of invalidity in DE-GAIL.
4.2 EXPLODING GRADIENTS DE-GAIL
Inspired by Lever & Stafford (0155) and Paternain et al. (020)) who employ multivariate Gaussian policy to approximate deterministic policy, we define the learned policy Th as
(a h(s))
E 'a h(s))
Th [a s)
exp
del
2
parameterized by deterministic functions E H. h: S + A and covariance matrix Σ. The function (.)) is an element of an RKHS H.s h(-)) Σ, x(8; E Hy. where 8; E and a, A. Note that nr (als) can be regarded as an approximation to the Dirac's impulse via covariance matrix approaching zero, i.c..
lim nA (als)) 5(a h(s)).
(5)
Eq. (5) means that when the covariance Σ 0, the stochastic policy (als) approaches the de- terministic policy h(s). Replacing with Ta in Eq. (2) Eq. and Eq. (4) respectively, the optimization problem GAIL under TT
min max (s.a) [log(Dss a)) + E (s.a))p [log(1 a)s,
Th
the optimum discriminator
D"(s,a)= (s a)/ ptth (s,aa),
(6)
and the policy optimization objective is
min Das (s, ( (s, Th
Before proceeding with our main result, we need a crucial definition.
Definition (Mismatched and Matched State-action Pair) The state-action pair (St, ((s))) in-
duced by the learned policy mismatches the expert demonstration (sr,0)), (f)) - 2 C Σ| for any C > Otherwise, (st, h(:))) matches the expert.
We utilize an event
E [(,,,, h(s:)) : [[((s)) azl12 2 CIE for any > 0}
characterize the mismatching. A descriptive example of mismatching and matching shown in Appendix A.3
Now present the following theorems the probability of exploding gradients in DE-GAIL..
Theorem
Let (-)ss)) (s) be the Gaussian stochastic policy with mean 1((s) and covariance Σ. When the discriminator is set to be optimal D(s, a) in Eq. 66. the gradient estimator of the policy loss with respect to the policy's 's parameter h satisfies
-= )1 -
with probability of Pr([[--1 (a; h(st))ll2 for any C 0) as where
1= (sx) nT (1+
2d (T (uelst))
DiDis(p" "M -m===
log d (st) (st) + d ( (S))Tg(a(ls.)
nd ( (s:)
and Vana(als) [ 1 h())))

---

where De and Under review as a conference paper at ICLR 2023
Proof. See Appendix A.4
Remark 1 When Σ - O, in other words, the policy is deterministic, we have
Pr( E -a h(ss)))ll 2 for any C > 0) 2 Pr[[]a: - (2s))l22 2 C| 22 any > 0)
Pr(EE).
The probability of mismatching Pr(E) is nontrivial since Σ - O. Theorem 1 implies that when the discriminator is set to be optimal, DE-GAIL will suffer from exploding gradients with the proba- bilistic lower bound Pr(E).
In contrast, for Gaussian stochastic policy (fixed E), we have that p ll2 is bounded referring the proof strategy of Theorem [ Thus, when the discriminator set to be optimal, the Gaussian stochastic policy in GAIL will not suffer from exploding gradients. Analo- gous conclusions can be drawn for non- Gaussian stochastic policies.
Theorem T reveals that the policy loss possibly suffers from exploding gradients when the discrim- inator is set to be optimal, subsequently, we will present more universal result the regular discriminator, which is defined as
(1+c)p" c)p= (s).
Dists az)
+ () (e) + (1 -e)p² =))x((r; u+)
where e E (-1, 1). Note that
D(s az) is monotonically increasing from to 1 as (-1,
"Regular" suggests that D(s,, at) ranges in (0, 1) stemmed from Eq. 2
D(st 0r) reaches its optimal when e O.
We next state the exploding gradients on D(se,a)).
Theorem (Main Result) Let (-(8))) be the Gaussian stochastic policy with mean h((s) and co-
variance E. When the discriminator is set to be regular D(s, a) in Eq. (7). iee. D(s,a) E 1), the gradient estimator of the policy loss with respect to the policy's 's parameter h satisfies
| (B a))l ((o,)) log(1 - D(s, D(s,,))))
with a probability of Pr(IIE-\aa, hss)))lle 2 C for any C > 0) as denote the expert demonstration and the replay buffer of Th respectively,
Vn Dy a))l + 1E (s,)) [log(1 D(s a)))
dFh (n te (St)TT log (1 (1+e)d (st) (s))T((a(ss)) + (1 - e) ( T (azlst)
(1- c) (s))
ne (8e) A
p=E + pTh (st, a1) c)
and (als(a)s) n. (als) E- h(s)))
Proof. See Appendix A.5
Analogous to Theorem
Theorem 2
implies
When the discriminator ranges in (0, DE-GAIL will also be at risk of exploding gradi- ents.
When the policy loss suffers from exploding gradients during many runs, the discriminator in DE-GAIL degenerates 0 or 1.
Differently, ST-GAIL will not suffer from exploding gradients when the discriminator ranges in
(0, 1).

---

Under review as a conference paper at ICLR 2023
5D3-GAIL Walker2d-22
5D3-GAIL Walker2d-v2
140
1.0)
3.00
Expart Discimmuloo
045
630
0.5
13 Number Samples
Number Samples
TSSG in Walker2d
TSSG Walker2d
0.11
Exyent (cdemsttatr
083
0.9
0.1
Number Samples
Number Samples
seed0
seedll
seed2
tee-d3
seedd
Seedd
Seed
seed?
seed8
seed9
seed20
Figure 5: The discriminators of SD3-GAIL and TSSG in Walker2d-v2 under seeds.
Experimental results in Fig.
20-22
5 and Fig. 6 support our the- oretical analysis. Seeds 5, 10 exhibit exploding gradi- ent performances (left figure in Fig. (6 and degenerating discriminator behaviors (first row in Fig. (5.. which are con- sistent with the invalid cases (r -> 0) in Fig. 4a). Note that the gradients of TSSG and the valid cases SD3- GAIL are in the same order of magnitude, far less than the invalid cases SD3-GAIL.
12020
Rickun of of Nadic Mebwwk
0000
0.20
6930-
2020
0.30
25
Number Sarpples
Numbbrr
Figure 6: The absolute gradients SD3-GAIL and TSSG policy networks in Walker2d-v2.
4.3 RELIEVING EXPLODING GRADIENTS WITH REWARD MODIFICATION
Kostrikov al. 2019) introduced the reward function ( 0x) log (og(D(ss, (())))))) log(1
Dist, a:)) of AIRL into GAIL, and illustrated reduction in instability through comparative ex- periments against GAIL with PLR (st, (s),a,) log(1 (st, a. )) The reward function of AIRL is defined as the combination reward function (CR) in Wang & Li (022). For written convenience,
DE-GAIL with PLR and CR are called PLR-DE DE--AE- and CR-DE BE-DE-CGO DE-GAIL respectively.
Further, in this section, we proceed with this reduction of instability from theoretical explanation, in other words, whether CR-DE-GAIL permits relief in exploding gradient probability over PLR- DE-GAIL.
Theorem points out that the mismatched aute-action pair results in exploding gradients. Further, what truly pertains to our discussion in the sequel is the behavior of the discriminator. This is due to its unified characteristic that lies in the interval [0, 1). Now we present the following Lemma on the mismatching in the view of the discriminator.
Proposition 1 When the discriminator is set to be optimal D* a) in Eq. 16. we have
Dd (St, ax) 1 h((ss) mismatches ot-
8

---

Under review as a conference paper at ICLR 2023
Proof. See Appendix A.6
Lemma [ indicates that exploding gradients depend the distance between the discriminator's value and I, or they depend on the degree of (,t,,, az) for i = 1, when r approaches infinity. This due to the monotonicity of both Tı (5t, 0e) and r2 (St, ae). thus obtain
r1 (st, az) oo and r2(st, ar) ~ Do
as ((,,, ar)
Naturally, to prevent exploding gradients, we make the constraints that (st,,, az) S C, i i= 1, 2, for some appropriate constant C. In contrast, the outlier of the discriminator can be characterized as C' for i = 1,2, which is referred to the following.
When the discriminator is set to be optimal D* a) in Eq. (6). the outliers of the discriminator are defined in (ax, 1] such that vı (s, a1) 2 C. Similarly, under the same upper bound C, the outliers of the discriminator are defined in [3, for r2(5t. oz).
Definition
We note that the training process will suffer from exploding gradients when the discriminator comes to rest in [a, 1]. The remission of exploding gradients in CR-DE-GAIL is presented in the following proposition.
Proposition 2 When the discriminator is set to be optimal D* (* a) in Eq. 6 have 8 2
Proof. See Appendix A.7
Proposition 2 reveals that the discriminator in CR-DE-GAIL exhibits smaller interval of outliers than that in PLR-DE-GAIL, decreasing the probability of gradient explosion. Our conclusion is consistent with the claim in Kostrikov al. 2019 (Fig. 5).
CONCLUSION
In this paper, we have explored the principle of instability in DE-GAIL. We first experimentally show extreme instability performance of DE-GAIL algorithms compared to ST-GAIL. Subsequently. our proof manifests that the gradient of the deterministic policy loss with respect the policy will suffer from exploding with some probability, thereby leading to training failure. In comparison, we present the compatibility between stochastic policy and GAIL. Finally, the modified reward function shown to remedy the exploding gradients.
Reducing the probability of exploding gradients is under consideration. By introducing the idea of clipping the reward into SD3-GAIL, we have discovered some interesting phenomena. For a preliminary verification of our algorithm, please refer to Appendix A.8] Specifically, the fact that clipping the reward that leads to the outlier of the discriminator shows its superiority in the stability of DE-GAIL.. Further analysis that improves the sample efficiency while keeping low exploding gradient probability is left for future works.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.0466. 2017.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances Neural Information Processing Systems, volume 32, pp. 8139-8148, 2019.
Lionel Blondé and Alexandros Kalousis. Sample-efficient imitation learning via generative adver- sarial nets. International Conference on Artificial Intelligence and Statistics, pp. 3138-3148, 2019.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In International Conference Artificial Intelligence and Statistics, pp. 182-189, 2011.

---

Under review as a conference paper at ICLR 2023
Minshuo Chen, Yizhou Wang, Tianyi Liu, Zhuoran Yang Yang, Xingguo Li, Zhaoran Wang, and Tuo Zhao. On computation and generalization of generative adversarial imitation learning. In International Conference Learning Representations, 2020.
Xiaocong Chen, Lina Yao, Aixin Sun, Xianzhi Wang, Xiwei Xu, and Liming Zhu. Generative in- verse deep reinforcement learning for online recommendation. In ACM International Conference on Information & Knowledge Management, pp. 201-210, 2021.
Wenqiang Chi, Giulio Dagnino, Trevor MY Kwok, Anh Nguyen, Dennis Kundrat, Mohamed EMK Abdelaziz, Celia Riga, Colin Bicknell, and Guang Zhong Yang. Collaborative robot-assisted endovascular catheterization with generative adversarial imitation learning, In IEEE International Conference on Robotics and Automation, pp. 2414-2420, 2020.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference Learning Representations, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error actor- critic methods. In International Conference Machine Learning, pp. 1587-1596, 2018.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pp. 1259-1277, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, volume 27, pp. 2672-2680, 2014.
Ziwei Guan, Tengyu Xu, and Yingbin Liang. When will generative adversarial imitation learning algorithms attain global convergence? International Conference on Artificial Intelligence and Statistics, pp. 11 1117-1125, 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, volume 29, pp.
2016.
Mohamed Khalil Jabri. Robot manipulation learning using generative adversarial imitation learning. In International Joint Conference on Artificial Intelligence, pp. 4893 4894, 2021.
Liyiming Ke, Sanjihan Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. In International Workshop on the Algorithmic Foundations of Robotics, pp. 313-329, 2020.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminnator-acoorcriii Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
Guy Lever and Ronnie Stafford. Modelling policies in MDPs in reproducing kernel hilbert space. In Artificial Intelligence and Statistics, pp. 590-598, 2015.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0244 2015.
Charles A Micchelli and Massimiliano Pontil. On learning vector-valued functions. Neural Compu- tation, 17(1):177-204, 2005.
Andrew Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In International Conference Machine Learning, pp. 663-670, 2000.
Dirk Ormoneit and Šaunak Sen. Kernel-based reinforcement learning. Machine Learning, 49(2): 161-178, 2002.
Ling Pan. Qingpeng Cai, and Longbo Huang. Softmax deep double deterministic policy gradients. In Advances in Neural Information Processing Systems, volume 33, pp. 11767-11777, 2020. 2020.
10

---

Under review as a conference paper at ICLR 2023
Santiago Paternain, Juan Andrés Bazerque, Austin Small, and Alejandro Ribeiro. Stochastic policy gradient ascent in reproducing kernel hilbert spaces. IEEE Transactions on Automatic Control, 66(8):3429-3444 -444, 2020.
Dean A Pomerleau.
Efficient training of artificial neural networks for autonomous navigation. Neu- ral Computation. 3(1):88-97 97, 1991.
Martin I. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John
Wiley & Sons, 2014.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
Jing-Cheng Shi, Yang Yu, Qing Da, Shi- Yong Chen, and An-Xiang Zeng. Virtual-taobao: Virtualiz- ing real-world online retail environment for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4902-4909, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: Introduction. Cambridge, MA, USA: MIT Press, 2018.
Advances in Neural Information Processing Systems, volume 20, pp. 1449-1455 1456, 2007. Umar Syed and Robert Schapire. A game-theoretic approach to apprenticeship learning, In
Lei Tai, Jingwei Zhang, Ming Liu, and Wolfram Burgard. Socially compliant navigation through raw depth inputs with generative adversarial imitation learning. In IEEE International Conference on Robotics and Automation, pp. 1111-1117, 2018.
Yawei Wang and Xiu Li. Reward function shape exploration in adversarial imitation learning: an empirical study. In IEEE International Conference on Artificial Intelligence and Computer Ap- plications, pp. 52-57, 2021.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Advances in Neural Information Processing Systems, volume 33, pp. 15737-15749, 2020.
Yirui Zhou, Yangchun Zhang, Xiaowei Liu, Wanying Wang, Zhengping Che, Zhiyuan Xu, Jian Tang, and Yaxin Peng. Generalization and computation for policy classes of generative adversarial imitation learning. In International Conference Parallel Problem Solving from Nature, pp- 385-399, 2022.
Brian D Ziebart, Andrew Maas, Andrew Bagnell, Anind K Dey, Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 8, pp. 1433-1438, 2008
Guoyu Zuo, Kexin Chen, Jiahao Lu, and Xiangsheng Huang. Deterministic generative adversarial imitation learning. Neurocomputing, 388:60-69, 2020.
A APPENDIX
A.1 DE-GAIL ALGORITHM
Algorithm GAIL with deterministic policy algorithms
1: Input: Expert demonstrations, initialize the learned policy no and discriminator network D, some RL deterministic policy algorithms, such as SD3, TD3 and DDPG.
2: for iteration ..... do
3: Update D by maximizing "s [oog( D(s, a))l + (s.)) [log(1 - D(s,)))).
4: Calculate PLR r(s, q) = log(1 ((s, a)).
5: Update by deterministic policy algorithm.
6: end for

---

Under review as a conference paper at ICLR 2023
A.2 THE INVALIDITY IN DE-GAIL
The instability of DE-GAIL is mainly imputed to the invalid case, i.e., the reward function ap- proaches zero. To illustrate this invalidity, consider a specific MDP MM contains (g + 1) states so: $1, .... Sg. where 8y is the terminal state. The action az-z in A is such that p(-]s,, ar-s)) satisfies that - jl S 9/4 for + j. The expert demonstration shown in Fig. 7
Agagge
The expert demonstration the designed demo
So
Input: S (5...... .80
Agis-aD
for 0.... 33/4 for ii
S g/4,j 0,1
Take the actions 04-15 and az-z
end for
So
Singe
Take the action a;-(it1)
Up-sl
end for
j g/2, 3g/4 - 1, 3g/4 +, 1,... .9
Take the actions 03g/1-23 and aj-33//1- end for Take the action 03g/4-49
S0
Figure 7: Expert demonstration. Each state transfers the neighboring g/4 states and back to itself from left to right. Left: The schematic of the expert demonstration, where the red arrow is the last action taken in each state. Right: The pseudocode of the expert demonstration.
For the state 8, 1 39/4), the action 04-14+1 occurs 3 times while others occur twice in the expert demonstration. Note that
0 TE (s,
D*
(a.a) (s + (s.(a
(*, a) is monotonically increasing with respect to ( a). Thus, the more frequent expert state-action pair tends attain higher value of the diseriminator, thereby resulting in higher PLR. Meanwhile, policy training in RI. aims to maximize the expected reward-to-go. Therefore, the most frequent action at-s1+1 the best choice for deterministic learned policy under each st. The detailed trajectory is shown in Fig. g
ag
a
So
S
S2
S g
Figure 8: The trajectory of the deterministic learned policy. From so to the terminal state sy sequen- tially.
We now show the behavior of rewards for state-action pairs in the deterministic policy.
Proposition 3 In the MDP M, when the discriminator is set to be optimal, PLR of (St, acset))
satisfies
r(8,, (r-s41) =- -log(1 - D(8r, a1-se4))) + + as
[S] + co,
where
<
+122- 161
1
(1-141)
169
3g
11 4009--00-- 10
=
0,
otherwise.
12

---

Under review as a conference paper at ICLR 2023
Proof. The length of the expert trajectory is
2 (4 + +1+
+ 1 4
2 g
- 1)
+
+
1)
1+
3g
2
2
4
1 + 24g - 16
16
Note that the first term 2(g/4 + (g/4 + 1) +.... ai-+j- Oj-ts in states Su, $1, ... 91 the second term 2(g/2 + 1)g/ 2 is derived by executing actions ai-aj, aj-st states 8//1; 8g/4+1;" 83//1 and the last term 3g/ obtained by executing the action 01-1+11-
1]) comes from executing actions
Denote the expert policy and the deterministic policy as тE and h, respectively. For the (st, a+-++1) 0 < 3g/ / in the trajectory of the deterministic policy, the optimal discriminator is (Kostrikov
et al. 2019)
- ( ag a1-1411 (st a[-++1) pun (5t, a+)++) + (82, a1-541)
3 I
48g
11g² + 72g 16
For / = 3g/4, we have
16g
((sgg/4: 03g/4
11g² + 40g - 16
While for any > 3g/ /4, the value D(s,, at-r+1) zero since it never appears in the expert demon- strations. To sum up, when the discriminator achieves its optimal,
r(s, amtt)) = - log(1 D((s, ((-(41))) + 0 as
S|] + oc.
Remark Note that under the deterministic policy, PLR of the best action approaches 0. As result, we have 0 for all state-action pairs.
In contrast, for stochastic policy reaching the expert level, the length of the expert trajectory and the stochastic policy have the same order (((S)2). Here |8] denotes the size of S. Referring the proof strategy, PLR will not decline to zero.
Proposition S illustrates the invalid case, implying the instability in DE-GAIL.
13

---

Under review as a conference paper at ICLR 2023
A.3 AN DESCRIPTIVE EXAMPLE OF MISMATCHING AND MATCHING
The probability density contour map of the expert action demonstration in state is shown in Fig.
0.05
0.55
at
0.04
0:235
6:225
0.085
6015
0.01
Figure 9: The process of the mismatched case and the matched case on a descriptive example of two-dimensional action space (x-axis and y-axis). Left: The mismatched case. Right: The matched case. Without loss of generality, the threshold of matching is set as 0,035. The trajectories used to train the learned policies are shown in red curves. The matching is that (sx)) lies in the neighborhood of a,, which has a high probabilistic density in the expert demonstration.
A.4 PROOF OF THBOREM 1
Theorem I Let n (-1s) be the Gaussian stochastic policy with mean h(s) and covariance Σ. When discriminator is set to be optimal D(s, a) in Eq. 16), the gradient estimator of the policy loss with respect to the policy's 's parameter h satisfies
( Dis(s 11 0o
with probability of (Pr([E--((4 h(st))l12 2 for any C > 0) as + where
2d ( Th d ( (arl8)) log n (s))sa( + d (T
2d (st (
and (ata(als) nh (a)ss)m
1ーー ) h(s)).
Proof. Through importance sampling which transfers the learned state-action distribution to the
expert demonstration distribution, the IS divergence can be rewritten from the definition in Eq.
(4)
as
nm + TR
1
Th + pm
+ 2
(n
2
2
1
2p (8
(s,
B(Ka)aD
log pT (s, a) + pr
E Els,aaDDz log pT (s, a) + pr (s,a)
2
(x, log nTh (s, a) + p
2h (8,
2 a)
B(a.0)DD
log
(9)
2
()
p=a p (s,
14

---

Under review as a conference paper at ICLR 2023
where DE and D1 denote the expert demonstration and the replay buffer of TT respectively. Then we can approximate the gradient of Eq. (9) with respect to with
V Dp ,pT) )
p (s) log pT (st, at) + pa p= (81, ar)
2pth (st, as)
2p (st, as) + log 1Th (8,, at)
1e (5t; ae))
p (St, as)
1
d VI log d (8) (81) + d (s)
2d (5) (aols)))
2
(1)18))
0 th (SL,
pTh (st, (s), p (s),
(1)
2pTh (sc: at) 2d (8t) VATT( (st + p. (sc, 0c)) 20 (5c, (n)Tanaaasss))) (pTh ax) + p (s,, ax))
MT ($), a1) + pTT (5t, at)
20TH (st, AT V
2 (8t, at)
(s), a1) + pm (8,, a.) 2d (SA) (ct d (s) (:) + d (T(((s)))
T (8t) VA (arl(t))
log
2 d (tt) (T (arl
1Th (8e) VATAA ( (1))
(8)) TT (a
pTh (s), a1) + pTT (5L, a1)
p (st, a1) pTT (st, a1)
(1) (8t) 7
nd (sx T (E) log d (s) (a) + uH (g(ess))))
(10)
ne (st (
where (ii) comes from Eq. 1 By the fact that
(als(a Tr (als) VA log (als) (a(als))(s.))) 'a (a--aha hhs)),
(11)
Eq. 10) can be shown that
V p
d (sx TT (cels)) )) (at h(st)) log d (8x TT (s)(s)) + d (sx (
nd (n
2d (8e) (ST) TE (x)
Then it follows that V Das(p p=t) + with probability of Pr( |- '(a h(se))ll C' for any C' > 0) as Σ + 0.
A.5 PROOF OF THEOREM 2
Theorem 2 (Main Result) Let (-(ss) be the Gaussian stochastic policy with mean h(s) and co-
variance E. When the discriminator is set to be regular D(s, a) in Eq. ( ie.. D(s, a) E 10, the gradient estimator of the policy loss with respect to the policy's parameter h satisfies
| Dogg ((s, ()))))) (iogg)
((s)
x
with probability of Pr((IE=-(aa h(+))) 2 C forany denote the expert demonstration and the replay buffer of Th respectively,
any C > 0) as Σ + O, where Dr. and D
Vn
a))] + +E - - (1 e) (s))
d=h (st h (arls)))
th 81 (m (.) log
(s) + e) t (st TH (arl3))
ne VaTA
n (s,, a1) + + nT (st, (1-c))
and (aTu(als) TT (als))<(( |Σ (a h(s)).
Proof. Referring to the proof strategy of Theorem [ the learned state-action distribution can be transferred to the expert demonstration distribution by importance sampling. Thus when the dis- criminator is set be regular D(8 a), can write the policy objective from the optimization
15

---

Under review as a conference paper at ICLR 2023
problem in Eq. 2
E De (logglog a))l + ED, [log - D(s.
(oaaDDD log
p (5, a1 +))
(=
+ c) + pTM (s, a)
pT (s,
+ B((aa)DD
log 1o ( +c) +)+ + p (0,
pa (5, (1 +e)
= (o
log (= a) + c) + pTM (8, (1
p Th (s,
0 (s)
log p (8, (1 + ) + pm (8, -6) (12)
p (8,a)
Then the gradient of Eq. 12) can be approximated with
nn (a Dog( D a])] + ED, [log(1 (s.
pTH (s,, a1)
Va log
1 a)) + e) +
(c)
p (m as)
0 nTh (s
log ( as (1+ + +m (s)
(TE at)
n (8,, a1)(1 +c) p a0 1)
p (sr (se V (a ()
p= ar) te)
(pre(s), (1) + + p (st; a:) o))
d (aa))) log (SE (1 c) pTh (st, (1) c)
(1- (s
(
(1 +1+ e)d= (5))=((a((s)) + (1 ( (s
nT (s),
p= (8)
pT (3,, ar (1 - E)
(1 e) cc(1+ccpppp)) (sc. a TM (8c (n
a1) (1 +c) p (s,, a1 - =))
d AA log (1++))=(ss)) + - e) (a))s))
(1- c)dh () (c()))
n (5t) (E)
(1 - c)d (
(1 + ( (st) (s)
(s), + pT (s), - a)) ~TM (st, 01) c)
d (s) (ac)se)
(1- (s)
log (1 + (s) (S))Tg((c((:) + (1- e)d (acls))
dTE (s TE (ael (tt)
2nd (st V AT (4t(8))
(13)
( (1)
T (st, az) )1 (1
Plugging Eq. into Eq.
13 when 1 (at h(ss))ll2 C for any C > 0, we have
v (( [log(
a)) (lg(((
ED, [log( D(ss,)))) || 00.
A.6 PROOF OF PROPOSITION T
Proposition
1
When the discriminator is set to be optimal D* a) in Eq. 6 we have
D (St, at) Re 1 + h(ss)) mismatches oz.
Proof. The optimal discriminator of (st, az) can be denoted by ( at)
D (St, 0z) - ( az) + TT (St. (z)
We can derive that the necessary and sufficient condition of D (*t,,, ax) so 1 is that p (st, ax) = 0,
i.e. (St, (((s)) mismatches (92,a)).
16

---

Under review as a conference paper at ICLR 2023
A.7 PROOF OF PROPOSITION
2
Proposition
2 When the discriminator is sel to be optimal (* a) in Eq. 1 we have 3 2 a.
Proof. When re(Se; C, i 1,2, obtain log 8 loog1 () log(1 a): which is followed by
B -- 2 2a +
2 2 0.
A.8 SD3-GAIL WITH CLIPPED REWARD
Clipping reward shows its superiority in the stability of DE-GAIL but at the expense of lower sample efficiency.
Hopper.22
Walker2d-v2
Wa Wallker2dvv2
6000
3000
3000
Return
Return
0036
2000
2000
1500
1000
1000
100
N03-
NO3
2.0
0.5
100
3.5
2.0
25
0.5
Number of Samples
Number Samples
Number Samples
Figure 10: Comparison of SD3-GAIL and SD3-GAIL with clipped reward in three different envi- ronments.
17


Reviews:
[{"review_id": "zQuposEUQRD", "reviewer": "Reviewer_59jo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "* This paper explores the mechanism behind the generative adversarial imitation learning, which provide an important conclusion that instability is caused by deterministic policies, rather than GANs.\n\n* It provides some existing methods relieve exploding gradients, but at the expense of “non-confidence”, and ST-GAIL has advantages in mitigating instability.\n", "summary_of_the_review": "I would tend to accept this paper as it is novel enough and supported by theoretical analysis.", "strength_and_weaknesses": "Pros:\n* I believe that there is well backed motivation for work based off of the plentiful literature review. \n*This article provides a theoretical basis for strengthening the study of learning stability and provides a plan for the design of future GAIL work.\n\nCons:\n*This paper is only analyzed under the toy data set but does not test it under the actual scene data set.\n* The composition of this article does not demonstrate carefully, and the writing is very obscure and difficult to understand.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a7LMFmwob8a", "reviewer": "Reviewer_F1Ws", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper appears to show that deterministic policies may cause exploding gradient when used with adversarial imitation learning (AIL), despite their improved sample efficiency. At the same time, the paper appears to show that stochastic policies do not suffer from exploding gradients. This led to the conclusion that deterministic policies are \"incompatible with AIL\".", "summary_of_the_review": "The paper is difficult to follow . While the observation/claim appears interesting, both the theoretical and empirical justification falls short. The manuscript may require significant revision before acceptance.", "strength_and_weaknesses": "Strength\n1. The paper raises an interesting observation that deterministic policies may cause exploding gradient with AIL. If properly proven/demonstrated, the result would have significant impact on future method design.\n\nWeakness\n1. The paper is poorly structured and unclear. Many aspects of the proof are not sufficiently motivated or connected to the overall conclusion. For instance, it is unclear to me how Prop. 1 is related to Theorem 1. In addition, the notion of \"non-confident\" reward for ARIL has very little explanation (e.g. how it is related to Prop. 2).\n\n2. The primary proof for the main claims is confusing and skip too many steps. Crucially, it is unclear how the exploding gradient is tied to $Pr(||\\Sigma^{-1}(a_t - h(s_t))||_2 \\ge C)$. In addition, it appears that $||\\Sigma^{-1}(a_t - h(s_t))||_2 > 0$ since $L_2$ norm is positive with non-zero $\\Sigma$. Therefore it is confusing why $Pr(||\\Sigma^{-1}(a_t - h(s_t))||_2 \\ge C)=0$ (Remark 2) for stochastic policies.\n\nIn addition, the proof relies on the assumption that for every gradient step, the discriminator is optimal. This is an unrealistic assumption and undermines the strength of the claim.\n\n3. The empirical analysis only shows that deterministic policies fail for certain seed. If exploding gradient is indeed an issue as the theoretical results suggested, the authors should explicitly detect exploding gradients rather than just showing learning failures. On the other hand, a large number of runs must be repeated for stochastic polices (much more than 9 seeds, and many runs for each seed) to corroborate that stochastic policies have no exploding gradients, since it's impossible to empirically prove the negative (i.e. stochastic policies have no exploding gradient).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YhXjlscIH3n", "reviewer": "Reviewer_EU3W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to show that using deterministic policy in a generative adversarial imitation learning algorithm is not proper, and results in a significant instability. This paper does the following:\n- empirical comparisons between SD3-GAIL, TD3-GAIL, TSSG\n- show a toy domain and prove that PLR approaches 0 as the size of state space increases\n- variance of the gradient of GAIL objective explodes in mismatched cases if we use deterministic policy\n- if we use CR instead of PLR, such a problem is addressed with some cost.\n", "summary_of_the_review": "Due to the weaknesses I listed above: low writing quality, not convincing analysis, and lack of algorithm as a result of the analysis, I recommend rejection of the paper.", "strength_and_weaknesses": "Strength:\n- this paper addresses a less explored problem of theoretically analyzing the \"instability\" of previous GAIL variants.\n\nWeaknesses:\n- bad writing quality:\n\nThere are a large number of grammatical errors and word misusages. The paper also refers to a number of other algorithms without any detailed explanation - e.g. TD3-GAIL, SD3-GAIL, and combination reward function. How they are defined is quite important in this analysis paper, so I think it had to be described in the paper for completeness. There are many theoretical results, but their implications are not well explained. These issues are combined to make the paper extremely hard to read.\n\n- not convincing analyses:\n\n1. The paper starts by arguing that some algorithms show pathological behavior (of high instability) when PLR and deterministic policy is combined. However, the paper simply makes the comparisons between SD3-GAIL, TD3-GAIL, and TSSG; by comparing different algorithms, there are so many different factors other than PLR and deterministic policy that affects the performance, and it is not accurate to make the hypothesis out of these comparisons. I believe the authors should have compared the consequence of different reward functions and policy choices in the same algorithm. Furthermore, it would have been better if other metrics other than simple returns are compared, e.g. gradient variance, minimum learned reward, etc, to show the validity of further technical analyses.\n\n2. In proposition 1, the authors argue that PLR is bad because, in this toy domain, the reward can approach 0 as the number of states grows to infinity. Why is that a problem? It will be fine unless we have $r=0$ for all state actions, and for a certain number of states $|\\mathcal{S}|$ we will always have positive rewards for some state actions. And it seems to be natural to have a small reward if we have such long expert trajectories.\n\n3. In theorem 1, the authors argue that the variance of gradient explodes as a variance of a policy approaches 0. However, it is not a new analysis and it is widely known if we have a low-variance stochastic policy, the gradient variance of usual policy gradient algorithms will explode, e.g. in REINFORCE algorithm. And it is also well known that DPG does not suffer from such a problem since it has defined the policy as deterministic in the first place. Therefore, for me, the author's argument on theorem 1 of saying GAIL with deterministic policies is unstable is not convincing unless any empirical evidence is provided, e.g. the actual variance of gradient during training.\n\n- the result is not very significant:\n\nThe paper does not propose any new algorithm based on the analysis. Since most of the findings in this paper do not seem to be very significant to me, I cannot highly value the contribution of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "R1cl9EuFN7", "reviewer": "Reviewer_Qudi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is about the instability in training when a deterministic learned policy is used in generative adversarial imitation learning (DE-GAIL) and how to alleviate the problem. The authors first show that the training of DE-GAIL is empirically unstable in Mujoco environments that have continuous action and state spaces. Then, they theoretically show that the reward of DE-GAIL becomes zero when the deterministic policy is used in finite MDP and when the cardinality of the state space goes to infinity in Proposition 1. Also, they argue that the instability is mainly due to the reward becoming zero. For MDPs with continuous action spaces, they theoretically show that DE-GAIL suffers the problem of exploding gradients due to a mismatch between actions from the expert and the learned policy. Furthermore, they use Lemma 1 to show that having a mismatch between the actions from the expert and the learned policy is equivalent to having an optimal discriminator function value of one which is also equivalent to having an infinite reward. Lastly, in Proposition 2, they prove that using CR instead of PLR can alleviate the problem. This is done by showing that PLR has a value larger than or equal to that of CR when their discriminator values are the same.", "summary_of_the_review": "The paper presents a novel theoretical analysis of the instability in the training of DE-GAIL and also theoretically shows how the modified reward function of AIRL can alleviate the problem. However, there seem to be some cases where statements or derivations are wrong, Furthermore, the paper lacks experiments that support propositions and theorems.", "strength_and_weaknesses": "**Strength**\n1. The authors theoretically show that the deterministic learned policy brings instability in training.\n\n2. The authors theoretically show that the problem can be alleviated by using the modified reward function of AIRL.\n\n**Weaknesses and Questions**\n\n1. There are no experiments that validate or support Proposition 1. The experimental results in Fig 1~3 are made from environments with continuous state and action spaces while Proposition 1 is about environments with finite action space. If the authors could empirically validate Proposition 1, that would strengthen the paper.\n\n2. There are no experiments that validate Proposition 2. The empirical results shown in the paper are only about PLR-DE-GAIL. The results of CR-DE-GAIL should be compared with that of PLR-DE-GAIL.\n\n3. Need more strong empirical validation for Theorem 1. Fig 2 and 3 may show instability in training DE-GAIL, but they don’t empirically show that the exploding gradients problem actually happened and that the problem is caused by the instability. If the authors could show that the low-performing seeds in Figure 3 indeed suffered from the exploding gradients while the other seeds did not, that would validate Theorem 1 strongly.\n\n4. The statement In pg 6: “For any state $s_t$, the action $a_{t \\rightarrow t+1}$ occurs 3 times while others occur at most twice in the expert demonstration.” seems to be wrong. If you follow the pseudo-code in Figure 4, when the agent is in $s_{\\frac{3g}{4}+1}, s_{\\frac{3g}{4}+2}, \\cdots, s_{g-1}$ the agent never executes the action $a_{t \\rightarrow t+1}, \\forall t\\in\\{ \\frac{3g}{4}+1, \\frac{3g}{4}+2, \\cdots, g-1  \\}$.\n\n5. On pg 7, authors mentioned that “The learned policy is exposure to be transferred to the mismatching case due to the limited area in the sub-optimal matching compared with the optimal; see seed 1 in Fig. 3(a) for instance”. But I think you cannot guarantee that the result from seed 1 in Fig.3 (a) is such a case. Could you show that the seed 1 experiment shows results similar to Fig 6?\n\n6. Proposition 1: The derivation of the “length of the expert trajectory” seems to be wrong. In Appendix A.1 in the proof of Proposition 1, the authors derived this equation :  $N=2\\left(\\frac{g}{4}+\\left(\\frac{g}{4}+1\\right)+\\cdots+\\left(\\frac{g}{4}+\\frac{g}{4}-1\\right)\\right)+2\\left(\\frac{g}{2}+1\\right) \\frac{g}{2}-1+\\frac{3 g}{4}=\\frac{11 g^2+24 g-16}{16}$. I may be wrong, but the derivation is different from mine. For the first for loop in the pseudo-code of Figure 4, the sub-trajectory length is $2\\left(\\frac{g}{4}+\\left(\\frac{g}{4}+1\\right)+\\cdots+\\left(\\frac{g}{4}+\\frac{g}{4}\\right)\\right)+\\frac{3 g}{4}$ where the first term comes from executing actions $a_{i\\rightarrow j}, a_{j\\rightarrow i}$, and the last term comes from executing the action $a_{i\\rightarrow i+1}$. For the second for loop, since the agent only traverses between $s_{\\frac{3g}{4}}$ and $s_{\\frac{g}{2}}, s_{\\frac{g}{2}+1}, \\cdots, s_{g-1}$ except $s_{3g/4}$, the length of sub-trajectory due to the second for loop should be $2\\left( (g-1)-g/2 \\right)$. For the last line, the sub-trajectory of length 1 comes from executing the action  $a_{3g/4 \\rightarrow g}$. By summing them up, I got $(g^2 + 36g)/16$.\n\n7. According to “Definition 1” on pg 7,   state-action pairs are in a mismatch when $|| \\Sigma ||_2 \\rightarrow 0$ and $a_t \\neq h(s_t)$. And the pairs match when $a_t = h(s_t)$. But Fig 6 draws actions in the neighborhood of a_t1 to show the matched case. And the paragraph under Fig 6 mentions that the matched case is made by having actions in the neighborhood of a_t1.\n\n8. In Appendix A.2 (proof of Theorem 1), the authors derived $\\nabla_h \\pi_h(a \\mid s)=\\pi_h(a \\mid s) \\kappa(s, \\cdot) \\boldsymbol{\\Sigma}^{-1}(a-h(s))$. Why is there $\\kappa$ when you define $\\pi_h$ as in the equation at the bottom of pg 6?\n\n9. In Remark 2, since $|| \\Sigma ||^{-1}_2 ||a_t-h(s_t)||_2 \\ge || \\Sigma^{-1} (a_t-h(s_t))||_2$, shouldn't be the inequality between probabilities (probability mentioned in Theorem 1 and $\\operatorname{Pr}(\\Xi)$) the other way around?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O_0S2wxFfdx", "reviewer": "Reviewer_rV27", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the case when deterministic policies are learned in a GAN framework. The authors describe occuring instabilities and attribute them to exploding gradients. ", "summary_of_the_review": "I'm not convinced of the validity of the key results of the paper (T1). However, more importantly, I found the writing at times too difficult to follow, so I am unable to appropriately evaluate the contributions of the paper. ", "strength_and_weaknesses": "# Strenghts\n\n* The authors try to thorougly understand existing algorithms and investigate their weaknesses, which is an interesting and value contribution.\n* The authors try to support their insights using experiments and toy models, which is helpful.\n\n# Weaknesses\n\nIn my opinion, the biggest weakness of the paper is insufficient clarity in writing. At times, wrong grammar or word-use make reading more difficult, but the intended meaning can still be inferred. Other times however, I found myself unable to follow the writing. Furthermore, the authors sometimes introduce concepts in quotations marks without ever clearly explaining their meaning (e.g. \"aggressive interval\", \"non-confidence\" or \"invalidity\"). \n\nMy other big concern is with respect to Theorem 1 in which they show the probability of exploding gradients. As far as I understand T1, the argument is that the gradient of a JSD divergence w.r.t a deterministic policy can explode. However, I'm not srue this can explain the experimental results, as this JSD is not the optimization target - instead an RL loss is being used. \n\nLastly, more experimental results would be needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject