Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is about the instability in training when a deterministic learned policy is used in generative adversarial imitation learning (DE-GAIL) and how to alleviate the problem. The authors first show that the training of DE-GAIL is empirically unstable in Mujoco environments that have continuous action and state spaces. Then, they theoretically show that the reward of DE-GAIL becomes zero when the deterministic policy is used in finite MDP and when the cardinality of the state space goes to infinity in Proposition 1. Also, they argue that the instability is mainly due to the reward becoming zero. For MDPs with continuous action spaces, they theoretically show that DE-GAIL suffers the problem of exploding gradients due to a mismatch between actions from the expert and the learned policy. Furthermore, they use Lemma 1 to show that having a mismatch between the actions from the expert and the learned policy is equivalent to having an optimal discriminator function value of one which is also equivalent to having an infinite reward. Lastly, in Proposition 2, they prove that using CR instead of PLR can alleviate the problem. This is done by showing that PLR has a value larger than or equal to that of CR when their discriminator values are the same.
strength_and_weaknesses: **Strength**
1. The authors theoretically show that the deterministic learned policy brings instability in training.

2. The authors theoretically show that the problem can be alleviated by using the modified reward function of AIRL.

**Weaknesses and Questions**

1. There are no experiments that validate or support Proposition 1. The experimental results in Fig 1~3 are made from environments with continuous state and action spaces while Proposition 1 is about environments with finite action space. If the authors could empirically validate Proposition 1, that would strengthen the paper.

2. There are no experiments that validate Proposition 2. The empirical results shown in the paper are only about PLR-DE-GAIL. The results of CR-DE-GAIL should be compared with that of PLR-DE-GAIL.

3. Need more strong empirical validation for Theorem 1. Fig 2 and 3 may show instability in training DE-GAIL, but they don’t empirically show that the exploding gradients problem actually happened and that the problem is caused by the instability. If the authors could show that the low-performing seeds in Figure 3 indeed suffered from the exploding gradients while the other seeds did not, that would validate Theorem 1 strongly.

4. The statement In pg 6: “For any state $s_t$, the action $a_{t \rightarrow t+1}$ occurs 3 times while others occur at most twice in the expert demonstration.” seems to be wrong. If you follow the pseudo-code in Figure 4, when the agent is in $s_{\frac{3g}{4}+1}, s_{\frac{3g}{4}+2}, \cdots, s_{g-1}$ the agent never executes the action $a_{t \rightarrow t+1}, \forall t\in\{ \frac{3g}{4}+1, \frac{3g}{4}+2, \cdots, g-1  \}$.

5. On pg 7, authors mentioned that “The learned policy is exposure to be transferred to the mismatching case due to the limited area in the sub-optimal matching compared with the optimal; see seed 1 in Fig. 3(a) for instance”. But I think you cannot guarantee that the result from seed 1 in Fig.3 (a) is such a case. Could you show that the seed 1 experiment shows results similar to Fig 6?

6. Proposition 1: The derivation of the “length of the expert trajectory” seems to be wrong. In Appendix A.1 in the proof of Proposition 1, the authors derived this equation :  $N=2\left(\frac{g}{4}+\left(\frac{g}{4}+1\right)+\cdots+\left(\frac{g}{4}+\frac{g}{4}-1\right)\right)+2\left(\frac{g}{2}+1\right) \frac{g}{2}-1+\frac{3 g}{4}=\frac{11 g^2+24 g-16}{16}$. I may be wrong, but the derivation is different from mine. For the first for loop in the pseudo-code of Figure 4, the sub-trajectory length is $2\left(\frac{g}{4}+\left(\frac{g}{4}+1\right)+\cdots+\left(\frac{g}{4}+\frac{g}{4}\right)\right)+\frac{3 g}{4}$ where the first term comes from executing actions $a_{i\rightarrow j}, a_{j\rightarrow i}$, and the last term comes from executing the action $a_{i\rightarrow i+1}$. For the second for loop, since the agent only traverses between $s_{\frac{3g}{4}}$ and $s_{\frac{g}{2}}, s_{\frac{g}{2}+1}, \cdots, s_{g-1}$ except $s_{3g/4}$, the length of sub-trajectory due to the second for loop should be $2\left( (g-1)-g/2 \right)$. For the last line, the sub-trajectory of length 1 comes from executing the action  $a_{3g/4 \rightarrow g}$. By summing them up, I got $(g^2 + 36g)/16$.

7. According to “Definition 1” on pg 7,   state-action pairs are in a mismatch when $|| \Sigma ||_2 \rightarrow 0$ and $a_t \neq h(s_t)$. And the pairs match when $a_t = h(s_t)$. But Fig 6 draws actions in the neighborhood of a_t1 to show the matched case. And the paragraph under Fig 6 mentions that the matched case is made by having actions in the neighborhood of a_t1.

8. In Appendix A.2 (proof of Theorem 1), the authors derived $\nabla_h \pi_h(a \mid s)=\pi_h(a \mid s) \kappa(s, \cdot) \boldsymbol{\Sigma}^{-1}(a-h(s))$. Why is there $\kappa$ when you define $\pi_h$ as in the equation at the bottom of pg 6?

9. In Remark 2, since $|| \Sigma ||^{-1}_2 ||a_t-h(s_t)||_2 \ge || \Sigma^{-1} (a_t-h(s_t))||_2$, shouldn't be the inequality between probabilities (probability mentioned in Theorem 1 and $\operatorname{Pr}(\Xi)$) the other way around?

clarity,_quality,_novelty_and_reproducibility: **Clarity**: poor

Below are the questions related to the clarity of the paper:

1. In Figure 6, adding the legend for the red line and also adding x-axis and y-axis labels will make the clarity of the paper higher. And rather than writing $a_{t1}$ and $a_{t2}$ in the figure, putting markers and adding legends showing that they are optimal and suboptimal actions would be more informative to the readers.

2. On pg 7, I found this sentence hard to understand: “The learned policy is exposure to be transferred to the mismatching case due to the limited area in the sub-optimal matching compared with the optimal”. I’d appreciate elaboration on the sentence.

3. I was unable to find information on how Fig 6 was drawn. Was it drawn as an illustrative example? or, is it an empirical result?

4. On pg 7, the authors mentioned that “The threshold of matching is set as 0.035.”. Could you elaborate on how 0.035 was computed? Also, the “threshold” was never mentioned before. What is the “threshold” and where is it applied?

5. In Appendix A.2 (proof of Theorem 1), the authors cited (Guan et al., 2021a) for the gradient of the JS divergence with respect to h, but I was not able to find the contents related to the gradients of JS divergence in the paper. Is the paper cited for policy gradient? If so, elaboration on the derivation of $\hat{\nabla}_h   D_J   (\rho^{\pi_h}, \rho^{\pi_E} )$ ($D_J$: JS divergence) would help the readers to understand.

6. Is the probabilistic lower bound related to the lower bound in Remark 2? If not, where is it mentioned in the paper?

**Reproducibility** : good
The details of the experiments for Fig 2 and 3 are mentioned in the paper

**Quality** : poor
Due to the lack of details in Fig 6 and hard-to-understand sentences.

**Novelty** : good



summary_of_the_review: The paper presents a novel theoretical analysis of the instability in the training of DE-GAIL and also theoretically shows how the modified reward function of AIRL can alleviate the problem. However, there seem to be some cases where statements or derivations are wrong, Furthermore, the paper lacks experiments that support propositions and theorems.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to show that using deterministic policy in a generative adversarial imitation learning algorithm is not proper, and results in a significant instability. This paper does the following:
- empirical comparisons between SD3-GAIL, TD3-GAIL, TSSG
- show a toy domain and prove that PLR approaches 0 as the size of state space increases
- variance of the gradient of GAIL objective explodes in mismatched cases if we use deterministic policy
- if we use CR instead of PLR, such a problem is addressed with some cost.

strength_and_weaknesses: Strength:
- this paper addresses a less explored problem of theoretically analyzing the "instability" of previous GAIL variants.

Weaknesses:
- bad writing quality:

There are a large number of grammatical errors and word misusages. The paper also refers to a number of other algorithms without any detailed explanation - e.g. TD3-GAIL, SD3-GAIL, and combination reward function. How they are defined is quite important in this analysis paper, so I think it had to be described in the paper for completeness. There are many theoretical results, but their implications are not well explained. These issues are combined to make the paper extremely hard to read.

- not convincing analyses:

1. The paper starts by arguing that some algorithms show pathological behavior (of high instability) when PLR and deterministic policy is combined. However, the paper simply makes the comparisons between SD3-GAIL, TD3-GAIL, and TSSG; by comparing different algorithms, there are so many different factors other than PLR and deterministic policy that affects the performance, and it is not accurate to make the hypothesis out of these comparisons. I believe the authors should have compared the consequence of different reward functions and policy choices in the same algorithm. Furthermore, it would have been better if other metrics other than simple returns are compared, e.g. gradient variance, minimum learned reward, etc, to show the validity of further technical analyses.

2. In proposition 1, the authors argue that PLR is bad because, in this toy domain, the reward can approach 0 as the number of states grows to infinity. Why is that a problem? It will be fine unless we have $r=0$ for all state actions, and for a certain number of states $|\mathcal{S}|$ we will always have positive rewards for some state actions. And it seems to be natural to have a small reward if we have such long expert trajectories.

3. In theorem 1, the authors argue that the variance of gradient explodes as a variance of a policy approaches 0. However, it is not a new analysis and it is widely known if we have a low-variance stochastic policy, the gradient variance of usual policy gradient algorithms will explode, e.g. in REINFORCE algorithm. And it is also well known that DPG does not suffer from such a problem since it has defined the policy as deterministic in the first place. Therefore, for me, the author's argument on theorem 1 of saying GAIL with deterministic policies is unstable is not convincing unless any empirical evidence is provided, e.g. the actual variance of gradient during training.

- the result is not very significant:

The paper does not propose any new algorithm based on the analysis. Since most of the findings in this paper do not seem to be very significant to me, I cannot highly value the contribution of the paper.
clarity,_quality,_novelty_and_reproducibility: While the arguments made in the paper are quite original, I believe that the quality and clarity of the paper are not good enough.
summary_of_the_review: Due to the weaknesses I listed above: low writing quality, not convincing analysis, and lack of algorithm as a result of the analysis, I recommend rejection of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: * This paper explores the mechanism behind the generative adversarial imitation learning, which provide an important conclusion that instability is caused by deterministic policies, rather than GANs.

* It provides some existing methods relieve exploding gradients, but at the expense of “non-confidence”, and ST-GAIL has advantages in mitigating instability.

strength_and_weaknesses: Pros:
* I believe that there is well backed motivation for work based off of the plentiful literature review. 
*This article provides a theoretical basis for strengthening the study of learning stability and provides a plan for the design of future GAIL work.

Cons:
*This paper is only analyzed under the toy data set but does not test it under the actual scene data set.
* The composition of this article does not demonstrate carefully, and the writing is very obscure and difficult to understand.

clarity,_quality,_novelty_and_reproducibility: * The writing is very obscure and difficult to understand.

*  It is interesting this paper explores the mechanism behind the generative adversarial imitation learning, which provide an important conclusion that instability is caused by deterministic policies, rather than GANs.
summary_of_the_review: I would tend to accept this paper as it is novel enough and supported by theoretical analysis.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper appears to show that deterministic policies may cause exploding gradient when used with adversarial imitation learning (AIL), despite their improved sample efficiency. At the same time, the paper appears to show that stochastic policies do not suffer from exploding gradients. This led to the conclusion that deterministic policies are "incompatible with AIL".
strength_and_weaknesses: Strength
1. The paper raises an interesting observation that deterministic policies may cause exploding gradient with AIL. If properly proven/demonstrated, the result would have significant impact on future method design.

Weakness
1. The paper is poorly structured and unclear. Many aspects of the proof are not sufficiently motivated or connected to the overall conclusion. For instance, it is unclear to me how Prop. 1 is related to Theorem 1. In addition, the notion of "non-confident" reward for ARIL has very little explanation (e.g. how it is related to Prop. 2).

2. The primary proof for the main claims is confusing and skip too many steps. Crucially, it is unclear how the exploding gradient is tied to $Pr(||\Sigma^{-1}(a_t - h(s_t))||_2 \ge C)$. In addition, it appears that $||\Sigma^{-1}(a_t - h(s_t))||_2 > 0$ since $L_2$ norm is positive with non-zero $\Sigma$. Therefore it is confusing why $Pr(||\Sigma^{-1}(a_t - h(s_t))||_2 \ge C)=0$ (Remark 2) for stochastic policies.

In addition, the proof relies on the assumption that for every gradient step, the discriminator is optimal. This is an unrealistic assumption and undermines the strength of the claim.

3. The empirical analysis only shows that deterministic policies fail for certain seed. If exploding gradient is indeed an issue as the theoretical results suggested, the authors should explicitly detect exploding gradients rather than just showing learning failures. On the other hand, a large number of runs must be repeated for stochastic polices (much more than 9 seeds, and many runs for each seed) to corroborate that stochastic policies have no exploding gradients, since it's impossible to empirically prove the negative (i.e. stochastic policies have no exploding gradient).
clarity,_quality,_novelty_and_reproducibility: As discussed, the paper is difficult to follow in the current form, given the relatively poor writing quality.
summary_of_the_review: The paper is difficult to follow . While the observation/claim appears interesting, both the theoretical and empirical justification falls short. The manuscript may require significant revision before acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates the case when deterministic policies are learned in a GAN framework. The authors describe occuring instabilities and attribute them to exploding gradients. 
strength_and_weaknesses: # Strenghts

* The authors try to thorougly understand existing algorithms and investigate their weaknesses, which is an interesting and value contribution.
* The authors try to support their insights using experiments and toy models, which is helpful.

# Weaknesses

In my opinion, the biggest weakness of the paper is insufficient clarity in writing. At times, wrong grammar or word-use make reading more difficult, but the intended meaning can still be inferred. Other times however, I found myself unable to follow the writing. Furthermore, the authors sometimes introduce concepts in quotations marks without ever clearly explaining their meaning (e.g. "aggressive interval", "non-confidence" or "invalidity"). 

My other big concern is with respect to Theorem 1 in which they show the probability of exploding gradients. As far as I understand T1, the argument is that the gradient of a JSD divergence w.r.t a deterministic policy can explode. However, I'm not srue this can explain the experimental results, as this JSD is not the optimization target - instead an RL loss is being used. 

Lastly, more experimental results would be needed.
clarity,_quality,_novelty_and_reproducibility: As discussed, in its current form, I found it very difficult to follow the writing of the paper. 
summary_of_the_review: I'm not convinced of the validity of the key results of the paper (T1). However, more importantly, I found the writing at times too difficult to follow, so I am unable to appropriately evaluate the contributions of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents the ill-posedness of using deterministic policies for the family of adversarial imitation learning algorithms. Albeit this is an important problem to be addressed and this work could be taking an important step towards a more robust algorithm, this paper has room for improvement. Reviewers all commented that the writing needs to be improved, and the idea has to be validated on more challenging tasks. I hope the comments below help the authors revise the paper for a future conference.
justification_for_why_not_higher_score: overall low score - contribution is low and writing needs to be improved
Decision: Reject