Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Review
Key points are written in bold font.

* Section 1:
  * You should maybe include a reference to "A Divergence Minimization Perspective on Imitation Learning" (Ghasemipour et al. 2019) since that work studies properties of different divergences used for match the joint state-action distributions (similar to Ke et al. which you cited) as well state-marginal matching using adversarial methods.
* Section 2:
  * For max-ent RL, there is an intuitive connection between RL and probability distributions over the space of trajectories. For example -- when we have deterministic transition dynamics -- max-ent RL's optimal policy is the policy that "samples" trajectories according to the (unnormalized) density $\hat{p}(\tau) = \sum_t r(s_t, a_t)$. I understand the motivation of the addition of this form of entropy term instead, but is there anything that can be said about what the optimal policy for this objective looks like (in a similar vein to the max-ent RL solution I described above)?
  * __Section 2.2, Theorem 1: Assumption 1 should definitely be included in the main text. This is quite a strong assumption (even if it holds in many physical/robotics problems), and the reader should not have to look into the appendix to realize this. Furthermore, I strongly believe that you should include a discussion in the main text of why you needed these assumptions (what goes wrong when these assumptions are not satisfied?). This is important so that 1) you provide intuition to the reader of what can go wrong if this is not satisfied, 2) future work can try to address the limitations.__ For example, if the transition dynamics was not deterministic, the intuition your provide when saying "$I\_{NWJ}$ encourages the agent to seek out states where...", is not necessarily correct. In a non-deterministic transition setting, the mutual information term could seek out states where the dynamics are more determinstic, but I don't see why that would be a good thing necessarily.
  * Section 2.2: "... Our bound will encourage the agent to continuously seek out...": I don't quite agree with this. You're also increasing entropy, which reduces the chances of getting to those states. Also, the initial low variance random policy might/probably won't get anywhere interesting anyways.
* Section 3:
  * You are using SAC as your RL algorithm. SAC itself is introducing entropy, so maybe comment on how that is playing with your entropy term. Is there any concerns one should be aware of?
  * __Top of page 6. You are saying that your method is not adversarial *<<even if you optimize $f$>>*. I believe, despite appendix A.3, this statement is incorrect. Consider the AIRL (Fu et al.) algorithm. According to Ghasemipour et al., 2019, the AIRL objective minimizes $KL(\rho^\pi(s,a)||\rho^{exp}(s,a))$.__
$-KL(\rho^\pi(s,a)||\rho^{exp}(s,a)) = E_{\rho^\pi(s,a)} [\log \frac{\rho^{exp}(s,a)}{\rho^\pi(s,a)}] = \text{RL with the loss }\quad \log \frac{\rho^{exp}(s,a)}{\rho^\pi(s,a)}$
__In works such as AIRL and GAIL there are two things to consider. 1) lower bounds similar to your $I\_{NWJ}$ lower bound are written to estimate quantities of interest (Jensen-Shannon for GAIL, and reverse KL for AIRL). These lower bounds are maximized to obtain tighter estimates of the quantity of interest. 2) Once a good estimate is obtained, an RL step is performed using the estimate. And this process is repeated.
In your work, you are trying to avoid directly estimating $\log \frac{\rho^\pi(s,a)}{\rho^{exp}(s,a)}$, and have decomposed the objective. If we try to write this $KL$ objective in a form that looks closer to your equations, we get:__
$-KL(\rho^\pi(s,a)||\rho^{exp}(s,a))
= E_{\rho^\pi(s,a)}[\log \frac{\rho^{exp}(s,a)}{\rho^\pi(s,a)}] = E_{\rho^\pi(s,a)}[\log \rho^{exp}(s,a) - \log \rho^\pi(s,a)]$
$= E_{\rho^\pi(s,a)}[\log \rho^{exp}(s,a)] - E_{\rho^\pi(s,a)}[\log \rho^\pi(s,a)] \text{(The J and H in your equation 2)}$
$= E_{\rho^\pi(s,a)}[\log \rho^{exp}(s,a)] - E_{\rho^\pi(s,a)}[\log \rho^\pi(a|s)] - E_{\rho^\pi(s,a)}[\log \rho^\pi(s)]$
__In your work, you are estimating $\log \rho^{exp}(s,a)$ directly through generative models, $\log \rho^\pi(a|s)$ is available from the policy, and in your derivations, it seems that you have been able to replace  $- E\_{\rho^\pi(s,a)}[\log \rho^\pi(s)]$ with the mutual information terms (due to Assumption 1 I believe). But this remaining mutual information terms is still being estimated using similar processes/lower bounds as prior work.__
__Hence, it is my view that your method is as "adversarial" as prior work. The reason you are suggesting stability of your optimization process (which I don't think you have quantified) is maybe because you are not actually learning $f(s\_{t+1}, s\_t)$ which is the property that makes those prior works "adversarial".__
* Section 5:
  * __Pipeline: Why did you checkpoint based on your augmented rewards? How were the checkpoints made for the other methods?__
  * Architecture: What is $\lambda_\pi$?
  * In GAIL are you using techniques from the GAN literatures (such as gradient penalty or spectral normalization) to regularize the discriminators? And are you using the original TRPO optimization or a newer technique such as PPO?
  * Do you have intuition for the poor performance or Ant NDI+MADE? For the 25 demos version in the appendix this value is also low.
  * __Section 5.1: Although this does not seem to be a central claim of contribution in your work, I am wondering if your claims of sample-efficiency are well-founded. You are using SAC, which is an off-policy algorithm, whereas GAIL uses on-policy RL (e.g. TRPO). To enable a proper judgement of sample-efficiency, please respond to the following questions: 1) What implementation of GAIL did you use? Does it use TRPO. If so, this might be a contributing factor since there are better on-policy algs now such as PPO. 2) For your method, in your SAC replay buffer are you only storing on-policy samples, or maintaining a large replay buffer or past interactions? 3) When you're estimating the mutual information, you are not using on policy samples for the marginal q distributions, correct? 4) What is the ratio of model update steps to environment steps in the various algorithms?__
  * With regards to the previous point, a more fair comparison might be to run GAIL with SAC and maintaining a full replay buffer, as done in Ghasemipour et al., 2019.
  * __Table 2: $\lambda\_f$ is tuning the knob for the stregth of the mutual information maximization term. Why should this be improving the KL?__
  * __You have a good set of experiments, but ( and I might be wrong) my biggest gripe with your experiments is that it doesn't seem that you have any experiments that would demonstrate a unique advantage of your method. Please clarify the unique advantages of your methodology, and how that is being shown in the experiments.__
* Section 6:
  * I don't think AIL works particularly poorly on visual domains (e.g. "InfoGAIL: Interpretable Imitation Learning from
Visual Demonstrations" or "Third Person Imitation Learning"). Do you have references for this statement?
  * I don't quite understand this statement: "We posit that...". Please clarify.
review_title: Considering my review, please clarify what you believe are your key contributions and which experiments demonstrate this

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This work proposes a novel density matching method for learning from demonstration, which achieves state-of-the-art demonstration efficiency. Prior density matching methods utilize the adversarial methods suffers from the instability of optimization. To overcome this issue, this work proposes to separate the imitation process into expert density estimation phase and density matching phase, where a model-free formulation is derived and provably served as the lower bound of reverse KL divergence between $\pi_\theta$ and expert policy $\pi_E$.

This work overcomes the instability issue of min-max optimization
 through transferring the objective to the lower bound leading to 
the model-free objective is attractive and novel as far as I am concerned. 

The experimental results are mostly complete and convincing. 
The improvement in the final performance over state-of-the-art is demonstrated. Although the sample-efficiency is not the focus of this 
work, it would be helpful to see the averaged learning curves as in the results
in related works, which can give a straight forward intuition on the
robustness of the proposed methods. 

Please specify the number of random seeds and, the number of evaluation trajectories [and what is the mechanism to choose those trajectories, such as the last 10 episodes?] of 
the reported results in Table 1 and other tables. 

Questions: 
1. Does the improvement come from the better density estimation of the expert's policy? It would be helpful to separate the two phases and conduct an ablation study to show how does each phase affects the performance. 
It would be nice to see the results of replacing density estimation with
oracle [density estimated by a sufficient amount of trajectries]. I didn't see too much difference in terms of final performance between 1 and 25 expert trajectories. Is there a large gap in the sample complexity?

2. Could the author explain the reason why one expert trajectory can provide a good density estimation of expert policy? Typically, we cannot expect one trajectory to cover the state-action space. It is less attractive
if the learned policy can reach expert-level performance but stick to
the original modality of the expert policy. 


Appendix: 
Lemma 1:
$-\lambda \sum_{x} \hat{p}(x) \log \hat{p}(x)+(1-\lambda) \sum_{x} \hat{q}(x) \log \hat{q}(x) $ $\rightarrow$ $-\left[\lambda \sum_{x} \hat{p}(x) \log \hat{p}(x)+(1-\lambda) \sum_{x} \hat{q}(x) \log \hat{q}(x)\right]$

Regarding theorem 1
Do we need to normalize $p_{\theta}(s)$ with $1/(1-\gamma)$ before applying concave property? 
$\left(\mathcal{H}\left(s_{0}\right)+\sum_{t=1}^{\infty} \gamma^{t} \mathcal{H}\left(s_{t}\right)\right)$ 


review_title: A principled novel imitation learning method with good empirical results. Need more results to demonstrate the robustness and better understand the properties of the proposed method

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: =====POST-REBUTTAL COMMENTS======== 

I thank the authors for the response and the efforts in the updated draft. Most of my concerns were clarified and I still think the paper should be accepted. However, I agree with Reviewer 4 that  additional experiments would be good to better tease out the reasons for this method working.



##########################################################################

Summary:
 

The paper proposes an interesting way to do imitation learning without using an adversarial framework. The proposed approach involves density estimation to learn a surrogate reward function that can be optimized via RL. The approach is motivated by recently proposed maximum occupancy entropy RL and extends this to the imitation learning setting. The authors derive an efficient policy optimization method that outperforms existing approaches for imitation learning.

##########################################################################

Reasons for score: 
 

This paper provides strong theory and strong empirical results validating the theory. GAIL-like methods are notorious for their instability so having non-adversarial IL methods is a significant improvement.
 

##########################################################################

Pros: 
 

1. Non-adversarial approach to IL that seems well suited policy optimization with any RL algorithm.

2. Significant improvement over state-of-the-art IL approaches. Also works with only one demonstration.

3. Nice theoretical results showing the objective lower bounds reverse KL between expert and imitator.

##########################################################################

Cons: 
 

1. Only results on mujoco tasks with state information. Most of these tasks can be solved reasonably well with just a bias toward longer episodes. It would be nice to show that the authors method qualitatively imitates a variety of behaviors rather than just being able to go really fast without falling down. Imitating something like the hopper back flip would be a nice addition (https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/). Some other kind of open ended task would also be interesting where learning from demonstrations actually makes sense.

2. Omits other papers that also perform efficient reward learning, then RL for non-adversarial imitation learning: 
e.g. 
Uchibe. "Model-free deep inverse reinforcement learning by logistic regression." Neural Processing Letters, 2018.
Brown et al. "Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations." CoRL, 2019. 

3. Quite a few knobs that need to be tuned in terms of hyperparameters. Perhaps I missed it, but it would be nice to have better intuition for how to set these and how imitation behavior changes based on them.


##########################################################################

Questions/Clarifications:

I'm confused about why the authors think this method will work well with hard exploration video games. I'm not sure I buy the claim that optimizing the authors bound will cause an agent to explore new levels.

After equation (9) the authors say this will cause the policy to seek out areas with low randomness and update the policy to be random there, but the alternative also seems equally likely given the objective, e.g., seek out states where there is lots of randomness and try and make the policy less random there.

The notation for the autoregressive section is confusing what does x = (x_1, ..., x_dim(S) + dim(A) ) =  (s,a) mean? What if S and A are inf dimensional? Also, autoregressive models are often used for time series data, but here is looks like q is conditioned on (s,a) pairs from some lexicographic ordering which does not seem to make sense since there may be no correlation between actions at far apart states.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces an approach for imitation learning based on density estimation. The approach uses the previously introduced idea of minimizing some divergence between policy and expert occupancy measures, state-action distributions induced by these policies. The authors propose to first estimate expert occupancy measure either using an autoregressive density model or an energy based model. Then authors use the Donsker-Varadhan KL representation to compute a log ratio between $p(s_{t+1}|s_t)$ and $p(s_t)$ where $s_{t}$ is a state at time $t$. Then, the expert occupancy and the KL representation are used as RL reward for imitation learning. 

In overall, I think that this approach is interesting and this direction is under-explored in the context of imitation learning. The paper is well written and easy to follow. However, there are some issues that require additional clarification (see my comments below). I think that experimental evaluation is adequate but limited, the paper requires an additional ablation study in order to justify certain design decisions. Moreover, I believe that since the method is using SAC as its underlying RL algorithm its necessary to perform additional comparisons with other methods in sample efficient RL ([1], [2] or [3]).

In conclusion, I think that at the moment this is a borderline paper. The approach is definitely interesting but right now the experimental evaluation is not solid enough to recommend this paper for acceptance. I highly recommend the authors to revise the paper and I'm looking forward to authors' response.

Comments:
1) How many additional interactions from the environments does this approach require? Some other papers ([1], [2], [3]) plot training performance vs number of additional environment interactions.
2) policy entropy term is included into rewards in algorithm 1 (line 6). Is it the same entropy term as in SAC? If yes, what is the benefit of including it twice?
3) algorithm 1 would benefit from explicitly stating how f is computed in practice;
4) what is the benefit of using the representation (7) to compute state density only? Could the same representation be used to compute $KL(\rho_\pi(s, a)|\rho_{expert}(s,a))$? I think that the paper could significantly benefit from additional experiments that demonstrate that the proposed formulation is better than simply using (7) for state-action distributions.
5) Table 2 demonstrates that smaller values of $\lambda_f$ lead to better performance (with an exception of Ant but the difference is within the standard deviation). The figure table 2 can benefit from including results for $\lambda_f=0$.


References:

[1] Sample Efficient Imitation Learning for Continuous Control, Sasaki et al., ICLR 2019: https://openreview.net/forum?id=BkN5UoAqF7

[2] Discriminator-Actor-Critic, Kostrikov et al., ICLR 2019: https://openreview.net/forum?id=Hk4fpoA5Km

[3] Sample-Efficient Imitation Learning via Generative Adversarial Nets, Blonde et al,. 2018: https://arxiv.org/abs/1809.02064
review_title: An interesting approach for imitation learning, but the paper requires additional experiments

Meta-review: This is a difficult borderline decision, with the reviewers evenly split in their final recommendation.  Overall, the authors provided good responses to the reviewer questions: this was much appreciated.  The reviewers requested additional ablations and explanations, which the authors provided.

A prevailing concern is that the experimental evaluation, restricted to a few standard MuJoCo environments, does not really demonstrated a distinctive advantage for the proposed approach.  In fact, one of the new ablations added raises concerns about the significance of the paper's main technical contributions: the \lambda_f=0 row added to Table 3 shows very strong reward results, which apparently obviates a key aspect of the proposed approach. 

This work is interesting, and would like to see it published, but the current state of the evaluation does not support the significance of the main contribution.  I think the authors need to expand their empirical evaluation, as suggested, to better highlight the effectiveness of the proposed approach over the \lambda_f=0 baseline.  In the end, I think the authors would be better served by broadening the evaluation, isolating scenarios where the key proposal shows significant (rather than marginal) benefits, and publishing a more compelling version.
Decision: Reject