Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes to mix reinforcement learning and imitation learning to boost the learning of an actor critic architecture. The authors use a regularized reward function that minimizes the divergence between the policy of the expert and the one followed by the agent. They use demonstrations obtained from a trained agent and experiment their method on several mujoco tasks. 

I have many concerns about this paper. First, The state of the art is missing important pre-deep-learning references such as: 

1. Direct Policy Iteration with Demonstrations: Chemali and Lazaric
2. Learning from limited demonstrations, Beomjoon et al.
3. Residual Minimization Handling Expert Demonstrations, Piot et al.

Then, they make a mistake by saying that DQfD only considers transitions from the expert as self-generated and placed in the replay buffer. DQfD actually uses the same additional structured classification loss than Piot et al. [3] (except that they use boosted trees instead of deep networks, DQfD and Piot et al. are the same algorithm).

Also, the proposed solution here is equivalent to regularizing the MDP with a KL divergence  w.r.t. to an initial policy that would be the one of the expert. It is already studied in several works and more generally it comes with some assumptions on the policy update. It is generally studied in 

4. A theory of regularized MDP, Geist et al

They actually propose exactly the same framework as a special case in the appendix of that paper. 

In addition to not be very novel, I think the method has some flaws. The authors use demonstrations coming from a pre-trained network which is known to make the imitation learning part much easier. Especially if it comes from an RL agent using similar deep RL algorithms (which is the case here). Finally, they only test on mujoco tasks which are very specific tasks with deterministic dynamics and very dense rewards  around states visited by the optimal strategy so initializing with an expert policy that is learned from demonstrations of a similar network of course helps. I would be more impressed by experiments on stochastic environments and sparse rewards. 

Finally, there is a concurrent work submitted to the same conference. Of course the authors could not know but I’d like to have their impression about how their work is different.

https://openreview.net/forum?id=BJg9hTNKPH&noteId=BJg9hTNKPH
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents a method for doing RL from demonstrations in continuous control tasks. It combines both an augmented reward for minimizing the KL between the policy and the expert actions as well as directly minimizing that KL in the policy. Results on 5 sparse reward mujoco tasks show that it out-performs other related methods.


The motivation for the paper is difficult to follow. They claim that using demonstration data in a supervised manner "cannot generalize supervision signal over those states unseen in the demonstrations," but most of these approaches are using neural networks and definitely are generalizing those signals to other states. Whether they're generalizing accurately or not is a different question. In contrast, they say that reward shaping approaches do not suffer that problem because they evaluate trajectories rather than states, but there will still be a problem of generalizing to new trajectories. 

The abstract is even more confusing as it tries to jump straight into the issues with these approaches without any explanation. I don't think there's enough space in the abstract to go into that level of detail.

The description of DQfD and DDPGfD in the related work is not accurate. They're described as "treating demonstration data as self-generated data," but in fact they both add supervised losses to more closely match the demonstrated data. https://ieeexplore.ieee.org/document/8794074 is another method built on DDPG that has both a critic and actor loss like yours and would make a useful comparison.

The related work section should also discuss and compare/contrast to GAIL, I was surprised that wasn't in there, especially since you also use a discriminator to differentiate expert and agent actions. 

The end of the related work section is not very clear, you say these methods are problematic because "the adopted shaping reward yields no direct dependence on the current policy" but there's no explanation or motivation for why that would be a problem. 

Assumption 1 seems like a very strong assumption that would not be true for many human experts. 

For the experiments, I wonder about the impact of only using sparse reward tasks. Converting the tasks to sparse reward in this way makes them partially observable, and then potentially the expert demonstrations are required to overcome that partial observability. How do the methods compare on the unmodified tasks? There was nothing specific in your algorithm that meant it should specifically address sparse reward tasks. What about tasks that are naturally sparse reward? 

Overall, the algorithm is interesting and the results are nice. The motivation and related work need to be made clearer to situate this work with the other related works. And the experiments should go beyond these tasks that have been modified to have sparse rewards. 


The revised version of the paper addresses many of my concerns about the motivation, related works, and comparisons with GAIL, so I'm updating my score to Weak Accept. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper studied reinforcement learning from demonstration. Given a set of 
expert demonstrations, this work provides a policy-dependent reward shaping objective that
can utilize demonstration information and preserves policy optimality, policy improvement,
and the convergence of policy iteration at the same time, under the assumption
that expert policy is optimal and stochastic.
The main advantage of the proposed method is that the reward shaping function
is related to the current policy. 
A practical algorithm based on theoretical derivation is provided. The authors conducted sufficient experimental results to demonstrate that the proposed method is effective,
comparing with a set of advanced baselines.

I recommend acceptance: 
Previous works on RLfD usually empirically incorporated a regularization
to the RL objective, while those works didn't discuss whether this
regularization will lead to sub-optimal policy or not. This paper discussed
how to use the demonstration information to do exploration and maintain
policy invariance at the same time, with a relatively strong assumption.
Using the framework from SAC, the
algorithm is shown to converge to the optimal via policy iteration, in
tabular case. This work also developed a practical expert policy
support estimation algorithm to measure the uncertainty of
expert policy. Utilizing the adversarial training framework, 
the explicit computation of expert policy is avoided. 
The authors conducted sufficient experiments to demonstrate
the effectiveness of the proposed method, compared with the state-of-the-art in RLfD. 


Technical concerns:
The stochasticity assumption of expert policy in Asm. 1 can be contradicted
with that expert policy is optimal in policy invariance proof.  
This paper works on a problem of infinite horizon discounted MDP. 
According to Puterman [1994], Theorem 6.2.7, there always exists a
deterministic stationary policy \pi that is optimal. Or intuitively,
if we find the optimal value function via Bellman optimality equation,
the optimal policy is acting greedily (deterministic). The provided theorems are
not compatible with the MDP where only deterministic optimal policy exists.
It is not clear that in what type of MDPs the optimal stochastic policy exists and
it can satisfy Asm. 1. 
Could the authors clearly specify the applicable problem settings?

If the asm 1 is satisfied, what is the necessity to incorporate the
indicator function in Eq 4.? Since p(s) > 0 for all s, following strong stochasticity policy.
For any trajectory \tau, p(\tau) = p(s)\Pi_t p(s_{t+1}|s_{t}, a_t)\pi_E(a_t|s_t) > 0. 

The proof of Theorem 2 is similar to the proof in Proposition 1, [1], though in a
different context. It would be better to have a citation?

Experiments:
It would be more convincing to show the performance of behavior cloning policy using
expert trajectories. 

[1] Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014.


review_title: Official Blind Review #3

Meta-review: The paper proposes to combine RL and Imitation Learning. It defines a regularized reward function that minimizes the KL distance between the policy and the expert action. The formulation is similar to the KL regularized MDPs, but with the difference that an additional indicator function based on the support of the expert’s distribution is multiplied to the regularized term.

Several issues have been brought up by the reviewers, including:
* Comparison with pre-deep learning literature on the combination of RL and imitation learning
* Similarity to regularized MDP framework
* Assumption 1 requiring a stochastic expert policy, contradicting the policy invariance claim
* Difficulty of learning the indicator function of the support of the expert’s data distribution

Some of these issues have been addressed, but at the end of the day, one of the expert reviewers was not convinced that the problem of learning an indicator function is going to be easy at all. The reviewer believes that learning such a function requires "learning a harsh approximation of the density of visits of the expert for every state which is a quite hard task, especially in stochastic environments.” 

Another issue is related to the policy invariance under the optimal expert policy. In most MDPs, the optimal policy is not stochastic and does not satisfy Assumption 1, so the optimal policy invariance proof seems to contradict Assumption 1.

Overall, it seems that even though this might become a good paper, it requires some improvements. I encourage the authors to address the reviewers’ comments as much as possible.
Decision: Reject