Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper is introducing a learning method which combines both Imitation Learning and Reinforcement Learning, such that an autonomous learner can leverage prerecorded expert knowledge. In comparison to previous work, this model has an expert cost function which gives priority to the expert behavior, not only using the expert demos (like in DQfD), but also with a model trained with those demos using  behavioral cloning, such that it could be evaluated in states that were not visited during the demonstrations. Additionally, new executions of the learner that have high performance are included in the buffer for training the policy imitating the expert, since they can also be considered new better demonstrations.

The paper presents an interesting idea with potential, although there are some aspects to consider mainly about the presentation of the paper, in order to improve its content, as listed below. However, the aspects that are not clear in the current version of the paper need to be fixed before considering acceptance, since there are clear open questions.

Section 2 Related work needs some improvement:
- First sentence of Section 2 could be improved for it to be more meaningful "Learning from demonstration (LfD) is a class of decision-making methods by learning from demonstration and  imitation  learning is a subset of LfD", additionally LfD are not exactly decision-making methods.
- It is stated that BC received extensive attention "due to its great performance", I'd say rather "due to its simplicity", since it is simply the application of supervised learning, there is extensive literature arguing that it does not always perform well, therefore several other methods have been proposed.
- "Behavioral cloning is an end-to-end learning method" is not a right statement.
- Authors mention about DAgger that "... address the shortcoming of limited state-action space covered by demonstration in traditional imitation learning". This is an indirect result, but actually it is intended to cope with the covariate shift problem.
- in "...adversarial learning method and has yielded impressive results...", impressive could be replaced by an objective word.

Regarding section 4:- In "As the transition quality of historical demonstrations is usually much higher than that associated with stochastic policies..." stochastic policies are not necessarily something meaningless, I guess the authors meant a completely random policy.
- In the line 20 of Algorithm 1. how the "get the best episode score" is computed? it is just taken from the final cumulative reward of each episode compared to the previous history? if that's the case how to deal with lucky episodes in which the conditions are simply easier for the agent, and not necessarily that the current policy is indeed better.
- It's not so clear why the self learning process is executed, my interpretation is that authors use it to update the BC policy towards \pi(s), such that suboptimal behaviors in the demonstrations don't keep pulling the learning policy.
- lines 8 and 18 have different notations for the same operation,  authors could use only one of them to be more consistent.
Regarding the experiments section- There are missing details of the experiments that do not allow for reproducing the procedure, how many repetitions were run for each case in Fig 2 and 3?, and also for the ablation study? It is not clear whether the results of Fig 2 and 3 are obtained from the same experiments that gave the results in Table 1, at least they do not seem to match. Why testing with the 2 variants of the CartPole environment instead of another kind of problem? since the difference between them is very simple

- Why is there no data in Table 1 for the cases of imperfect demonstrations for the CartPole-V0 environment?
- Why the learning curves of DDQN for the CartPole-V0 and CartPole-V1 cases are not similar at least during the first 200 episodes, looking at the mean and variance, both environments are not close to reach the maximum time steps, therefore in that region they should behave exactly the same.
- In Table 1, it is not so clear what average stop episode means, if it is the episode number in which the stop condition was reached, why are there negative numbers for the acrobot? if they are rewards obtained in the last episode, why are rewards higher than the possible to obtain in CartPole-V0.
- It is mentioned  "...but the DDQN and DQfD fail to meet the experimental termination condition with imperfect demonstration", However this is completely wrong, DDQN do not use demonstrations.

- It would be interesting to additionally mention how to define the size of the margin for the expert cost function, because this may depend on the kind of reward functions/range of value functions.
 

review_title:  

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper proposes a new approach (DQfDD-BC) to leverage demonstrations in the framework of deep Q-learning. The new approach augments the previous approach called DQfD by 
a) adding self-generated trajectories with high rewards as additional demonstrations (a feature the authors called “dynamic demonstrations”), 
b) replacing large-margin loss with a cross-entropy loss between the learner agent and a helper policy learned by behavior cloning demonstrations. 
The new approach improves upon the previous by better leveraging demonstrations and being more robust against imperfect demonstrations. The authors demonstrated these improvements in experiments by showing the new approach leads to accelerated training speed (in terms of episodes) when using either perfect or imperfect demonstrations, and higher performance (in terms of rewards) when using imperfect demonstrations. 

==============

Positives

This paper is well motivated. The two intended benefits of the new approach are meaningful and important.

The experiments back up the authors’ claim on the benefits, especially on the improvement of training speed. In the CartPole tasks, the training speed is improved by 2 folds.

The paper is written clearly. It was easy to understand the new approach and the differences on top of the previous one. 

==============

Negatives

The extensiveness of experiments is underwhelming, both in the diversity and complexity of the tasks chosen. The four tasks are among the most simple ones typically used for comparable algorithms, having very low degrees of freedom and very simple dynamics. For example, DQfD was validated on a broader and more challenging set of Atari tasks, including Montezuma Revenge where (human) demonstrations are critical for a learner agent to tackle the task.

By introducing a BC-learned helper policy, the approach introduces dependencies on how well this helper policy is (in terms of approximating the demonstrations) in order to provide better loss signals to large-margin loss. It’d be good to design additional experiments to showcase shortcomings, if any, of this proposed approach. For example, the authors used ten thousands transitions as initial demonstrations. It’d be interesting to see how the proposed approach would fare when different amounts of demonstrations are available.

==============

Recommendation

Overall the approach proposed is interesting and promising. I’d recommend a weak accept. The authors are strongly encouraged to further improve the experiments. 

==============

Minor Comments

Table 1 is confusing to read. Especially in the top half, “Demo” rewards are juxtaposed with other numbers in a completely different units (number of episodes).

There are some typos. In section 2, it’s not clear what determination is meant in “determination of DAGGER”. In Section 1 “difficult consistent” -> “difficult to be consistent”. 

review_title: Official Blind Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 

This paper proposes integrating deep Q-learning from dynamic demonstrations with a behavioral cloning model (DQfDD-BC). Compared with DQfD, the proposed approach introduces a behavior cloning model, which was first pre-trained by leveraging historical demonstrations and then updated using generated dynamic demonstration. The BC model is used in the expert loss function, where the DRL model's actions are compared with those obtained from the BC model for policy improvement guidance. The experimental results in OpenAI Gym environments show that the proposed approach adapts well to different demonstrations' imperfection levels and accelerates the learning processes. The ablation study also indicates that the new method improves the learning convergence performance compared with the original DQfD model.

========================

Reasons for score:
Overall, based on the description of the paper, the proposed approach works well.

========================

Detailed comments:
- In the proposed approach, the BC model is fine-tuned if the model achieves a relative high-performance score. With this mechanism,  high-quality transition samples are included, and it helps avoid adverse impacts caused by imperfect demonstrations.
- The experiments show that the performance of DQfDD-BC is better than DQfQ, and it works well for imperfect demonstration scenarios.
- In this paper, I think the primary idea is about how to leverage high-quality data in the generated trajectory well. This paper aims to build a BC model, initialized by demonstration dataset, and later finetuned with newly generated high quality data. One thing I do not understand well is the necessity of building the BC model. What if we use some heuristic approaches to replace the replay buffer with higher quality data? Can some simple strategies like this achieve similar performance as building a BC model?

========================

Some obvious typos:
- Section 5 Experimental results: environmets -> environments 
-Section 5.3, paragraph 2: combins -> combines
review_title: Tend to accept

final_score: rating: 7: Good paper, accept
review_content: =====POST-REBUTTAL COMMENTS======== 

I thank the authors for the response and the efforts in the updated draft. Most of my concerns were addressed. This is a simple, but nice idea. After reading the rebuttal and the other reviews I am recommending to accept the paper.

##########################################################################

Summary:
 

This paper seeks to improve deep reinforcement learning from demonstrations via a supervised learning loss based on behavioral cloning. The results show that adding the BC loss stabilizes training and shows improvements over baselines for OpenAI Gym environments. 

##########################################################################

Reasons for score: 
 

I think the idea is nice, but it is hard to follow as written and the experiments are on simple problems. It would be better to compare against more complex benchmarks such as Atari. The writing needs improvement for clarity.
 

##########################################################################
Pros: 
 

1. The ablation study in Figure 4 shows the benefit of the proposed method.

2. The ability to outperform a suboptimal demonstrator is important for imitation learning in the real world.
 



##########################################################################

Cons: 
 
1. The idea of adding new trajectories from high scores was interesting, but I was left wondering why this works. Won't this add a lot of really bad trajectories in the beginning of learning which seem like they would have a negative impact on the BC policy. Also, the function "get the best episode score" was never rigorously defined.

2. Equations should be introduced before they are included. Equations 1-4 come before the text that talks about them and this makes the paper more confusing that it needs to be.

3. Equation (5) seems incomplete. Shouldn't l(\pi, \pi_bc, s_t) be a function of the action? Otherwise it is just a constant per state, but it is in the max_a brackets. Also it is unclear what the difference between s_t and s are in Eqn (5).

4. Average stop episode is not clearly defined in the text. In the appendix it explains more but the thresholds are not justified.


review_title: Review

Meta-review: The reviewer acknowledged that the proposed method is simple and seems to work well on the chosen benchmarks. Yet the expressed several concerns that were not fully addressed by the authors in their responses. The major concern is about the experimental setup. The chosen tasks have been judged too simple and quite different from those where the baselines were tested initially (e.g. DQfD was demonstrated on a diverse set of Atari games). 

The clarity of the paper should also be improved. For instance, the way the number of trajectories that are added to the replay buffer increases with time is not well explained and it seems to be crucial for the algorithm to outperform the baselines. The authors also seemed to select the experiments so that the "results are more persuasive", discarding experiments where you can be unlucky. This looks very much like cherry-picking and didn't convince the reviewers. 
Decision: Reject