Deep Q Learning from Dynamic Demonstration with Behavioral Cloning
Authors: Xiaoshuang Li
Junchen Jin
Xiao Wang
Fei-Yue Wang
Venue:  (2021)

Abstract:     Although Deep Reinforcement Learning (DRL) has proven its capability to learn optimal policies by directly interacting with simulation environments, how to combine DRL with supervised learning and leverage additional knowledge to assist the DRL agent effectively still remains difficult. This study proposes a novel approach integrating deep Q learning from dynamic demonstrations with a behavioral cloning model (DQfDD-BC), which includes a supervised learning technique of instructing a DRL model to enhance its performance. Specifically, the DQfDD-BC model leverages historical demonstrations to pre-train a supervised BC model and consistently update it by learning the dynamically updated demonstrations. Then the DQfDD-BC model manages the sample complexity by exploiting both the historical and generated demonstrations. An expert loss function is designed to compare actions generated by the DRL model with those obtained from the BC model to provide advantageous guidance for policy improvements. Experimental results in several OpenAI Gym environments show that the proposed approach adapts to different performance levels of demonstrations, and meanwhile, accelerates the learning processes. As illustrated in an ablation study, the dynamic demonstration and expert loss mechanisms with the utilization of a BC model contribute to improving the learning convergence performance compared with the origin DQfD model. 

Under review as a conference paper at ICLR 2021
DEEP Q LEARNING FROM DYNAMIC DEMONSTRATION WITH BEHAVIORAL CLONING
Anonymous authors Paper under double-blind review
ABSTRACT
Although Deep Reinforcement Learning (DRL) has proven its capability to learn optimal policies by directly interacting with simulation environments, how to combine DRL. with supervised learning and leverage additional knowledge to as- sist the DRL agent effectively still remains difficult. This study proposes a novel approach integrating deep Q learning from dynamic demonstrations with behav- ioral cloning model (DQfDD-BB (CF) which includes supervised learning technique of instructing DRL model to enhance its performance. Specifically, the DQFDD- BC model leverages historical demonstrations to pre-train supervised BC model and consistently update it by learning the dynamically updated demonstrations. Then the DQMDD-BC model manages the sample complexity by exploiting both the historical and generated demonstrations. An expert loss function is designed compare actions generated by the DRL model with those obtained from the BC model to provide advantageous guidance for policy improvements. Experimental results in several OpenAl Gym environments show that the proposed approach adapts to different performance levels of demonstrations, and meanwhile, accel- erates the learning processes. As illustrated in an ablation study, the dynamic demonstration and expert loss mechanisms with the utilization of a BC model contribute to improving the learning convergence performance compared with the origin DQAD model.
INTRODUCTION
Deep reinforcement learning (DRL) methods have made great progress (Mnih et al. 2013; 2015 Silver al. 2017) when applied in several rule-based applications such the Go game Silver et al.] 2016). However, due to the diversity and uncertainty of complex systems, the establishment of simulation environment difficult be consistent with the real-world system. Therefore, DRL algorithms usually fail in direct application to many real-world scenarios. Mcanwhile, a DRL model may produce an action that is sampled from a random policy when exploring the state-action space. However, random actions are not allowed in many real-world circumstances. For example, autonomous driving experiments (Kiran et al., 2020), a random policy may contribute to traffic congestion, even road accidents. Therefore, fitting complex situations becomes one of the most urgent tasks for applying a DRL model for complicated decision
Decision-making making tasks.
It is noted that human experts have great advantages in learning efficiency and decision-making performance (Tsividis et al. 2017). Incorporating expert knowledge is a potential solution to en- hance the adaptability of DRL models for complex tasks (Hester al. 2018 Matas et al. 2018). Nevertheless, the knowledge and experience of an expert are difficult to be modeled and described directly. One solution, attracting more and more attention, is to indirectly learn expert strategies by learning their decision trajectories, also known as demonstrations (Schaal) 1997; Behbahani et al.) 2019 Ravichandar al. 2020). Particularly, deep Q learning from demonstrations (DQfD) is a typ- ical algorithm that succeeded in combining DRL with demonstrations (Hester et al. 2018), which combines the temporal difference (TD) eтror in the traditional DDQN algorithm with supervised expert loss by constructing hybrid loss function. Through specially designed large margin super- vised loss function (Piot al. 2014a b). the DQfI) method can guide and assist an agent to learn the expert's is knowledge by constantly steering the agent learning strategies closer to those represented by the demonstration.

---

Under review as a conference paper at ICLR 2021
However, the DQfD model suffers from three major issues: (1) In the DQfD learning process, the trajectory data in the historical demonstra- tion dataset the only source for contributing expert loss values, which does not include the self-generated transitions of the trained agents, As result, the DQID algorithm merely re- lies on TD errors to improve the policy but the demonstration is idle when the self- self-generated transitions are sampled from the experience re- play buffer, which reduces the efficiency of utilizing demonstrations. (2) According to the learning mechanism, static demonstrations are too limited to cover sufficient state-action space during the agent training process, espe- cially when it is difficult or expensive to col- lect demonstrations in real-world applications.
Loes function
TD ltm)
Supervised loss
largets
Target Q
network
copy every
network
model
S. (S..)
slepps
S.a.
demunaballin
buffer
demonstration
upiited
Priorttized Replay buffer
a..
The technique framework of the pro- posed DQIDD-BC method where IL model rep- resents an imitation learning model.
Figure 1:
Also, with more newly-generated transitions added into the experience replay buffer, historical demonstrations would make smaller contributions to the policy improvement as their sampling prob- ability becomes lower and lower. (3) The DQfD algorithm requires the learned policy approximate the demonstration but ignores the imperfection of historical demonstrations and it is an imperfect demonstration that commonly exists in real-world applications. Perfect demonstration can always provide an appropriate guide but the imperfect demonstration is detrimental to the improvement of the policy when the learned policy superior to the policy represented by the imperfect demonstra- tion.
To address the above issues, we propose a novel deep Q learning from dynamic demonstration with the behavioral cloning (DQrDD-Bi) BC) method. Fig. illlstrates the structure of the proposed approach. DQIDD-BC shares the same basic components of the DDQN algorithm (Hasselt et al. 2016), including two Q networks and an experience replay buffer. There learning (IL.) (Hussein et al., 2017) module and an alterable demonstration in the replay buffer. Besides, the loss value is calculated considering both the output of the trained Q network and the BC model. Two main contributions are summarized as follows:
additional imitation
(1) An IL model, named behavioral cloning (BC) Torabi al. 2018 Büihler al., 2020), is pro- posed to generate expert loss to utilize all the transitions in the experience replay buffer. It first attempts extract the experts' policy from the initial demonstration and allows the agent to provide reasonable actions when facing newly generated states. During the self-learning process, the agent's s actions are compared with those generated by the BC model through a self-designed expert loss function. The inclusion of the BC model allows the knowledge in the demonstrations to be suffi- ciently utilized in the training process and enables the model to cope with the states which experts have not ever encountered. The supervised learning process and self-learning process can promote each other. The supervised model provides a basic reference to guide the model adjustment. Mean- while, the self-learning process keeps improving and overcomes the limitation of the BC model and suboptimal samples during the learning process.
(2) An automatic update mechanism is proposed to adaptively enhance the BC model. In particular, new transitions are generated by the trained agents to tee-tune the BC model if the model achieves relatively gigh-performanc score. Such mechanism tries to include more high-quality transition samples to improve the demonstration and avoids potential adverse impacts caused by imperfect demonstrations.
In this study, we evaluate the proposed DQPDD-BC method on several gym environments (Brock- man et al. 2016)) For comparison purposes, the DDQN and DQID methods are used as the baselines (Hasselt et al. 2016, Hester et al. 2018). The experiments clearly demonstrate that DQIDD-BC surpasses all the baselines in terms of convergence speed as well as decision-making performance. The ablation experiments also show that both the proposed expert loss function together with the BC model and dynamic demonstration contribute significantly the performance superior to the DQIDD-BC algorithm.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORKS
In most cases, demonstration comes from the human or other kinds of "experts", who can provide valuable information for different hard problems, such as robot control (Schaal 1997 Ravichandar
et al. 2020), self-driving (Bojarski et al. (016) and so on. As a kind of approaches which can make use of the demonstration, behavioral cloning Torabi et al.) 2018), one of imitation learning (Hus- sein et al. 2017 Brrwnn et al.) (020), has received extensive attention (Zhang et al.) 2018 Bühler ct al., 2020) due to its significant advantages, such as fast learning speed, simplicity, and effective utilization of demonstration and so It constructs mappings from states to actions (or distribu- tions of actions) in a supervised manner to model the policies represented by the demonstrations and achieves the goal of learning from demonstration by minimizing various supervised losses. In a study by NVIDIA, the model obtained interesting results by minimizing the mean squared error between the steering command output by the network and the command of human driver (Bojarski) et al. 2016). Chauffeurnet further enhances the performance of imitation learning in autopilot by synthesizing the worst scenarios (Bansal al. 2018).
DAgger (Ross et al. 201T) is proposed to cope with the covariate shift problem and experts are required to respond to self- gelf-gene states so that totally new transitions are generated and they can expand the state-action space covered by demonstration. The goal of performance improvement can be achieved by adding new samples continuously during the learning process. However, the DAgger method requires an always-available expert to assist in labeling the data, which reduces the practical- ity of the method. Deeply AggreVaTeD (Sun et al., 2017) enables the DAgger to handle continuous action spaces by using deep neural networks but the weakness of DAgger was also preserved.
Another typical class of imitation learning algorithms is generative adversarial imitation learning (GAIL) (Ho & Ermon 2016: Wang et al.) 2019). Instead of mapping from state to action directly, GAIL learns from the demonstrations by introducing the generator and discriminator. The generator is used to generate state-action pairs and learn policy, while the discriminator is used to distinguish whether a state-action pair is from the expert or the learned policy. The information contained in the demonstration is indirectly combined with policy improvement through the process of adversar- ial learning. GAIL shows good performance on the high-dimensional continuous-control problem (Kang al. 2018 Song et al., 2018). Policy optimization with demonstrations (POID) (Kang al.) 2018) utilizes the demonstration by an adversarial learning method and has made efforts on sparse and dense reward environments.
Recently, demonstrations have been leveraged to tutor DRL to achieve better performance. Since the DRL reward is immediate feedback from the environment for evaluating the performance of the policy, reshaping the reward function through the demonstration is an effective approach to improve DRL performance (Brys et al.] 2015] Suay et al. 2016). By transforming the form of the reward function, the difficulty of agent training has been reduced. Model-Based DRL (MBRL) has shown great power in some difficult tasks (Kaiser et al. 2019) and demonstration is also leveraged to deal
with the MBRL problems (Lambert et al. 2019 Thananjeyan et al. 2020).
All of the above methods yielded impressive results but most of them utilized demonstrations through reward shaping imitating the demonstration. Another idea is to learn from demonstrations directly to improve the policy. The transitions in the experience replay buffer and demonstrations both extracted by DRL agents (Hester et al., 2018 Oh et al. 2018 Xiong et al.) 2019). These approaches put the demonstration into the experience replay buffer and sample from the hybrid ex- perience replay buffer. The DQID algorithm enhances the DDQN (Hasselt al., 2016) algorithm by keeping the demonstration in the replay buffer at all times. A designed supervised expert loss generated when the transitions in the demonstration are sampled to update the parameters of the neural networks. The self- self-imitation learning (SIL) method chooses different loss functions and adds new trajectories to the demonstration. It updates the same neural network twice with the A2C loss function and the SIL loss function (Oh et al.) 2018). The DDPGfD algorithm (Vecerik et al.. 2017) is similar to the DQID and they both assist in enhancing the original algorithms through a hybrid loss function with the "expert loss". The difference between the two is that the DDPGID algorithm is based on the DDPG algorithm (Lillicrap et al. 2015)). which is designed to solve continuous action space problems. However, both DQfD and DDPGfD suffer from the problem of under- under-exploiting demonstration data (Kang et al. 2018) and our DQfDD-Be method has made new and comprehen- sive progress solving the problem.
3

---

Under review as a conference paper at ICLR 2021
3 PROBLEM STATEMENT
We consider the standard Markov Decision Process (MDP) (Sutton & Barto, 2018). which is de- fined by tuple AS r),),, where S,A.P,r, represent the state space, action space, state transition distribution, reward function and discount factor, respectively. The transition distribution P (s' 8, a) describes the process of taking action a at state s, s' is the next state and reward func- tion r(s. a) gives the feedback. A policy (aass) specifies how the agent responses to each state. The goal of the agent to find the policy x wheih maps states to actions that maximizes the expected discounted total reward E |Et= > (st, (x)] over the agent's 's lifetime.
In the DQND algorithm, the complete loss (Eq. I function contains four parts: (QQ (Q). (E( and JL2 (Q), which are the double Q learning TD loss, n-step double Q learning TD loss, expert loss and L.2 regularization loss, respectively. The parameters control the weighting between the losses. The expert loss (Eq. 2 in DQID is the most significant loss among the losses, allows the model to satisfy the Bellman equation, and also enables the agent to learn from the demonstrations. The large margin supervised loss ((ag, a) is 0 when demonstration, and a positive constant otherwise. This loss forces the Q values of the other actions to be at least margin lower than the value of the demonstrator's action (Hester al.) (018). It is noted that, when the sampled transitions are not in the demonstration, X2 = 0 and the expert loss JE (Q) is non-functional.
ag with ag being the expert action in the
+ J(Q)= (Q) + + AgJa())
Jx(Q) = max ( (8.
(2)
aEA
4 METHODOLOGY
4.1 DYNAMIC DEMONSTRATIONS FOR DQFD
As the transition quality of historical demonstrations is usually much higher than that associated with random policies, DQID agent learns its initial knowledge from preset demonstrations, One of the major improvements of our proposed DQIDD-BC approach is adding the newly generated interactions into the demonstrations. These newly generated transitions, who have a higher per- formance score for the same task, are collected from trained model. Particularly, after the early stage of the training process, the transition samples are automatically generated and inserted into the demonstration dataset when the final cumulative reward of each episode reaches a new high score. By adding new and better transition samples after the performance of the model has been improved effectively, the quality of the transition samples is continuously improved while attempting to cover the full state-action space. Such mechanism can continuously improve the performance of the policy represented by the demonstration, and generate a continuously positive effect on the model performance.
The idea of the proposed approach is close to the DAgger algorithm, especially for improving the model's makking capability by adding new transitions to the experience replay buffer and continuously using the added data to optimize the model parameters. However, unlike DAgger, our method does not rely on manual labeling and automatically determines whether new data need be added to the demonstration, which can significantly reduce the computationally cost and improves the applicability of the method.
4.2 EXPERT LOSS WITH SUPERVISED BC MODEL.
The DQID method originally leverages binary supervised large margin loss function ((ag, a) to compare the generated actions with those in demonstrations under the same environment states. However, the values of the large margin loss function are prone to cause volatile gradient and lead to the instability of the training process. In this study, DQfDD-BC is designed for discrete action spaces such that the BC model is considered to solve a multi-label classification problem.
In detail, the proposed DQIDD-BC method contains a deep neural network-basee BC model and maintains two different experience replay buffers: Dreplay and Deemo In particular, Dreploy refers
4

---

Under review as a conference paper at ICLR 2021
to the common experience replay buffer in DRL model while Ddemo consisting of both historical and self self-generated demonstrations. The BC model is first pre-trained with demonstrations D to obtain its initial decision- decision-making ability. Prioritized experience replay mechanism (Schaul al. 2013) is also applied to both of the two replay buffers to improve the sample efficiency. TTee perfor- mance of the model improves all the time during the learning process, so newly generated transition has always been given the highest priority to ensure that transitions with higher performance are ex- tracted timely, and the value of complete loss function is used update the priority of the transition samples.
The DQIDD-BC model takes full advantage of the demonstration by generating expert losses for all self- generated transitions instead of directly using historical demonstrations. Eq. 3 is used to pre-train the BC model and Eq. 4 evaluates the difference between and (s), where a the action provided by the Q network and (s) is the action generated by the latest BC model. In Eqs. 3 and 4 (g refers the action in the demonstration and The represents the learned policy of the BC model. The supervised loss ((a, (s)) is zero when the action of the Q network output is the same as the output of the BC model and otherwise a positive number. Consequently, the complete expert loss function of DQUDD-BC is shown as Eq.
LBC = Crossentropy(ag, The (8t)) I(a, The (s)) Crossentropy(a, Tbc(s)
(4)
(=( = max (Q(s,a) a) + I(a, The(s)) Q (8,ar) a
(5)
Algorithm 1: DQIDD-BC; Deep Q learning from Dynamic Demonstration with Behavioral Cloning
Initialization: Dreplay and Ddeeo; both initialized with the incipient demonstration data set, O: weights for initial Q network, 0': weights for the target network, т: frequency at which to update the target network, E: maximum number of episodes, M: maximum number of newly generated trajectories, k: number of pre-training gradient updates;
Train the BC model with Ddemo with the behavioral cloning cross cross-entropy loss (Eq. 3
for step / to A:
Sample mini-batch of transitions from Dreplay with prioritization;
Calculate expert loss (Q(Q) using the BC model;
Calculate loss JQ)) using target net;
Perform gradient descent step to update 0:
every 7 steps update the target network 0' 0:
for episode = / to E do while not done do
10
11
Sample action from e e- greedy policy;
12
Play action a and observe (8', done);
Store (s, 8', done) into Dreplog;
13
Sample mini-batch of transitions from Dreplay with prioritization;
15
Calculate expert g (Q) using the BC model;
Calculate (Q) using target net;
17
Perform gradient descent step to update 0:
every 7 steps update the target network 0' 0;
19
s = s";
29
if get the best episode score then for i= / to min(episode/ /2, M) do
21
22
Agent interacts with the environment until done;
Store the trajectory into Deemo,
23
Fine-tune the BC model with new Ddeeoo
24
This expert loss function aims to smooth the output of the loss function and stabilize the training process. More importantly. the BC model can be used to provide reasonable actions and generate supervised losses for all self-generated transitions the experience replay buffer. Compared with the DQfD method, the sample efficiency is significantly improved.
5

---

Under review as a conference paper at ICLR 2021
The pseudo-code of the proposed DQIDD-BC method is sketched in Algorithm I and consists of three main stages: BC model pre- training.. agent pre-training, and joint model self-learning. Firstly, the BC model is pre-trained with Eq. to gain the initial decision-male making ability, and then the Q network is also pre-trained in a supervised manner. Through the trained BC model, the information in the demonstrations indirectly contributes the improvement of the agent's performance. During the self-learning process, the trained BC model is leveraged to generate an expert loss to construct the complete loss function. Combined with TD errors, the self-learning ability is retained and the L2 regularization losses on the network weights to prevent over-fitting. After each episode, we evaluate the policy and apply the dynamic demonstration method to D to update the demonstration. The best episode score is taken from the final cumulative reward of each episode compared to the previous history. To reduce the influence the lucky episode, the demonstration will not be updated in the early stage of the self-learning process. We begin to update the demonstration after the self- learning process has last for at least 10 episodes. Fine- sane-ttning of the BC model is conducted after Ddemo is updated, thus enabling the BC model to constantly improve the supervised strategy and providing better target for the agent.
5 EXPERIMENTS AND RESULTS
Experiments setting We evaluated DQIDD-BC in the classic OpenAl Gym environments: CartPole-vO, CartPole-vI, Acrobot-vl and LunarLander-v2 (Brockman al. 2016). We used the original reward functions provided by the gym environments without giving any additional rewards or penalties during the learning process. At the end of each episode, the rewards for all steps in the episode are summed up to form an episode score. The classical DDQN (Hasselt et al., (016) and DQfI) (Hester et al., 2018) algorithms are chosen as the benchmark algorithms to compare with the proposed approach.
Two experimental conditions were set up: perfect demonstrations and imperfect demonstrations. All demonstrations were obtained from pre-trained DDQN models and the major difference between perfect and imperfect demonstrations is as follows. The perfect demonstrations are obtained from state transitions data of well-trained DDQN agent, while the imperfect demonstrations are derived from the model during the training process. In all demonstrations, the initial amount of transitions is around thousand before the training and the number will increase when transitions added into the demonstrations. The average episode scores of perfect demonstration collected from CartPole-vO. vO, CartPole-vl, Acrobot-v1, LunarLander-v2 are 200, 500, -84, 246. and they are 357, More imple-
-238, 14 in imperfect demonstrations of CartPole-vl, Acrobot-vl, mentation and experimental details, such as hyperparameters, termination conditions are presented in the appendix.
Experimental results Table I shows the comparison between DDQN, DQID and DQIDD-BC in different gym environments. The average stop episode means the average stop episode number among different runs and roughly represents the learning speed and the average final score illustrates the final performance of these methods.
Table 1: Performance of different methods in different OpenAl Gym environments
Perfect Demonstration
Imperfect demonstration
DDQN
Env
DQID
DQIDD BC
DQID DQIDD-BC
Average stop episode
CartPole-v0
310-185
162-E170
161::170
CartPole-vl
500+:0
121=19
91:
50010
411::179
Acrobot-vl
198::123
51::
49155
261
143 11
LunarLander-v2
263:12
65113
65:19
402-1192
17
Eny
Average training score for last 10 episodes
CartPole-vO
193:10.6
19.7
194.19.77
CartPole-vl
133--12.1
498.13.9
500-1-.33
348-1-25.1
46.0
Acrobot-vl
.9715.3
-95-4.0
-91+2.5
.96:5..3
-95
LunarLander-v2
2311:6.5
219488.7
2231-6.1
234
211+2004
In all environments, our DQIDD-BC method requires the smallest number of episodes to stop train- ing and obtains almost the highest final performance, which demonstrates the significant advantages

---

Under review as a conference paper at ICLR 2021
of learning speed and performance. It needs be pointed out that the termination condition of CartPole-vO is much casier to be satisfied compared to the CartPole-vl. In some lucky episodes, the termination condition Thus the imperfect demonstration of CartPole-vO is hard to he distinguished from the random pol- icy and trained policy. To make the results more persuasive, only test the proposed method in CartPole-vl 1 with the imperfect demonstration.
be triggered in CartPole-vO,, which is almost impossible in CartPole-vI. 1.
5.1 DQFDD-BO BC WITH PERFECT DEMONSTRATION
Compared with the DQfD algorithm, the proposed DQIDD-BC algorithm achieved better results in terms of convergence speed and learning capability. Fig. shows the episode rewards of various methods in three gym environments. In CartPole-vO, the DQIDD-BC approach has slightly faster convergence speed than DQID during early learning stage. Although some runs DQfDD- BC and DQfD failed to satisfy the termination condition, the table [ shows that the DQIDD-BC has a smaller average stop episode and also gained a small advantage in final performance score. In CartPole-vl and Lunarl Laneer2 environments, the DQfDD-BC shows obvious faster-learning speed and fewer episodes are needed for DQIDD-BC to get a higher score than DQfD.
Averge 8 1 3
Dight
Avecag
1900
250
100
354
150
Epheddo
Egleddo
Sppinedr
(a) CartPole-vO
(b) CartPole-vl
(c) Lunarl Landrrladder-22
Figure 2: The training episode scores of different methods with perfect demonstrations in different environments. The green line is the average scores of the perfect demonstrations.
Although the DDQN failed to trigger the termination condition before the experiment was forced stop CartPole-vl, both DQID and DQIDD-BC can obtain full score, which proves that the demonstration improves the learning ability of DRL agent. The DQFDD-BC algorithm retains the advantages of the DQFD algorithm and further improves the supervised loss function so that all self-generated data can obtain supervised losses, thus improving the learning speed.
5.2 DQFDD-BC WITH IMPERFECT DEMONSTRATIONS
We also validated the adaptability of the DQIDD-BB BC method through experiments with imperfect demonstrations. Since perfect demonstrations are difficult to define and acquire many scenar- ios, it is more relevant to train the agent with imperfect demonstrations which are relatively better than the random policy. For example, the off-line optimized method can be used to generated im- perfect demonstration. Fig. (a) shows the episode scores of different method with imperfect demonstrations in the CartPole-vl environment. The results show that the DQIDD-BC method out- performs DQID by significant margin terms of convergence speed. By keeping the demonstra- tions updated, the proposed DQIDD-BC method allows the newly updated demonstration to avoid the negative influence brought by the imperfect demonstration and eventually converge better performance level.
The proposed DQ method can adapt to imperfect demonstrations well and obtain higher average final performance than DQND) with imperfect demonstration. DQfD method shows great dependence on the demonstration and ultimately converges to the vicinity the demonstration and imperfect demonstrations harm the performance of the DQID method. We further explored the variation of each component of the complete loss function to figure out the reason why DQFDD-BC able to get superior performance. Fig. 3 (b) shows that the TD losses (TD-1, TD-n) of DQfD have been stabilized within a small fluctuation range in the later stages of self-learning but the expert loss fluctuating over wide range. This is partly due to the reason that the action output by the learned

---

Under review as a conference paper at ICLR 2021
rsportinn
401
Avrrer Woweadd
por
PQrDD-cc
Donse
100
159
100
200
360
300
400
Fyledds
(a) Average training scores
(b) Losses of DQfD
(c) Losses of DQIDD-BC
Figure 3: Experimental result with imperfect demonstration. (a) The training score of different
methods in the CartPole-vl environment. The green line is the average scores of the demonstrations. (b) and (c) show the loss function value during the training process.
policy can be quite different from those sampled from the imperfect demonstrations. The agent can obtain better policy through the TD losses and provides an appropriate action. However, the supervised loss draws the learned policy near the policy represented by the imperfect demonstration and inapposite actions are provided.
Additionally, the fixed imperfect demonstration is not immune to its own negative effects, result- ing in the inability to converge to the highest score. Fig- 3 (c) illustrates that the expert loss of DQfDD-BC decreases more steadily and the agent achieves better final performance. The dynamic demonstration has the ability to modify the disadvantage of imperfect demonstration and expand the state-action space covered by the demonstration.
5.3 ABLATIONS
To figure out the importance of dynamic demonstrations and expert loss with BC model in the proposed DQFDD-BC method, we also test the method without dynamic demonstra- tion or BC model in the CartPole-vl environ- ment. Fig. 4 shows the effect of integrat- ing dynamic demonstrations and BC model to the original DQFD model. The figure clearly shows that when the dynamic demonstration is added to the DQID algorithm (DQIDD), the model present a significant increase in training speed during the early stage. The DQ1DD can also converge to the highest score and ended the training process in short time while the DQfD) needs some more time. This indicates that by adding new samples the demonstra- tions, a large number of tiih-ppeffrmance tran- sitions can improve the generalization ability and training stability of the model.
400
Reward
300
DQD
100
DQMD
DQID-BE
DDMD-BC
20
40
68
30
1.00
120
140
160
Epiaodos
Figure 4: Ablation experimental results. The dy- namic demonstration (DQfDD) and expert loss with behavioral cloning model (DQFD-BC) are added to the original DQAD method.
In the case where the large margin expert loss is replaced by the loss generated by the BC model (DQID-BC), the training speed of the model is significantly improved. Fig. Thowws that the average training score of the DQfD algorithm reaches only about 80% of the convergence state, while the DQ model finishes the training process. The DQQD-BC has a slightly lower learning speed than the DQIDD-BC method during the learning process but the DQID-BC ends the learning process earlier. By introducing a supervised model to generate an expert loss, the learning speed is improved significantly. Since the self gelf-gennrated transitions can also be utilized to generate TD losses and ex- pert loss with the supervised model, higher sample efficiency is achieved. The final DQfDD-BC model. which combines dynamic demonstrations and the expert loss with the BC model, simultane- ously improves the learning speed and exhibits excellent performance.
8

---

Under review as a conference paper at ICLR 2021
6 CONCLUSION
In this work, we present an approach, namely the deep Q learning from dynamic demonstration integrated with behavioral cloning (DQfDD-BC), to enhance the learning efficiency and quality of DQfD. The experimental results demonstrate that regardless of the degree of perfection of histori- cal demonstrations, DQJDD-BC can effectively improve the learning speed and final performance compared with DQfD in generic environments. From ablation experiment, both the dynamic demonstration and expert loss with the BC model showed their own potential to improve the learn- ing quality.
As our approach provides a feasible solution to bridge the DRL methods and the decision-making problems in complex real-world environments, will later apply model to real-world settings, such as a large-scale urban traffic control system. While considering the process recording human demonstrations, will explore and improve model for the ability to handle engineering issues in data sparsity, human diversity, and so on.
REFERENCES
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03077 2018.
Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, João Gomes, Supratik Paul, Frans A Oliehoek, João Messias, et al. Learning from demonstration in the wild. In 20/9 International Conference Robotics and Automation (ICRA), pp. 775-781. IEEE, 2019.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv: (16660041166 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Daniel S Brown, Wonjoon Goo, and Scott Nickum. Better-than-demmosttaato imitation learning via automatiiaall--ranked demonstrations. In Conference on Robot Learning, pp. 330-359, 2020.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In Twenty- interna- tional joint conference on artificial intelligence, 2015.
Andreas Bühler, Adrien Gaidon, Andrei Cramariue, Rares Ambrus, Guy Rosman, and Wolfram Burgard. Driving through ghosts: Behavioral cloning with false positives. arXiv preprint arXiv:2008. 12969, 2020.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 2094- 2100, 2016.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations, In Association for the Advancement of Artificial Intelligence, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565 4573, 2016.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR). 50(2):1-35, 2017.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: 1903.0090000000 2019.

---

Under review as a conference paper at ICLR 2021
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations, In International Conference on Machine Learning, pp.
2018.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: survey. arXiv preprint arXiv:2002.00444 2020.
Nathan Lambert, Daniel Drew, Joseph Yaconelli, Sergey Levine, Roberto Calandra, and Kristofer SJ Pister. Low-level control of quadrotor with deep model- reinforcement learn- ing. IEEE Robotics and Automation Letters, 4(4):4224-45) 4230, 2019.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint -XXxv:5509002977 2015.
Jan Matas, Stephen James, and Andrew J Davison. Sim-to-real reinforcement learning for de- formable object manipulation. In Conference Robot Learning, pp- 734-743, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference on Machine Learning, pp. 3878-3887, 2018.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. In Joint European Conference on Machine Learning and Knowledge Dis- covery in Databases, pp. 549-564. Springer, 2014a.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regulariveed classification for apprenticeship learning. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp. 1249-1256 International Foundation for Autonomous Agents and Multiagent Systems, 2014b.
Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3, 2020.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics. pp. 627-635, 2011.
Stefan Schaal. Learning from demonstration. In Advances neural information processing systems, pp. 1040-1046, 1997.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint
0.5952, 2015.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwesser, Ioannis Antonoglou, Veda Panneershelvam, and Marc Lanctot. Master- ing the game Go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, and Adrian Bolton. Mastering the game of Go without human knowledge. Nature, 350,, 2017.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. In Advances neural information processing systems, pp. 7461-7472 2018.
10

---

Under review as a conference paper at ICLR 2021
Halit Bener Suay. Tim Brys, Matthew E Taylor, and Sonia Chernova. Learning from demonstration for shaping through inverse reinforcement learning. In Proceedings of the 2016 International Conference Autonomous Agents & Multiagent Systems, pp. 429 437, 2016.
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and Andrew Bagnell. Deeply ag- grevated: Differentiable imitation learning for sequential prediction. In International Conference on Machine Learning, pp- 3309-3318, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph E Gon- zalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Safety augmented value estima- tion from demonstrations (saved): Safe deep model-basee rl for sparse cost robotic tasks. IEEE Robotics and Automation Letters, 5(2):3612-33))) 2020.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954. 2018.
Pedro A Tsividis, Thomas Pouncy, Jacqueline Xu, Joshua Tenenbaum, and Samuel J Gershman. Human learning in atari. 2017.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 2017.
Borui Wang, Ehsan Adeli, Hsu-kuang Chiu, De-An Huang, and Juan Carlos Niebles. Imitation learning for human pose prediction. In Proceedings of the IEEE International Conference Computer Vision, pp. 7124-7133, 2019.
Yuanhao Xiong, Guanjie Zheng, Kai Xu, and Zhenhui Li. Learning traffic signal control from demonstrations. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2289-2292. 2019.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldherg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality telcoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
A APPENDIX
Hyperparameters There are numerous hyperparameters in the experiments and different hyperpa- rameters have significant effect on the results. In all experiments, the in step return is 10, the batch size = 64, the buffer 100e=0000000, the learning ratc=0,00001, the learning rate of supervised model is 0.001, and the discount factor 7 0.95 in the CartPole environments (0.99 for other environments). We find that different random seeds have different performances and we choose [100, 200, 300, ..., 1000] as the random seed list. In the greedy policy, linearly de- creases from maximum value of 0.99 to minimum value of 0.001 and remains at the minimum value. In the CartPole-vl environment, the descent is maintained for 10000 interaction steps and in other environments 5000. In the prioritized experience replay buffers, new transitions are given the highest probability of being sampled and are later updated based on the absolute values of TD-1 enror.. The parameter 33 controls the amount of importance sampling and it is annealed linearly from 30 to In DQND and DQADD-BC. À decides how to combine different losses. In the experiments, A1 = A2 0.5, A3 0.001 and the L2 regularization is provided by the optimizer in the PyTorch. The large margin in DQID. is fine-tuned and set to be 1. A margin that is too large will cause the model rely heavily the supervised learning process and lose its ability to learn independently, while a margin that is too small will cause the advantages of supervised learning to be unused.
Model architecture Since the state of experiment environments are vectors, fully- fully-connected layers are appropriate for the requirements and they are leveraged to construct the Q network and the BC model. In the BC model, the hidden unit number of all three fully-connected layers are [64, 24, 12)
11

---

Under review as a conference paper at ICLR 2021
and Leaky Relu is selected the activation function. The Q network which has [312, 512, 256, 128) hidden units in every hidden layer and the activation function is Relu.
Termination condition In the experiments, the termination condition was following. Within 1000 episodes (it is 500 in CartPole-vO vO and v1), the average episode score of last 30 consecutive episodes greater than certain scores (CartPole-vO: 190, CartPole-vI: 490, Acrobot-vl: -100, Lunarl_ander v2: 200). The models were considered to be successfully converged if the termination conditions satisfied. Each method is repeated 10 times (it is in CartPole-vO and vl) with different random seeds.
12


Reviews:
[{"review_id": "tZdEeok0gxC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes integrating deep Q-learning from dynamic demonstrations with a behavioral cloning model (DQfDD-BC). Compared with DQfD, the proposed approach introduces a behavior cloning model, which was first pre-trained by leveraging historical demonstrations and then updated using generated dynamic demonstration. The BC model is used in the expert loss function, where the DRL model's actions are compared with those obtained from the BC model for policy improvement guidance. The experimental results in OpenAI Gym environments show that the proposed approach adapts well to different demonstrations' imperfection levels and accelerates the learning processes. The ablation study also indicates that the new method improves the learning convergence performance compared with the original DQfD model.\n\n========================\n\nReasons for score:\nOverall, based on the description of the paper, the proposed approach works well.\n\n========================\n\nDetailed comments:\n- In the proposed approach, the BC model is fine-tuned if the model achieves a relative high-performance score. With this mechanism,  high-quality transition samples are included, and it helps avoid adverse impacts caused by imperfect demonstrations.\n- The experiments show that the performance of DQfDD-BC is better than DQfQ, and it works well for imperfect demonstration scenarios.\n- In this paper, I think the primary idea is about how to leverage high-quality data in the generated trajectory well. This paper aims to build a BC model, initialized by demonstration dataset, and later finetuned with newly generated high quality data. One thing I do not understand well is the necessity of building the BC model. What if we use some heuristic approaches to replace the replay buffer with higher quality data? Can some simple strategies like this achieve similar performance as building a BC model?\n\n========================\n\nSome obvious typos:\n- Section 5 Experimental results: environmets -> environments \n-Section 5.3, paragraph 2: combins -> combines", "extra_scores": {}, "extra_text": {}}, {"review_id": "d275UPzeZcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=====POST-REBUTTAL COMMENTS======== \n\nI thank the authors for the response and the efforts in the updated draft. Most of my concerns were addressed. This is a simple, but nice idea. After reading the rebuttal and the other reviews I am recommending to accept the paper.\n\n##########################################################################\n\nSummary:\n \n\nThis paper seeks to improve deep reinforcement learning from demonstrations via a supervised learning loss based on behavioral cloning. The results show that adding the BC loss stabilizes training and shows improvements over baselines for OpenAI Gym environments. \n\n##########################################################################\n\nReasons for score: \n \n\nI think the idea is nice, but it is hard to follow as written and the experiments are on simple problems. It would be better to compare against more complex benchmarks such as Atari. The writing needs improvement for clarity.\n \n\n##########################################################################\nPros: \n \n\n1. The ablation study in Figure 4 shows the benefit of the proposed method.\n\n2. The ability to outperform a suboptimal demonstrator is important for imitation learning in the real world.\n \n\n\n\n##########################################################################\n\nCons: \n \n1. The idea of adding new trajectories from high scores was interesting, but I was left wondering why this works. Won't this add a lot of really bad trajectories in the beginning of learning which seem like they would have a negative impact on the BC policy. Also, the function \"get the best episode score\" was never rigorously defined.\n\n2. Equations should be introduced before they are included. Equations 1-4 come before the text that talks about them and this makes the paper more confusing that it needs to be.\n\n3. Equation (5) seems incomplete. Shouldn't l(\\pi, \\pi_bc, s_t) be a function of the action? Otherwise it is just a constant per state, but it is in the max_a brackets. Also it is unclear what the difference between s_t and s are in Eqn (5).\n\n4. Average stop episode is not clearly defined in the text. In the appendix it explains more but the thresholds are not justified.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U2thcg5eBr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is introducing a learning method which combines both Imitation Learning and Reinforcement Learning, such that an autonomous learner can leverage prerecorded expert knowledge. In comparison to previous work, this model has an expert cost function which gives priority to the expert behavior, not only using the expert demos (like in DQfD), but also with a model trained with those demos using  behavioral cloning, such that it could be evaluated in states that were not visited during the demonstrations. Additionally, new executions of the learner that have high performance are included in the buffer for training the policy imitating the expert, since they can also be considered new better demonstrations.\n\nThe paper presents an interesting idea with potential, although there are some aspects to consider mainly about the presentation of the paper, in order to improve its content, as listed below. However, the aspects that are not clear in the current version of the paper need to be fixed before considering acceptance, since there are clear open questions.\n\nSection 2 Related work needs some improvement:\n- First sentence of Section 2 could be improved for it to be more meaningful \"Learning from demonstration (LfD) is a class of decision-making methods by learning from demonstration and  imitation  learning is a subset of LfD\", additionally LfD are not exactly decision-making methods.\n- It is stated that BC received extensive attention \"due to its great performance\", I'd say rather \"due to its simplicity\", since it is simply the application of supervised learning, there is extensive literature arguing that it does not always perform well, therefore several other methods have been proposed.\n- \"Behavioral cloning is an end-to-end learning method\" is not a right statement.\n- Authors mention about DAgger that \"... address the shortcoming of limited state-action space covered by demonstration in traditional imitation learning\". This is an indirect result, but actually it is intended to cope with the covariate shift problem.\n- in \"...adversarial learning method and has yielded impressive results...\", impressive could be replaced by an objective word.\n\nRegarding section 4:- In \"As the transition quality of historical demonstrations is usually much higher than that associated with stochastic policies...\" stochastic policies are not necessarily something meaningless, I guess the authors meant a completely random policy.\n- In the line 20 of Algorithm 1. how the \"get the best episode score\" is computed? it is just taken from the final cumulative reward of each episode compared to the previous history? if that's the case how to deal with lucky episodes in which the conditions are simply easier for the agent, and not necessarily that the current policy is indeed better.\n- It's not so clear why the self learning process is executed, my interpretation is that authors use it to update the BC policy towards \\pi(s), such that suboptimal behaviors in the demonstrations don't keep pulling the learning policy.\n- lines 8 and 18 have different notations for the same operation,  authors could use only one of them to be more consistent.\nRegarding the experiments section- There are missing details of the experiments that do not allow for reproducing the procedure, how many repetitions were run for each case in Fig 2 and 3?, and also for the ablation study? It is not clear whether the results of Fig 2 and 3 are obtained from the same experiments that gave the results in Table 1, at least they do not seem to match. Why testing with the 2 variants of the CartPole environment instead of another kind of problem? since the difference between them is very simple\n\n- Why is there no data in Table 1 for the cases of imperfect demonstrations for the CartPole-V0 environment?\n- Why the learning curves of DDQN for the CartPole-V0 and CartPole-V1 cases are not similar at least during the first 200 episodes, looking at the mean and variance, both environments are not close to reach the maximum time steps, therefore in that region they should behave exactly the same.\n- In Table 1, it is not so clear what average stop episode means, if it is the episode number in which the stop condition was reached, why are there negative numbers for the acrobot? if they are rewards obtained in the last episode, why are rewards higher than the possible to obtain in CartPole-V0.\n- It is mentioned  \"...but the DDQN and DQfD fail to meet the experimental termination condition with imperfect demonstration\", However this is completely wrong, DDQN do not use demonstrations.\n\n- It would be interesting to additionally mention how to define the size of the margin for the expert cost function, because this may depend on the kind of reward functions/range of value functions.\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ElwfW-0Maw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a new approach (DQfDD-BC) to leverage demonstrations in the framework of deep Q-learning. The new approach augments the previous approach called DQfD by \na) adding self-generated trajectories with high rewards as additional demonstrations (a feature the authors called “dynamic demonstrations”), \nb) replacing large-margin loss with a cross-entropy loss between the learner agent and a helper policy learned by behavior cloning demonstrations. \nThe new approach improves upon the previous by better leveraging demonstrations and being more robust against imperfect demonstrations. The authors demonstrated these improvements in experiments by showing the new approach leads to accelerated training speed (in terms of episodes) when using either perfect or imperfect demonstrations, and higher performance (in terms of rewards) when using imperfect demonstrations. \n\n==============\n\nPositives\n\nThis paper is well motivated. The two intended benefits of the new approach are meaningful and important.\n\nThe experiments back up the authors’ claim on the benefits, especially on the improvement of training speed. In the CartPole tasks, the training speed is improved by 2 folds.\n\nThe paper is written clearly. It was easy to understand the new approach and the differences on top of the previous one. \n\n==============\n\nNegatives\n\nThe extensiveness of experiments is underwhelming, both in the diversity and complexity of the tasks chosen. The four tasks are among the most simple ones typically used for comparable algorithms, having very low degrees of freedom and very simple dynamics. For example, DQfD was validated on a broader and more challenging set of Atari tasks, including Montezuma Revenge where (human) demonstrations are critical for a learner agent to tackle the task.\n\nBy introducing a BC-learned helper policy, the approach introduces dependencies on how well this helper policy is (in terms of approximating the demonstrations) in order to provide better loss signals to large-margin loss. It’d be good to design additional experiments to showcase shortcomings, if any, of this proposed approach. For example, the authors used ten thousands transitions as initial demonstrations. It’d be interesting to see how the proposed approach would fare when different amounts of demonstrations are available.\n\n==============\n\nRecommendation\n\nOverall the approach proposed is interesting and promising. I’d recommend a weak accept. The authors are strongly encouraged to further improve the experiments. \n\n==============\n\nMinor Comments\n\nTable 1 is confusing to read. Especially in the top half, “Demo” rewards are juxtaposed with other numbers in a completely different units (number of episodes).\n\nThere are some typos. In section 2, it’s not clear what determination is meant in “determination of DAGGER”. In Section 1 “difficult consistent” -> “difficult to be consistent”. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewer acknowledged that the proposed method is simple and seems to work well on the chosen benchmarks. Yet the expressed several concerns that were not fully addressed by the authors in their responses. The major concern is about the experimental setup. The chosen tasks have been judged too simple and quite different from those where the baselines were tested initially (e.g. DQfD was demonstrated on a diverse set of Atari games). 

The clarity of the paper should also be improved. For instance, the way the number of trajectories that are added to the replay buffer increases with time is not well explained and it seems to be crucial for the algorithm to outperform the baselines. The authors also seemed to select the experiments so that the "results are more persuasive", discarding experiments where you can be unlucky. This looks very much like cherry-picking and didn't convince the reviewers. 
Decision: Reject