Title: REINFORCEMENT LEARNING FROM IMPERFECT DEMONSTRATIONS

Abstract: Robust real-world learning should benefit from both demonstrations and interaction with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on reward from the environment. These tasks have divergent losses which are difficult to jointly optimize; further, such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstration and refines the policy in a real environment. Crucially, both learning from demonstration and interactive refinement use exactly the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data, since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.

1. INTRODUCTION:

Deep reinforcement learning (RL) has achieved significant success on many complex sequential decision-making problems. However, RL algorithms usually require a large amount of interaction with an environment to reach good performance (Kakade et al., 2003) ; initial performance may be nearly random, clearly suboptimal, and often rather dangerous in real-world settings such as autonomous driving. Learning from demonstration is a well-known alternative, but typically does not leverage reward, and presumes relatively small-scale noise-free demonstrations. We develop a new robust algorithm for Reinforcement-Learning-from-Demonstrations (RLfD) that can learn value and policy functions from imperfect demonstration data as well as environmental reward signals.

Recent efforts toward policy learning which does not suffer from a suboptimal initial performance generally perform RLfD by leveraging an initial phase of supervised learning and/or auxiliary task learning. Several authors have shown that demonstrations can speed RL by mimicking expert data with a temporal difference regularizer (Hester et al., 2017) or via gradient-free optimization (Ebrahimi et al., 2017) , yet these methods presume near-optimal demonstrations. Shelhamer et al. (2016) and Jaderberg et al. (2016) obtained better initialization via auxiliary task losses (e.g., predicting environment dynamics) in a self-supervised manner; performance is still initially random with these approaches.

Our approach, Normalized Actor-Critic (NAC), is based on the maximum entropy reinforcement learning framework (Toussaint, 2009; Haarnoja et al., 2017b; Schulman et al., 2017) . Our method performs RLfD using exactly the same algorithm to process both offline demonstration data and online experience, without any additional supervised loss function. This enables robust learning even from very corrupted (or even partially adversarial) demonstrations that contains (s, a, r, s ), since no assumption on the optimality of the data is required. This is enabled by a normalized formulation of the soft Q-learning gradient, which can also be regarded as a variant of the policy gradient.

We evaluate our result in realistic 3D simulated environments, Torcs and Grand Theft Auto V (GTA V), using access only to raw image inputs and game-generated rewards. Experimental results con-firm that our approach outperforms previous approaches on driving tasks with only a modest amount of demonstration and that significant noise in the demonstrations are well-tolerated, as our method utilizes rewards rather than simply imitating demonstrated behaviors.

2. PRELIMINARIES:

In this section, we will briefly review the reinforcement learning techniques that our method builds on, including maximum entropy reinforcement learning and the soft Q-learning.

2.1. MAXIMUM ENTROPY REINFORCEMENT LEARNING:

The reinforcement learning problem we consider is defined by a Markov decision process(MDP) (Thie, 1983) . Specifically, the MDP is characterized by a tuple < S,A,R,T,γ >, where S is the set of states, A is the set of actions, R(s, a) is the reward function, T (s, a, s ) = P (s |s, a) is the transition function and γ is the reward discount factor. The agent interacts with the environment by taking action at some state, receiving the reward, and transiting to the next state.

In the conventional reinforcement learning setting (Sutton & Barto, 1998) , the goal of an agent is to learn a policy π std , such that it maximizes the future discounted reward:

EQUATION

Maximum entropy policy learning (Ziebart, 2010; Haarnoja et al., 2017b) uses an entropy augmented reward, so that the optimal policy will not only optimize for discounted future rewards, but also maximize the discounted future entropy of the action distribution:

EQUATION

where α is a weighting term to balance the importance of the entropy. Unlike previous attempts of only adding the entropy term at a single time step, maximum entropy policy learning maximizes the discounted future entropy over the whole trajectory. Maximum entropy reinforcement learning has many benefits, such as better exploration in multi-modal problems, and establishing connections between Q learning and the actor-critic method (Haarnoja et al., 2017b; Schulman et al., 2017) .

2.2. SOFT VALUE FUNCTIONS:

Since the maximum entropy RL paradigm augments the reward with an entropy term, the definition of the value functions naturally changes to

EQUATION

where π is some policy that value functions evaluate on. Given the state-action value function Q * (s, a) of the optimal policy, Ziebart (2010) show that the optimal state value function and the optimal policy could be expressed as:

V * (s) = α log a exp(Q * (s, a)/α) (5) π * (a|s) = exp((Q * (s, a) -V * (s))/α) (6) 2.3 SOFT Q-LEARNING AND POLICY GRADIENT

With the entropy augmented reward, one can derive the soft versions of Q learning and policy gradient (Haarnoja et al., 2017a) . The soft Q learning gradient is given by

EQUATION

where Q(s, a) is a bootstrapped Q-value estimate obtained by R(s, a) + γV Q (s ). Here, R(s, a) is the reward received from the environment, V Q is computed from Q θ (s, a) with Equation (5).

We can also derive a policy gradient variant of this method, by parameterizing as the policy as log π θ (a|s) = Q θ (s, a) -V θ (s), which induces a policy gradient of the form

EQUATION

where b(s t ) is some arbitrary baseline (Schulman et al., 2017) .

3. ROBUST LEARNING FROM DEMONSTRATION AND REWARD:

Given a set of demonstrations that contains (s, a, r, s ) and the corresponding environment, an agent should perform appropriate actions when it starts the interaction, and continue to improve. Although a number of off-policy RL algorithms could in principle be used to learn directly from off-policy demonstration data, standard methods such as Q-learning can suffer from extremely poor performance when trained entirely on demonstration data, as shown in our experiments. The intuition behind this problem is that, if the Q-function is trained only on good data, it has no way to understand why the action in the data is good: it will assign it a high Q-value, but will not necessarily assign a low Q-value to other alternative actions. The framework of soft optimality provides us with a very natural mechanism to mitigate this problem by normalizing the Q-function over the actions.

Our approach, Normalized Actor-Critic (NAC), combines the soft Q-learning and soft policy gradient formulations described in the previous section to obtain a Q-function gradient that naturally reduces the Q-values of actions that were not observed along the demonstration. Put another way, without data to indicate otherwise, NAC will opt to follow the demonstration. This method is derived from the policy gradient in Equation ( 8), which introduces a normalization term ∇ θ V (s) that reduces the value of unobserved actions. Since this method is still a well-defined RL algorithm without any auxiliary supervised loss, it is able to learn without bias in the face of low-quality demonstration data. We will first describe our algorithm, and then further discuss why it performs well when trained on off-policy demonstration data.

3.1. NORMALIZED ACTOR-CRITIC:

We propose a unified learning from demonstration approach, which applies the normalized actorcritic updates to both off policy demonstrations and in-environment transitions. To better facilitate the comparison with the Q learning method, we re-parametrize the normalized actor-critic with Q function. As shown in the appendix A.1, we have the updates for the actor and the critic being:

EQUATION

where

EQUATION

As discussed by Schulman et al. (2017) ; Haarnoja et al. (2017b) , the method can be interpreted as either a Q-learning algorithm or a policy gradient method. The two interpretations lead to slightly different variants of the update equations. The soft Q-learning variant omits the ∇ θ V (s) in Equation (( 9)). In our normalized actor critic method, we emphasize the normalization effect from the ∇ θ V (s) term; this term enables our method to learn from off-policy data. Our method also requires importance sampling to use off-policy data. As discussed in Section 3.2, our method critically relies on the ∇ θ V (s) term to attain good performance on demonstration data, which means that, in order to formalize it as a policy gradient method, we should include importance weights. To be specific,

when estimating E (s,a)∼π Q [f (s, a)], we estimate E (s,a)∼µ [f (s, a)β] , where β = min π Q (a|s)

µ(a|s) , c and c is some constant that prevents the importance ratio from being too large.

Although the importance weights are needed to formalize our method as a proper policy gradient algorithm, we find in our empirical evaluation that the inclusion of these weights consistently reduces the performance of our method even on demonstrations. For this reason, our final algorithm does not use importance sampling.

We summarize the proposed method in Algorithm 1. Our method uses samples from the demonstrations and the replay buffer, rather than restricting the samples to be on policy as in standard actor-critic methods. Similar to DQN, we utilize a target network to compute Q(s, a) and V (s), which stabilizes the training process.

Algorithm 1 The Normalized Actor-Critic Algorithm θ: parameters for the rapid Q network, θ : parameters for the target Q network, D: demonstrations collected by human or a trained policy network, T : target network update frequency, M: replay buffer, k: number of steps to train on the demonstrations for step t ∈ {1, 2, ...} do if t ≤ k then Sample a mini-batch of transitions from D else

Start from s, sample a from π, execute a, observe (s ,r) and store (s, a, r, s

) in M Sample a mini-batch of transitions from M end if Update θ with gradient: ∇ θ J P G + ∇ θ J V if t mod T = 0 then θ ← θ end if end for

3.2. ANALYSIS OF THE METHOD:

We provide an intuitive analysis in this section to reason why our method can learn from demonstrations while other reinforcement learning methods, such as Q-learning, can not. The states and actions in the demonstrations generally have higher Q-values than other states. Q-learning will therefore push up Q(s, a) values in a sampled state s. However, if the values for the bad actions are not observed, then the Q-function has no way of knowing whether the action itself is good, or whether all actions in that state are good, so the demonstration action will not necessarily have a higher Q-value than other actions in the demonstration state. Comparing the actor update (Eq. ( 9)) of our method with the soft Q-learning (Eq. 7) update, our method includes an extra term in the gradient: -∇ θ V Q (s). This term falls out naturally when we consider the method as a policy gradient algorithm, rather than a Q-learning algorithm (Haarnoja et al., 2017b) . Intuitively, this term will decrease V Q (s) when increasing Q(s, a) and vice versa, since

∇ θ Q(s, a) and -∇ θ V Q (s) have different signs. Since V Q (s) = α log a exp(Q(s, a)/α), decreasing V Q (s) will prevent Q(s, a)

from increasing for the actions that are not in the demonstrations.

4. RELATED WORK:

Recently, deep reinforcement learning has achieved great success on many tasks. Notable examples include deep Q-learning (Mnih et al., 2015) , deep visuomotor policy learning (Levine et al., 2016) and trust region policy optimization (Schulman et al., 2015) . Other recent successes include expert action prediction in Go (Silver et al., 2016) and intelligent agents in Starcraft II and vizDoom (Vinyals et al., 2017; Kempka et al., 2016) . In this section, we review the related work in deep RL as well as imitation learning.

4.1. LEARNING FROM DEMONSTRATION:

Imitation learning can learn meaningful behaviors from expert demonstrations. For example, Xu et al. (2016) ; Bojarski et al. (2016) learn driving policies from human demonstrations. One popular paradigm of imitation learning is DAGGER (Ross et al., 2011) , which improves an agent's performance with the expert's feedback in the loop. Recently, Ho & Ermon (2016) ; Wang et al. (2017) ; Ziebart et al. (2008) utilize an adversarial paradigm to further boost the performance of the behavior cloning method. Another popular paradigm is Inverse Reinforcement Learning (IRL) (Ng et al., 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) . IRL learns a reward model which explains the demonstrations as optimal behaviors. The reward function can be represented as the distance to a demonstrated trajectory, as a weighted combination of features.

Recently, Subramanian et al. (2016) showed the exploration strategy in an environment can be guided by human demonstrations. Taylor et al. ( 2011) directly transfer knowledge from expert demonstrations to the learning agent. Guided policy search can be seen as a form of imitation learning, where the expert is the optimal control policy (Levine & Koltun, 2013) . To have a smoother transition from imitation learning to reinforcement learning, Hester et al. (2017) propose to use an objective that combines a supervised loss and a temporal difference loss. Both Hester et al. (2017) and our algorithm follow the Reinforcement Learning with Expert Demonstrations (RLED) framework (Chemali & Lazaric, 2015; Kim et al., 2013; Piot et al., 2014) , where both rewards and actions are available in the demonstration. We follow the terminology from Hester et al. (2017) and use Learning from Demonstration to describe our task but rewards are available from the demonstrations in our setting. Different from previous works, our method does not require an explicit mechanism to determine expert data because it learns to maximize the reward using arbitrary data.

4.2. OFF-POLICY LEARNING:

Reinforcement learning methods learn a policy while interacting with the environment. Some methods require the data being used to update the current policy π t to be roll-outs of π t in the environment, while other methods allow the updates to be computed from data generated by arbitrary policy. Those methods are classified as on-policy methods and off-policy methods respectively. The most well known off-policy method is Q-learning (Watkins & Dayan, 1992) . Policy gradient (Sutton et al., 2000) methods require on-policy data, but recently some works have proposed variants of policy gradient that can learn from off-policy data (Gu et al., 2017; 2016) . Off-PAC (Degris et al., 2012) is the first off-policy actor-critic method. Munos et al. (2016) proposed Retrace(λ), a method that allows multiple step return to being used with off-policy updates. ACER (Wang et al., 2016) combines the advances of Off-PAC, Retrace(λ), trust region update and leads to a stable and sample efficient actor-critic method. However, some methods, such as Retrace, require the off-policy data to be close to the current policy. The other category of off-policy methods consists of Q-learning methods, which we care to in our evaluation and find that they generally do not perform well on purely off-policy data.

Maximum entropy reinforcement learning has been explored in a number of prior works Todorov (2008) ; Toussaint (2009) ; Ziebart et al. (2008) , including several recent works that extend it into a deep reinforcement learning setting Nachum et al. (2017) ; Haarnoja et al. (2017b) . Our work builds on the connection between policy gradient and soft Q-learning discussed by Haarnoja et al. (Haarnoja et al., 2017b) and Schulman et al. (Schulman et al., 2017) . However, unlike these prior works, we are specifically concerned with learning from demonstrations, and we show how a novel algorithm that combines elements of policy gradient and Q-learning is especially well suited for learning from demonstrations, while still optimizing the entropy-augmented maximum reward objective, without any explicit imitation loss.

5. RESULTS:

We address several questions with our experiments: (1) Can NAC benefit from both demonstrations and rewards? (2) Is NAC robust to ill-behaved demonstrations? (3) Can NAC learn meaningful behaviors with a limited amount of demonstration? We compare our algorithm with DQfD Figure 1 : Sample frames from Torcs (left) and GTA (right). (Hester et al., 2017) , which has been shown to learn efficiently from demonstrations and to preserve performance while acting in an environment. Other baselines include supervised behavioral cloning method, Q-learning, soft Q-learning, and the version of our method with importance sampling weighting.

5.1. ENVIRONMENTS:

We evaluate our algorithm on Torcs and Grand Theft Auto V (GTA) shown in Fig 1 .

Torcs::

Torcs is an open-source racing game that has been used widely as an experimental environment for driving. The goal of the agent is to drive as fast as possible on the track, while avoiding crashes. We use an oval two-lane racing venue in our experiments. The input to the agent is an 84×84 gray scale image. The agent controls the vehicle at 5Hz, and at each step, it chooses from a set of 9 actions which is a Cartesian product between {left, no-op, right} and {up, no-op, down}. We design a dense driving reward function that encourages the car to follow the lane and to avoid collision with obstacles.1 GTA: Grand Theft Auto is an action-adventure video game with goals similar in part to the Torcs game, but with a more diverse and realistic surrounding environment, including other vehicles, buildings, and bridges. The agent observes 84×84 RGB images from the environment. It chooses from 7 possible actions from {left-up, up, right-up, left, no-op, right, down} at 6Hz. We use the same reward function as in Torcs.

5.2. BASELINES:

We compare our approach with the following baselines:

• DQfD: the method proposed by Hester et al. (2017) . For the learning-from-demonstration phase, DQfD combines a hinge loss with a temporal difference (TD) loss. For the finetuningin-environment phase, DQfD combines a hinge loss on demonstration and a TD loss on both the demonstration and the policy-generated data. • Q learning: the classic DQN method (Mnih et al., 2015) . We first train DQN with the demonstrations in a replay buffer and then finetune in the environment with regular Q learning. Similar to DQfD, we use a constant exploration ratio of 0.01 in the finetuning phase to preserve the performance obtained from the demonstration. • Soft Q learning: similar to the Q learning baseline, but with an entropy regularized reward. This is the method proposed by Haarnoja et al. (2017a) ; Schulman et al. (2017) . • Behavior cloning with Q learning: the naive way of combining cross entropy loss with Q learning. First we perform behavior cloning with cross entropy loss on the demonstrations. Then we treat the logit activations prior the softmax layer as an initialization of the Q function and finetune with regular Q learning in the environment. • Normalized actor-critic with importance sampling: the NAC method with the importance sampling weighting term mentioned in Sec 3.1. The importance weighting term is used to correct the action distribution mismatch between the demonstration and the current policy. • Path Consistency Learning (PCL): the PCL method that minimizes soft path consistency losses extracted from action sequences from both off-policy data and on-policy data. Different from original implementation Nachum et al. (2017) , we only consider one step ahead the path.

5.3. COMPARISON TO BASELINES:

We first compare our NAC method with all baselines on 300k transitions. The demonstration is collected by a trained Q-learning expert policy. We execute the policy in the environment to collect demonstrations. To avoid deterministic executions of the expert policy, we uniformly sample an action with probability 0.01.

To explicitly compare different methods, we show separate figures for performances on demonstrations and in environments. In Fig 2 , we show that our method performs better than other methods on demonstrations. When we start finetuning, the performance of our method continues to increase and reaches peak performance faster than the baselines. In comparison, Q learning, PCL and soft Q learning can barely learn from demonstrations. DQfD (Hester et al., 2017) has similar behavior to ours but has lower performance. Behavior cloning learns well on demonstrations, but it has a significant performance drop while interacting with environments. All the methods can ultimately learn by interacting with the environment but only our method and DQfD start from a relatively high performance. Empirically, we found that the importance weighted NAC method does not perform as well as NAC. The reason might be the decrease in the gradient bias is not offset sufficiently by the increase in the gradient variance.

We also test our method on the challenging GTA environment, where both the visual input and the game logic are more complex. Due to the limit of environment execution speed, we only compare our method with DQfD. As shown in Fig. 3a , our method outperforms DQfD both on demonstration and in environment.

5.4. LEARNING FROM HUMAN DEMONSTRATIONS:

For many practical problems, such as autonomous driving, we might have a large number of human demonstrations, but no demonstration available from a trained agent at all. In contrast to a scripted agent, humans usually act diversely, both in terms of multiple individuals (e.g. conservative players will slow down before a U-turn; aggressive players will not) and a single individual (e.g. a player may randomly turn or go straight at an intersection). We study how different methods perform with diverse demonstrations. To collect human demonstrations, we asked 3 non-expert human players to play TORCS for 3 hours each. Human players control the game with the combination of four arrow keys, at 5Hz, the same rate as the trained agent. The experiment is run for 800k training steps, as indicated from the experiments in Sec 5.3. In total, we collected around 150k transitions.

Comparing with data collected from trained agent, the data is more diverse and the performance of the demonstration improves naturally when the players get familiar with the game. In Fig. 3b , we show the comparison among our NAC method, DQfD, and behavior cloning; we omit Q learning, soft Q learning, and the importance weighted NAC are shown to perform poorly in the demonstration Figure 4 : Left: Learning from imperfect data when the imperfectness is 30%. Our NAC method does not clone suboptimal behaviors and thus outperforms DQfD and behavior cloning. Right: Learning from a limit amount of demonstrations. Even with only 30 minutes (10k transitions) of experience, our method could still learn a policy that is comparable with supervise learning method.

More results are available in appendix Fig. 5 and Fig. 6 .

phase. We observe that the behavior cloning method performs much worse than the other two methods. DQfD initially is better than our method but later overfits quickly, which might be caused by the supervised hinge loss being harmful when demonstrations are not optimal. NAC does not assume either perfect or consistent demonstrations, and performs better than DQfD and behavior cloning at convergence.

5.5. EFFECTS OF IMPERFECT DEMONSTRATIONS:

In the real world, collected demonstrations might be far from optimal. The human demonstrations above have already shown that imperfect demonstrations could have a large effect on performance.

To study this phenomenon in a principled manner, we collect a few versions of demonstrations with varying degrees of noise. When collecting the demonstrations with the trained Q agent, we corrupt a certain percentage of the demonstrations by choosing actions regarding (min a Q(s, a)). The data corruption process is conducted while interacting with the environment; therefore, the error accumulated after the corrupted frames will also influence the quality of the dataset in a similar way as human demonstrations. We get 3 sets of {30%, 50% and 80%} percentage of imperfect data. In the left of Fig. 4 , we show that our method performs quite well compared with DQfD and behavior cloning baselines. Supervised behavior cloning method is heavily influenced by the imperfect demonstrations. DQfD is also heavily affected, but not as severely as the behavior cloning.

NAC is robust because it does not imitate the suboptimal behaviors.

5.6. EFFECTS OF DEMONSTRATION SIZE:

In this section, we show comparisons among our method and baseline methods with different amounts of demonstration data. We use a trained agent to collect three sets of demonstrations which include 10k, 150k, and 300k transitions each. In the experiments we find that our algorithm per-forms well when the amount of data is large and is comparable to supervised methods even with a limited amount of data. In Fig. 4 (right), we show when there are extremely limited amounts of demonstration data (10k transitions or 30 minutes of experience), our method performs on par with supervised methods. In Fig. 6 , we show the results for 150k and 300k transitions: our method outperforms the baselines by a large margin with 300k transitions. In summary, our method can learn from small amounts of demonstration data and dominates in terms of performance when there is a sufficient amount of data.

5.7. INTUITIVE ANALYSIS:

In this section, we analyze the reason why other reinforcement learning methods can hardly learn from demonstrations while our method can. We use a concrete example to clarify and complement the mathematical discussion in Section 3.2. Suppose at state s i , we observe the demonstrated behavior choosing action a 1 , and receiving a positive reward of 1.0. The action space is a 0 ,a 1 , a 2 for each state. Soft Q learning method takes an update of ∇Q(s i , a 1 )(Q(s i , a 1 ) -Q(s i , a 1 )) for the parameter θ of the Q function, where Q(s 0 , a 1 ) = 1.0 + γ * V (s i ) and s i is the next state after action a 1 . This update pushes the value of Q(s i , a 1 ) close to 1.0 + γV (s 0 ). However, at the state s i , we did not observe other actions, as it is usually the case in real world. Hence, there is no regressing target for Q(s i , a 0 ) and Q(s i , a 2 ). When we parametrize the Q function with a neural network, there is no guarantee of how Q(s i , a 0 ) and Q(s i , a 2 ) will compare to Q(s i , a 1 ). As a result, the learning algorithm cannot give meaningful Q values for the unobserved actions. The results in Figure 2 (left) has confirmed our analysis. A similar analysis applies to the hard Q method as well.

However, in the NAC method, the policy gradient update part is (∇Q(s i , a 1 )-∇V (s i ))(Q(s i , a 1 )-Q(s i , a 1 )), where V (s i ) = log a exp(Q(s i , a)). After manipulating the terms, the update is equal

EQUATION

)). Note that the first term becomes the gradient of a cross entropy loss between a softmax layer output and a synthetic label of action a 1 . The logits are Q(s i , •). Intuitively, if the Q(s i , a 1 ) is under-estimated, the method tries to increase Q(s i , a 1 ) while pushing down Q(s i , a 0 ) and Q(s i , a 2 ). When Q(s i , a 1 ) is over-estimated, the reverse process happens. It is worth to note that the proposed method naturally has a prior of reducing the unobserved Q(s i , a 0 ) and Q(s i , a 2 ) values, when the current action a 1 is good.

6. CONCLUSION:

We proposed a Normalized Actor-Critic algorithm for reinforcement learning from demonstrations. Our algorithm provides a unified approach for learning from reward and demonstration, and is robust to potentially suboptimal demonstration data. An agent can be fine-tuned with rewards after training on demonstrations by simply continuing to perform the same algorithm on on-policy data. Our algorithm preserves and then improves the behaviors learned from demonstration while receiving reward through interaction with an environment. Based on our experiments with the Torcs and GTA environments, we show that our algorithm is able to learn effectively and is robust to the arbitrary quality of demonstrations.

Appendices:



A APPENDIX:

A.1 NORMALIZED ACTOR-CRITIC WITH Q PARAMETRIZATION Usually the actor-critic method parametrizes π(s, a) and V (s) with a neural network that has two heads. In this section, we explore an alternative parametrization: Q-parametrization. Instead of outputting π and V directly, the neural network computes Q(s, a). We parametrize π and V based on Q by specifying a fixed mathematical transform: 

V Q (s) = α log

Note that the Q-parametrization we propose here can be seen as a specific design of the network architecture. Instead of allowing the net to output arbitrary π(s, a) and V (s) values, we restrict the network to only output π(s, a) and V (s) pairs that satisfy the above relationship. This extra restriction will not harm the network's ability to learn since the above relationship has to be satisfied at the optimal solution (Schulman et al., 2017; Haarnoja et al., 2017b; Nachum et al., 2017) .

Based on the Q-parametrization, we can derive the update of the actor. Note that we assume the behavioral policy is π Q , and we sample one step out of a trajectory, thus dropping the subscript t.

The goal is to maximize expected future reward, thus taking gradient of it we get: 

As in previous work, an entropy-regularized policy gradient simply add the gradient of the entropy of the current policy with a tunable parameter α in order to encourage exploration. The entropy term is:

EQUATION

putting the two terms together and using the energy-based policy formulation (Eq. ( 14)) : 

If we let the baseline b(s) be V Q (s), we get the update:

EQUATION

where Q(s, a) could be obtained through bootstrapping by R(s, a) + γV Q (s ). In practice Q(s, a) is a target Q value and V Q (s ) can be obtained from Eq. 13 and the parameters of the target network.

For the critic, the update is:

EQUATION

where V (s) could be obtained by V (s) = E a [R(s, a) + γV Q (s )] + αH(π Q (•|s)).

:

reward = (1 -1 damage ) [(cos θ -sin θ -lane ratio) × speed] + 1 damage[-10], where 1 damage is an indicator function of whether the vehicle is damaged at the current state. lane ratio is the ratio between distance to lane center and lane width. θ is the angle between the vehicle heading direction and the road direction.

Figures and Tables:

Figure 2: Performances on the Torcs game. The x-axis shows the training iterations. The y-axis shows the average total rewards. Solid lines are average values over 10 random seeds. Shaded regions correspond to one standard deviation. The left figure shows the performance for each agent when they only learn from demonstrations, while the right one shows the performance for each agent when they interact with the environments after learning from demonstrations. Our method consistently outperforms other methods in both cases.

(a) The on-demonstration and in-environment performance of the NAC and DQfD methods on GTA. The vertical line separates the learning from demonstration phase and finetuning in environment phase. Our method consistently outperforms DQfD in both phases.(b) Performances on the Torcs game with human demonstrations. DQfD performs well in the beginning, but overfits in the end. The behavior cloning method is much worse than NAC and DQfD. Our NAC method performs best at convergence.

Figure 3: Performance on GTA (left) and performance on Torcs with human demonstrations (right)

π Q (a|s) = exp((Q(s, a) -V Q (s))/α)

∇E s,a˜π Q [R(s, a)] =E s,a˜π Q [R(s, a)∇ log θ p(a, s|π Q )] ≈E s,a˜π Q [R(s, a)∇ θ log π Q (a|s)] (15)where the last step ignores the state distribution, thus an approximation. By adding some baseline functions, it turns to the following format, where Q(s, a) = R(s, a) + γV Q (s ):E s,a ∇ θ log π Q (a|s)( Q(s, a) -b(s)) =E s a π Q (a|s)∇ θ log π Q (a|s)( Q(s, a) -b(s))

s,a ∇ θ log π Q (a|s)( Q(s, a) -b(s)) + α∇ θ H(π Q (•|s)) =E s a π Q (s, a)∇ θ log π Q (s, a)( Q(s, a) -b(s) -(Q(s, a) -V Q (s)))

Peer review (iclr2018):
 SUMMARY: The motivation for this work is to have an RL algorithm that can use imperfect demonstrations to accelerate learning. The paper proposes an actor-critic algorithm, called Normalized Actor-Critic (NAC), based on the entropy-regularized formulation of RL, which is defined by adding the entropy of the policy as an additional term in the reward function. Entropy-regularized formulation leads to nice relationships between the value function and the policy, and has been explored recently by many, including [Ziebart, 2010], [Schulman, 2017], [Nachum, 2017], and [Haarnoja, 2017]. The paper benefits from such a relationship and derives an actor-critic algorithm. Specifically, the paper only parametrizes the Q function, and computes the policy gradient using the relation between the policy and Q function (Appendix A.1). Through a set of experiments, the paper shows the effectiveness of the method. EVALUATION: I think exploring and understanding entropy-regularized RL algorithm is important. It is also important to be able to benefit from off-policy data. I also find the empirical results encouraging. But I have some concerns about this paper: - The derivations of the paper are unclear. - The relation with other recent work in entropy-regularized RL should be expanded. - The work is less about benefiting from demonstration data and more about using off-policy data. - The algorithm that performs well is not the one that was actually derived. * Unclear derivations: The derivations of Appendix A.1 is unclear. It makes it difficult to verify the derivations. To begin with, what is the loss function of which (9) and (10) are its gradients? To be more specific, the choices of .hat{Q} in (15) and .hat{V} in (19) are not clear. For example, just after (18) it is said that “.hat{Q} could be obtained through bootstrapping by R + gamma V_Q”. But if it is the case, shouldn’t we have a gradient of Q in (15) too? (or show that it can be ignored?) It appears that .hat{Q} and .hat{V} are parameterized independently from Q (which is a function of theta). Later in the paper they are estimated using a target network, but this is not specified in the derivations. The main problem boils down to the fact that the paper does not start from a loss function and compute all the gradients in a systematic way. Instead it starts from gradient terms, each of which seems to be from different papers, and then simplifies them. For example, the policy gradient in (8), which is further decomposed in Appendix A.1 as (15) and (16) and simplified, appears to be Eq. (50) of [Schulman et al., 2017] (https://arxiv.org/abs/1704.06440). In that paper we have Q_pi instead of .hat{Q} though. I suggest that the authors start from a loss function and clearly derive all necessary steps. * Unclear relation with other papers: What part of the derivations of this work are novel? Currently the novelty is not obvious. For example, having the gradient of both Q and V, as in (9), has been stated by [Haarnoja et al., 2017] (very similar formulation is developed in Appendix B of https://arxiv.org/abs/1702.08165). An algorithm that can work with off-policy data has also been developed by [Nachum, 2017] (in the form of a Bellman residual minimization algorithm, as opposed to this work which essentially uses a Fitted Q-Iteration algorithm as the critic). I think the paper could do a better job differentiating from those other papers. * The claim that this paper is about learning from demonstration is a bit questionable. The paper essentially introduces a method to use off-policy data, which is of course important, but does not cover the important scenario where we only have access to (state,action) pairs given by an expert. Here it appears from the description of Algorithm 1 that the transitions in the demonstration data have the same semantic as the interaction data, i.e., (s,a,r,s’). This makes it different from the work by [Kim et al., 2013], [Piot et al., 2014], and [Chemali et al., 2015], which do not require such a restriction on the demonstration data. * The paper mentions that to formalize the method as a policy gradient one, importance sampling should be used (the paragraph after (12)), but the performance of such a formulation is bad, as depicted in Figure 2. As a result, Algorithm 1 does not use importance sampling. This basically suggests that by ignoring the fact that the data is collected off-policy, and treating it as an on-policy data, the agent might perform better. This is an interesting phenomenon and deservers further study, as currently doing the “wrong” things is better than doing the “right” thing. I think a good paper should investigate this fact more.