Deep Deterministic Policy Gradient
By: Amit Patel
Tags: Machine Learning, Reinforcement Learning, Artificial Intelligence

Reinforce Algorithm

Please note, in the below analyses, the discount factor γ is assumed to be 1 for simplicity. But all the analyses can be easily extended to cases where γ is not 1.

The basic objective in all of Reinforcement Learning (RL) is to maximize the expected total utility Uθ, which is defined as follows [1]:

After doing some Math, it can be shown that Uθ is equal to the expected value of Q(s₀,a₀). And if the initial state distribution is uniform, then it means the goal in RL is to find a policy which maximizes the q-values of all possible states.

Using the definition of expectation, the above equation 1a can be re-written as:

Using Policy gradient method, we can maximize Uθ by first computing its gradient with respect to θ, which (using the Reinforce log likelihood trick [3]) can be derived to be:

One approach to improving the expected total reward is to randomly add noise to the current θ and if it results in better total reward, then we keep it, otherwise we ignore it, and we keep repeating this process. This method is called the random shooting method. There are other more sophisticated methods in the same vein such as the Cross Entropy Method. All these methods fall under the domain of stochastic optimization algorithms. However, while these methods are very simple to implement, they are not efficient and don’t scale well with high dimensional space. A more efficient approach is to change θ in the direction of the gradient using Stochastic Gradient Ascent as follows:

A basic policy gradient algorithm making use of the above gradient is known as the Reinforce algorithm, and here is how it works:

A Basic Reinforce Algorithm:

Start with a random vector θ and repeat the following 3 steps until convergence:

1. Use the policy Pθ(at|st) to collect m trajectories {τ1, τ2, …, τm}, where each trajectory is as defined above.

2. Use these trajectories to compute the Monte-Carlo estimator of the gradient as follows:

Note that the reason why the above estimator is valid is because the trajectories are generated by following the policy being learned, i.e. Pθ(τ) — i.e. it is an on-policy algorithm. Another way to say it is that we sample each of the trajectories in {τ1, τ2, …, τm} from the probability distribution Pθ(τ).

3. Update the weights/parameters of the policy network using the above estimator of the gradient:

The intuition behind the reinforce algorithm is that if the total reward is positive, then all the actions taken in that trajectory are reinforced whereas if the total reward is negative, then all the actions taken in the trajectory are inhibited. Moreover, to be computationally efficient, typically m is set to 1.

While better than stochastic optimization methods, the Reinforce algorithm suffers from a few drawbacks:

1. The gradient estimator is pretty noisy, especially for the case m=1, because a single trajectory maynot be representative of the policy.

2. There is no clear credit assignment. A trajectory may contain many good and bad actions, and whether those actions are reinforced or not depend only on the total reward achieved starting from the initial state.

3. It is very sensitive to the absolute value of the rewards. For example, adding a fixed constant to all the rewards can drastically change the behavior of the algorithm. Such a trivial transformation should have no effect on the optimal policy.

By the definition of the gradient, ∇θUθ points in the direction of the maximum change in Uθ. However, at a fundamental level, the above drawbacks of Reinforce algorithm are due to the fact that the Monte-Carlo estimator of ∇θUθ (i.e. ĝ) has high variance. If we can reduce its variance, then our estimate of gradient (ĝ) will be closer to the true gradient ∇θUθ.

While the Monte-Carlo estimator of the gradient (ĝ) is unbiased, it exhibits high variance. As discussed below, there are a few ways of reducing variance without introducing bias: 1) using causality and 2) using a baseline.

Actor-Critic Algorithm

One way to reduce variance is by taking advantage of causality: ĝ updates all the actions in a trajectory based upon total rewards and not the rewards to go. That is to say, future actions affect past rewards, which is not possible in our causal Universe. So we can make the gradient estimator more realistic by using rewards to go as shown in the below equation.

Note that using the rewards to go instead of the total rewards still results in an unbiased estimator of ∇θUθ because causality is handled in the expectation in Equation 3 using Pθ(τ). Moreover, doing so reduces variance because the rewards to go expression has fewer terms (and thus lower uncertainty) than the total rewards expression.

An important aside to note is that the rewards to go is really an estimate of the the q-value of (st, at). This is because the q-value is defined as follows:

And so, if the trajectory τ is sampled from Pθ(τ), then the single-sample Monte-Carlo estimate of QPθ(st, at) is just:

As shown above, instead of using the Monte-Carlo estimator of the rewards to go as in Equation 7, we can use the Q-value estimator of the rewards to go. As a result, Equation 7 can be re-written as:

If Qhat Pθ(st, at) is modeled using a neural network (parameterized by w), then we get:

Note that because the state-action space can be very high dimensional, it quickly runs into Bellman’s curse of dimensionality; and thus, in most practical situations with complex state-transition dynamics, Qhat Pθ(st, at) is modeled using a neural network based function approximator.

Then Equation 10 can be re-written as:

Whereby, Pθ(at | st) is the actor network that is parameterized by θ and Qhat Pθ(st, at) is the critic network that is parameterized by w. This is essentially what is known as the actor-critic algorithm.

For any visited state-action pair (s,a), the actor network is updated using Equation 6 (utilizing ĝ from Equation 12), and the critic network is typically updated using Temporal-Difference learning (due to its lower variance than Monte-Carlo learning) using the following update equation:

Whereby the weight vector w is updated to reduce the loss L(w), which is defined as:

and using Q-learning (so that the critic is based of off an off-policy algorithm):

and so

whereby

This is the basics of the actor-critic algorithm. While there are many variants of it, as we will see below, this is the basic core of it.

Advantage Actor-Critic Algorithm

In addition to using the rewards to go (due to causality), another approach to minimizing the variance of ĝ is by subtracting out a baseline b that is not dependent on θ or action a — and this combined term is known as the Advantage function. It can be mathematically proved that such a transformation is not only unbiased, but it reduces variance. An intuitive explanation for why it reduces variance is because the term multiplying ∇θlog(Pθ(a|s)) has smaller magnitude, which essentially reduces the variance of the overall expression.

There are many choices for the baseline b, and in theory, the optimal value of b can also be computed. However, in the interest of simplicity and to be intuitive, a commonly used baseline is the q-value averaged over all the actions, i.e. the state-value.

The Advantage function is then written as follows:

The basic idea with using this advantage function is that actions with higher q-value than the average (i.e. state-value) are reinforced where as other actions are inhibited. This makes a lot more intuitive sense than the gradient equation used in the original Reinforce algorithm. And so it’s not totally surprising that Mathematically it results in lower variance. Moreover, now the gradient is no longer dependent on the absolute value of the rewards.

One problem with the above Equation is that, in practice, it is very difficult to compute the above expectation — especially for continuous actions or high dimensional action space. Hence, the state-value function is modeled with a separate neural network that is parameterized by wᵥ as follows:

The advantage function now becomes:

The issue with this advantage function is that it requires two separate neural networks. With some clever re-ordering, we can re-write the Advantage function using a single neural network. However, inorder to do so, let us first re-visit the above analysis. Basically, the ideal Advantage function we would like to have is:

As defined in Equation 8 above, state-action value can be further simplified interms of the state-value function as:

The single-sample Monte-Carlo estimate of QPθ(st, at) as defined in the Equation above is:

And so now we just need to represent the state-value function using a neural network parameterized by wv as follows:

And thus the Advantage function can now be represented using a single neural network parameterized with wᵥ. Note with the above equation for the Advantage function, it is really just the one-step TD error (i.e. TD(0) error). Additionally, it is also possible to represent it using TD(λ) error.

The gradient equation for Advantage Actor Critic is now going to be:

And this is going to be a much better estimator of the expected gradient (Equation 3), i.e. with lower variance and still be unbiased, even for m=1. As a result, the algorithm will learn much faster.

wv is updated as follows:

whereby using one-step TD learning (i.e. TD(0)):

Using the gradient estimator from Equation 29, the weight update from Equation 30, and the remaining steps from the basic Reinforce algorithm results in what is known as the Advantage Actor-Critic algorithm.

To briefly summarize the above discussion, the main downside of the Reinforce algorithm is that the gradient estimator is based upon the Monte-Carlo estimator of the expected total reward from the initial state-action pair — which while has low bias, it has high variance. By using causality and subtracting out a baseline from the Monte-Carlo estimator, we can reduce the variance. The variance is further reduced by using TD estimator of the expected total reward to go instead of Monte-Carlo estimator.

Deterministic Policy Gradient (DPG) Algorithm

For stochastic policies in continuous environments, the actor outputs the mean and variance of a Gaussian distribution. And an action is sampled from this Gaussian distribution. For deterministic actions, while this approach still works as the network will learn to have very low variance, it involves complexity and computational burden that unnecessarily slows down the learning algorithm. To address these short comings, for deterministic actions, we can use what is known as the deterministic policy gradient.

In stochastic case, the policy gradient algorithm integrates over both state and action spaces, whereas in the deterministic case it only integrates over the state space. As a result, computing the deterministic policy gradient can potentially require fewer samples. But in order to fully explore the state space, the basic idea is to choose actions according to a stochastic behavior policy and learn about a deterministic target policy (i.e. needs to be an off-policy algorithm).

DPG is essentially a deterministic version of Actor-Critic algorithm. For a basic DPG algorithm, we have two neural networks, one network (parameterized by θ) is estimating the optimal target policy and the second network (parameterized by w) is estimating the action-value function corresponding to the target policy. The below equations formalize this.

As mentioned above, because the target policy is deterministic, the actor may not explore the state-space very well to find the optimal policy. To address it, we use a behavior policy (b(st)) that is different from the target policy. It is basically the target policy with some additional noise. For simplicity, we will use a Normal distribution as our noise source. But note that this term is like a hyper parameter, and in the below implementation for the Reacher environment, a different noise process is used.

Deterministic Policy Gradient Update:

1. Actor network is updated as follows:

which by chain rule, it becomes:

2. The critic network is updated as follows:

The TD error is given by:

and the weight update is:

To reiterate, in order to properly balance exploration-exploitation tradeoff, while the target policy μ is deterministic, the behavior policy is stochastic. So this is an off-policy version of the DPG algorithm. While stochastic off-policy actor-critic algorithms typically use importance sampling for both the actor and the critic, because the deterministic policy gradient removes expectation over the actions, and given the state transition dynamics are same for both the target and behavior policies as they operate in the same environment, importance sampling ratio is not needed. So we can avoid having to use importance sampling in the actor, and with same reasoning, we avoid using importance sampling in the critic [2]. For those who are wondering, similar reasoning applies as to why we don’t use importance sampling with Q-learning.

Deep Deterministic Policy Gradient (DDPG) Algorithm

DDPG is basically DPG with a few training changes adopted from the DQN architecture.

One challenge when using neural networks for reinforcement learning is that most optimization algorithms assume the samples are independently and identically distributed. Obviously this assumption doesn’t hold true because the samples are generated by exploring sequentially in an environment. Because DDPG is an off policy algorithm, we can use the replay buffer (a finite sized cache) as in DQN to address this issue. At each timestep the actor and critic are updated by sampling a minibatch uniformly from the buffer [2].

For the critic, since the network being updated is also used in calculating the target, this can potentially lead to training instabilities for highly nonlinear function approximators like neural networks. One solution to address this is using a separate target network, as with DQN [2]. Given the target values are determined using both the critic and actor networks, we create a copy of both of these networks and soft update their weights to the respective learned networks. Please refer to my github code for details.