DDPG gradient with respect to action

username: DrerD
cooked: <p>Hi, I am trying to implement DDPG with pytorch, I am sure most of my implementation is right, but the policies don’t converge so I am not too certain about the gradient part. The paper shows that the gradient of Q(s,a) is with respect to action a. I am defining the loss to be Q(s,a), but how do I know that the gradient will be taken derivative with respect to a? Since the loss = Q(s,a), I am setting loss.requires_grad = True, and loss.volatile = False.</p>

username: alexis-jacq
cooked: <p>Q should never be the loss function. DDPG is a case of Deep Actor-Critic algorithm, so you have two gradients: one for the actor (the parameters leading to the action (<code>mu</code>)) and one for the critic (that estimates the value of a state-action (<code>Q</code>) – this is our case – , or sometimes the value of a state (<code>V</code>) ).</p>
<p>In DDPG, the critic loss is the temporal difference (as in classique deep Q learning):<br>
<code>critic_loss = (R - gamma*Q(t+1) - Q(t))**2</code><br>
Then the critic’s gradient is obtained by a simple backward of this loss.</p>
<p>For the actor gradient, things are more complex: it’s an estimation of the policy gradient, given by:<br>
<code>actor_grad = Q_grad * mu_grad</code><br>
Where <code>mu</code> is the output of the network, estimating the optimal mean of the action’s Gaussian distribution.</p>

username: DrerD
cooked: <p>sorry, I should say I am trying to find the actor gradient. I am referencing <a href="https://github.com/ghliu/pytorch-ddpg/blob/master/ddpg.py" rel="nofollow noopener">this implementation</a> and he does<br>
policy_loss = -self.critic([to_tensor(state_batch), self.actor(to_tensor(state_batch))].mean()<br>
which is just simply defining the loss = -mean(critic(state,actor(state)))<br>
My belief is that the autograd will find the gradient with respect to action, which is actor_grad = Q_grad * mu_grad given the chain rule property.</p>

username: alexis-jacq
cooked: <p>Ok I see, it makes sens to directly derivates <code>Q(S, pi(S))</code> wrt <code>pi</code> 's parameters.</p>
<p>In the paper, the gradient is wrt a, because they decompose the derivative (hence the <code>mu_grad</code> in my equation above). If you directly take <code>Q</code> as a loss, you must derivate it wrt policy’s parameters.</p>
<p>So, just doing</p>
<pre><code class="lang-auto">pi_loss = -Q(state, pi(state))
pi_loss.backward()
pi_optimizer.step()
</code></pre>
<p>should be ok.</p>
<p>In the paper, look at equation 6: it’s a mater of computing the first line (Q directly with derivative wrt pi) or the second line (decomposition, with a derivative wrt a)</p>

username: DrerD
cooked: <p>so back to my question, how is the gradient wrt action specified?  Q(s,a) has both state and action as variable, and the gradient can be taken wrt to state if it’s not specified. This is what’s confusing me.</p>

username: alexis-jacq
cooked: <p>As I said, you don’t want to derivate wrt action in your case, but wrt the parameters of your policy. The states must be detached from the graph, and the code in my post above should do what you want.</p>

username: DrerD
cooked: <p>Oh I see, so is there anyway to actually see that the gradient is actually with respect to policy parameters?</p>

username: alexis-jacq
cooked: <p>When you are using an optimizer on some parameters, only these parameter will be affected by the gradient step. So, if you did something like</p>
<pre><code class="lang-python">policy_optim = nn.Optimizer(policy.parameters(), lr)
</code></pre>
<p>above in your code, and then</p>
<pre><code class="lang-python">policy_loss = -Q(s, policy(s))
policy_loss.backward()
policy_optim.step()
</code></pre>
<p>… then, only the parameter of the policy will be affected, with the gradient of your loss wrt these parameters. It’s that simple!</p>

username: fabrice
cooked: <p><a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> I am coming a bit late in the discussion and actually I would like to know why do you need to detach the state from the graph when you take directly Q as a loss?<br>
DDPG is a bit complicated to understand regarding the gradient update of the actor.</p>

username: alexis-jacq
cooked: <p>You don’t need to detach the state. My last point was : taking Q as a loss, only the parameters given to the optimizer will be optimized.</p>
<p>The gradient update in DDPG is not that complicated. In fact, it’s even simpler than stochastic policy gradients:</p>
<p>Since Q approximates your returns, you want to maximize E_a~mu(s) [Q(a,s)] = E_s [Q(mu(s), s)]<br>
So, you take the derivative of this thing : E_s [mu’(s) * Q’(mu(s), s)] and do a gradient descent on the parameters of mu.</p>
<p>With an automatic derivation tool like pytorch, you don’t even care about going that far, you just take -Q(mu(s), s) as a loss to maximize (wrt mu’s parameters).</p>

username: fabrice
cooked: <p><a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> Thank you for your answer, it is clear now.</p>

username: Rance
cooked: <p><a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> I was also confused with this actor gradient problem and not any more with your replies. Thanks a lot!</p>

username: MngFrc
cooked: <p>Hi!<br>
Sorry I am coming definitely late, but I’m facing your identical problem. I have implemented the actor loss as:</p>
<pre><code class="lang-auto">pred_a1 = self.actor.forward(s1)
actor_Loss = -1*torch.mean(self.critic.forward(s1, pred_a1))
</code></pre>
<p>but this makes the actor diverge.<br>
I do not how to fix that problem… Did you solve it?</p>
<p>Thank you a lot in advance!</p>

username: yxz77777
cooked: <p>I am coming a bit late in the discussion .I have implemented the actor loss as:</p>
<ul>
<li>policy_Q = torch.mean(self.critic(state_actor_batch))<br>
but my actor-net unable to  backpropagate.How should I handle this issue?</li>
</ul>