Inverting Gradients - Gradient of critic network output wrt action

frknayk: <p>Hello there,<br>
I want to implement the inverting gradients as shown in the paper " Deep Reinforcement Learning in Parametrized Action Space". But firstly I need to have gradient of the critic network’s output ( Q(s,a)  ) with respect to the action ( dQ(s,a) / da ) which is (6) equation in the paper.</p>
<p>I can write the loss as,<br>
loss = value_net(state, policy_net(state))</p>
<p>But how can I differentiate this network with only respect to the action instead state ?</p>

soulless: <p><a class="mention" href="/u/frknayk">@frknayk</a> I am trying to implement this paper too and having trouble implement its gradient inversion. Have you figured it out?</p>

frknayk: <p>I have found tensorflow implementation<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/stevenpjg/ddpg-aigym" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/15150767?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/stevenpjg/ddpg-aigym" target="_blank" rel="nofollow noopener">stevenpjg/ddpg-aigym</a></h3>

<p>Continuous control with deep reinforcement learning - Deep Deterministic Policy Gradient (DDPG) algorithm implemented in OpenAI Gym environments - stevenpjg/ddpg-aigym</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>But later I decided to use loss = value_net(state,policy_net(state)) for some reasons</p>

Rahul_Patel: <p><a class="mention" href="/u/frknayk">@frknayk</a> Were you able to implement Inverting Gradients? If yes can you please share the snippet?</p>
<p>Thanks</p>

jomavera: <p>Is this correct?</p>
<p>Invert gradient function for actions defined in the interval [-1,1]:</p>
<pre><code class="lang-auto">def invert_gradient(grads, action):

    pdiff_max = torch.div(-action+1.0,2.0)
    pdiff_min = torch.div(action+1.0,2.0)

    zeros_grad    = torch.zeros_like(grads)
    grad_inverter = torch.zeros_like(grads)

    grad_inverter = torch.where(torch.gt(grads, zeros_grad), torch.mul(grads,pdiff_max), torch.mul(grads,pdiff_min))

    return grad_inverter
</code></pre>
<p>Outside training loop:</p>
<pre><code class="lang-auto">params_grads = {}
for name, param in actor.named_parameters():
    params_grads[name] = torch.zeros_like(param.data)
</code></pre>
<p>Inside training loop to update actor network parameters:</p>
<pre><code class="lang-auto">actor_optimizer.zero_grad()
action_Q = actor(state_batch)
policy_output = -critic(state_batch,action_Q)

for i in range(policy_output.size()[0]):
      policy_output[i].backward(retain_graph=True)
      for name, param in actor.named_parameters():
            params_grads[name] += invert_gradient(param.grad.data, action_Q.detach()[i])

     for name, param in actor.named_parameters():
          param.grad.data = params_grads[name]/batch_size
          params_grads[name] = torch.zeros_like(param.data)

actor_optimizer.step()
</code></pre>

gemsanyou: <p>Hi jomavera, thanks so much for the example. However, i have a follow-up question.</p>
<ol>
<li>will this affect the discrete parameters also? because in the original paper if I’m not mistaken, the bounded action space is only for the continuous parameter.</li>
<li>What if each parameter has different bound or limits? how do we track it?<br>
I think both of these questions are greatly influenced by my limited knowledge of how gradient storing and backpropagation works in pytorch.</li>
</ol>
<p>However, I know that we can treat all action as bounded by -1 to 1, even the discrete, and for the parameters we can then map them to their respective limit. But will this approach be valid?<br>
Thank you so much beforehand</p>

gemsanyou: <p>Another question if I may, turns out it produce error of mismatch dimension<br>
pdiff_max 's dimension is n_actions<br>
while grads is of n_states</p>
<p>is it maybe because n_actions &gt; 1</p>
<p>and if n_actions &gt; 1, do we just sum pdiff_max and pdiff_min ?</p>
<pre><code class="lang-auto">pdiff_max = torch.sum(torch.div(-action+1.0,2.0))
pdiff_min = torch.sum(torch.div(action+1.0,2.0))
</code></pre>
<p>because we are doing it for every action and add all.<br>
Thank you once again</p>

jomavera: <p>Hi <a class="mention" href="/u/gemsanyou">@gemsanyou</a>,</p>
<p>I think my code is not correct. I forgot to zero gradients for each action. This should be correct:</p>
<pre><code class="lang-auto">actor_optimizer.zero_grad()
action_Q = actor(state_batch)
policy_output = -critic(state_batch,action_Q)

for i in range(policy_output.size()[0]):
    policy_output[i].backward(retain_graph=True)
    for name, param in actor.named_parameters():
        params_grads[name] += invert_gradient(param.grad.data, action_Q.detach()[i])
    actor_optimizer.zero_grad()

for name, param in actor.named_parameters():
    param.grad.data = params_grads[name]/batch_size
    params_grads[name] = torch.zeros_like(param.data)

actor_optimizer.step()
</code></pre>
<p>Regarding your questions:</p>
<ol>
<li>I see no problem to apply it to discrete actions as long as you know the action bounds</li>
<li>If each action have different bounds you can change the code of <code>invert_gradient</code> function to have as argument the bounds, as follows:</li>
</ol>
<pre><code class="lang-auto">def invert_gradient(grads, action, max_b, min_b):

    pdiff_max = torch.div(-action+max_b, max_b - min_b)
    pdiff_min = torch.div(action-min_b, max_b - min_b)

    zeros_grad    = torch.zeros_like(grads)
    grad_inverter = torch.zeros_like(grads)

    grad_inverter = torch.where(torch.gt(grads, zeros_grad), torch.mul(grads,pdiff_max), torch.mul(grads,pdiff_min))

    return grad_inverter
</code></pre>
<p>Then, somewhere outside the trainning you should store the action bounds for example in a dict called <code>actions_bounds</code>. Then the trainning loop should be</p>
<pre><code class="lang-auto">actor_optimizer.zero_grad()
action_Q = actor(state_batch)
policy_output = -critic(state_batch,action_Q)

for i in range(policy_output.size()[0]):
    policy_output[i].backward(retain_graph=True)
    max_b, min_b = actions_bounds[i]
    for name, param in actor.named_parameters():
        params_grads[name] += invert_gradient(param.grad.data, action_Q.detach()[i], max_b, min_b)
    actor_optimizer.zero_grad()

for name, param in actor.named_parameters():
    param.grad.data = params_grads[name]/batch_size
    params_grads[name] = torch.zeros_like(param.data)

actor_optimizer.step()
</code></pre>
<p>Regarding the final remarks, the inverting gradient formula tries to downscale the gradient if the action is close to the upper or lower limit in order to avoid the behaviour of a policy on actions on the bounds.</p>

gemsanyou: <p>Hi jomavera,<br>
thank you so much for the super fast reply, even for necroing this post <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>I will try this right now and follow up with the results. In particular this line of code, i think the i iteration is for each sample in batch, not for each action. But i will confirm it right now</p>
<aside class="quote no-group" data-username="jomavera" data-post="8" data-topic="37518">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e9bcb4/48.png" class="avatar"> jomavera:</div>
<blockquote>
<p><code>for i in range(policy_output.size()[0]):</code></p>
</blockquote>
</aside>
<p>Ah yes, the i is for sample iteration and the previous error i mentioned is this:</p>
<pre><code class="lang-auto">   grad_inverter = T.where(T.gt(grads, zeros_grad), T.mul(grads,pdiff_max), T.mul(grads,pdiff_min))
RuntimeError: The size of tensor a (99) must match the size of tensor b (8) at non-singleton dimension 1
</code></pre>
<p>when i print</p>
<pre><code class="lang-auto">print(policy_output.size()) = torch.Size([64, 1])
</code></pre>

jomavera: <p>I assumed in the code that the action is only one component/scalar. The <code>invert_gradient</code> function should be called for each action component independently so the <code>sum</code> I don’t think is appropiate. So if we “compute” an inverted gradient for each action component and each at batch sample and sum it to <code>params_grads[name]</code>  then I think the gradient should be the mean w.r.t. components and batch</p>
<pre><code class="lang-auto">param.grad.data = params_grads[name]/(batch_size*action_size)
</code></pre>

gemsanyou: <p>So currently here are my code, knowing that i is for each sample<br>
the part in learn method:</p>
<pre><code class="lang-auto">self.actor.optimizer.zero_grad()
action_q = self.actor.forward(states)
policy_output = -self.critic.forward(states, action_q)
        
for i in range(policy_output.size()[0]):
    policy_output[i].backward(retain_graph=True)
    for name, param in self.actor.named_parameters():
        self.params_grads[name] += invert_gradient(param.grad.data, action_q.detach()[i])
    ??? self.actor.optimizer.zero_grad() ???

for name, param in self.actor.named_parameters():
    param.grad.data = self.params_grads[name]/(self.batch_size*self.n_actions)
    self.params_grads[name] = T.zeros_like(param.data)
</code></pre>
<p>invert gradient</p>
<pre><code class="lang-auto">def invert_gradient(grads, action, lower_bound=-1, upper_bound=1):
    bound_range = upper_bound-lower_bound
    pdiff_max = T.sum(T.div(upper_bound-action, bound_range))
    pdiff_min = T.sum(T.div(action-lower_bound, bound_range))
    zeros_grad = T.zeros_like(grads)
    grad_inverter = T.where(T.gt(grads, zeros_grad), T.mul(grads,pdiff_max), T.mul(grads,pdiff_min))
    return grad_inverter
</code></pre>
<p>do you think this is correct?<br>
so what is the zeroing gradient for each i will do, now knowing that i is for iterating sample size? will it zero gradients for each action?</p>
<p>or should I have another iteration inside here like this:</p>
<pre><code class="lang-auto">for i in range(policy_output.size()[0]):
    policy_output[i].backward(retain_graph=True)
    for a in action_q.detach()[i]:
        for name, param in self.actor.named_parameters():
            self.params_grads[name] += invert_gradient(param.grad.data, a)
        self.actor.optimizer.zero_grad()
</code></pre>
<p>without summing inside invert grad method.<br>
thank you so much beforehand</p>
<p>EDIT: i think i found an example repo of this for you guys also who needed it, turns out it’s with all other paramterized RL agent in pytorch too : <a href="https://github.com/cycraig/MP-DQN" class="inline-onebox" rel="noopener nofollow ugc">GitHub - cycraig/MP-DQN: Source code for the dissertation: "Multi-Pass Deep Q-Networks for Reinforcement Learning with Parameterised Action Spaces"</a></p>
<p>I’m going to read it and try it out</p>

idiotpotato: <p>Hi everyone,<br>
I know it’s been too long since the last update but I just had the problem with inverting gradient so I wanted to ease this subject for anyone else.<br>
as <a class="mention" href="/u/gemsanyou">@gemsanyou</a> said the inverting gradient is implemented in <a href="https://github.com/cycraig/MP-DQN" rel="noopener nofollow ugc">mp-dqn</a>. For anyone who encounters this problem in future i will explain the code here.</p>
<p>this being the inverting gradient function:</p>
<pre><code class="lang-auto">	def invert_gradient(self,delta_a,current_a):
		index = delta_a&gt;0
		delta_a[index] *=  (index.float() * (self.max_p - current_a)/self.rng)[index]
		delta_a[~index] *= ((~index).float() * (current_a- self.min_p)/self.rng)[~index]

		return delta_a	
</code></pre>
<p>gets the gradient(wrt actions) and actions themselves. index is a masking variable indicating gradient is suggesting increment of action or not. gradients are then updated accordingly.<br>
Now in order to use this function:</p>
<pre><code class="lang-auto">current_a = Variable(self.actor(state))
current_a.requires_grad = True
actor_loss = self.critic(state, current_a).mean()
self.critic.zero_grad()
actor_loss.backward()
delta_a = deepcopy(current_a.grad.data)
delta_a = self.invert_gradient(delta_a,current_a)
current_a = self.actor(state)
out = -torch.mul(delta_a,current_a)
self.actor.zero_grad()
out.backward(torch.ones(out.shape).to(device))
</code></pre>
<p>First you get the gradient wrt actions then after applying inverting gradient you will end up with modified dQ/da now what did we want in the first place? -(dQ/da) * (da/dw)<br>
we changed dQ/da part now simply consider dQ/da = c. having -c<em>a calculating gradient wrt weights we’ll end up with -c</em>(da/dw) = -(dQ/da) * (da/dw).</p>
<p>One point worth mentioning here is about out.backward and why it has an ones vector as its input. it is because “a” is a vector and in backward we will end up with Jacobian matrix however for updating the weights we need one vector of gradients so we simply give the linear combination of J matrix’s columns as the input(one could pass delta_a to backward however it is not implemented like this in the original code) for more info read <a href="https://zhang-yang.medium.com/the-gradient-argument-in-pytorchs-backward-function-explained-by-examples-68f266950c29" rel="noopener nofollow ugc">gradient arg</a></p>

TheUnnamed22: <p>Thanks a lot for the explenation! I think i understood it now.</p>
<p>But i have one question to be more certain. Why do you have to multiply the gradient with the current action?</p>
<pre><code class="lang-auto">out = -torch.mul(delta_a,current_a)
</code></pre>
<p>if delta_a is dQ/da or c then you would have to multiply with da/dw and not the action. Well i think its because you first do -c*a and then you differenciate that which leads to:</p>
<p>d(-c*a)/dw</p>
<p>now you can take out the c from the equation and get:</p>
<p>-c*(da/dw)</p>
<p>am i right with that?</p>

idiotpotato: <p>Yes that is exactly right</p>