Grad is None in Sequential Model

username: StaleChexMix
cooked: <p>I’ve looked at many articles and have been Googling for a few days now without being able to fix the issue I’m having. I am having an issue with the autograd PyTorch function in my sequential models. I am trying to implement the PPO algorithm, but for some reason, the gradients don’t propagate to my networks after calling backward on the loss term as seen below in my critic network<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2ff5fbb2b3fd3682fcedfec7adb48b357f2c8478.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ff5fbb2b3fd3682fcedfec7adb48b357f2c8478" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2ff5fbb2b3fd3682fcedfec7adb48b357f2c8478_2_690x280.png" alt="Capture" data-base62-sha1="6QhsnjeQZIRjJvn7gSakAIRZOnS" width="690" height="280" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2ff5fbb2b3fd3682fcedfec7adb48b357f2c8478_2_690x280.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2ff5fbb2b3fd3682fcedfec7adb48b357f2c8478_2_1035x420.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2ff5fbb2b3fd3682fcedfec7adb48b357f2c8478_2_1380x560.png 2x" data-dominant-color="2B2B29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1489×606 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In fact, the gradients propagate all the way to the vector of outputs from the network, but they don’t actually propagate through the network (currCriticVals is an array storing values directly from the critic network):</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://ibb.co/hY4SqQH">
  <header class="source">
      <img src="https://simgbb.com/images/favicon.png" class="site-icon" width="" height="">

      <a href="https://ibb.co/hY4SqQH" target="_blank" rel="noopener nofollow ugc">ImgBB</a>
  </header>

  <article class="onebox-body">
    <img width="180" height="180" src="https://i.ibb.co/hY4SqQH/Capture2.png" class="thumbnail onebox-avatar">

<h3><a href="https://ibb.co/hY4SqQH" target="_blank" rel="noopener nofollow ugc">Capture2 hosted at ImgBB</a></h3>

  <p>Image Capture2 hosted in ImgBB</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
(Sorry for the link. As a new user, I can only have one embed and two links)</p>
<p>Also, note that I am running this on my CPU at the moment if that may cause any problems.</p>
<p>The full code can be found at the following GitHub link:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/gmongaras/PPO_CartPole">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/gmongaras/PPO_CartPole" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/7b5e122ae67b80cded57a5eb51a107c59a73c0f374d2cbc599af8b386d324888/gmongaras/PPO_CartPole" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/gmongaras/PPO_CartPole" target="_blank" rel="noopener nofollow ugc">GitHub - gmongaras/PPO_CartPole: Using PPO, I am attempting to solve the...</a></h3>

  <p>Using PPO, I am attempting to solve the cartpole environment - GitHub - gmongaras/PPO_CartPole: Using PPO, I am attempting to solve the cartpole environment</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have a few debugging lines near the backward function in case you realize it looks a bit messy.</p>
<p>Any help would be greatly appreciated! Thank you!</p>

username: ptrblck
cooked: <p>I’m not sure which parameters should get gradients but don’t so could you explain the issue a bit more?<br>
PS: you can post code snippets by wrapping them into three backticks ``` which allows to debug the code easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Based on the linked code you have to check if you are (accidentally) detaching the tensors form the computation graph vi:</p>
<ul>
<li>using <code>numpy</code> (Autograd won’t be able to track these operations so you should use PyTorch methods instead. If that’s not possible, write a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>)</li>
<li>rewrapping tensors via <code>x = torch.tensor(x)</code> where <code>x</code> was a tensor attached to a graph</li>
<li>using non-differentiable functions such as <code>torch.argmax</code>
</li>
</ul>

username: StaleChexMix
cooked: <p>Thank you for your reply! I’m hoping to update both the Actor and Discriminator networks in terms of the L_Final loss function where the Actor network is updated through the L_CLIP loss function and the Critic network is updated through the L_VF function. Both L_CLIP and L_VF are added together to make L_Final which is what .backward() is called on. S is just some noise added on to the loss which shouldn’t affect the backward pass and is why I set it to “requires_grad=False”</p>
<pre><code class="lang-auto">L_CLIP = torch.min(r_ts*advantages, torch.clip(r_ts, 1-epsilon, 1+epsilon)*advantages).mean()

# Calculate the critic loss (L_VF)
L_VF = torch.pow(rewards-currCriticVals, 2).mean()

# Get the entropy bonus from a normal distribution
S = torch.tensor(np.random.normal(), dtype=torch.float, requires_grad=False)
 
#Calculate the final loss (L_CLIP_VF_S)
L_Final = L_CLIP - self.c1*L_VF + self.c2*S
</code></pre>
<p>What I’m hoping is that the Critic and Actor networks get a gradient for all parameters. In a simple case, L_VF is the loss for the critic and is what I’m mostly paying attention to when debugging. L_VF is composed of the values received from the critic. So, the process to calculate L_VF is as follows:</p>
<ol>
<li>collect n critic values and put them in an array<br>
<code>currCriticVal = self.critic(observation) # Get value from critic</code><br>
<code>self.storeMemory(...,currCriticVal,...) # Store value in list</code>
</li>
<li>turn the array into a tensor and remove the last value:<br>
<code>currCriticVals = torch.tensor(currCriticVals[0:advantages.shape[0]], dtype=torch.float, requires_grad=True, device=device)</code>
</li>
<li>Calculate the L_VF loss for the critic network<br>
<code>L_VF = torch.pow(rewards-currCriticVals, 2).mean()</code>
</li>
</ol>
<p>The interesting part is that currCriticVals receives a grad, but the Critic parameters do not and this is the problem I am having.</p>
<p>I reviewed my code again, I didn’t notice any of the following potential causes you mentioned. I used numpy once, but that was for input into the Critic network, and that value shouldn’t need a gradient. Before calculating the loss, I did rewrap a few tensors, but changing the values from say <code>x = torch.tensor(x)</code> to <code>x2 = torch.tensor(x)</code> didn’t solve the gradient issue. I also used argmax once, but the value received from argmax didn’t need a gradient as it’s just input into one of the networks.</p>

username: ptrblck
cooked: <p>You are recreating the tensor here without any gradient history:</p>
<pre><code class="lang-python">currCriticVals = torch.tensor(currCriticVals[0:advantages.shape[0]], dtype=torch.float, requires_grad=True, device=device)
</code></pre>
<p><code>currCriticVals</code> is a new tensor, which requires gradients without any history and will thus detach the computation graph.<br>
This fits into the “rewrapping tensors” point. Assuming <code>currCriticVals</code> is a tensor attached to the computation graph and you want to create a new tensor with its slices, try to use <code>torch.cat</code> or <code>torch.stack</code> instead.</p>

username: StaleChexMix
cooked: <p>Oh I see now. I didn’t realize changing the list to a torch tensor with torch.tensor removes it from the gradient history. Instead of using<br>
<code>currCriticVals = torch.tensor(currCriticVals[0:advantages.shape[0]], dtype=torch.float, requires_grad=True, device=device)</code><br>
I used<br>
<code>currCriticValss = torch.cat(currCriticVals[0:advantages.shape[0]])</code><br>
Where currCriticVals is a list of tensors.<br>
Now that I changed it, the gradients propagate through the network. Looking back at torch.tensor vs. torch.cat makes perfect sense as to why it wasn’t working before.</p>
<p>Thank you for your help with this issue!</p>