Grad None when using custom loss function for a2c implementation

username: codeflux
cooked: <p>Hi,</p>
<p>I am implementing an A2C algorithm, the loss function for the critic is simply the advantage function and that works.<br>
For the actor I use this:</p>
<pre><code class="lang-auto">loss = -distribution.log_prob(sample)*advantage.detach()
loss.requires_grad = True
loss.backwards()
</code></pre>
<p>where advantage is calculated using the critic and sample is from sampling from the distribution I am trying to learn.</p>
<pre><code class="lang-auto">variance_energy = 0.01
variance_decision = 0.01
mean_matrix = torch.FloatTensor([mean_energy, mean_decision])
covariance_matrix = torch.FloatTensor([[variance_energy, 0],[0, variance_decision]])
distribution = MultivariateNormal(mean_matrix, covariance_matrix)
sample = distribution.sample()
</code></pre>
<p>Here, mean_energy and mean_decision are the output of my NN and are restricted to positive values (so I don’t have problems with the log)<br>
The gradient is always None.<br>
I have already checked that required_grad = True and is_leaf = True for the loss function also.</p>
<p>I calculate the gradients of the weights manually (so I can update them that way) after calling loss.backward()  with:</p>
<pre><code class="lang-auto"> for p in self.model.parameters():
            gradient_list.append(p.grad)
</code></pre>
<p>Does  anybody have an idea how to troubleshoot this? Thanks in advance!</p>

username: KFrank
cooked: <p>Hi Codeflux!</p>
<aside class="quote no-group" data-username="codeflux" data-post="1" data-topic="137240" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e19b73/48.png" class="avatar"> codeflux:</div>
<blockquote>
<pre><code class="lang-auto">loss = -distribution.log_prob(sample)*advantage.detach()
loss.requires_grad = True
loss.backwards()
</code></pre>
</blockquote>
</aside>
<p><code>advantage.detach()</code> “breaks the computation graph,” and<br>
<code>loss.requires_grad = True</code> doesn’t repair the damage.</p>
<p>Regardless of whether your code can be tweaked to (appear to)<br>
work, <code>loss</code> must be usefully differentiable with respect to the<br>
<code>parameters</code> you are trying to train in order for backpropagation<br>
and training to work.</p>
<p>Best.</p>
<p>K. Frank</p>

username: codeflux
cooked: <p>Thanks for the reply! Turns out I don’t need loss.required_grad = True and the .detach() call on the advantage function for the loss function to have grad enabled.<br>
But now I receive the error message:</p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.</p>
</blockquote>
<p>Seems like backpropagation goes through my network two times?</p>
<p>I’ve also tried something else, removing the advantage function (which is ofc not correct for actor-critic). Then for some reason the loss function does not have grad_enabled, and using grad_enabled = True results in the gradient of my weights being zero again.<br>
Is_Leaf is true again though.</p>
<p>I think here is already something wrong.</p>

username: codeflux
cooked: <p>It appears that .backward does not seem to work with the MultivariateNormal function… when I use Normal instead for just one parameter, I can calculate the gradient.</p>
<p>Can anyone confirm if this is true?</p>

username: KFrank
cooked: <p>Hi Codeflux!</p>
<aside class="quote no-group" data-username="codeflux" data-post="4" data-topic="137240" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e19b73/48.png" class="avatar"> codeflux:</div>
<blockquote>
<p>It appears that .backward does not seem to work with the MultivariateNormal function… when I use Normal instead for just one parameter, I can calculate the gradient.</p>
</blockquote>
</aside>
<p>I believe that <code>MultivariateNormal</code> and <code>Normal</code> have the same<br>
behavior in this regard.</p>
<p>In general, you can’t differentiate or back propagate through calling<br>
<code>.sample()</code> on a <code>Distribution</code>.  This is true for <code>Normal</code>, as well as<br>
<code>MultivariateNormal</code>.  In contrast, you typically can backpropagate<br>
through non-sampling methods such as <code>.log_prob()</code>.</p>
<p>Here are some illustrative results:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt;
&gt;&gt;&gt; meanN = torch.zeros (1, requires_grad = True)
&gt;&gt;&gt; stdN = torch.ones (1, requires_grad = True)
&gt;&gt;&gt; distN = torch.distributions.Normal (meanN, stdN)
&gt;&gt;&gt;
&gt;&gt;&gt; xN = distN.sample()
&gt;&gt;&gt; xN.grad_fn
&gt;&gt;&gt;
&gt;&gt;&gt; yN = distN.log_prob (torch.zeros (1))
&gt;&gt;&gt; yN.grad_fn
&lt;SubBackward0 object at 0x00000188F9459CF8&gt;
&gt;&gt;&gt; yN.backward()
&gt;&gt;&gt; meanN.grad
tensor([0.])
&gt;&gt;&gt; stdN.grad
tensor([-1.])
&gt;&gt;&gt;
&gt;&gt;&gt; meanMV = torch.zeros (1, requires_grad = True)
&gt;&gt;&gt; stdMV = torch.eye (1, requires_grad = True)
&gt;&gt;&gt; distMV = torch.distributions.MultivariateNormal (meanMV, stdMV)
&gt;&gt;&gt;
&gt;&gt;&gt; xMV = distMV.sample()
&gt;&gt;&gt; xMV.grad_fn
&gt;&gt;&gt;
&gt;&gt;&gt; yMV = distMV.log_prob (torch.zeros (1))
&gt;&gt;&gt; yMV.grad_fn
&lt;SubBackward0 object at 0x00000188FA12CA58&gt;
&gt;&gt;&gt; yMV.backward()
&gt;&gt;&gt; meanMV.grad
tensor([0.])
&gt;&gt;&gt; stdMV.grad
tensor([[-0.5000]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>