Gradients are None Policy Gradient

username: btil
cooked: <p>I wish to update my PolicyNet network based on a loss from two variables not on the graph. So, any idea how I can only update PolicyNet based on a loss that does not require grad? Thanks.</p>
<pre><code class="lang-auto">actions = PolicyNet.forward(state)
loss = criterion(a,b) # a and b have requires_grad=False
loss = Variable(loss, requires_grad=True)
loss.backward()
print(list(PolicyNet.parameters())[0].grad)
optimizer.step()
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You should not call the <code>.forward()</code> function of Modules directly but just call them like: <code>actions = PolicyNet(state)</code>.</p>
<p>If your PolicyNet’s params have requires_grad=True, then <code>actions</code> will also have requires_grad=True and it will propagate all the way to the loss.<br>
So you want to make sure that the PolicyNet params do require grads and that you don’t <code>.detach()</code> (or <code>.data</code>) anything during the loss computation.</p>

username: btil
cooked: <p>Hi thanks for responding. Within RegressNet I am having to return the loss as a Variable with requires_grad=True or else I get the “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”. Using this Variable loss is breaking the graph, it works when I just use the actions of the PolicyNet. I am breaking the computation graph inside RegressNet with a criterion between two variables that do not have grad. I edited my post.</p>

username: albanD
cooked: <aside class="quote no-group" data-username="btil" data-post="1" data-topic="99252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/82dd89/48.png" class="avatar"> btil:</div>
<blockquote>
<p><code>loss = Variable(loss, requires_grad=True)</code></p>
</blockquote>
</aside>
<p>Doing this is 100% wrong. You break the graph and create a new Tensor with no history. So your backward will just stop at <code>loss</code>.</p>
<p>If you properly use the actions that require grads to compute the loss. Then the loss will require gradients naturally and all will work.<br>
You need to make sure you don’t do any op that breaks the link.</p>

username: btil
cooked: <p>Thanks, the variables a and b I am getting the loss with are outputs of the eval network, and so when I try to call .backward() I get the error of no grad or grad_fn.</p>

username: albanD
cooked: <p>Right,<br>
But does a and b depend on actions? If so, they should require gradients.</p>

username: btil
cooked: <p>They depend on actions and are being input to an eval network so I do not know how to make them require gradients and pass them through an eval network at the same time.</p>

username: albanD
cooked: <p>If your eval network is just a net with requires_grad=False. It won’t present the output from requiring gradients if the input does. So if the input to the eval net requires grad, then the output will as well.<br>
And if there is a function that computes these values based on the action, then they should require grad. Unless you detach/unpack Variable which you shouldn’t do.</p>