Gradient w.r.t. input image is None

G.Elham: <p>Hi everybody,</p>
<p>I am trying to return the gradient of an output score w.r.t. the input image. While I am setting requires_grad()=True, the gradient is None. Here are two equivalent codes to get the gradients, Code1 raises the error <strong>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior</strong>, and Code2 returns None. Does anybody have some ideas about what I missed?</p>
<p>Code1:<br>
net_d.eval()<br>
im = images[0, …].reshape(1, images.shape[1], images.shape[2], images.shape[3]).clone().cuda()<br>
im.requires_grad_()<br>
d_x = net_d(im)<br>
d_x = torch.max(torch.sigmoid(d_x))<br>
d_x.requires_grad_()<br>
grad_1 = torch.autograd.grad(d_x, im, create_graph=True)<br>
print(grad_1)</p>
<p>Code2:<br>
net_d.eval()<br>
im = images[0, …].reshape(1, images.shape[1], images.shape[2], images.shape[3]).clone().cuda()<br>
im.requires_grad_()<br>
d_x = net_d(im)<br>
d_x = torch.max(torch.sigmoid(d_x))<br>
out = Variable(d_x, requires_grad=True)<br>
out.backward(retain_graph=True)<br>
print(im.grad)</p>

ptrblck: <p>Your first approach works fine and you wouldn’t need to call <code>.requires_grad_()</code> on the output:</p>
<pre><code class="lang-python">im = torch.randn(1, 1, device='cuda')
im.requires_grad_()
net_d = nn.Linear(1, 1).cuda()

d_x = net_d(im)
d_x = torch.max(torch.sigmoid(d_x))
grad_1 = torch.autograd.grad(d_x, im, create_graph=True)
print(grad_1)
# (tensor([[-0.0331]], device='cuda:0', grad_fn=&lt;TBackward0&gt;),)
</code></pre>
<p>assuming you are not detaching the computation graph.</p>
<p>If case you get an error after removing <code>d_x.requires_grad_()</code> then check where the computation graph is detached in your model (e.g by rewrapping an intermediate activation into a new tensor).</p>

G.Elham: <p>Thanks for the quick response <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<p>I loaded the model state_dict a line before net_d.eval():<br>
‘net_d.load_state_dict(torch.load(PATH))’<br>
Removing d_x.requires_grad_() raises error ‘element 0 of tensors does not require grad and does not have a grad_fn’. Shoud I calculate the saliency map from a model which weights are loaded by a saved state_dict weights in another way?</p>

ptrblck: <p>No, loading the <code>state_dict</code> should work and the error points to a detached computation graph.<br>
Could you post the model definition as well as the input shape you are using so that I could reproduce the issue, please?</p>

G.Elham: <p>Sure, the model has been defined on src/model.py on the github page ‘<a href="https://github.com/ricbl/eye-tracking-localization" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ricbl/eye-tracking-localization: This repository contains code for the paper "Localization supervision of chest x-ray classifiers using label-specific eye-tracking annotation".</a>’. The input size is [bsx3x512x512]. The function ‘get_highlight(image, self.opt.get_saliency)’ has been written in a separate script from train.py; Please let me know if you need further information or you need the checkpoint.</p>
<p>import torch<br>
import numpy as np<br>
import torchvision.transforms as transforms # import ToPILImage<br>
from torch.autograd import Variable<br>
import opts<br>
opt = opts.get_opt()<br>
import model<br>
from torch.autograd import Variable</p>
<p>def get_highlight(images, method):<br>
net_d = model.Thoracic(opt.grid_size, pretrained=opt.use_pretrained, calculate_cam=opt.calculate_cam, last_layer_index=opt.last_layer_index).cuda()     net_d.load_state_dict(torch.load(PATH))<br>
net_d.eval()<br>
im = images[0, …].reshape(1, images.shape[1], images.shape[2], images.shape[3]).clone().cuda()<br>
im.requires_grad_()<br>
d_x = net_d(im)<br>
d_x = torch.max(torch.sigmoid(d_x))<br>
out = Variable(d_x, requires_grad=True)<br>
out.backward(retain_graph=True, requires_grad=True)<br>
saliency, _ = im.grad.data.abs()</p>

ptrblck: <p>I’m unable to reproduce the issue using:</p>
<pre><code class="lang-python">model = Thoracic().cuda()
bs = 2
x = torch.randn(bs, 3, 512, 512).cuda().requires_grad_()

out = model(x)
loss = out[0].mean() + out[1].mean()

grad_1 = torch.autograd.grad(loss, x, create_graph=True)
print(grad_1)
</code></pre>
<p>and get a valid gradient. Could you check what the difference between the codes could be?</p>

G.Elham: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="165404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">model = Thoracic().cuda()
bs = 2
x = torch.randn(bs, 3, 512, 512).cuda().requires_grad_()

out = model(x)
loss = out[0].mean() + out[1].mean()

grad_1 = torch.autograd.grad(loss, x, create_graph=True)
print(grad_1)
</code></pre>
</blockquote>
</aside>
<p>Running your code I’m getting this error ‘RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn’.</p>

ptrblck: <p>Did you disable the gradient calculation globally by chance? E.g. via <code>torch.autograd.set_grad_enabled(False)</code>?</p>

G.Elham: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="165404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>set_grad_enabled</p>
</blockquote>
</aside>
<p>I did, I’m sorry for not concidering that. Thanks a lot for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>