Element 0 of tensors does not require grad and does not have a grad_fn

sweets: <p>Hi everybody,</p>
<p>I’ve been trying to debug what is happening but don’t know what’s wrong.</p>
<p>If you need more info let me know.</p>
<p>Regards!</p>
<p>epochs = 10<br>
steps = 0<br>
running_loss = 0<br>
print_every = 5<br>
for epoch in range(epochs):<br>
for inputs, labels in train_loader:<br>
steps += 1</p>
<pre><code>    inputs, labels = inputs.to(device), labels.to(device)

    optimizer.zero_grad()
    logps = model.forward(inputs)
    loss = criterion(logps, labels)
    loss.backward()
    optimizer.step()

    running_loss += loss.item()
    
    if steps % print_every == 0:
        val_loss = 0
        accuracy = 0
        model.eval()
        
        with torch.no_grad():
            for inputs, labels in val_loader:
                inputs, labels = inputs.to(device), labels.to(device)
                logps = model.forward(inputs)
                batch_loss = criterion(logps, labels)
                
                val_loss += batch_loss.item()
                
                ps = torch.exp(logps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.tpye(torch.FloatTensor)).item()
        print('Epoch {}/{}'.format(epoch + 1, epochs))
        print('Train loss: {}'.format(running_loss/print_every))
        print('Val loss: {}'.format(val_loss/len(val_loader)))
        print('Val accuracy: {}'.format(accuracy/len(val_loader)))
        running_loss = 0
        model.train()</code></pre>

ptrblck: <p>Could you post the <code>forward</code> method of your model?<br>
It seems your output gets detached somehow, e.g. by calling <code>detach()</code> directly on a tensor or by leaving PyTorch and using some other library like numpy.<br>
If it’s possible the whole model code would be interesting to see, too.</p>
<p>As a small side note, you shouldn’t call the <code>forward</code> method of your model, but the model directly instead: <code>model(inputs)</code>.</p>

sweets: <p>Hi ptrblck,</p>
<p>I didn’t know that, thanks!</p>
<p>model = models.resnet50(pretrained=True)</p>
<p>for param in model.parameters():<br>
param.requires_grad = False</p>
<p>model.classifier = nn.Sequential(nn.Linear(2048, 1024),<br>
nn.ReLU(),<br>
nn.Linear(1024, 102),<br>
nn.LogSoftmax(dim=1))</p>
<p>criterion = nn.NLLLoss()<br>
optimizer = optim.Adam(model.classifier.parameters(), lr=0.003)<br>
model.to(device)</p>

ptrblck: <p>Thanks for the code.<br>
It looks like you would like to swap the last linear layer of the pretrained ResNet with your <code>nn.Sequential</code> block.<br>
However, resnet does not use <code>self.classifier</code> as its last layer, but <code>self.fc</code>. This also explains the error, since you are currently setting the <code>required_grad</code> flag to <code>False</code> for each layer, while your newly created <code>model.classifier</code> is not being used.</p>
<p>Try to assign your <code>nn.Sequential</code> block to <code>model.fc</code> instead, which should fix the error.</p>

sweets: <p>That makes sense.</p>
<p>It works, thank you for your help!!!</p>

vbcpascal: <p>Hello, I have a similar problem here. And I don’t know how to solve it.</p>
<pre><code class="lang-python">model = models.resnet50(pretrained=True)
num_in_features = model.fc.in_features
cls_num = 5
model.fc.out_features = cls_num

for param in model.parameters():
    param.requires_grad = False

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)

best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0

phase = 'train'
for epoch in range(30):
    print('Epoch {}/{}'.format(epoch+1, 30))
    print('-' * 10)

    # Each epoch has a training and validation phase
    for phase in ['train', 'valid']:
        if phase == 'train':
            model.train()  # Set model to training mode
        else:
            model.eval()   # Set model to evaluate mode

        running_loss = 0.0
        running_corrects = 0

        # Iterate over data.
        for inputs, labels in dataloaders[phase]:
            inputs = inputs.to(device)
            labels = labels.to(device)
            
            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    #sched.step()
                    loss.backward()
                    optimizer.step()
                    ...
</code></pre>
<p>and it shows</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-11-4b93aa6367ee&gt; in &lt;module&gt;
      1 epochs = 40
      2 model.to(device)
----&gt; 3 model = train_model(model, criterion, optimizer, epochs)

&lt;ipython-input-10-608674d4de7a&gt; in train_model(model, criterion, optimizer, num_epochs)
     37                     if phase == 'train':
     38                         #sched.step()
---&gt; 39                         loss.backward()
     40                         optimizer.step()
     41 

D:\Anaconda\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    105                 products. Defaults to ``False``.
    106         """
--&gt; 107         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    108 
    109     def register_hook(self, hook):

D:\Anaconda\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 93         allow_unreachable=True)  # allow_unreachable flag
     94 
     95 

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>many thanks</p>

ptrblck: <p>You are setting all parameters to <code>requires_grad = False</code>, thus freezing the model.<br>
If you want to train (some) parameters, you should set <code>requires_grad = True</code> for them (or just skip them as this is the default).</p>

vbcpascal: <p>It works. I had a wrong understanding of <code>requires_grad</code>, thanks for your help!</p>

vijaytida: <p>I’m using vgg16 model. I got this error from the block of code<br>
num_ftrs = model_ft.classifier[0].out_features<br>
model_conv.fc = nn.Linear(num_ftrs, 2)</p>
<p>model_conv = model_conv.to(device)</p>
<p>criterion = nn.CrossEntropyLoss()</p>
<h1>Observe that only parameters of final layer are being optimized as</h1>
<h1>opposed to before.</h1>
<p>optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)</p>
<h1>Decay LR by a factor of 0.1 every 7 epochs</h1>
<p>exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)</p>
<p>######################################################################</p>
<h1>Train and evaluate</h1>
<h1>^^^^^^^^^^^^^^^^^^</h1>
<h1></h1>
<h1>On CPU this will take about half the time compared to previous scenario.</h1>
<h1>This is expected as gradients don’t need to be computed for most of the</h1>
<h1>network. However, forward does need to be computed.</h1>
<h1></h1>
<p>model_conv = train_model(model_conv, criterion, optimizer_conv,<br>
exp_lr_scheduler, num_epochs=25)</p>
<p>######################################################################</p>
<h1></h1>
<p>visualize_model(model_conv)</p>
<p>plt.ioff()<br>
plt.show()<br>
RuntimeError                              Traceback (most recent call last)</p>
<p> in ()<br>
24<br>
25 model_conv = train_model(model_conv, criterion, optimizer_conv,<br>
—&gt; 26                          exp_lr_scheduler, num_epochs=25)<br>
27<br>
28 ######################################################################</p>
<p>2 frames</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
91     Variable._execution_engine.run_backward(<br>
92         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 93         allow_unreachable=True)  # allow_unreachable flag<br>
94<br>
95</p>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn<br>
Can you please help me how to solve this issue?<br>
Thanks</p>

ptrblck: <p>Could you post the code of <code>train_model</code>?<br>
Maybe the output was detached accidentally.</p>
<p>PS: You can add code snippets by wrapping them in three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

vijaytida: <p>I used the example code mentioned in the tutorial except the model I changed to vgg16<br>
Here is the link<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a><br>
I changed the line</p>
<p>model_ft = models.vgg16(pretrained=True)<br>
-&gt;num_ftrs = model_ft.classifier[0].out_features<br>
model_ft.fc = nn.Linear(num_ftrs, 2)</p>
<p>model_ft = model_ft.to(device)<br>
Do I need to change any other lines to run VGG16 model ?</p>
<p>Do I need to change anoth</p>

ptrblck: <p><code>vgg16</code> uses <code>.classifier</code> as the last classification block, not <code>.fc</code>.<br>
Have a look at the architecture using <code>print(model_ft)</code>:</p>
<pre><code class="lang-python">...
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)
</code></pre>
<p>You should thus change the code to:</p>
<pre><code class="lang-python">model_ft.classifier[6] = nn.Linear(4096, 2)
</code></pre>

HamedH25: <p>Hi<br>
I have a same issue and I haven’t put any “requies_grad=false”. would you please advise about my codes.</p>
<pre><code># Model.
model = my_model()


criterium = nn.MSELoss()

# Adam optimizer with learning rate 0.1 and L2 regularization with weight 1e-4.
optimizer = torch.optim.Adam(model.parameters(),lr=0.1, weight_decay=1e-4)
# Set gradient to 0.
optimizer.zero_grad()

# Feed forward.
pred = model(data)
 _, output = torch.max(pred, 1)
output = output.type(torch.FloatTensor)

# Loss calculation.
loss = criterium(output, target)


# Gradient calculation.
loss.backward()

# Print loss every 10 iterations.
if k % 10 == 0:
    print('Loss {:.4f} at iter {:d}'.format(loss.item(), k))

# Model weight modification based on the optimizer.
optimizer.step()</code></pre>

ptrblck: <p>You are detaching <code>output</code> from the computation graph by assigning it to the max. index here:</p>
<pre><code class="lang-python"> _, output = torch.max(pred, 1)
</code></pre>
<p>Could you explain your use case a bit?<br>
Would it be possible  to reformulate your loss function, e.g. to return a single output value in your model and pass it the <code>MSELoss</code>?</p>

HamedH25: <p>Actually, It was a example code for my original code. I want to define a custom loss function in my original code but I couldn’t. Thus, I have tried to manipulate the loss function in a simple classification problem to solve the issue. may I put the original codes here? I need your kind support regarding the loss function. however, it is too long.</p>

ptrblck: <p>Maybe creating a new topic would be a good idea, as the original issue should be answered. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

poojitharamachandra: <p>hi,</p>
<p>i have the same error! but i dont want to train my model from scratch. i want to use the pretrained model as it is expect only for the fc layer . please let me know how to solve this</p>

ptrblck: <p>Did you freeze some parameters by setting their <code>requires_grad</code> attribute to <code>False</code>?<br>
If so, make sure that your last layer, which you would like to train, contains parameters which require gradients.<br>
If you get stuck, could you post a code snippet so that we can have a look?</p>

poojitharamachandra: <p>ah! i was wrongly freezing the parameters of fc also i guess.</p>
<pre><code>    self.model = models.resnet50(pretrained=True)
    for param in self.model.parameters():
      param.requires_grad = False
    # 30 classes
    num_ftrs = self.model.fc.in_features
    self.model.fc = nn.Linear(num_ftrs, 30)
</code></pre>
<p>this works</p>

tinhchuquang: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>. I use Lovasz_highe (link <a href="https://github.com/bermanmaxim/LovaszSoftmax/blob/master/pytorch/lovasz_losses.py" rel="nofollow noopener">https://github.com/bermanmaxim/LovaszSoftmax/blob/master/pytorch/lovasz_losses.py</a>  )<br>
I have problem RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn<br>
I change function with no error, can you fix error of Lovasz_loss in pytorch 1.12.0. Thanks</p>