Loss.backward(): element 0 of tensors does not require grad and does not have a grad_fn

MidnightChaos: <p>Hey all,</p>
<p>I’m a beginner experimenting with resnet50 transfer learning, and I’ve been getting the runtime error “element 0 of tensors does not require grad and does not have a grad_fn” when attempting to do a training run. I’m doing this in a google colab environment, and here’s the code:</p>
<pre data-code-wrap="python"><code class="lang-python">train_session_epochs = 12

with torch.enable_grad():
  for epoch in range(train_session_epochs):
    running_loss = 0.0
    cur_epochs = cur_epochs + 1

    for inputs, labels in trainloader:
      i = i+1

      inputs = inputs.to(device)
      labels = labels.to(device)
      
      optimizer.zero_grad()

      outputs = model(inputs)
      loss = criterion(outputs, labels)
      loss.backward()
      optimizer.step()
</code></pre>
<p>Criterion is cross entropy loss, and I’m using SGD as the optimizer. I’ve managed to identify that the issue is that <code>loss.requires_grad</code> for torch is false, and I’ve tried setting <code>loss = Variable(criterion(outputs, labels), requires_grad=True)</code>, and using other methods to force requires_grad to be true, however, those are only workarounds that make the error go away and not result in loss reduction.</p>
<p>The strange thing is that I have a separate colab with nearly the exact same setup of training data, optimizer, model, etc., and that colab does not throw any errors and results in learning progress when I use it to train. The primary difference being that this colab file that I’m currently working on is attempting to use <code>optimizer.load_state_dict</code> when loading the model to train further.</p>
<p>Am I missing anything? I’ll be happy to provide more code if needed.</p>

ptrblck: <p>Did you freeze all trainable parameters or disabled gradient computation globally? If not, could you post a minimal and executable code snippet reproducing the issue?</p>

MidnightChaos: <p>I don’t think I have frozen anything, and I haven’t run <code>torch.set_grad_enabled(False)</code>. The only possible place where I have gradient computation disabled is when I’m checking the accuracy where I run it <code>with torch.no_grad()</code>.</p>
<p>Here’s my notebook: <a href="https://colab.research.google.com/drive/1w-_N1ogvHzwxGOJpkAKJwefw-mTsP1vP?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

ptrblck: <p>In your code you are freezing all parameters in:</p>
<pre data-code-wrap="python"><code class="lang-python">for param in model.parameters():
  param.requires_grad=False
</code></pre>
<p>However, you are replacing the <code>.fc</code> layer afterwards. Did you execute the cells in a different order by mistake as I cannot reproduce the issue locally?</p>

MidnightChaos: <p>I see. After changing requires_grad to True, it works like normal. I believe I froze the layers some time ago because of a CUDA out of memory error kept getting thrown during training runs. Thanks for the help!</p>

vietvo: <p>I got the same error with my code here <a href="https://www.kaggle.com/code/vietvo89/finetune-open-clip-torch-peft-randlora" class="inline-onebox" rel="noopener nofollow ugc">Finetune_open_clip_torch-peft-RandLoRA | Kaggle</a>. I tried to fine-tune open-clip but not sure why I got the error although I enabled requires_grad of some parameters. It’s great if you can give me a hand and tell me what’s wrong with my code. Thanks</p>

ptrblck: <p>Check if intermediate activations show a valid <code>.grad_fn</code> to isolate if and where you might have detached the computation graph.</p>