RuntimeError: "element 0 of tensors does not require grad and does not have a grad_fn"

Ouni_Hamza: <p>Hello PyTorch community,</p>
<p>I hope this message finds you well. I am facing an issue while training a comment classification model using PyTorch Lightning with a pre-trained BERT model.</p>
<p>I encountered the following error during the training process:</p>
<p>javascriptCopy code</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>To provide some context, I have already enabled gradients for all parameters of the model using the function <code>enable_gradients(model)</code>. However, the error still persists.</p>
<p>The model I am using is based on the <code>aubmindlab/bert-base-arabertv02-twitter</code> pre-trained model, and I noticed that some weights of the BERT model were not initialized properly upon loading. I have ensured that I am using the latest versions of PyTorch, Transformers, and PyTorch Lightning.</p>
<p>I attempted to pretrain the BERT model on a downstream task before training my specific model, but the error remains unresolved.</p>
<p>I am reaching out to the community for any guidance or insights on how to resolve this issue. I would be more than happy to provide additional details about my training environment, code, and model setup if needed.</p>
<p>Thank you in advance for your support and advice. Your expertise and suggestions are highly appreciated.</p>
<p>Best regards,<br>
Hamza</p>

ptrblck: <p>I’m unsure but your issue might also be related to <a href="https://discuss.pytorch.org/t/fine-tune-model-runtimeerror-element-0-of-tensors-does-not-require-grad-and-does-not-have-a-grad-fn/185067/2">this one</a> where it seems that HF models are either detaching the computation graph or are disabling gradient calculations. I don’t see any updates, so unsure if these issues are already forwarded or fixed.<br>
To debug the issue you could check the <code>.grad_fn</code> of intermediate activations and make sure a valid backward function is returned. If the <code>.grad_fn</code> is set to <code>None</code> the tensor is not attached to a computation graph.</p>

Ouni_Hamza: <p>this is my code :</p>
<p>from pytorch_lightning import Trainer</p>
<p>def enable_gradients(model):<br>
for param in model.parameters():<br>
param.requires_grad = True</p>
<h1><a name="datamodule-1" class="anchor" href="#datamodule-1"></a>datamodule</h1>
<p>ucc_data_module = UCC_Data_Module(train_path, val_path, test_path, attributes=attributes, batch_size=config[‘batch_size’])<br>
ucc_data_module.setup()</p>
<h1><a name="model-2" class="anchor" href="#model-2"></a>model</h1>
<p>model = UCC_Comment_Classifier()</p>
<p>enable_gradients(model)</p>
<h1><a name="trainer-and-fit-3" class="anchor" href="#trainer-and-fit-3"></a>trainer and fit</h1>
<h1><a name="instantiation-of-the-lightning-trainer-4" class="anchor" href="#instantiation-of-the-lightning-trainer-4"></a>Instantiation of the Lightning Trainer</h1>
<p>trainer = Trainer(max_epochs=config[‘n_epochs’], accelerator=‘gpu’, num_sanity_val_steps=1)</p>
<p>try:<br>
trainer.fit(model, ucc_data_module)<br>
torch.save(model.state_dict(), PATH)<br>
except RuntimeError as e:<br>
print(e)</p>