Getting element 0 error while fine tuning t5 llm

sank: <p>This is the error I am getting<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn.</p>
<p>I am trying to fine tune the t5 llm model here.<br>
The Model:</p>
<pre><code class="lang-auto">class LModel(pl.LightningModule):
    def __init__(self):
        super(LModel, self).__init__()
        self.model = T5ForConditionalGeneration.from_pretrained("t5-base", return_dict=True)
  
    def forward(self, input_ids, attention_mask, labels=None, decoder_attention_mask=None):
        outputs = self.model(input_ids=input_ids,
                        attention_mask=attention_mask,
                        labels=labels,
                        decoder_attention_mask=decoder_attention_mask)
        return outputs.loss, outputs.logits
  
    def training_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["summary_ids"]
        decoder_attention_mask = batch["summary_mask"]

        loss, output = self(input_ids, attention_mask, labels, decoder_attention_mask)
        return loss

    def validation_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["summary_ids"]
        decoder_attention_mask = batch["summary_mask"]

        loss, output = self(input_ids, attention_mask, labels, decoder_attention_mask)
        return loss
 
    def test_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        loss, output = self(input_ids=input_ids, 
                      attention_mask=attention_mask)
        return loss
    
    def configure_optimizers(self):
        optimizer = AdamW(self.model.parameters(), lr=0.0001)
        scheduler = get_linear_schedule_with_warmup(
                optimizer, num_warmup_steps=0,
                num_training_steps=EPOCHS*len(df))
        return {'optimizer': optimizer, 'lr_scheduler': scheduler}
</code></pre>
<p>And the trainer looks like this:</p>
<pre><code class="lang-auto">device  = 'cuda' if torch.cuda.is_available() else "cpu"
trainer = pl.Trainer(
    max_epochs=EPOCHS,
    accelerator=device
)


trainer.fit(model,module)
</code></pre>
<p>can anyone let me know why am I getting this element 0 error and how to fix it?</p>

ptrblck: <p>This error is raised if you are detaching a tensor in the <code>forward</code> pass explicitly, if you are using non-differentiable operations, or if you are disabling gradient calculations globally.<br>
I don’t know how the <code>forward</code> method is implemented in your model but you could <code>print</code> the <code>.grad_fn</code> attribute of each intermediate tensor to check if a valid backward method is returned to isolate where the detaching takes place.</p>

sank: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I tried printing the grad_fn :</p>
<pre><code class="lang-auto">def forward(self, input_ids, attention_mask, labels=None, decoder_attention_mask=None):
        outputs = self.model(input_ids=input_ids,
                        attention_mask=attention_mask,
                        labels=labels,
                        decoder_attention_mask=decoder_attention_mask)
        print("outputs.logits.grad_fn:",outputs.logits.grad_fn)
        print("total params reuires grad",sum(p.numel() for p in self.model.parameters() if p.requires_grad))
        print("model.training:",model.training)
        return outputs.loss, outputs.logits
</code></pre>
<p>I got this output:</p>
<pre><code class="lang-auto">outputs.logits.grad_fn: None
total params reuires grad 222903552
model.training:True
</code></pre>

ptrblck: <p>What does <code>output.loss.grad_fn</code> return? Maybe the <code>.logits</code> were explicitly detached but the <code>.loss</code> is still differentiable?<br>
If the <code>.loss.grad_fn</code> is also set to <code>None</code> you would need to check the model’s <code>forward</code> pass.</p>

sank: <p>outputs.logits.grad_fn: None<br>
outputs.loss.grad_fn: None</p>
<p>Yes the loss grad_fn is also shown as None.</p>
<p>Its strange, because I am not deliberately changing anything inside the model, and directly using the pretrained version.</p>

ptrblck: <p>Yes, it’s indeed strange and you would thus need to check the internals of this model and how it was implemented as either the model or script might disable gradient calculations globally or via <code>with torch.no_grad()</code> for some reason that I’m unaware of.</p>

sank: <p>yes, I rewrote the code in PyTorch and it worked fine, maybe some issue with PyTorch lightning wrapper.</p>

ptrblck: <p>To debug it further, you could use <code>print(torch.is_grad_enabled())</code> to see if something disabled gradient calculation (by mistake).</p>

sank: <p>sure <a class="mention" href="/u/ptrblck">@ptrblck</a> , will try it. Thanks a lot, I will update the thread with solution if found.</p>