RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
Author: KaifAhmad1
Pull request: no

### System Info

Cuda : 12.1
OS : Windows x64
pip : 24.0
python : 3.10.10
transformers : 4.40.0
bitsandbytes: 0.43.1

### Who can help?

Hi @younesbelkada @muellerz, could you assist me with this issue? Thanks!




### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

``` Python 
# Training Arguments
training_arguments = TrainingArguments(
    output_dir="./results",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=1,
    optim="paged_adamw_32bit",
    save_steps=0,
    logging_steps=25,
    learning_rate=2e-4,
    weight_decay=0.001,
    fp16=True,
    bf16=False,
    max_grad_norm=0.3,
    max_steps=-1,
    warmup_ratio=0.03,
    group_by_length=True,
    lr_scheduler_type="cosine",
    report_to="tensorboard"
)
     

# SFTTrainer Arguments
trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    peft_config=peft_config,
    dataset_text_field='text',
    args=training_arguments,
    tokenizer=tokenizer,
    packing=False,
    max_seq_length=None
)
     
/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:246: UserWarning: You didn't pass a `max_seq_length` argument to the SFTTrainer, this will default to 1024
  warnings.warn(
Map:   0%|          | 0/2774 [00:00<?, ? examples/s]

trainer.train()
```


```
/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:90: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-22-3435b262f1ae> in <cell line: 1>()
----> 1 trainer.train()

/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py in train(self, *args, **kwargs)
    359             self.model = self._trl_activate_neftune(self.model)
    360 
--> 361         output = super().train(*args, **kwargs)
    362 
    363         # After training we make sure to retrieve back the original forward pass method

/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1857                 hf_hub_utils.enable_progress_bars()
   1858         else:
-> 1859             return inner_training_loop(
   1860                 args=args,
   1861                 resume_from_checkpoint=resume_from_checkpoint,

/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   2201 
   2202                 with self.accelerator.accumulate(model):
-> 2203                     tr_loss_step = self.training_step(model, inputs)
   2204 
   2205                 if (

/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
   3145                 scaled_loss.backward()
   3146         else:
-> 3147             self.accelerator.backward(loss)
   3148 
   3149         return loss.detach() / self.args.gradient_accumulation_steps

/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py in backward(self, loss, **kwargs)
   2009             return
   2010         elif self.scaler is not None:
-> 2011             self.scaler.scale(loss).backward(**kwargs)
   2012         else:
   2013             loss.backward(**kwargs)

/usr/local/lib/python3.10/dist-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    520                 inputs=inputs,
    521             )
--> 522         torch.autograd.backward(
    523             self, gradient, retain_graph, create_graph, inputs=inputs
    524         )

/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    264     # some Python versions print out the first line of a multi-line function
    265     # calls in the traceback and some print out the last line
--> 266     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    267         tensors,
    268         grad_tensors_,

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

```

### Expected behavior

finetuning will start!

Comments:
Hi @KaifAhmad1, in order to be able to help we need a full reproducer i.e. something we can copy, paste and run to get the same error. Without the model and dataset (or public equivalents) or peft config there's not much we can do here 

cc @younesbelkada @pacman100 

Hey, @amyeroberts I cannot share the full code here. Here is the link of colab notebook so you can refer. 
https://github.com/KaifAhmad1/code-test/blob/main/Phi_3_Fine_Tuned_on_Indic_Lanuage.ipynb

Hi @KaifAhmad1 
Can you try to force-set `gradient_checkpointing`:

```diff
# Training Arguments
training_arguments = TrainingArguments(
    output_dir="./results",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=1,
    optim="paged_adamw_32bit",
    save_steps=0,
    logging_steps=25,
    learning_rate=2e-4,
    weight_decay=0.001,
    fp16=True,
    bf16=False,
    max_grad_norm=0.3,
    max_steps=-1,
    warmup_ratio=0.03,
    group_by_length=True,
    lr_scheduler_type="cosine",
    report_to="tensorboard",
+   gradient_checkpointing=True
)
     

# SFTTrainer Arguments
trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    peft_config=peft_config,
    dataset_text_field='text',
    args=training_arguments,
    tokenizer=tokenizer,
    packing=False,
    max_seq_length=None
)
   
```

I'm reasonably confident this is because SFTTrainer seems to have changed how it expects args to be passed.

SFT_config must be used.

Also, it seems that peft_config cannot be passed within SFT_config. See [here](https://huggingface.co/docs/trl/main/en/sft_trainer).

It took me a while to track this down but it appeared in an error message.

Kind of annoying if it is this, because a lot of my SFT scripts have now broken (and I didn't have frozen versioning because I was using the dev version of SFT in quite a few cases).

Thanks, @younesbelkada  for fixing it! 

Thanks @RonanKMcGovern for the feedback, indeed the SFTConfig might have broken things .. I will look into that, can you help me providing more details here on what are the silent bugs you faced ? In our CI where we extensively test many things, all seemed green for us, this will help improve our testing infra 
Can you elaborate on why peft_config cannot be passed within SFTConfig ? 

I think this can be closed. As long as one uses `training_arguments = SFTConfig(...`, there are no issues.

I'm getting the same error, when i'm trying to finetune an finetuned model ( the peft version merged with baseline).