Unable to resume LoRA training with PEFT
Author: JohnnyRacer
Pull request: no

### System Info

- `transformers` version: 4.38.1
- Platform: Linux-5.15.0-97-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.21.2
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.0a0+81ea7a4 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import Trainer, DataCollatorForLanguageModeling
from peft import PeftModel, LoraConfig, get_peft_model

#tokenizer and model have already been loaded from pretrained with AutoModelForCausalLM and AutoTokenizer 

model = PeftModel.from_pretrained(model, lora_save_path)  # Load the LoRA from the saved directory, this works correctly and produces the expected generated outputs

trainer = trainer = Trainer(
    model=model,
   data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
   resume_from_checkpoint=True,
    **training_args
)
trainer.train()

### Expected behavior

Hello, I am following [this notebook](https://github.com/huggingface/peft/blob/main/examples/int8_training/Finetune_opt_bnb_peft.ipynb) which correctly when training the LoRA models. However when I try to load a LoRA adapter that I have trained and resume training using the code snippet above, I get the following error: 
```python
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
```
Which forces me to LoRA from scratch everytime if I want to train further steps.

Comments:
cc @pacman100 and @younesbelkada 

Hi @JohnnyRacer 
you need to call `prepare_model_for_kbit_training` whenever you want to re-train your LoRA, can you double check?

@younesbelkada 

I just tried that and now I'm getting a different error, the full traceback pretty long so I just pasted where the error was thrown. Let me know if you need the full traceback and I will post it below. What's very strange is the GPU seems to be busy for few minutes (100% GPU utilization and 95% power limit) as if it were training but then throws the following error. Changing the optimizer does not seem to help as I've tried "adamw_torch", "adamw_bnb_8bit", and "adamw_apex_fused" but they all throw the same error.

Traceback: 
```python
File /usr/local/lib/python3.10/dist-packages/torch/cuda/amp/grad_scaler.py:446, in GradScaler.step(self, optimizer, *args, **kwargs)
    442 if optimizer_state["stage"] is OptState.READY:
    443     self.unscale_(optimizer)
    445 assert (
--> 446     len(optimizer_state["found_inf_per_device"]) > 0
    447 ), "No inf checks were recorded for this optimizer."
    449 retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
    451 optimizer_state["stage"] = OptState.STEPPED

AssertionError: No inf checks were recorded for this optimizer.
```
I am loading the model like this:

```python
from peft import prepare_model_for_kbit_training
from peft import PeftModel
from transformers import AutoModelForCausalLM

peft_save_path = "/models/Llama2-AlpacaGPT4-FT" # Loading from the LoRA directory
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
model = PeftModel.from_pretrained(model, peft_save_path)  # Resume training from path
model = prepare_model_for_kbit_training(model)
```
This the training arguments:
```python
trainer = Trainer(
    model=model,
    train_dataset=lm_datasets["train"],
    args=transformers.TrainingArguments(
        per_device_train_batch_size=8,
        gradient_accumulation_steps=16,
        warmup_steps=100,
        max_steps=800,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        output_dir="/logging/Llama2-AlpacaGPT4-FT",
        auto_find_batch_size=True,
        report_to="tensorboard"
    ),
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)
trainer.train()
```
Everything works fine if I train the LoRA from scratch using `model = get_peft_model(model, config)` with no errors. 

Hmm that's strange, perhaps your adapter weights have corrupted values (e.g. `Nan`), can you double check that?

@younesbelkada  I don't see any `NaN` in weights. The adapter works fine if I load via `PeftModel.from_pretrained` and generates the correct outputs if I call `model.generate(**args)` .e I am saving the LoRA model via `model.save_pretrained` . Seems like this [PR](https://github.com/huggingface/transformers/pull/24274) was suppose to fix resuming training for LoRA but it still has issues for others too.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.