Bug in Trainer for PeftModel multi-gpu training by accelerate
Author: Orion-Zheng
Pull request: no

### System Info

When using the accelerate library to fine-tune LLaMA2 by QLoRA on multiple GPUs, I met an error when I set load_best_model_at_end=True in the TrainingArguments. 
<img width="1115" alt="image" src="https://github.com/huggingface/transformers/assets/62330719/a2e7a49b-3627-4249-8fb2-bc8a61f0ede1">
It seems the function '_load_best_model' try to find the weight of the complete model under the peft model's checkpoint dir, but actually there is only adapter's weight saved in this directory.
<img width="350" alt="image" src="https://github.com/huggingface/transformers/assets/62330719/ecbd2f7c-8230-4e63-8265-11ba790e03e7">
I try to find the reason behind the error. I notice that the best peft model should be loaded by these lines of code in '_load_best_model',
<img width="1273" alt="image" src="https://github.com/huggingface/transformers/assets/62330719/e06fbcee-035d-458a-b348-82f783d1a073">
but in the multi-gpu training, the trainer execute another code, which is not intended for the peft model's checkpoint.
<img width="1110" alt="image" src="https://github.com/huggingface/transformers/assets/62330719/21bf760b-a208-4a91-b718-d6aec13dbf39">
But why the correct code is skipped? I finally found the reason.
In multi-GPU code, the Peft Model was wrapped into DistributedDataParallel that looks like:
DistributedDataParallel( 
  (module): PeftModelForCausalLM(
which makes 'model' is not a PeftModel anymore, so this condition can't be satisfied and the correct code is skipped
<img width="1113" alt="image" src="https://github.com/huggingface/transformers/assets/62330719/52035c34-5c19-4bfd-a68c-83a7b542ca82">
To solve this problem, I add some code to this function. I know this is not a elegant way, but it works for me.
<img width="1106" alt="image" src="https://github.com/huggingface/transformers/assets/62330719/3bbbb438-4b69-46c8-9c48-a7dbd1dcd956">
Besides, I think the original Trainer didn't consider training PeftModel on multiple GPUs well, which also lead to other bugs in the Trainer.save_model when doing multi-gpu training for peft model. Because I don't have enough time to read all of the code of Trainer, I can't solve this problem by myself. So could anyone work on this? Thank you very much!😃

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Transformer Version: 4.31.0
(sorry, my code consists of several files. I'll attach them to this issue once I've organized them into one.)

### Expected behavior

The PeftModel can be saved and load correctly by the Trainer during multi gpu training.

Comments:
cc @pacman100 :)

Hey! Would recommend you to update to the latest version of both packages and see if this is fixed! 

Hello, 

I think @younesbelkada should have a better idea about this.

Thanks! Will look into it

Hi @Orion-Zheng 
Thanks for the issue, I was not able to reproduce the issue with the newest transformers version (from source), it seems the model gets un-wrapped from DDP before the call to `_load_best_model()`. 

Here is the snippet I used:

```python
from datasets import load_dataset
from transformers import AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer
from accelerate import PartialState

from peft import LoraConfig

dataset = load_dataset("timdettmers/openassistant-guanaco", split="train[:1%]")

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

args = TrainingArguments(
    output_dir="./out-test",
    max_steps=2,
    load_best_model_at_end=True,
    save_steps=1,
    save_strategy="steps",
    evaluation_strategy="steps",
    eval_steps=1,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1
)

model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/gpt-neo-125m",
    load_in_4bit=True,
    device_map={"": PartialState().process_index},
)

trainer = SFTTrainer(
    model,
    args=args,
    train_dataset=dataset,
    eval_dataset=dataset,
    dataset_text_field="text",
    peft_config=peft_config,
)

trainer.train()
```
Can you try your script with the latest versions of transformers and accelerate? `pip install -U transformers accelerate`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.