[peft] model will have no weight with `require_grads=True` if gradient checkpointing is enabled after PeftModel is created
Author: yundai424
Pull request: no

### System Info

peft==0.5.0
transformers==4.33.2
torch==2.0.1

### Who can help?

trainer: @muellerzr and @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

code snippet:

```python
import transformers
import datasets
import trl
from dataclasses import dataclass, field
import peft
import torch
import time
import callbacks


@dataclass
class CustomArguments():
    model_path: str
    data_path: str = field(default="alpaca_data.json")
    max_seq_length: int = field(default=512)
    lora: bool = field(default=False)
    lora_r: int = field(default=32)
    lora_alpha: int = field(default=16)
    lora_dropout: float = field(default=0.1)


def formatting_func(example):
    output_texts = []
    for i in range(len(example['instruction'])):
        output_texts.append(f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{example["instruction"][i]}

### Input:
{example["input"][i]}

### Response:
{example["output"][i]}""")
    return output_texts

def main():
    parser = transformers.HfArgumentParser((transformers.TrainingArguments, CustomArguments))
    training_args, custom_args = parser.parse_args_into_dataclasses()

    dataset = datasets.load_dataset("json", data_files=custom_args.data_path, split=[
        'train'])[0].train_test_split(test_size=0.2, shuffle=True)
    dataset_train, dataset_eval = dataset['train'], dataset['test']

    if torch.distributed.get_rank() == 0:
        print(custom_args, training_args)

    model = transformers.AutoModelForCausalLM.from_pretrained(custom_args.model_path,
            trust_remote_code=True,
            use_cache=False)

    peft_config = peft.LoraConfig(task_type=peft.TaskType.CAUSAL_LM,
                inference_mode=False,
                r=custom_args.lora_r,
                lora_alpha=custom_args.lora_alpha,
                lora_dropout=custom_args.lora_dropout) if custom_args.lora else None

    tokenizer = transformers.AutoTokenizer.from_pretrained(custom_args.model_path)
    tokenizer.pad_token = tokenizer.eos_token

    trainer = trl.SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=dataset_train,
        eval_dataset=dataset_eval,
        formatting_func=formatting_func,
        max_seq_length=custom_args.max_seq_length,
        peft_config=peft_config,
        args=training_args,
    )

    if torch.distributed.get_rank() == 0:
        for name, param in trainer.model.named_parameters():
            print(name, param.requires_grad)

    trainer.train()
    trainer.evaluate()
    eval_results = trainer.evaluate()
    print(f"Evaluation result", eval_results)

    tokenizer.save_pretrained(training_args.output_dir)
    trainer.save_model()

if __name__ == "__main__":
    main()
```

command:
```
torchrun --nnodes=1 --nproc-per-node=$NUM_GPUS src/training.py \
      --model_path <any decoder model such as falcon> \
      --data_path src/alpaca_data.json \
      --bf16 True \
      --num_train_epochs 1 \
      --per_device_train_batch_size 32 \
      --per_device_eval_batch_size 32 \
      --gradient_accumulation_steps 8 \
      --gradient_checkpointing \
      --save_strategy "no" \
      --learning_rate 2e-5 \
      --logging_steps 1 \
      --lora True \
      --output_dir "$OUTPUT_FULL_PATH" \
      --deepspeed src/ds_config_no_offload.json
```

This will result in error saying `[RuntimeError: element 0 of variables does not require grad and does not have a grad_fn](https://discuss.pytorch.org/t/runtimeerror-element-0-of-variables-does-not-require-grad-and-does-not-have-a-grad-fn/11074)`. the reason is that `peft` will call `model.enable_input_require_grads` [here](https://github.com/huggingface/peft/blob/v0.5.0/src/peft/peft_model.py#L118) if it finds the model is enabled with gradient checkpointing. However, **this requires the model to be MANUALLY set with `model.gradient_checkpointing_enable()` before wrapping it with `get_peft_model`**. If the user doesn't explicitly do this before but instead delegates the activation checkpointing enable to `transformers.Trainer` [here](https://github.com/huggingface/transformers/blob/v4.33.1/src/transformers/trainer.py#L1660), then none of the model will require grad.

### Expected behavior

if the model is `PeftModel` and if `args.gradient_checkpointing` is True, `transformers.Trainer` should check if the model has been configured with grad ckpt enabled. If not, it should throw an error with explicit message. 

i.e. something like
```python
if args.gradient_checkpointing and not getattr(model, "is_gradient_checkpointing", True):
    if isinstance(model, PeftModel):
        throw RuntimeError(...)
```

Would be better if this can be automatically gracefully handled as well. 

Comments:
should be fixed by https://github.com/huggingface/transformers/pull/25846/ already