saving model fails with deepspeed 
Author: shahules786
Pull request: no
Labels: name: solved

### System Info

System Info
transformers v4.30.0
python 3.8

There is a bug [here](https://github.com/huggingface/transformers/blob/0b7b4429c78de68acaf72224eb6dae43616d820c/src/transformers/trainer.py#LL2257C59-L2257C59), No `PretrainedModel` does not have `save_checkpoint` method. 
Error trace 

```
Traceback (most recent call last):
  File "funtuner/trainer.py", line 98, in train
    trainer.train()
  File "/nfshome/store03/users/c.scmse/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1540, in train
    return inner_training_loop(
  File "/nfshome/store03/users/c.scmse/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1884, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/nfshome/store03/users/c.scmse/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2196, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/nfshome/store03/users/c.scmse/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2257, in _save_checkpoint
    self.model_wrapped.save_checkpoint(output_dir)
  File "/nfshome/store03/users/c.scmse/venv/lib/python3.8/site-packages/peft/peft_model.py", line 289, in __getattr__
    return getattr(self.base_model, name)
  File "/nfshome/store03/users/c.scmse/venv/lib/python3.8/site-packages/peft/tuners/lora.py", line 206, in __getattr__
    return getattr(self.model, name)
  File "/nfshome/store03/users/c.scmse/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'GPTNeoXForCausalLM' object has no attribute 'save_checkpoint'
```

### Who can help?

@pacman100 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


My code is [here](https://github.com/explodinggradients/Funtuner/blob/main/funtuner/trainer.py)
Run python3 funtuner/trainer.py

- export PYTHONPATH="${PYTHONPATH}:/your-path/Funtuner"
-  please change the log_dir to your folder [here](https://github.com/explodinggradients/Funtuner/blob/c4e66209d5ee276a7eb8caf582435f1eaafbf18f/funtuner/config/config.yaml#L4) also you might want to set log_wandb=False
- `dev-train` branch


### Expected behavior

Please ensure that model training is running atleast 1000 steps without any errors. 

Comments:
Run it on 2  or more GPUs and it is working as expected.
<img width="1439" alt="Screenshot 2023-06-16 at 7 08 58 AM" src="https://github.com/huggingface/transformers/assets/13534540/d2ed6d49-886a-4798-a493-81d1984b1f39">


So, the issue is that the model isn't getting wrapped in DeepSpeedEngine when run on a single GPU. Running on single GPU makes little sense to me with stage 2 because even with offloading, you don't get any considerable vram savings as the optimizer states and gradients in your case will be tiny as you are using PEFT.

As seen above optimizer state is 30MB compared to 11.2GB of Model

It is working for single GPU for the official example scripts. So, some issue with your codebase.

```
cd transformers
export TASK_NAME=mrpc
export CUDA_VISIBLE_DEVICES="0,1"

torchrun --nnodes 1 --nproc-per-node 1 ./examples/pytorch/text-classification/run_glue.py --model_name_or_path bert-base-cased  --task_name $TASK_NAME  --do_train  --do_eval  --max_seq_length 128  --per_device_train_batch_size 16  --learning_rate 5e-5  --num_train_epochs 3  --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --deepspeed ~/transformers/tests/deepspeed/ds_config_zero2.json --save_steps 10 --evaluation_strategy "epoch"
```

Okay, it is also working with your code but you need to use distributed launcher `torchrun`/`accelerate launch`/`deepspeed` instead of just `python`

command I am running:
```
torchrun --nproc-per-node 1 funtuner/trainer.py
```


<img width="1430" alt="Screenshot 2023-06-16 at 7 18 46 AM" src="https://github.com/huggingface/transformers/assets/13534540/91994c8c-4980-4155-97c6-17e05e04aff0">


Marking this as solved. Feel free to close this. Thank you for giving a clear reproducer with correct steps detailed avoiding a lot of back and forth; helping us resolve the issue faster.

Thanks for your reply @pacman100 . But I'm still facing the same issue even with multiple GPUs. Doesn't Deepspeed automatically use all the available GPUs? I even tried with `torchrun` it gets errors on the same line. 

No, it doesn't automatically use all the available GPUs

As seen above, I'm able to save ckpts properly even with a single GPU when launching via torchrun. 

Hey @pacman100, It was a mistake from my side (disk space was full), but the error didn't show up properly. It works fine now. You're the best :)