seems to be a bug related to saving model
Author: jeffchy
Pull request: no

### System Info

I use pytorch==2.0 fsdp fully-shard
If I use transformers==4.29.1, accelerate==0.19.0,  things works well:
```
[INFO|trainer.py:2904] 2023-06-09 10:35:25,236 >> Saving model checkpoint to ../outputs/tigerbot-7b/full/2023-06-09-10-33-49/ckpt/checkpoint-4
[INFO|configuration_utils.py:458] 2023-06-09 10:35:25,237 >> Configuration saved in ../outputs/tigerbot-7b/full/2023-06-09-10-33-49/ckpt/checkpoint-4/config.json
[INFO|configuration_utils.py:364] 2023-06-09 10:35:25,237 >> Configuration saved in ../outputs/tigerbot-7b/full/2023-06-09-10-33-49/ckpt/checkpoint-4/generation_config.json
```
When I switch to transformers==4.30 accelerate==0.20.0 when I save the model, I got the following error
```
│    285                                                                                           │
│    286 class _open_zipfile_writer_file(_opener):                                                 │
│    287 │   def __init__(self, name) -> None:                                                     │
│ ❱  288 │   │   super().__init__(torch._C.PyTorchFileWriter(str(name)))                           │
│    289 │                                                                                         │
│    290 │   def __exit__(self, *args) -> None:                                                    │
│    291 │   │   self.file_like.write_end_of_file()                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Parent directory ../outputs/tigerbot-7b/full/2023-06-09-10-21-30/ckpt/checkpoint-4 does not exist.
```

It seems like, when I save fsdp model, transformers/accelerator don't help me to create the parent folder 'xxxx/checkpoint-4'. When I downgrade the transformers and the accelerate's version, it works, and when I manually create the 'xxx/checkpoint-4' before saving, it also works.

### Who can help?

 @sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

pytorch==2.0
transformers==4.30.0
accelerate==0.20.3

Trainer using FSDP fully shard, modified from train_clm.py example
```
CUDA_VISIBLE_DEVICES=0,1,2,3,7 $BASE_ENV/torchrun --nproc_per_node 5 --nnodes=1 --node_rank=0 --master_port $MASTER_PORT main_sft.py \
    --model_name_or_path $MODEL \
    --model_type $MODEL_TYPE \
    --dataset_config_file config/data/tiger.yaml \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --do_train \
    --do_eval \
    --output_dir $OUTPUT_DIR \
    --fp16 \
    --cutoff_len 2048 \
    --save_steps 500 \
    --logging_steps 50 \
    --max_steps 6000 \
    --eval_steps 500 \
    --warmup_steps 5 \
    --gradient_accumulation_steps 32 \
    --lr_scheduler_type "cosine" \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'BloomBlock' \
    --gradient_checkpointing True \
    --overwrite_cache \
    --learning_rate 1e-5 \
    | tee $LOG_DIR/train.log \
    2> $LOG_DIR/train.err
```

### Expected behavior

I use pytorch==2.0 fsdp fully-shard
If I use transformers==4.29.1, accelerate==0.19.0,  things works well:
```
[INFO|trainer.py:2904] 2023-06-09 10:35:25,236 >> Saving model checkpoint to ../outputs/tigerbot-7b/full/2023-06-09-10-33-49/ckpt/checkpoint-4
[INFO|configuration_utils.py:458] 2023-06-09 10:35:25,237 >> Configuration saved in ../outputs/tigerbot-7b/full/2023-06-09-10-33-49/ckpt/checkpoint-4/config.json
[INFO|configuration_utils.py:364] 2023-06-09 10:35:25,237 >> Configuration saved in ../outputs/tigerbot-7b/full/2023-06-09-10-33-49/ckpt/checkpoint-4/generation_config.json
```
When I switch to transformers==4.30 accelerate==0.20.0 when I save the model, I got the following error
```
│    285                                                                                           │
│    286 class _open_zipfile_writer_file(_opener):                                                 │
│    287 │   def __init__(self, name) -> None:                                                     │
│ ❱  288 │   │   super().__init__(torch._C.PyTorchFileWriter(str(name)))                           │
│    289 │                                                                                         │
│    290 │   def __exit__(self, *args) -> None:                                                    │
│    291 │   │   self.file_like.write_end_of_file()                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Parent directory ../outputs/tigerbot-7b/full/2023-06-09-10-21-30/ckpt/checkpoint-4 does not exist.
```

It seems like, when I save fsdp model, transformers/accelerator don't help me to create the parent folder 'xxxx/checkpoint-4'. When I downgrade the transformers and the accelerate's version, it works, and when I manually create the 'xxx/checkpoint-4' before saving, it also works.

Comments:
cc @pacman100 

Hello @jeffchy, Thank you for the thorough issue, can you please confirm if the above PR resolves your issue?

still not work in transformers==4.42.4, accelerate==0.32.1

@00ffcc Could you share a reproducible code snippet and more details about the error?

> @00ffcc Could you share a reproducible code snippet and more details about the error?

![image](https://github.com/user-attachments/assets/bd67a625-5ed2-4e19-bde1-1595a4f64ef1)
it is tested on accelerate==0.26.1, transformers==4.37.2

@00ffcc Please share copy-pasted text instead of screenshots. This makes debugging easier for us, as we can copy the snippet directly and ensures that the information is searchable. 

With regards to the issue, the error is indicating that you're trying to save the checkpoint into a directory `"./ck` which doesn't exist

> @00ffcc Please share copy-pasted text instead of screenshots. This makes debugging easier for us, as we can copy the snippet directly and ensures that the information is searchable.
> 
> With regards to the issue, the error is indicating that you're trying to save the checkpoint into a directory `"./ck` which doesn't exist

ok, here is the code.
```python
import torch
import accelerate
model = torch.nn.Linear(10, 10)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
accelerator = accelerate.Accelerator()
model, optimizer = accelerator.prepare(model, optimizer)
accelerator.save(model.state_dict(), r"./ck/my_model.pt")
```