HF trainer training args: save_only_model does not work together with load_best_model_at_end when using deepspeed
Author: welsh01
Pull request: no

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Linux-5.15.0-25-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.24.1
- PyTorch version (GPU?): 2.1.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@pacman100 

Setup: 
Training via  _examples/pytorch/language-modeling/run_clm.py_ using hf trainer together with deepspeed. When combining the arguments _"save_only_model"_ and  _"load_best_model_at_end"_,  the final checkpoint cannot be loaded.

Error message:

```
transformers/integrations/deepspeed.py", line 409, in deepspeed_load_checkpoint
            raise ValueError(f"Can't find a valid checkpoint at {checkpoint_path}")
```

I guess deepspeed is expecting optimizer and scheduler states that are missing now.

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
deepspeed --num_gpus=4 run_clm.py \
    --deepspeed ~/ds_config_zero2.json \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --output_dir ~/res \
    --validation_split_percentage 10 \
    --num_train_epochs 1 \
    --evaluation_strategy epoch \
    --save_strategy epoch \
    --do_train \
    --do_eval \
    --logging_first_step \
    --save_total_limit 1 \
    --logging_steps 10 \
    --metric_for_best_model eval_loss \
    --greater_is_better False \
    --save_safetensors False \
    --load_best_model_at_end \
    --save_only_model
```


### Expected behavior

Final (in that case best) checkpoint should be loaded successfully, deepspeed checkpoint checks are probably too strict in that case.

Comments:
Hello, thank you for bringing this to the notice. We will look into this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Gently pinging @pacman100 and @muellerzr 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @SunMarc as well 😉 

Hello, please see the above PR for the reason why this isn't possible and the above PR will raise an error when such config args are passed.