Deepspeed Integration multi-gpu example does not work as written 
Author: sei-amellinger
Pull request: no

### System Info

ds_report shows this 

```
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 2.0.1+cu117
deepspeed install path ........... ['/opt/conda/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.9.2, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.6
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7
```

I am running in a container built by:

```
FROM nvcr.io/nvidia/pytorch:22.04-py3

RUN pip install  git+https://github.com/huggingface/peft.git
RUN pip install "transformers==4.29.1" "datasets==2.9.0" "accelerate==0.19.0" "evaluate==0.4.0"  loralib --upgrade --quiet
RUN pip install bitsandbytes rouge-score tensorboard py7zr einops
RUN pip install deepspeed --upgrade
RUN pip install jupyter
RUN pip uninstall -y apex
RUN pip uninstall -y apex
```

And then adding in the transformers and examples as described when running examples.

I am running this on a system with 2 x 40G A100s.


### Who can help?

@pacman100 
@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am following the deepspeed integration example here:

https://huggingface.co/docs/transformers/main/en/main_classes/deepspeed

I have a latest clone of transformers have installed transformers and the pytorch examples a described in the examples documentation.  Then I run this snippet from the example:

```
deepspeed examples/pytorch/translation/run_translation.py \
--deepspeed tests/deepspeed/ds_config_zero3.json \
--model_name_or_path t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
--dataset_name wmt16 --dataset_config "ro-en" \
--source_lang en --target_lang ro
```

When I do I get the complaint:

```
Traceback (most recent call last):
  File "examples/pytorch/translation/run_translation.py", line 666, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 581, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1661, in train
    return inner_training_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1788, in _inner_training_loop
    model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
  File "/opt/conda/lib/python3.8/site-packages/accelerate/accelerator.py", line 1139, in prepare
    result = self._prepare_deepspeed(*args)
  File "/opt/conda/lib/python3.8/site-packages/accelerate/accelerator.py", line 1360, in _prepare_deepspeed
    raise ValueError(
ValueError: You cannot specify an optimizer in the config file and in the code at the same time. Please remove the optimizer from the config file or create `accelerate.utils.DummyOptim` in the code.
```

I tried removing the optimizer stanza, then it complains about bits in the scheduler, etc. At that point I felt like I was in the weeds. I suspect there is a better ds_config for this reference. Basically, I am trying to run a basic deepspeed test to verify my installation.

This is using the unmodified config and the run_translation.

### Expected behavior

That the example works as described in the documentation,

Comments:
Looking into this

hello, I'm unable to reproduce the issue when using deepspeed 0.9.3, transformers pr #23914  and main branch of accelerate

Awesome, I'll try updating to those versions and see if I can get it working.
Thanks!

I am using these versions and I still get:

`ValueError: You cannot specify an optimizer in the config file and in the code at the same time. Please remove the optimizer from the config file or create `accelerate.utils.DummyOptim` in the code.`

I am using these versions.

accelerate: 0.20.0.dev0
deepspeed: 0.9.3+e02b8d0b
transformers: 4.30.0.dev0

Am I missing another lib?

Hello, with exact same library versions, I am able to run the above `run_translation.py` example. Are you making any changes to the example code?

Command:
```
deepspeed examples/pytorch/translation/run_translation.py --deepspeed tests/deepspeed/ds_config_zero3.json --model_name_or_path t5-small --per_device_train_batch_size 1 --output_dir output_dir --overwrite_output_dir --fp16 --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro
```

```
accelerate               0.20.0.dev0      /home/sourab/accelerate
deepspeed                0.9.3+e02b8d0b   /home/sourab/DeepSpeed
transformers             4.30.0.dev0      /home/sourab/transformers
```
![Screenshot 2023-06-01 at 8 55 34 PM](https://github.com/huggingface/transformers/assets/13534540/b673f397-4ed6-4d82-b675-816a31e2f254)


After cleaning/reinstalling I can confirm it works!
Thanks for the quick turnaround. :)