deepspeed multi-gpu inference
Author: liuchengyuan123
Pull request: no

### System Info

I'm using transformers.Trainer with deepspeed. During training, Zero 2 is adopted. For evaluation, I just want to accelerate with multi-GPU inference like in normal DDP, while deepspeed raises ValueError: "ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config".
I find that in evaluation it works, but direct calling "trainer.predict" after initialize the trainer would cause the above error.

I understand the error, that means optimizer and scheduler of ZeRO 2 is not optimized during train, but is there an approach that allows multiple-GPUs inference given configure of ZeRO 2?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
deepspeed test.py ... --deepspeed deepspeed.json
```
where deepspeed.json is a configure file of ZeRO 2, and `test.py` is the code to call `trainer.predict` after initialize trainer.

### Expected behavior

ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config

Comments:
cc @pacman100 

Hello, I don't understand what the issue is here. As the error suggests, DeepSpeed ZeRO only makes sense for prediction if it is stage 3 because the parameters would get sharded. DeepSpeed Stage 2 would not be leading to any memory savings and in that case it would be better to not use DeepSeed at all as it would be same as DDP inference.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Hello, I don't understand what the issue is here. As the error suggests, DeepSpeed ZeRO only makes sense for prediction if it is stage 3 because the parameters would get sharded. DeepSpeed Stage 2 would not be leading to any memory savings and in that case it would be better to not use DeepSeed at all as it would be same as DDP inference.

Hi @pacman100 , I ran into same problem here.
I want model to evaluate before my train process, so I add` trainer.evaluate()` before `trainer.train()`  as  [sgugger](https://discuss.huggingface.co/t/how-to-evaluate-before-first-training-step/18838/2?u=chandler-m) suggests.
however my run script is still `deepspeed run.py --args` ,  it triggered a `ValueError("ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config") `[deepspeed_init](https://github.com/huggingface/transformers/blob/v4.39.1/src/transformers/integrations/deepspeed.py#L397)


I tried to comment the code here [ _, _ = deepspeed_init] (https://github.com/huggingface/transformers/blob/v4.39.1/src/transformers/trainer.py#L3480), the evaluate works fine, but  in `trainer.train()`, it gives me
```
Traceback (most recent call last):
  File "/app/nfs_share_dir/5/boruipeng/finance_llm/pretraining.py", line 325, in <module>
    main()
  File "/app/nfs_share_dir/5/boruipeng/finance_llm/pretraining.py", line 297, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1780, in train
    return inner_training_loop(
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2118, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 3045, in training_step
    self.accelerator.backward(loss)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/accelerator.py", line 1958, in backward
    self.deepspeed_engine_wrapped.backward(loss, **kwargs)
AttributeError: 'NoneType' object has no attribute 'backward'
```
ops... seems not a feasible solution...

why would this happen? how should I do if I want my model evaluate before train with Zero-2?


Thank u

> Hello, I don't understand what the issue is here. As the error suggests, DeepSpeed ZeRO only makes sense for prediction if it is stage 3 because the parameters would get sharded. DeepSpeed Stage 2 would not be leading to any memory savings and in that case it would be better to not use DeepSeed at all as it would be same as DDP inference.

Hi @pacman100, I think the issue lies in why running this is normal:
```
# using stage2
trainer.train()
trainer.evaluate()
```
but commenting the train() function out here results in an error. If forcing stage 2 cannot predict, then why doesn't the code above throw an error as well?  There is some ambiguity.

cc @muellerzr @SunMarc 

Hi all,
I think the problem is probably at line 3784 at transformers/trainer.py

```
        if self.is_deepspeed_enabled and self.deepspeed is None:
            _, _ = deepspeed_init(self, num_training_steps=0, inference=True)
```
When you first call trainer.train(), self.deepspeed will be initialised by the inner_training_loop, (at line 2133, same file):
```
        if self.is_deepspeed_enabled:
            self.deepspeed = self.model_wrapped
```
Therefore, calling evaluate() after self.train() can work as normal since self.deepspeed is not None, but a DeepSpeedEngine object pointing from self.model_wrapped. Otherwise, self.deepspeed will be None and it will trigger deepspeed_init in the evaluation process.

I tried to disable deepspeed_init in evaluation_loop but it would cause `AssertionError: zero stage 2 requires an optimize` at `
File "/dir/envs/myenv/lib/python3.11/site-packages/accelerate/accelerator.py", line 1318, in prepare
[rank4]:     result = self._prepare_deepspeed(*args)
`

Looking forward to possible solutions!


@GilesLuo  transformers has add `eval_on_start` para , u can set it to `True `to make model eval before `model.train()`

[related issue](https://github.com/huggingface/transformers/issues/30205)