ver 4.35.2 transformers.Trainer, deepspeed backward(loss) is not used if distributed_state is NO
Author: haixpham
Pull request: no

### System Info

transformers 4.35.2
deepspeed 0.12.3
torch 2.1
CUDA 12.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

In 4.35.2 `Trainer.training_step()`:

```
if self.use_apex:
    with amp.scale_loss(loss, self.optimizer) as scaled_loss:
        scaled_loss.backward()
else:
    self.accelerator.backward(loss)
```
` self.accelerator.backward(loss)` only uses deepspeed engine's `backward(loss)` if distributed_state is Deepspeed, which is only set if launched with torchrun. If this is not the intended behavior, the fix is to either
- modify Trainer to detect this case
- modify Accelerator to handle this case in `backward()`

### Expected behavior

The below piece of code is from transformers 4.28.1, which always uses deepspeed when training_args.deepspeed is present.
```
if self.do_grad_scaling:
    self.scaler.scale(loss).backward()
elif self.use_apex:
    with amp.scale_loss(loss, self.optimizer) as scaled_loss:
        scaled_loss.backward()
elif self.deepspeed:
    # loss gets scaled under gradient_accumulation_steps in deepspeed
    loss = self.deepspeed.backward(loss)
else:
    loss.backward()

```

Comments:
I ended up rewriting `Trainer._inner_training_loop()` and `Trainer.training_step()` to force deepspeed when distributed training is not used, as follows:
- _inner_training_loop

```
# prepare using `accelerator` prepare
if use_accelerator_prepare:
  self.model.train()
  if hasattr(self.lr_scheduler, "step"):
      if self.use_apex:
          model = self.accelerator.prepare(self.model)
      else:
          #model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
          ## add the workaroud for deepspeed
          if not self.accelerator.use_distributed and self.is_deepspeed_enabled:
              ## the workaround to force using deepspeed
              model, self.optimizer = self.accelerator._prepare_deepspeed(self.model, self.optimizer)
          else:
              # old behavior
              model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  else:
      # to handle cases wherein we pass "DummyScheduler" such as when it is specified in DeepSpeed config.
      # model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
      #     self.model, self.optimizer, self.lr_scheduler
      # )
      if not self.accelerator.use_distributed and self.is_deepspeed_enabled:
          ## the workaround to force using deepspeed
          model, self.optimizer, self.lr_scheduler = self.accelerator._prepare_deepspeed(
              self.model, self.optimizer, self.lr_scheduler
          )
      else:
          # old behavior
          model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
              self.model, self.optimizer, self.lr_scheduler
          )
```
- training_step
```
# back-prop
if self.do_grad_scaling:
    self.scaler.scale(loss).backward()
elif self.use_apex:
    with amp.scale_loss(loss, self.optimizer) as scaled_loss:
        scaled_loss.backward()
## this is a hack to force accelerator to backward loss with deepspeed in non-distributed setting
elif self._check_deepspeed_no_distributed():
    self.accelerator.deepspeed_engine_wrapped.backward(loss)
else:
    self.accelerator.backward(loss)
```
I think the problem should be addressed in accelerator - in case the code is not launched with distributed launcher like torchrun or deepspeed or accelerate

cc @muellerzr and @pacman100 

Hello, as per the documentation in https://huggingface.co/docs/transformers/main_classes/deepspeed#deployment-with-one-gpu and https://www.deepspeed.ai/getting-started/#launching-deepspeed-training, you need to use `deepspeed`, `torchrun` or `accelerate` launcher when using DeepSpeed. Just pass `--num_gpus=1` (with deepspeed launcher) or `nproc_per_node=1` (with torchrun launcher) or `num_processes=1` (with accelerate launcher). 

I knew it would work if launched with distributed launcher. However, in ver 4.28.1, deepspeed would work WITHOUT using launcher - I could actually debug with deepspeed invoked. I'm not sure is the current way (using launcher) is intended by design or a stop gap measure.

Let me look into it a bit more but the recommended way is to use the distributed launcher irrespective of whether or not it was working before without it.

Hello, could you please let us know if the above PR resolves your issue?

> Hello, could you please let us know if the above PR resolves your issue?

yes that would solve it at first glance. I will report back after running the whole training/eval process