[BUG] bf16 incorrectly configured in src/transformers/deepspeed.py
Author: michaelroyzen
Pull request: no
Labels: name: DeepSpeed

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Ubuntu
- Python version: 3.8
- PyTorch version (GPU?): 8x A10
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:

### Who can help
@stas00 

Models: T5

## Information

I am trying to fine-tune T5 using the Huggingface Trainer in bf16 using its built-in DeepSpeed integration. While I added `bf16=True` and `"bf16": {
      "enabled": true
    },` to the TrainingArguments and DeepSpeed config respectively, this flag makes no difference in GPU memory usage or training speed. Hence, I found a typo in src/transformers/deepspeed.py: At line 253,
`if self.is_true("bfoat16.enabled"):
            self._dtype = torch.bfloat16`

Instead of `bfoat16.enabled`, it should be `bfloat16.enabled`. Yet that in itself is also outdated, as the latest DeepSpeed docs say that `bf16` not `bfloat16` should be in the DeepSpeed config.

Comments:
Oh, my bad, thank you for noticing the misspelling, Michael. 

And it indeed was renamed at the very last moment to `bf16` to match `fp16`, but the old `bfloat16` config should work too.

Please try with this fix https://github.com/huggingface/transformers/pull/16597

Can you help me with where to add the 'bf16' configuration? 
I met following situation when training T5. "ValueError: `bfloat16.enabled` not found in kwargs. Please specify `bfloat16.enabled` without `auto`(set to correct value) in the DeepSpeed config file or pass it in kwargs."

The information are listed below:
  File "./ds_train.py", line 378, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/hechenghua/anaconda3/envs/swiftsage/lib/python3.8/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/home/hechenghua/anaconda3/envs/swiftsage/lib/python3.8/site-packages/transformers/trainer.py", line 1659, in _inner_training_loop
    model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
  File "/home/hechenghua/anaconda3/envs/swiftsage/lib/python3.8/site-packages/accelerate/accelerator.py", line 1178, in prepare
    result = self._prepare_deepspeed(*args)
  File "/home/hechenghua/anaconda3/envs/swiftsage/lib/python3.8/site-packages/accelerate/accelerator.py", line 1486, in _prepare_deepspeed
    deepspeed_plugin.deepspeed_config_process(must_match=False, **config_kwargs)
  File "/home/hechenghua/anaconda3/envs/swiftsage/lib/python3.8/site-packages/accelerate/utils/dataclasses.py", line 624, in deepspeed_config_process
    self.deepspeed_config_process(
  File "/home/hechenghua/anaconda3/envs/swiftsage/lib/python3.8/site-packages/accelerate/utils/dataclasses.py", line 628, in deepspeed_config_process
    self.fill_match(prefix + key, mismatches, must_match=must_match, **kwargs)
  File "/home/hechenghua/anaconda3/envs/swiftsage/lib/python3.8/site-packages/accelerate/utils/dataclasses.py", line 603, in fill_match
    raise ValueError(
ValueError: `bfloat16.enabled` not found in kwargs. Please specify `bfloat16.enabled` without `auto`(set to correct value) in the DeepSpeed config file or pass it in kwargs.



Please note that https://github.com/huggingface/accelerate took over Deepspeed integration in HF Transformers so you should ask there.

But this should work:
```
    "bf16": {
        "enabled": true
    },
```

thanks for replying.
However, I don't know where to add this configuration.
The variable "Trainer" has type "transformers.trainer_seq2seq.Seq2SeqTrainer", I don't know how to pass in the configuration mentioned in the "Trainer.train" method.
Could you please give me some hints?

You never shared how you were launching the program, so until you do I can't help with that.

HF Trainer used to require a config file launched as `--deepspeed ds_config.json`, which is where you'd add/edit what I proposed - but I haven't used HF Trainer in a while and I don't know if perhaps it was modified to not require the config file any longer in which case I defer to the current maintainers of this project.

In any case I suggest you open a new Issue as this Issue is very old.

lots of thanks for your patience !