loss 0.0 or NaN when training T5 or Flan-T5 models with bf16 on multiple GPUs
Author: cchen-dialpad
Pull request: no

### System Info

- `transformers` version: 4.28.0
- Platform: Linux-5.19.0-1022-gcp-x86_64-with-Ubuntu-22.04-jammy
- Python version: 3.7.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@stas00

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:
1. Launch the example summarization training pipeline with `--bf16` and one of the Flan-T5 or T5 models such as `google/flan-t5-base` and `t5-large`. 
```
python3.7 examples/pytorch/summarization/run_summarization.py \
    --report_to none \
    --bf16 \
    --model_name_or_path google/flan-t5-base \
    --evaluation_strategy steps \
    --logging_strategy steps \
    --logging_steps 10 \
    --save_strategy steps \
    --save_steps 30000 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --max_train_samples=10000 \
    --max_eval_samples=100 \
    --overwrite_output_dir \
    --overwrite_cache
```
Then you will see the following logs:
```
{'loss': 0.0, 'learning_rate': 4.893503727369542e-05, 'epoch': 0.06}
{'eval_loss': nan, 'eval_runtime': 1.6413, 'eval_samples_per_second': 60.927, 'eval_steps_per_second': 2.437, 'epoch': 0.06}
```

### Expected behavior

Loss and eval_loss are wrong during training. Only T5 model works properly is `t5-small`, same as what was mentioned here: https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139, but other `T5` or `Flan-T5` models (including `flan-t5-small`) still suffer from this issue. I can train with FP32 without this problem, but would like to know if the fix (maybe not relevant here? since bf16 is different from fp16) mentioned has been incorporated.

Comments:
BTW, if I use deepspeed to launch the script then `loss` and `eval_loss` are normal.
In addition, the 0.0 or NaN only happens when it is trained on multiple GPUs in the DataParallel mode, which is aligned with this issue: https://github.com/huggingface/transformers/issues/18899

Thank you for the good report and the link to the other similar issue, @cchen-dialpad 

I think hardly anybody uses DP since DDP was introduced. Is there a reason to use DP when DDP is by far more superior?

Switching to DDP was the resolution of the ticket you linked to: https://github.com/huggingface/transformers/issues/18899#issuecomment-1249262873



@stas00 Oh, I guess I just got curious why that is the case, DDP works but DP doesn't  :)

You're more than welcome to try to figure it out, @cchen-dialpad - I haven't used DP in many years, perhaps it's not being well maintained because it's rarely used? It'd be an optimizer issue most likely if you want a place to start.

lol I see, thanks for the pointers!

this happened to me! ddp, bf16, t5-large, also FSDP