m2m100 model do not support eval / predict on deepspeed + fp16 enviroment?
Author: wenlai-lavine
Pull request: no

Hello everyone! Recently, I am finetuning m2m100 model, but find using ```deepspeed``` + ```fp16``` can caculate loss in train_step, but got ```nan``` loss in eval_step and predict_step. The ```run_translation.py``` is normally in ```mbart``` and ```t5``` model and is also normarlly in ```m2m100``` model when do not using ```fp16```. The ```zero-2``` and ```zero-3``` got the same results. Has anyone encountered this issues? Thank you very much!

##### Version
+ Deepspeed: 0.5.10
+ transformers: 4.16.2
+ pytorch: 1.8

##### Command to repro
```
deepspeed examples/pytorch/translation/run_translation.py \
--deepspeed tests/deepspeed/ds_config_zero2.json \
--model_name_or_path facebook/m2m100_418M \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--output_dir output_dir --overwrite_output_dir \
--fp16 \
--do_train --do_eval --do_predict \
--max_train_samples 500 --max_eval_samples 50 --max_predict_samples 50 \
--num_train_epochs 3 \
--dataset_name wmt16 --dataset_config "ro-en" \
--source_lang en --target_lang ro \
--predict_with_generate --forced_bos_token ro
```

<img width="459" alt="image" src="https://user-images.githubusercontent.com/86561756/153156016-12b48a08-b717-47f5-a500-919f787e604d.png">



Comments:
Maybe of interest to @patil-suraj (M2M100) and @stas00 (DeepSpeed)

I bet that if you remove deepspeed the problem won't go away, 

How was m2m100 pre-trained? if it's bf16 mixed precision (on TPUs) then it suffers the same problem as most other bf16-pretrained models (e.g. t5) - i.e. it most likely can't be finetuned/eval'ed under fp16 as it overflows.

I'm collecting info on how the models were trained [here](https://github.com/stas00/ml-ways/blob/master/numbers/detect-model-pretrained-in-bf16-fp16-fp32.ipynb). I don't have `m2m100` so if you can find that info it'd be very helpful.

So if that's the case then you have 2 choices:

1. switch to fp32
2. switch to bf16 if you have access to Ampere GPUs (you'd need pytorch >= 1.9 for that) - you will get about the same performance as fp16.

If these suggestions don't work, first we need to isolate the problem alone, w/o deepspeed and just using bare HF transformers. 


@stas00 Thanks for your reply, I am switching to fp32, we can finetune a ```m2m100``` model with deepspeed. 

> How was m2m100 pre-trained? 

@stas00 `M2M100` is trained on GPUs with `fp16`. Here is the original model https://github.com/pytorch/fairseq/tree/main/examples/m2m_100 

Thank you for letting me know it wasn't the bf16-pretrained model, @patil-suraj

I checked that it works w/o deepspeed:
```
PYTHONPATH=src CUDA_VISIBLE_DEVICES=0 python examples/pytorch/translation/run_translation.py --model_name_or_path facebook/m2m100_418M --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --output_dir output_dir --overwrite_output_dir --fp16 --do_train --do_eval --do_predict --max_train_samples 500 --max_eval_samples 50 --max_predict_samples 50 --num_train_epochs 3 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro --predict_with_generate --forced_bos_token ro
[...]
  train_loss               =     0.5012
  train_runtime            = 0:00:32.22
  train_samples            =        500
  train_samples_per_second =     46.542
  train_steps_per_second   =      5.864
02/10/2022 09:27:53 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:2387] 2022-02-10 09:27:53,558 >> ***** Running Evaluation *****
[INFO|trainer.py:2389] 2022-02-10 09:27:53,559 >>   Num examples = 50
[INFO|trainer.py:2392] 2022-02-10 09:27:53,559 >>   Batch size = 8
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:05<00:00,  1.23it/s]02/10/2022 09:28:02 - INFO - datasets.metric - Removing /home/stas/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:06<00:00,  1.01it/s]
***** eval metrics *****
  eval_loss               =     1.5899
  predict_loss               =     1.7644
```

I reproduced the problem under deepspeed and here is the fix:

```
diff --git a/tests/deepspeed/ds_config_zero2.json b/tests/deepspeed/ds_config_zero2.json
index dec097dd1..a9f9a5805 100644
--- a/tests/deepspeed/ds_config_zero2.json
+++ b/tests/deepspeed/ds_config_zero2.json
@@ -3,7 +3,7 @@
         "enabled": "auto",
         "loss_scale": 0,
         "loss_scale_window": 1000,
-        "initial_scale_power": 16,
+        "initial_scale_power": 32,
         "hysteresis": 2,
         "min_loss_scale": 1
     },
```

same for zero3 - I will explain why shortly.

```
  train_loss               =     0.7379
  eval_loss               =     1.8916
  predict_loss               =     2.0059
```

the loss is worse I think because it takes some 10 steps before it figures out the right loss scaling factor and there are very few steps here.

So what happened:

If you look at the log, you can see it's trying to find the right loss scaling factor - scroll to the right to see the "Attempted loss scale:" numbers.
```
[2022-02-10 09:32:30,702] [INFO] [stage_1_and_2.py:1652:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 65536
  1%|▌                                                                                                                    | 1/189 [00:00<01:23,  2.26it/s]
  [2022-02-10 09:32:31,160] [INFO] [stage_1_and_2.py:1652:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 32768.0
  2%|█▊                                                                                                                   | 3/189 [00:02<02:47,  1.11it/s]
  [2022-02-10 09:32:33,056] [INFO] [stage_1_and_2.py:1652:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 32768.0, reducing to 16384.0
  2%|██▍                                                                                                                  | 4/189 [00:02<02:14,  1.37it/s]
  [2022-02-10 09:32:33,518] [INFO] [stage_1_and_2.py:1652:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 16384.0, reducing to 8192.0
  3%|███                                                            
[...]  
   17%|███████████████████▋                                                                                                | 32/189 [00:16<01:12,  2.18it/s]
   [2022-02-10 09:32:47,241] [INFO] [stage_1_and_2.py:1652:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 17%|████████████████████▎                                                                                               | 33/189 [00:16<01:12,  2.16it/s]
 [2022-02-10 09:32:47,718] [INFO] [stage_1_and_2.py:1652:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 18%|████████████████████▊                                                                                               | 34/189 [00:17<01:12,  2.14it/s]
 [2022-02-10 09:32:48,157] [INFO] [stage_1_and_2.py:1652:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
  
```
and it fails to do so.

So in this particular case, configuring deepspeed.fp16 settings to start from a higher initial scale power overcomes the problem.

Please let me know if this resolves this issue for you, @lavine-lmu 

it works after increase the ```initial_scale_power ``` from 16 to 32. Thank you very much! @stas00 

Thank you for confirming that the suggestion worked for you, closing this issue.

also added this use case to the troubleshooting section here: https://github.com/huggingface/transformers/pull/15585

@lavine-lmu were you able to achieve good results when fine-tuning M2M100? I'm trying to do that but unfortunately when I try to do that the loss decreases but the performance does not improve (it goes from around 30BLEU points without fine-tuning to 0.1 BLEU points after fine-tuning)