[wip] diagnose MT metrics regression from pl 0.8.5 upgrade
Author: sshleifer
Pull request: yes
Labels: name: Help wanted

name: lightning

**Base Command:**
```bash
BS=8
GAS=4
MAX_LEN=128
python finetune.py \
    --learning_rate=3e-5 \
    --do_train \
    --val_check_interval=0.25 \
    --adam_eps 1e-06 \
    --num_train_epochs 6 --src_lang en_XX --tgt_lang ro_RO \
    --data_dir $ENRO_DIR \
    --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
    --train_batch_size=$BS --eval_batch_size=$BS --gradient_accumulation_steps=$GAS \
    --task translation \
    --warmup_steps 500 \
    --freeze_embeds \
    --model_name_or_path=facebook/mbart-large-cc25 \
	--label_smoothing 0.1  --freeze_embeds --gpus 1 --logger_name wandb --sortish_sampler \
	$@
```

**Clues:**
- many more steps per epoch in wandb on `distillmbart` branch
- lr reasonable on both branches.
- loss much **higher** on `distillmbart` branch.
- val_avg_bleu after ¼ epoch much higher  (23 vs 19)
- fp32 loss goes to NaN after .25 epochs. (  `bru_baseline_pl85_fp32`)

**Suspects:**
- not lr scheduler, though lr schedules differ (because steps differs I presume)
- **early stopping**
- maybe fp16_opt_level being used regardless of `--fp16`?
	(maybe scaler line in PL?)
- optimizer_step unchanged besides `lr_scheduler.step` and the scheduler is clearly stepping. Feels wrong.
- dataloader shuffle/`setup`
- src_lens change in LineByLine ds
- just a change in the way val metrics are computed?


**TLDR**:
can get test BLEU = 26.27 with gradient accumulation steps=1 and no early stopping:
```bash
./train_mbart_cc25_enro.sh --output_dir bru_pl85_long  --label_smoothing 0.1  --freeze_embeds --logger_name wandb --sortish_sampler --fp16_opt_level O1 --gpus 1
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6166?src=pr&el=h1) Report
> Merging [#6166](https://codecov.io/gh/huggingface/transformers/pull/6166?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c0b93a1c7a961e30b30d02d641c9d22120ef5d73&el=desc) will **decrease** coverage by `1.38%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6166/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6166?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6166      +/-   ##
==========================================
- Coverage   79.82%   78.44%   -1.39%     
==========================================
  Files         146      146              
  Lines       26597    26597              
==========================================
- Hits        21231    20863     -368     
- Misses       5366     5734     +368     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6166?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/6166/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `23.38% <0.00%> (-73.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6166/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.18% <0.00%> (-34.62%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6166/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+1.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6166/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `95.32% <0.00%> (+23.67%)` | :arrow_up: |
| [src/transformers/tokenization\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/6166/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmFydC5weQ==) | `95.77% <0.00%> (+35.21%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6166/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `88.19% <0.00%> (+63.97%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6166?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6166?src=pr&el=footer). Last update [c0b93a1...68319f0](https://codecov.io/gh/huggingface/transformers/pull/6166?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).