transformers 4.39.1 caused all my previous model training loss hard to converge
Author: lucasjinreal
Pull request: no
Labels: name: Core: Modeling

Am not sure it in some where wrong with 4.39.1, but my all training now can not converge normally (previously all OK).

Am training llava model, in older version, same model same data loss can converge about to 0.8, but after upgrade, the loss only gets 1.6  and can not descrease.

How to debug this weired behavior? Am not sure what 4.39.1 introduced new.

I have tested with 2 machine all with A100, the only difference is transformers verison.

4.39.1:

```
{'loss': 3.5624, 'grad_norm': 79.45402526855469, 'learning_rate': 1.1961722488038277e-06, 'epoch': 0.0}                                                                                                                          
{'loss': 3.9967, 'grad_norm': 82.91897583007812, 'learning_rate': 2.3923444976076554e-06, 'epoch': 0.0}                                                                                                                          
{'loss': 4.017, 'grad_norm': 80.83908081054688, 'learning_rate': 3.5885167464114835e-06, 'epoch': 0.0}                                                                                                                           
{'loss': 4.0142, 'grad_norm': 74.29833221435547, 'learning_rate': 4.784688995215311e-06, 'epoch': 0.0}                                                                                                                           
{'loss': 4.414, 'grad_norm': 64.46388244628906, 'learning_rate': 5.980861244019139e-06, 'epoch': 0.0}                                                                                                                            
{'loss': 4.298, 'grad_norm': 45.83267593383789, 'learning_rate': 7.177033492822967e-06, 'epoch': 0.0}                                                                                                                            
{'loss': 3.4189, 'grad_norm': 26.755836486816406, 'learning_rate': 8.373205741626795e-06, 'epoch': 0.0}                                                                                                                          
{'loss': 3.0565, 'grad_norm': 23.13935661315918, 'learning_rate': 9.569377990430622e-06, 'epoch': 0.0}                                                                                                                           
{'loss': 2.8657, 'grad_norm': 18.022300720214844, 'learning_rate': 1.0765550239234451e-05, 'epoch': 0.0}                                                                                                                         
{'loss': 2.8427, 'grad_norm': 22.7450008392334, 'learning_rate': 1.1961722488038278e-05, 'epoch': 0.0}                                                                                                                           
{'loss': 2.9401, 'grad_norm': 21.4661808013916, 'learning_rate': 1.3157894736842104e-05, 'epoch': 0.0}                                                                                                                           
{'loss': 2.6572, 'grad_norm': 17.573665618896484, 'learning_rate': 1.4354066985645934e-05, 'epoch': 0.0}                                                                                                                         
{'loss': 3.0221, 'grad_norm': 13.378881454467773, 'learning_rate': 1.555023923444976e-05, 'epoch': 0.0}                                                                                                                          
{'loss': 2.6944, 'grad_norm': 10.9893217086792, 'learning_rate': 1.674641148325359e-05, 'epoch': 0.0}                                                                                                                            
{'loss': 2.6247, 'grad_norm': 11.024161338806152, 'learning_rate': 1.7942583732057417e-05, 'epoch': 0.0}                                                                                                                         
{'loss': 2.5326, 'grad_norm': 9.668575286865234, 'learning_rate': 1.9138755980861243e-05, 'epoch': 0.0}                                                                                                                          
{'loss': 2.2975, 'grad_norm': 10.578173637390137, 'learning_rate': 2.033492822966507e-05, 'epoch': 0.0}                                                                                                                          
{'loss': 2.4432, 'grad_norm': 6.985546588897705, 'learning_rate': 2.1531100478468903e-05, 'epoch': 0.0}                                                                                                                          
{'loss': 2.4433, 'grad_norm': 9.492457389831543, 'learning_rate': 2.272727272727273e-05, 'epoch': 0.0}                                                                                                                           
{'loss': 2.5511, 'grad_norm': 14.73047924041748, 'learning_rate': 2.3923444976076556e-05, 'epoch': 0.0}      
```

4.37.0

```
{'loss': 2.9511, 'learning_rate': 1.1961722488038277e-06, 'epoch': 0.0}                                                                                                                                                          
{'loss': 3.0343, 'learning_rate': 2.3923444976076554e-06, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.9587, 'learning_rate': 3.5885167464114835e-06, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.9541, 'learning_rate': 4.784688995215311e-06, 'epoch': 0.0}                                                                                                                                                           
{'loss': 3.0184, 'learning_rate': 5.980861244019139e-06, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.8609, 'learning_rate': 7.177033492822967e-06, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.7076, 'learning_rate': 8.373205741626795e-06, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.9336, 'learning_rate': 9.569377990430622e-06, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.7018, 'learning_rate': 1.0765550239234451e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.4909, 'learning_rate': 1.1961722488038278e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.9486, 'learning_rate': 1.3157894736842104e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.5025, 'learning_rate': 1.4354066985645934e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.4773, 'learning_rate': 1.555023923444976e-05, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.5077, 'learning_rate': 1.674641148325359e-05, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.4559, 'learning_rate': 1.7942583732057417e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.5975, 'learning_rate': 1.9138755980861243e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.4911, 'learning_rate': 2.033492822966507e-05, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.5445, 'learning_rate': 2.1531100478468903e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.3081, 'learning_rate': 2.272727272727273e-05, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.6962, 'learning_rate': 2.3923444976076556e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.6305, 'learning_rate': 2.5119617224880382e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.5737, 'learning_rate': 2.631578947368421e-05, 'epoch': 0.0}                                                                                                                                                           
{'loss': 2.1082, 'learning_rate': 2.7511961722488042e-05, 'epoch': 0.0}                                                                                                                                                          
{'loss': 2.2048, 'learning_rate': 2.8708133971291868e-05, 'epoch': 0.0} 
```

Am not sure what caused tiny difference on loss discrease, but Now I found using latest verison, the loss very hard to converge.

I got this when using along with deepspeed zero2

Comments:
Hey! This might be related to #29836. It's gonna be hard to know what broke it... cc @NielsRogge 
If you can checkout some commits in the middle / some versions like 4.38.2 would help. That's not expected

Ok thanks for reporting, we should investigate this. Which model are you training specifically? Are you adding any special tokens to the model?

Does it still work on v4.39 for you? v4.39.1 was just a patch release: https://github.com/huggingface/transformers/releases.

Am not sure it's tokens related, but for llava it actually added special tokens.

May I ask which tokenizers would effected or all of them? Am using LLM model of Qwen2

It should not have a direct effect on any tokenizers, unless you force the re-conversion using `from_slow=True`. The effect is consistency between slow and fast

I upgraded transformers to 4.39.2 from 4.31.0, and it had a breaking effect on my loss when using your run_glue.py to train roberta. There's little reproducibility possible across hf versions.
<img width="719" alt="image" src="https://github.com/huggingface/transformers/assets/87317405/7b1dcf49-426e-41d8-b3dd-b34a90b2554f">


I don't know how you are training, but you mus be aware that training adds a lot of randomness: shuffling, seeds, torch version, datasets used, pre-processing, etc etc. 
It's fairly eassy to blame `transformers` but know that we try our best to always improve and fix bugs. Would love to have your feedback if you find a potential issue!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.