Accuracy not increasing with BERT Large model
Author: ghost
Pull request: no
Labels: name: wontfix

I experimented with  `BERT_base_cased` and `BERT_large_cased` model for multi class text classification. With BERT_base_cased, I got satisfactory results. When I tried with BERT_large_cased model, the accuracy is same for all the epochs

```
Epoch: 01 | Epoch Time: 0m 57s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.200893470219204	2.790178544819355	4.977107011354887	3.6057692021131516
Epoch: 02 | Epoch Time: 0m 57s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.085730476038797	2.287946455180645	4.954357807452862	3.6057692021131516
Epoch: 03 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.019492668764932	2.901785634458065	4.961122549497164	3.6057692021131516
Epoch: 04 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.0052995937211175	3.57142873108387	4.9535566843473	3.6057692021131516
Epoch: 05 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.003523528575897	3.23660708963871	4.9652618261484	3.6057692021131516
Epoch: 06 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.010107040405273	3.29241082072258	4.96296108686007	3.6057692021131516
Epoch: 07 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.028377030576978	2.678571455180645	4.94510478239793	3.6057692021131516
Epoch: 08 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.04387321642467	2.901785634458065	4.9411917466383715	3.6057692021131516
Epoch: 09 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.027528064591544	3.18080373108387	4.940045246711144	3.6057692021131516
Epoch: 10 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.023407867976597	3.29241082072258	4.940378886002761	3.6057692021131516
Epoch: 11 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.015415557793209	3.125	4.939220135028545	3.6057692021131516
Epoch: 12 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.018008896282741	3.29241082072258	4.9386150653545675	3.6057692021131516
Epoch: 13 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.003824523517063	2.957589365541935	4.938107490539551	3.6057692021131516
Epoch: 14 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.003440124647958	3.069196455180645	4.93824944129357	3.6057692021131516
Epoch: 15 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.012082431997571	3.069196455180645	4.9383643590486965	3.6057692021131516
Epoch: 16 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.009286454745701	3.01339291036129	4.93832148038424	3.6057692021131516
Epoch: 17 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.006769972188132	2.901785634458065	4.937925778902494	3.6057692021131516
Epoch: 18 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.006464583533151	3.125	4.937762847313514	3.6057692021131516
Epoch: 19 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.004164610590253	2.957589365541935	4.937491783728967	3.6057692021131516
Epoch: 20 | Epoch Time: 0m 57s
*******train_loss,train_acc,valid_loss,valid_ac********** 
5.013612789767129	2.957589365541935	4.937890896430383	3.6057692021131516
Epoch: 21 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac********** 
4.997398240225656	2.511160634458065	4.937900726611797	3.6057692021131516
```
With `BERT_base_cased`, there is no such problem. But with `BERT_large_cased`, why accuracy is same in all the epochs? Any help is really appreciated............. @thomwolf  @nreimers

Comments:
Hi, I experienced this also in several experiments. BERT large is extremely sensitive to the random seed. Try some other seeds and you will likely get a performance at least on oar with the base model.

I haven't it studied further why the large model is so sensitive to the random seed, but it appears that the gradient for some step destroys the model, from which on you only get bad scores. Might be some exploding gradient or some Nan issues.

Best, 
Nils Reimers 

I find this issue as well -- no convergence with the large model.

Potentially related:

https://github.com/huggingface/transformers/issues/753
https://github.com/huggingface/transformers/issues/92

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.