run_glue.py, CoLA : MCC goes to 0, in some hyperparameter cases
Author: drcdr
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using: roberta-large
Language I am using the model on: English

The problem arise when using:
* [x] the official example scripts: run_glue.py

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: CoLA

## To Reproduce
Steps to reproduce the behavior:
1. Running the following command is one way to create an mcc of 0.  Possible 'non-standard' items here are: LR up to 3e-5, different seeds, warmup, and evaluate-during-training.
```
python run_glue_orig.py --model_type roberta --model_name_or_path roberta-large --task_name CoLA --do_train --do_eval --do_lower_case --evaluate_during_training --logging_steps 50 --save_steps 1000000 --data_dir /home/_DBS/GLUE/CoLA --max_seq_length 128 --per_gpu_eval_batch_size 8 --per_gpu_train_batch_size 8 --learning_rate 3e-5 --num_train_epochs 10 --warmup_steps 641 --output_dir try_fail_1 --seed 3
```
For reference, the args are:
```
Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir='/home/_DBS/GLUE/CoLA', do_eval=True, do_lower_case=True, do_train=True, eval_all_checkpoints=False, evaluate_during_training=True, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=3e-05, local_rank=-1, logging_steps=50, max_grad_norm=1.0, max_seq_length=128, max_steps=-1, model_name_or_path='roberta-large', model_type='roberta', no_cuda=False, num_train_epochs=10.0, output_dir='try_fail_1', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=8, per_gpu_train_batch_size=8, save_steps=1000000, seed=3, server_ip='', server_port='', task_name='CoLA', tokenizer_name='', warmup_steps=641, weight_decay=0.0)
```
 ## Expected behavior
Non-zero mcc values for CoLA, after warmup, even for LR = 3e-5.  (Agree?)

## Environment
* OS: Ubuntu 18.04   Platform Linux-4.15.0-74-generic-x86_64-with-debian-buster-sid
* Python version:    Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) 
* PyTorch version:  1.2.0.dev20190702 py3.6_cuda10.0.130_cudnn7.5.1_0
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU : TitanXP, qty. 1
* Distributed or parallel setup : no
* Any other relevant information:

## Additional context

* I am trying to reproduce the CoLA results for BERT and RoBERTa.  I've built a test harness that calls a lightly modified version of run_glue.py, following the approach of paragraph 5.1 [here](https://arxiv.org/pdf/1907.11692.pdf).
* Once I saw the mcc=0 values in my version, I went back and reproduced the problem in the standard run_glue.py version.
* I have not seen this (mcc->0) behavior when running the hyperparameter search on bert-base-cased, or roberta-base.  Just on roberta-large.
* The output of the standard run_glue.py is too large to include here, but here is the output from my version, showing what mcc looks like during the run, with the same parameters as above (you need to scroll to the right to see the mcc values):
```
2020-01-19 13:14:10 Ep  0/10 Gstep    50 Step    49:  Loss=  0.680712  LR=2.340094e-06   mcc=  0.000000,
2020-01-19 13:14:45 Ep  0/10 Gstep   100 Step    99:  Loss=  0.610892  LR=4.680187e-06   mcc=  0.000000,
2020-01-19 13:15:19 Ep  0/10 Gstep   150 Step   149:  Loss=  0.627466  LR=7.020281e-06   mcc=  0.000000,
2020-01-19 13:15:54 Ep  0/10 Gstep   200 Step   199:  Loss=  0.603764  LR=9.360374e-06   mcc=  0.000000,
2020-01-19 13:16:28 Ep  0/10 Gstep   250 Step   249:  Loss=  0.576851  LR=1.170047e-05   mcc=  0.222055,
2020-01-19 13:17:03 Ep  0/10 Gstep   300 Step   299:  Loss=  0.581703  LR=1.404056e-05   mcc=  0.046356,
2020-01-19 13:17:37 Ep  0/10 Gstep   350 Step   349:  Loss=  0.580515  LR=1.638066e-05   mcc=  0.176324,
2020-01-19 13:18:12 Ep  0/10 Gstep   400 Step   399:  Loss=  0.499536  LR=1.872075e-05   mcc=  0.452767,
2020-01-19 13:18:46 Ep  0/10 Gstep   450 Step   449:  Loss=  0.586593  LR=2.106084e-05   mcc=  0.459277,
2020-01-19 13:19:21 Ep  0/10 Gstep   500 Step   499:  Loss=  0.619785  LR=2.340094e-05   mcc=  0.465695,
2020-01-19 13:19:55 Ep  0/10 Gstep   550 Step   549:  Loss=  0.554341  LR=2.574103e-05   mcc=  0.480823,
2020-01-19 13:20:29 Ep  0/10 Gstep   600 Step   599:  Loss=  0.573441  LR=2.808112e-05   mcc=  0.272740,
2020-01-19 13:21:04 Ep  0/10 Gstep   650 Step   649:  Loss=  0.581185  LR=2.997313e-05   mcc=  0.347692,
2020-01-19 13:21:38 Ep  0/10 Gstep   700 Step   699:  Loss=  0.662212  LR=2.982386e-05   mcc=  0.000000,
2020-01-19 13:22:13 Ep  0/10 Gstep   750 Step   749:  Loss=  0.601005  LR=2.967459e-05   mcc=  0.000000,
2020-01-19 13:22:47 Ep  0/10 Gstep   800 Step   799:  Loss=  0.648288  LR=2.952533e-05   mcc=  0.000000,
2020-01-19 13:23:22 Ep  0/10 Gstep   850 Step   849:  Loss=  0.637533  LR=2.937606e-05   mcc=  0.000000,
2020-01-19 13:23:56 Ep  0/10 Gstep   900 Step   899:  Loss=  0.633061  LR=2.922679e-05   mcc=  0.000000,
2020-01-19 13:24:30 Ep  0/10 Gstep   950 Step   949:  Loss=  0.647681  LR=2.907752e-05   mcc=  0.000000,
2020-01-19 13:25:05 Ep  0/10 Gstep  1000 Step   999:  Loss=  0.608788  LR=2.892825e-05   mcc=  0.000000,
2020-01-19 13:25:39 Ep  0/10 Gstep  1050 Step  1049:  Loss=  0.620082  LR=2.877898e-05   mcc=  0.000000,
2020-01-19 13:26:14 Ep  1/10 Gstep  1100 Step    30:  Loss=  0.604326  LR=2.862971e-05   mcc=  0.000000,
2020-01-19 13:26:48 Ep  1/10 Gstep  1150 Step    80:  Loss=  0.599047  LR=2.848045e-05   mcc=  0.000000,
2020-01-19 13:27:22 Ep  1/10 Gstep  1200 Step   130:  Loss=  0.635962  LR=2.833118e-05   mcc=  0.000000,
2020-01-19 13:27:57 Ep  1/10 Gstep  1250 Step   180:  Loss=  0.576796  LR=2.818191e-05   mcc=  0.000000,
2020-01-19 13:28:31 Ep  1/10 Gstep  1300 Step   230:  Loss=  0.627478  LR=2.803264e-05   mcc=  0.000000,
2020-01-19 13:29:06 Ep  1/10 Gstep  1350 Step   280:  Loss=  0.584176  LR=2.788337e-05   mcc=  0.000000,
2020-01-19 13:29:40 Ep  1/10 Gstep  1400 Step   330:  Loss=  0.604549  LR=2.773410e-05   mcc=  0.000000,
2020-01-19 13:30:14 Ep  1/10 Gstep  1450 Step   380:  Loss=  0.619358  LR=2.758483e-05   mcc=  0.000000,
2020-01-19 13:30:49 Ep  1/10 Gstep  1500 Step   430:  Loss=  0.582581  LR=2.743557e-05   mcc=  0.000000,
2020-01-19 13:31:23 Ep  1/10 Gstep  1550 Step   480:  Loss=  0.599344  LR=2.728630e-05   mcc=  0.000000,
2020-01-19 13:31:58 Ep  1/10 Gstep  1600 Step   530:  Loss=  0.628334  LR=2.713703e-05   mcc=  0.000000,
2020-01-19 13:32:32 Ep  1/10 Gstep  1650 Step   580:  Loss=  0.619170  LR=2.698776e-05   mcc=  0.000000,
2020-01-19 13:33:07 Ep  1/10 Gstep  1700 Step   630:  Loss=  0.647814  LR=2.683849e-05   mcc=  0.000000,
2020-01-19 13:33:41 Ep  1/10 Gstep  1750 Step   680:  Loss=  0.573997  LR=2.668922e-05   mcc=  0.000000,
2020-01-19 13:34:16 Ep  1/10 Gstep  1800 Step   730:  Loss=  0.591928  LR=2.653995e-05   mcc=  0.000000,
2020-01-19 13:34:50 Ep  1/10 Gstep  1850 Step   780:  Loss=  0.636912  LR=2.639069e-05   mcc=  0.000000,
2020-01-19 13:35:25 Ep  1/10 Gstep  1900 Step   830:  Loss=  0.647883  LR=2.624142e-05   mcc=  0.000000,
2020-01-19 13:35:59 Ep  1/10 Gstep  1950 Step   880:  Loss=  0.612844  LR=2.609215e-05   mcc=  0.000000,
2020-01-19 13:36:34 Ep  1/10 Gstep  2000 Step   930:  Loss=  0.661776  LR=2.594288e-05   mcc=  0.000000,
2020-01-19 13:37:08 Ep  1/10 Gstep  2050 Step   980:  Loss=  0.648593  LR=2.579361e-05   mcc=  0.000000,
2020-01-19 13:37:42 Ep  1/10 Gstep  2100 Step  1030:  Loss=  0.628153  LR=2.564434e-05   mcc=  0.000000,
2020-01-19 13:38:17 Ep  2/10 Gstep  2150 Step    11:  Loss=  0.589748  LR=2.549507e-05   mcc=  0.000000,
2020-01-19 13:38:51 Ep  2/10 Gstep  2200 Step    61:  Loss=  0.583273  LR=2.534581e-05   mcc=  0.000000,
2020-01-19 13:39:26 Ep  2/10 Gstep  2250 Step   111:  Loss=  0.631948  LR=2.519654e-05   mcc=  0.000000,
2020-01-19 13:40:00 Ep  2/10 Gstep  2300 Step   161:  Loss=  0.628461  LR=2.504727e-05   mcc=  0.000000,
2020-01-19 13:40:35 Ep  2/10 Gstep  2350 Step   211:  Loss=  0.644976  LR=2.489800e-05   mcc=  0.000000,
2020-01-19 13:41:09 Ep  2/10 Gstep  2400 Step   261:  Loss=  0.612494  LR=2.474873e-05   mcc=  0.000000,
2020-01-19 13:41:44 Ep  2/10 Gstep  2450 Step   311:  Loss=  0.628107  LR=2.459946e-05   mcc=  0.000000,
2020-01-19 13:42:18 Ep  2/10 Gstep  2500 Step   361:  Loss=  0.663605  LR=2.445019e-05   mcc=  0.000000,
2020-01-19 13:42:52 Ep  2/10 Gstep  2550 Step   411:  Loss=  0.586811  LR=2.430093e-05   mcc=  0.000000,
2020-01-19 13:43:27 Ep  2/10 Gstep  2600 Step   461:  Loss=  0.620628  LR=2.415166e-05   mcc=  0.000000,
2020-01-19 13:44:01 Ep  2/10 Gstep  2650 Step   511:  Loss=  0.608283  LR=2.400239e-05   mcc=  0.000000,
2020-01-19 13:44:36 Ep  2/10 Gstep  2700 Step   561:  Loss=  0.620366  LR=2.385312e-05   mcc=  0.000000,
2020-01-19 13:45:10 Ep  2/10 Gstep  2750 Step   611:  Loss=  0.631265  LR=2.370385e-05   mcc=  0.000000,
2020-01-19 13:45:44 Ep  2/10 Gstep  2800 Step   661:  Loss=  0.636871  LR=2.355458e-05   mcc=  0.000000,
2020-01-19 13:46:19 Ep  2/10 Gstep  2850 Step   711:  Loss=  0.639280  LR=2.340531e-05   mcc=  0.000000,
2020-01-19 13:46:53 Ep  2/10 Gstep  2900 Step   761:  Loss=  0.603878  LR=2.325605e-05   mcc=  0.000000,
2020-01-19 13:47:28 Ep  2/10 Gstep  2950 Step   811:  Loss=  0.647503  LR=2.310678e-05   mcc=  0.000000,
2020-01-19 13:48:02 Ep  2/10 Gstep  3000 Step   861:  Loss=  0.580940  LR=2.295751e-05   mcc=  0.000000,
2020-01-19 13:48:37 Ep  2/10 Gstep  3050 Step   911:  Loss=  0.612520  LR=2.280824e-05   mcc=  0.000000,
2020-01-19 13:49:11 Ep  2/10 Gstep  3100 Step   961:  Loss=  0.607909  LR=2.265897e-05   mcc=  0.000000,
2020-01-19 13:49:46 Ep  2/10 Gstep  3150 Step  1011:  Loss=  0.596036  LR=2.250970e-05   mcc=  0.000000,
2020-01-19 13:50:20 Ep  2/10 Gstep  3200 Step  1061:  Loss=  0.597132  LR=2.236043e-05   mcc=  0.000000,
2020-01-19 13:50:55 Ep  3/10 Gstep  3250 Step    42:  Loss=  0.611773  LR=2.221117e-05   mcc=  0.000000,
2020-01-19 13:51:29 Ep  3/10 Gstep  3300 Step    92:  Loss=  0.605382  LR=2.206190e-05   mcc=  0.000000,
2020-01-19 13:52:03 Ep  3/10 Gstep  3350 Step   142:  Loss=  0.619225  LR=2.191263e-05   mcc=  0.000000,
2020-01-19 13:52:38 Ep  3/10 Gstep  3400 Step   192:  Loss=  0.628465  LR=2.176336e-05   mcc=  0.000000,
2020-01-19 13:53:12 Ep  3/10 Gstep  3450 Step   242:  Loss=  0.626908  LR=2.161409e-05   mcc=  0.000000,
2020-01-19 13:53:47 Ep  3/10 Gstep  3500 Step   292:  Loss=  0.636135  LR=2.146482e-05   mcc=  0.000000,
2020-01-19 13:54:21 Ep  3/10 Gstep  3550 Step   342:  Loss=  0.624727  LR=2.131555e-05   mcc=  0.000000,
2020-01-19 13:54:55 Ep  3/10 Gstep  3600 Step   392:  Loss=  0.620360  LR=2.116629e-05   mcc=  0.000000,
2020-01-19 13:55:30 Ep  3/10 Gstep  3650 Step   442:  Loss=  0.578948  LR=2.101702e-05   mcc=  0.000000,
2020-01-19 13:56:04 Ep  3/10 Gstep  3700 Step   492:  Loss=  0.644757  LR=2.086775e-05   mcc=  0.000000,
2020-01-19 13:56:39 Ep  3/10 Gstep  3750 Step   542:  Loss=  0.599062  LR=2.071848e-05   mcc=  0.000000,
2020-01-19 13:57:13 Ep  3/10 Gstep  3800 Step   592:  Loss=  0.623814  LR=2.056921e-05   mcc=  0.000000,
2020-01-19 13:57:48 Ep  3/10 Gstep  3850 Step   642:  Loss=  0.607594  LR=2.041994e-05   mcc=  0.000000,
2020-01-19 13:58:22 Ep  3/10 Gstep  3900 Step   692:  Loss=  0.636492  LR=2.027067e-05   mcc=  0.000000,
2020-01-19 13:58:57 Ep  3/10 Gstep  3950 Step   742:  Loss=  0.596911  LR=2.012141e-05   mcc=  0.000000,
2020-01-19 13:59:31 Ep  3/10 Gstep  4000 Step   792:  Loss=  0.615585  LR=1.997214e-05   mcc=  0.000000,
2020-01-19 14:00:06 Ep  3/10 Gstep  4050 Step   842:  Loss=  0.583996  LR=1.982287e-05   mcc=  0.000000,
2020-01-19 14:00:40 Ep  3/10 Gstep  4100 Step   892:  Loss=  0.616359  LR=1.967360e-05   mcc=  0.000000,
2020-01-19 14:01:15 Ep  3/10 Gstep  4150 Step   942:  Loss=  0.610478  LR=1.952433e-05   mcc=  0.000000,
2020-01-19 14:01:49 Ep  3/10 Gstep  4200 Step   992:  Loss=  0.604057  LR=1.937506e-05   mcc=  0.000000,
2020-01-19 14:02:23 Ep  3/10 Gstep  4250 Step  1042:  Loss=  0.646733  LR=1.922579e-05   mcc=  0.000000,
2020-01-19 14:02:58 Ep  4/10 Gstep  4300 Step    23:  Loss=  0.603115  LR=1.907653e-05   mcc=  0.000000,
2020-01-19 14:03:32 Ep  4/10 Gstep  4350 Step    73:  Loss=  0.569349  LR=1.892726e-05   mcc=  0.000000,
2020-01-19 14:04:07 Ep  4/10 Gstep  4400 Step   123:  Loss=  0.603698  LR=1.877799e-05   mcc=  0.000000,
2020-01-19 14:04:41 Ep  4/10 Gstep  4450 Step   173:  Loss=  0.616832  LR=1.862872e-05   mcc=  0.000000,
2020-01-19 14:05:15 Ep  4/10 Gstep  4500 Step   223:  Loss=  0.638386  LR=1.847945e-05   mcc=  0.000000,
2020-01-19 14:05:50 Ep  4/10 Gstep  4550 Step   273:  Loss=  0.587255  LR=1.833018e-05   mcc=  0.000000,
2020-01-19 14:06:24 Ep  4/10 Gstep  4600 Step   323:  Loss=  0.616687  LR=1.818091e-05   mcc=  0.000000,
2020-01-19 14:06:59 Ep  4/10 Gstep  4650 Step   373:  Loss=  0.593398  LR=1.803164e-05   mcc=  0.000000,
2020-01-19 14:07:33 Ep  4/10 Gstep  4700 Step   423:  Loss=  0.612735  LR=1.788238e-05   mcc=  0.000000,
2020-01-19 14:08:07 Ep  4/10 Gstep  4750 Step   473:  Loss=  0.612634  LR=1.773311e-05   mcc=  0.000000,
2020-01-19 14:08:42 Ep  4/10 Gstep  4800 Step   523:  Loss=  0.597561  LR=1.758384e-05   mcc=  0.000000,
2020-01-19 14:09:16 Ep  4/10 Gstep  4850 Step   573:  Loss=  0.630737  LR=1.743457e-05   mcc=  0.000000,
2020-01-19 14:09:51 Ep  4/10 Gstep  4900 Step   623:  Loss=  0.639448  LR=1.728530e-05   mcc=  0.000000,
2020-01-19 14:10:25 Ep  4/10 Gstep  4950 Step   673:  Loss=  0.607357  LR=1.713603e-05   mcc=  0.000000,
2020-01-19 14:11:00 Ep  4/10 Gstep  5000 Step   723:  Loss=  0.601610  LR=1.698676e-05   mcc=  0.000000,
2020-01-19 14:11:34 Ep  4/10 Gstep  5050 Step   773:  Loss=  0.604433  LR=1.683750e-05   mcc=  0.000000,
2020-01-19 14:12:08 Ep  4/10 Gstep  5100 Step   823:  Loss=  0.643115  LR=1.668823e-05   mcc=  0.000000,
2020-01-19 14:12:43 Ep  4/10 Gstep  5150 Step   873:  Loss=  0.639950  LR=1.653896e-05   mcc=  0.000000,
2020-01-19 14:13:17 Ep  4/10 Gstep  5200 Step   923:  Loss=  0.638993  LR=1.638969e-05   mcc=  0.000000,
2020-01-19 14:13:51 Ep  4/10 Gstep  5250 Step   973:  Loss=  0.633510  LR=1.624042e-05   mcc=  0.000000,
2020-01-19 14:14:26 Ep  4/10 Gstep  5300 Step  1023:  Loss=  0.581198  LR=1.609115e-05   mcc=  0.000000,
2020-01-19 14:15:00 Ep  5/10 Gstep  5350 Step     4:  Loss=  0.592722  LR=1.594188e-05   mcc=  0.000000,
2020-01-19 14:15:34 Ep  5/10 Gstep  5400 Step    54:  Loss=  0.614371  LR=1.579262e-05   mcc=  0.000000,
2020-01-19 14:16:09 Ep  5/10 Gstep  5450 Step   104:  Loss=  0.607973  LR=1.564335e-05   mcc=  0.000000,
2020-01-19 14:16:43 Ep  5/10 Gstep  5500 Step   154:  Loss=  0.605945  LR=1.549408e-05   mcc=  0.000000,
2020-01-19 14:17:18 Ep  5/10 Gstep  5550 Step   204:  Loss=  0.620083  LR=1.534481e-05   mcc=  0.000000,
2020-01-19 14:17:52 Ep  5/10 Gstep  5600 Step   254:  Loss=  0.625456  LR=1.519554e-05   mcc=  0.000000,
2020-01-19 14:18:27 Ep  5/10 Gstep  5650 Step   304:  Loss=  0.633456  LR=1.504627e-05   mcc=  0.000000,
2020-01-19 14:19:01 Ep  5/10 Gstep  5700 Step   354:  Loss=  0.608953  LR=1.489700e-05   mcc=  0.000000,
2020-01-19 14:19:35 Ep  5/10 Gstep  5750 Step   404:  Loss=  0.590640  LR=1.474774e-05   mcc=  0.000000,
2020-01-19 14:20:10 Ep  5/10 Gstep  5800 Step   454:  Loss=  0.608632  LR=1.459847e-05   mcc=  0.000000,
2020-01-19 14:20:44 Ep  5/10 Gstep  5850 Step   504:  Loss=  0.615661  LR=1.444920e-05   mcc=  0.000000,
2020-01-19 14:21:19 Ep  5/10 Gstep  5900 Step   554:  Loss=  0.602201  LR=1.429993e-05   mcc=  0.000000,
2020-01-19 14:21:53 Ep  5/10 Gstep  5950 Step   604:  Loss=  0.593200  LR=1.415066e-05   mcc=  0.000000,
2020-01-19 14:22:27 Ep  5/10 Gstep  6000 Step   654:  Loss=  0.623690  LR=1.400139e-05   mcc=  0.000000,
2020-01-19 14:23:02 Ep  5/10 Gstep  6050 Step   704:  Loss=  0.608412  LR=1.385212e-05   mcc=  0.000000,
2020-01-19 14:23:36 Ep  5/10 Gstep  6100 Step   754:  Loss=  0.637673  LR=1.370286e-05   mcc=  0.000000,
2020-01-19 14:24:11 Ep  5/10 Gstep  6150 Step   804:  Loss=  0.606503  LR=1.355359e-05   mcc=  0.000000,
2020-01-19 14:24:45 Ep  5/10 Gstep  6200 Step   854:  Loss=  0.575204  LR=1.340432e-05   mcc=  0.000000,
2020-01-19 14:25:19 Ep  5/10 Gstep  6250 Step   904:  Loss=  0.619873  LR=1.325505e-05   mcc=  0.000000,
2020-01-19 14:25:54 Ep  5/10 Gstep  6300 Step   954:  Loss=  0.587598  LR=1.310578e-05   mcc=  0.000000,
2020-01-19 14:26:28 Ep  5/10 Gstep  6350 Step  1004:  Loss=  0.637348  LR=1.295651e-05   mcc=  0.000000,
2020-01-19 14:27:03 Ep  5/10 Gstep  6400 Step  1054:  Loss=  0.613608  LR=1.280724e-05   mcc=  0.000000,
2020-01-19 14:27:37 Ep  6/10 Gstep  6450 Step    35:  Loss=  0.631628  LR=1.265798e-05   mcc=  0.000000,
2020-01-19 14:28:12 Ep  6/10 Gstep  6500 Step    85:  Loss=  0.640207  LR=1.250871e-05   mcc=  0.000000,
2020-01-19 14:28:46 Ep  6/10 Gstep  6550 Step   135:  Loss=  0.620651  LR=1.235944e-05   mcc=  0.000000,
2020-01-19 14:29:20 Ep  6/10 Gstep  6600 Step   185:  Loss=  0.620827  LR=1.221017e-05   mcc=  0.000000,
2020-01-19 14:29:55 Ep  6/10 Gstep  6650 Step   235:  Loss=  0.600443  LR=1.206090e-05   mcc=  0.000000,
2020-01-19 14:30:29 Ep  6/10 Gstep  6700 Step   285:  Loss=  0.617364  LR=1.191163e-05   mcc=  0.000000,
2020-01-19 14:31:04 Ep  6/10 Gstep  6750 Step   335:  Loss=  0.640479  LR=1.176236e-05   mcc=  0.000000,
2020-01-19 14:31:38 Ep  6/10 Gstep  6800 Step   385:  Loss=  0.636902  LR=1.161310e-05   mcc=  0.000000,
2020-01-19 14:32:12 Ep  6/10 Gstep  6850 Step   435:  Loss=  0.596365  LR=1.146383e-05   mcc=  0.000000,
2020-01-19 14:32:47 Ep  6/10 Gstep  6900 Step   485:  Loss=  0.608958  LR=1.131456e-05   mcc=  0.000000,
2020-01-19 14:33:21 Ep  6/10 Gstep  6950 Step   535:  Loss=  0.619341  LR=1.116529e-05   mcc=  0.000000,
2020-01-19 14:33:56 Ep  6/10 Gstep  7000 Step   585:  Loss=  0.626361  LR=1.101602e-05   mcc=  0.000000,
2020-01-19 14:34:30 Ep  6/10 Gstep  7050 Step   635:  Loss=  0.613354  LR=1.086675e-05   mcc=  0.000000,
2020-01-19 14:35:05 Ep  6/10 Gstep  7100 Step   685:  Loss=  0.612054  LR=1.071748e-05   mcc=  0.000000,
2020-01-19 14:35:39 Ep  6/10 Gstep  7150 Step   735:  Loss=  0.613521  LR=1.056822e-05   mcc=  0.000000,
2020-01-19 14:36:13 Ep  6/10 Gstep  7200 Step   785:  Loss=  0.577611  LR=1.041895e-05   mcc=  0.000000,
2020-01-19 14:36:48 Ep  6/10 Gstep  7250 Step   835:  Loss=  0.605257  LR=1.026968e-05   mcc=  0.000000,
2020-01-19 14:37:22 Ep  6/10 Gstep  7300 Step   885:  Loss=  0.611978  LR=1.012041e-05   mcc=  0.000000,
2020-01-19 14:37:57 Ep  6/10 Gstep  7350 Step   935:  Loss=  0.595855  LR=9.971141e-06   mcc=  0.000000,
2020-01-19 14:38:31 Ep  6/10 Gstep  7400 Step   985:  Loss=  0.585623  LR=9.821873e-06   mcc=  0.000000,
2020-01-19 14:39:05 Ep  6/10 Gstep  7450 Step  1035:  Loss=  0.590831  LR=9.672604e-06   mcc=  0.000000,
2020-01-19 14:39:40 Ep  7/10 Gstep  7500 Step    16:  Loss=  0.621975  LR=9.523336e-06   mcc=  0.000000,
2020-01-19 14:40:14 Ep  7/10 Gstep  7550 Step    66:  Loss=  0.602145  LR=9.374067e-06   mcc=  0.000000,
2020-01-19 14:40:49 Ep  7/10 Gstep  7600 Step   116:  Loss=  0.620748  LR=9.224798e-06   mcc=  0.000000,
2020-01-19 14:41:23 Ep  7/10 Gstep  7650 Step   166:  Loss=  0.602158  LR=9.075530e-06   mcc=  0.000000,
2020-01-19 14:41:58 Ep  7/10 Gstep  7700 Step   216:  Loss=  0.573956  LR=8.926261e-06   mcc=  0.000000,
2020-01-19 14:42:32 Ep  7/10 Gstep  7750 Step   266:  Loss=  0.585606  LR=8.776993e-06   mcc=  0.000000,
2020-01-19 14:43:06 Ep  7/10 Gstep  7800 Step   316:  Loss=  0.585316  LR=8.627724e-06   mcc=  0.000000,
2020-01-19 14:43:41 Ep  7/10 Gstep  7850 Step   366:  Loss=  0.643286  LR=8.478456e-06   mcc=  0.000000,
2020-01-19 14:44:15 Ep  7/10 Gstep  7900 Step   416:  Loss=  0.607292  LR=8.329187e-06   mcc=  0.000000,
2020-01-19 14:44:50 Ep  7/10 Gstep  7950 Step   466:  Loss=  0.599829  LR=8.179918e-06   mcc=  0.000000,
2020-01-19 14:45:24 Ep  7/10 Gstep  8000 Step   516:  Loss=  0.651290  LR=8.030650e-06   mcc=  0.000000,
2020-01-19 14:45:58 Ep  7/10 Gstep  8050 Step   566:  Loss=  0.631367  LR=7.881381e-06   mcc=  0.000000,
2020-01-19 14:46:33 Ep  7/10 Gstep  8100 Step   616:  Loss=  0.602171  LR=7.732113e-06   mcc=  0.000000,
2020-01-19 14:47:07 Ep  7/10 Gstep  8150 Step   666:  Loss=  0.649055  LR=7.582844e-06   mcc=  0.000000,
2020-01-19 14:47:42 Ep  7/10 Gstep  8200 Step   716:  Loss=  0.588507  LR=7.433575e-06   mcc=  0.000000,
2020-01-19 14:48:16 Ep  7/10 Gstep  8250 Step   766:  Loss=  0.649030  LR=7.284307e-06   mcc=  0.000000,
2020-01-19 14:48:50 Ep  7/10 Gstep  8300 Step   816:  Loss=  0.622789  LR=7.135038e-06   mcc=  0.000000,
2020-01-19 14:49:25 Ep  7/10 Gstep  8350 Step   866:  Loss=  0.586203  LR=6.985770e-06   mcc=  0.000000,
2020-01-19 14:49:59 Ep  7/10 Gstep  8400 Step   916:  Loss=  0.597735  LR=6.836501e-06   mcc=  0.000000,
2020-01-19 14:50:34 Ep  7/10 Gstep  8450 Step   966:  Loss=  0.636263  LR=6.687233e-06   mcc=  0.000000,
2020-01-19 14:51:08 Ep  7/10 Gstep  8500 Step  1016:  Loss=  0.612294  LR=6.537964e-06   mcc=  0.000000,
2020-01-19 14:51:42 Ep  7/10 Gstep  8550 Step  1066:  Loss=  0.576644  LR=6.388695e-06   mcc=  0.000000,
2020-01-19 14:52:17 Ep  8/10 Gstep  8600 Step    47:  Loss=  0.627220  LR=6.239427e-06   mcc=  0.000000,
2020-01-19 14:52:51 Ep  8/10 Gstep  8650 Step    97:  Loss=  0.613375  LR=6.090158e-06   mcc=  0.000000,
2020-01-19 14:53:26 Ep  8/10 Gstep  8700 Step   147:  Loss=  0.555401  LR=5.940890e-06   mcc=  0.000000,
2020-01-19 14:54:00 Ep  8/10 Gstep  8750 Step   197:  Loss=  0.593526  LR=5.791621e-06   mcc=  0.000000,
2020-01-19 14:54:34 Ep  8/10 Gstep  8800 Step   247:  Loss=  0.659518  LR=5.642352e-06   mcc=  0.000000,
2020-01-19 14:55:09 Ep  8/10 Gstep  8850 Step   297:  Loss=  0.626100  LR=5.493084e-06   mcc=  0.000000,
2020-01-19 14:55:43 Ep  8/10 Gstep  8900 Step   347:  Loss=  0.634845  LR=5.343815e-06   mcc=  0.000000,
2020-01-19 14:56:18 Ep  8/10 Gstep  8950 Step   397:  Loss=  0.608417  LR=5.194547e-06   mcc=  0.000000,
2020-01-19 14:56:52 Ep  8/10 Gstep  9000 Step   447:  Loss=  0.631753  LR=5.045278e-06   mcc=  0.000000,
2020-01-19 14:57:27 Ep  8/10 Gstep  9050 Step   497:  Loss=  0.596399  LR=4.896010e-06   mcc=  0.000000,
2020-01-19 14:58:01 Ep  8/10 Gstep  9100 Step   547:  Loss=  0.612060  LR=4.746741e-06   mcc=  0.000000,
2020-01-19 14:58:35 Ep  8/10 Gstep  9150 Step   597:  Loss=  0.612795  LR=4.597472e-06   mcc=  0.000000,
2020-01-19 14:59:10 Ep  8/10 Gstep  9200 Step   647:  Loss=  0.615351  LR=4.448204e-06   mcc=  0.000000,
2020-01-19 14:59:44 Ep  8/10 Gstep  9250 Step   697:  Loss=  0.643801  LR=4.298935e-06   mcc=  0.000000,
2020-01-19 15:00:19 Ep  8/10 Gstep  9300 Step   747:  Loss=  0.598972  LR=4.149667e-06   mcc=  0.000000,
2020-01-19 15:00:53 Ep  8/10 Gstep  9350 Step   797:  Loss=  0.624094  LR=4.000398e-06   mcc=  0.000000,
2020-01-19 15:01:28 Ep  8/10 Gstep  9400 Step   847:  Loss=  0.628975  LR=3.851129e-06   mcc=  0.000000,
2020-01-19 15:02:02 Ep  8/10 Gstep  9450 Step   897:  Loss=  0.589227  LR=3.701861e-06   mcc=  0.000000,
2020-01-19 15:02:36 Ep  8/10 Gstep  9500 Step   947:  Loss=  0.609399  LR=3.552592e-06   mcc=  0.000000,
2020-01-19 15:03:11 Ep  8/10 Gstep  9550 Step   997:  Loss=  0.599416  LR=3.403324e-06   mcc=  0.000000,
2020-01-19 15:03:45 Ep  8/10 Gstep  9600 Step  1047:  Loss=  0.583097  LR=3.254055e-06   mcc=  0.000000,
2020-01-19 15:04:20 Ep  9/10 Gstep  9650 Step    28:  Loss=  0.609004  LR=3.104787e-06   mcc=  0.000000,
2020-01-19 15:04:54 Ep  9/10 Gstep  9700 Step    78:  Loss=  0.599495  LR=2.955518e-06   mcc=  0.000000,
2020-01-19 15:05:28 Ep  9/10 Gstep  9750 Step   128:  Loss=  0.627960  LR=2.806249e-06   mcc=  0.000000,
2020-01-19 15:06:03 Ep  9/10 Gstep  9800 Step   178:  Loss=  0.641110  LR=2.656981e-06   mcc=  0.000000,
2020-01-19 15:06:37 Ep  9/10 Gstep  9850 Step   228:  Loss=  0.594384  LR=2.507712e-06   mcc=  0.000000,
2020-01-19 15:07:12 Ep  9/10 Gstep  9900 Step   278:  Loss=  0.639790  LR=2.358444e-06   mcc=  0.000000,
2020-01-19 15:07:46 Ep  9/10 Gstep  9950 Step   328:  Loss=  0.606976  LR=2.209175e-06   mcc=  0.000000,
2020-01-19 15:08:21 Ep  9/10 Gstep 10000 Step   378:  Loss=  0.591287  LR=2.059906e-06   mcc=  0.000000,
2020-01-19 15:08:55 Ep  9/10 Gstep 10050 Step   428:  Loss=  0.591274  LR=1.910638e-06   mcc=  0.000000,
2020-01-19 15:09:29 Ep  9/10 Gstep 10100 Step   478:  Loss=  0.611863  LR=1.761369e-06   mcc=  0.000000,
2020-01-19 15:10:04 Ep  9/10 Gstep 10150 Step   528:  Loss=  0.623777  LR=1.612101e-06   mcc=  0.000000,
2020-01-19 15:10:38 Ep  9/10 Gstep 10200 Step   578:  Loss=  0.631314  LR=1.462832e-06   mcc=  0.000000,
2020-01-19 15:11:13 Ep  9/10 Gstep 10250 Step   628:  Loss=  0.641188  LR=1.313564e-06   mcc=  0.000000,
2020-01-19 15:11:47 Ep  9/10 Gstep 10300 Step   678:  Loss=  0.634291  LR=1.164295e-06   mcc=  0.000000,
2020-01-19 15:12:22 Ep  9/10 Gstep 10350 Step   728:  Loss=  0.609055  LR=1.015026e-06   mcc=  0.000000,
2020-01-19 15:12:56 Ep  9/10 Gstep 10400 Step   778:  Loss=  0.609741  LR=8.657578e-07   mcc=  0.000000,
2020-01-19 15:13:30 Ep  9/10 Gstep 10450 Step   828:  Loss=  0.617953  LR=7.164892e-07   mcc=  0.000000,
2020-01-19 15:14:05 Ep  9/10 Gstep 10500 Step   878:  Loss=  0.628554  LR=5.672206e-07   mcc=  0.000000,
2020-01-19 15:14:39 Ep  9/10 Gstep 10550 Step   928:  Loss=  0.567698  LR=4.179520e-07   mcc=  0.000000,
2020-01-19 15:15:14 Ep  9/10 Gstep 10600 Step   978:  Loss=  0.603719  LR=2.686835e-07   mcc=  0.000000,
2020-01-19 15:15:48 Ep  9/10 Gstep 10650 Step  1028:  Loss=  0.587463  LR=1.194149e-07   mcc=  0.000000,
```
* Finally, here is what the results look like over a lot of hyperparameter choices, for roberta-large.  See the 3rd column from the right, 'last_mcc', and notice that several (but not all) finish with mcc = 0.
* The command for run_glue.py I included at the top corresponds with the last line in this table.  Note that mcc finishes =  0 for all seeds for LR=3e-5, but changing to LR=2e-5, and things are better. (But: see line 7)
```
#     bs       LR  seed  best_mcc  warmup  best_mcc_step  last_mcc  last_mcc_step  last_loss
0   12.0  0.00001   1.0  0.681680   427.0         3600.0  0.623916         7100.0   0.066959
1   12.0  0.00001   2.0  0.678343   427.0         1800.0  0.643239         7100.0   0.028075
2   12.0  0.00001   3.0  0.683793   427.0         3550.0  0.648869         7100.0   0.069469
3   12.0  0.00002   1.0  0.680998   427.0         5450.0  0.635618         7100.0   0.021568
4   12.0  0.00002   2.0  0.651091   427.0         4350.0  0.631009         7100.0   0.042258
5   12.0  0.00002   3.0  0.661668   427.0         4500.0  0.645933         7100.0   0.091654
6   12.0  0.00003   1.0  0.650776   427.0         5150.0  0.598263         7100.0   0.056770
7   12.0  0.00003   2.0  0.403664   427.0          300.0  0.000000         7100.0   0.641559
8   12.0  0.00003   3.0  0.637731   427.0         1800.0  0.597579         7100.0   0.078028
9    8.0  0.00001   1.0  0.667611   641.0         7400.0  0.640759        10650.0   0.046092
10   8.0  0.00001   2.0  0.685714   641.0         4050.0  0.639093        10650.0   0.035902
11   8.0  0.00001   3.0  0.691234   641.0         2650.0  0.625711        10650.0   0.016033
12   8.0  0.00002   1.0  0.670903   641.0         6100.0  0.655837        10650.0   0.071015
13   8.0  0.00002   2.0  0.660777   641.0         8700.0  0.623216        10650.0   0.183001
14   8.0  0.00002   3.0  0.633514   641.0         8450.0  0.585573        10650.0   0.061994
15   8.0  0.00003   1.0  0.521248   641.0          500.0  0.000000        10650.0   0.622017
16   8.0  0.00003   2.0  0.494827   641.0          600.0  0.000000        10650.0   0.614100
17   8.0  0.00003   3.0  0.480823   641.0          550.0  0.000000        10650.0   0.587463
```





Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.