Classifier example not training on CoLa data
Author: ironflood
Pull request: no

Hi,

I obtained strange classification eval results (always predicting the same label) when trying out the `run_classifier.py` after cloning the repo (no modif) so to dig a bit more I rebalanced the CoLA dataset (train.tsv and dev.tsv) to have a better understanding of what is happening. When running the classifier example the network still doesn't learn and keeps predicting the same label. Any idea why? Was thinking the issue might be in saving / loading the state_dict after training so I bypassed it by using directly the model trained in train loop but to no avail, the results are the same. Any pointers? Am I missing something big here?

The rebalanced & randomized CoLA dataset (simply sampled down the majority class to minority one).
https://drive.google.com/file/d/1_QjnknEusQZgbhTqJcFhBLDrRZXQQRZ6/view?usp=sharing

My training command:
`run_classifier.py --task_name CoLA --do_train --do_eval --data_dir ../data/CoLA-BALANCED/ --bert_model bert-base-multilingual-cased --max_seq_length 128 --train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir output/cola`

The output digits are always in favor of one class, example:
 ```
eval batch#0 print(digits)
[[ 0.05559488  0.00027706]
 [ 0.05565088  0.00031819]
 [ 0.05568472  0.00032058]
 [ 0.05567677  0.00027802]
 [ 0.05567478  0.00028492]
 [ 0.05566313  0.00030545]
 [ 0.05558664  0.00028202]
 [ 0.05569095  0.0002955 ]]

eval batch#1 print(digits)
[[ 0.05566129  0.00032648]
 [ 0.05567207  0.00029943]
 [ 0.05569698  0.00030764]
 [ 0.05563145  0.00030007]
 [ 0.05566984  0.00032966]
 [ 0.05565657  0.00032679]
 [ 0.05569271  0.00030621]
 [ 0.05561762  0.00030394]]
```

Some training examples:
```
01/18/2019 16:28:12 - INFO - __main__ -   *** Example ***
01/18/2019 16:28:12 - INFO - __main__ -   guid: train-0
01/18/2019 16:28:12 - INFO - __main__ -   tokens: [CLS] Ent ##hus ##ias ##tic golf ##ers with large hand ##icap ##s can be good company . [SEP]
01/18/2019 16:28:12 - INFO - __main__ -   input_ids: 101 63412 15471 15465 13275 32288 10901 10169 12077 15230 73130 10107 10944 10347 15198 12100 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   label: 1 (id = 1)
01/18/2019 16:28:12 - INFO - __main__ -   *** Example ***
01/18/2019 16:28:12 - INFO - __main__ -   guid: train-1
01/18/2019 16:28:12 - INFO - __main__ -   tokens: [CLS] The horse jump ##ed over the fe ##nce . [SEP]
01/18/2019 16:28:12 - INFO - __main__ -   input_ids: 101 10117 30491 54941 10336 10491 10105 34778 12150 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   label: 1 (id = 1)
01/18/2019 16:28:12 - INFO - __main__ -   *** Example ***
01/18/2019 16:28:12 - INFO - __main__ -   guid: train-2
01/18/2019 16:28:12 - INFO - __main__ -   tokens: [CLS] Brown equipped Jones a camera . [SEP]
01/18/2019 16:28:12 - INFO - __main__ -   input_ids: 101 12623 41880 12298 169 26665 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   label: 0 (id = 0)
01/18/2019 16:28:12 - INFO - __main__ -   *** Example ***
01/18/2019 16:28:12 - INFO - __main__ -   guid: train-3
01/18/2019 16:28:12 - INFO - __main__ -   tokens: [CLS] I destroyed there . [SEP]
01/18/2019 16:28:12 - INFO - __main__ -   input_ids: 101 146 24089 11155 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:28:12 - INFO - __main__ -   label: 0 (id = 0)
```

Some eval examples:
```
01/18/2019 16:32:04 - INFO - __main__ -   *** Example ***
01/18/2019 16:32:04 - INFO - __main__ -   guid: dev-0
01/18/2019 16:32:04 - INFO - __main__ -   tokens: [CLS] Dana walk ##ed and Leslie ran . [SEP]
01/18/2019 16:32:04 - INFO - __main__ -   input_ids: 101 27149 33734 10336 10111 25944 17044 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   label: 1 (id = 1)
01/18/2019 16:32:04 - INFO - __main__ -   *** Example ***
01/18/2019 16:32:04 - INFO - __main__ -   guid: dev-1
01/18/2019 16:32:04 - INFO - __main__ -   tokens: [CLS] The younger woman might have been tall and , and the older one def ##inite ##ly was , bl ##ond . [SEP]
01/18/2019 16:32:04 - INFO - __main__ -   input_ids: 101 10117 27461 18299 20970 10529 10590 36243 10111 117 10111 10105 18757 10464 100745 100240 10454 10134 117 21484 26029 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   label: 0 (id = 0)
01/18/2019 16:32:04 - INFO - __main__ -   *** Example ***
01/18/2019 16:32:04 - INFO - __main__ -   guid: dev-2
01/18/2019 16:32:04 - INFO - __main__ -   tokens: [CLS] What the water did to the bot ##tle was fill it . [SEP]
01/18/2019 16:32:04 - INFO - __main__ -   input_ids: 101 12489 10105 12286 12172 10114 10105 41960 16406 10134 20241 10271 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/18/2019 16:32:04 - INFO - __main__ -   label: 0 (id = 0)
```



Comments:
Try this for CoLA: `--bert_model bert-base-uncased --do_lower_case`. 
You may also need to increase `num_train_epochs` or `learning_rate` a little. 


Thanks for your suggestions, but when following them nothing changes, it always predict one class regardless.
Tried:
- 3 and 10 epochs
- different learning rates: 5e-3, 5e-4, 5e-5
- all done with --bert_model bert-base-uncased --do_lower_case (but at the same time being able to use the multi language + Cased input would be an important plus)

One of the new command tested:
`run_classifier.py --task_name CoLA --do_train --do_eval --data_dir ../data/CoLA-BALANCED/ --bert_model bert-base-uncased --do_lower_case --max_seq_length 128 --train_batch_size 32 --learning_rate 5e-4 --num_train_epochs 10.0 --output_dir output/cola/`

The console output:
https://gist.github.com/ironflood/150618e6f9cb56572729bf282c9cd2aa

The rebalanced train & test dataset didn't change from first message and I checked its validity.

I think 5e-3 and 5e-4 are too high; they are even higher than pre-training. 

I'm getting eval_accuracy = 0.76 (i.e. not one class) with `--learning_rate 5e-5 --num_train_epochs 3.0` (other arguments and dataset are same as yours), so I have no idea why you didn't. 


Thanks for testing it out. I tried again 5e-5 and I'm getting close to your result as well. As I tried earlier this LR could it be that sometimes it doesn't converge properly because of random seed? Another question: testing out 5e-5 and 5e-6 on the cased multi language model I'm getting approx 0.68 acc instead of 0.78 with lowercased standard model on 3 and 10 epoch training, is it only because of the case adding more difficulty?

No, random seed is always same. See [this line](https://github.com/huggingface/pytorch-pretrained-BERT/blob/0a9d7c7edb20a3e82cfbb4b72515575543784823/examples/run_classifier.py#L421) in the code. 
For multilingual models, you should refer [multilingual.md](https://github.com/google-research/bert/blob/master/multilingual.md#results).

Closing since there is no recent activity. Feel free to re-open if needed.