Poor Training and evaluation accuracy even with low loss
Author: amithadiraju1694
Pull request: no
Labels: name: wontfix

@spolu @cynthia @thomwolf @davidefiocco  

I initially used command line arguments to run the `run_classifier.py`, using `cola` as a task, for `Sequence Classification` on  a custom data set, I was able to execute and get the results , but they were very poor: an evaluation accuracy of 0.0 and loss of close to .9.

Then I decided to write a small wrapper class similar to `BertForSequenceClassificaiton` in `modeling.py`, to invoke a basic `BERT Model` from set of pre-trained classes using `BERT weights`:

ex: 
`model = BERT_MULTILABEL_SEQ_Classify.from_pretrained('bert-base-uncased', num_labels = 8)`

- > BERT_MULTILABEL_SEQ_Classify is my wrapper class


up on executing this line of code I see the following log information:


```
06/25/2019 20:40:12 - INFO - pytorch_pretrained_bert.modeling -   loading archive file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz from cache at /home/dbi/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba
06/25/2019 20:40:12 - INFO - pytorch_pretrained_bert.modeling -   extracting archive file /home/dbi/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba to temp dir /tmp/tmp8hjgcb4r
06/25/2019 20:40:17 - INFO - pytorch_pretrained_bert.modeling -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

06/25/2019 20:40:34 - INFO - pytorch_pretrained_bert.modeling -   Weights of BERT_MULTILABEL_SEQ_Clasfify not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
06/25/2019 20:40:34 - INFO - pytorch_pretrained_bert.modeling -   Weights from pretrained model not used in BERT_MULTILABEL_SEQ_Clasfify: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
```


I ignored this log information and proceeded with training my data, the results were similar to my initial attempts ( which were really poor ). Then I realized that, because this model is not using any of the pre-trained weights; it's performing poorly on my classification task.

I do have the `uncased_L-12_H-768_A-12` file in the same path as this program is running. It's contents look something like this
![bertpls](https://user-images.githubusercontent.com/23751321/60150916-804a9600-978f-11e9-804f-a131dad0a4d3.png)

So, my question's how do I properly invoke the `BERTModel ` so that it's pre-trained weights are also loaded along with it ? I understand that I might have either messed up with signature of the BERT Model or may need to point the bert model to look for weights, but not sure how. Every time I use the `from_pretrained` method to load pre-trained weights, I can see some files being downloaded from s3 buckets but looks like those files do not contain pre-trained weights.  It might also be the case that the conversion from `pytorch.bin` (downloaded from s3 buckets) to checkpoint files is not working as expected in `ubuntu 18.04`. Any help is much appreciated. TIA



Note: I thought including code for my entire wrapper class would be irrelevant to this question, so I didn't do so. 

      I have:

      `
        Python 3.6.8
        Pytorch 1.1.0
      `

Comments:
@amit8121  can you just sent the run_classifier.py python command that you are using in terminal to run this model

@himanshututeja1998 

Thanks for the response. This is a similar command to what I used for `run_classifier.py`

```
export BERT_BASE_DIR=./path/to/uncasedweightsfolder/

python bert/run_classifier.py \
--task_name=cola \
--do_train=true \
--do_eval=true \
--data_dir=./data \
--vocab_file=$BERT_BASE_DIR/vocab.txt \
--bert_config_file=$BERT_BASE_DIR/bert_config.json \
--init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
--max_seq_length=128 \
--train_batch_size=32 \
--learning_rate=2e-5 \
--num_train_epochs=3.0 \
--output_dir=./bert_output/
```

BTW, I also had to change the `get_labels` function of this code to support multi label classification ( 8 in my case). 

I believe the issue here has more to do with the kind of weights file I downloaded as a part of my uncased folder. From my knowledge, I believe task of `cola` might not be too suitable for multi label classification. And our team felt this command line methodology is not ideal for production grade scenario, so we decided to develop a `BERT wrapper` and train it.

@amit8121   Use this ::::::::
python/python3 run_classifier.py --task_name cola --do_eval --do_lower_case --data_dir DATA_DIR PATH/ --bert_model PATH TO PRETRAINNED WEIGHT (uncased_L-12_H-768_A-12)/ --max_seq_length 128 --train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir MODEL_OUTPUT

@himanshututeja1998 

I appreciate your willingness to help, but even if the above command runs and trains successfully, of which I have some reservations on, it still doesn't entirely solve our problem i.e., we ultimately want to make production grade code using the pre-trained weights rather than using command line tools. Thanks for the response.

> @amit8121 Use this ::::::::
> python/python3 run_classifier.py --task_name cola --do_eval --do_lower_case --data_dir DATA_DIR PATH/ --bert_model PATH TO PRETRAINNED WEIGHT (uncased_L-12_H-768_A-12)/ --max_seq_length 128 --train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir MODEL_OUTPUT


Still getting the same results:

```
INFO:tensorflow:***** Eval results *****
INFO:tensorflow:  eval_accuracy = 0.0
INFO:tensorflow:  eval_loss = 3.9577928
INFO:tensorflow:  global_step = 25
INFO:tensorflow:  loss = 3.9577928
```

for the command:
`
python3 bert/run_classifier.py --task_name=cola --do_train=true --do_eval=true --data_dir=./data --vocab_file=$BERT_BASE_DIR/vocab.txt --bert_model=$BERT_BASE_DIR/pytorch_model.bin --bert_config_file=$BERT_BASE_DIR/bert_config.json --max_seq_length=128 --train_batch_size=32 --learning_rate=2e-5 --num_train_epochs=3.0 --output_dir=./bert_output/`

just an FYI.

@spolu @cynthia @thomwolf 

I think I might have figured some parts of the issue out.

When I load bert model using `BERT_CLASS.from_pretrained('bert-base-uncased')` only the `pytorch.bin` file gets downloaded form S3 bucket in to a temp folder under system's /tmp folder. Interestingly, `BertForSequenceClassification` class couldn't load pre-trained weights form this file. 

But, when I copied the `pytorch.bin` file to folder `uncased_L-12_H-768_A-12`, with all the above mentioned files in tact and changed the path to `from_pretrained` file to point to uncased folder; I was able to load the model with pre-trained weights. My results are much better  than previous rounds (I only ran it for 4 epochs though):

```
For each batch in an epoch:
Training -   ~ 70 % accuracy , with loss ~0.10; 
Evaluation -  About  ~ 10% accuracy, with loss ~ 0.35

```

I did use `BertAdam` along with decent learning rate scheduling. But the results are a bit under whelming and there's a huge difference in training and evaluation accuracies, does it have anything to do with the size of the data ? I only have ~ 350 rows of training data and ~ 100 rows of testing data.

@amit8121  on which dataset you are training this because this can also due to mismatched task name "cola / mnli " etc. 

@himanshututeja1998 

Underlying data set shouldn't matter as long as you've transformed it in a format required for `Sequence Classification`, I made sure that I `pre-processed` my data as per `BertForSeqeunceClassification` requirements ; my initial issue was not being able to use pre-trained weights which I figured out and results were much better, yet not close to SOTA. My training and evaluation losses go as low as 0.06 on average for each epoch, yet accuracy hovers around 65 - 70 % which is strange. This issue seems to be quite common for both GPT and BERT models, there are multiple `issues` on these topics .I believe the issue is with the way we're trying to transform our target variables. Hope the authors would find time to respond.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.