Strange difference between NER results in 2.8.0 and previous versions
Author: pvcastro
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using: Albert

Language I am using the model on: English and Portuguese

The problem arises when using:
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

## To reproduce

Not sure if this is a bug, but for the same seed and hyperparameters, I'm getting quite different results for different benchmarks on transformers 2.8.0 and 2.7.0.

Steps to reproduce the behavior:

1. For 2.7.0: `python examples/ner/run_ner.py --data_dir <data-dir> --labels <labels-path> --model_type albert --model_name_or_path <albert-model> --output_dir <output-dir> --max_seq_length 512 --overwrite_output_dir --do_train --do_eval --do_predict --per_gpu_train_batch_size 16 --per_gpu_eval_batch_size 8 --learning_rate 5e-05 --num_train_epochs 5 --save_steps 878 --seed 1`

2. For 2.8.0: `python examples/ner/run_ner.py config.json`. Here's the config file for each:

2.1. For CoNLL: 
```
{
    "data_dir": "/home/corpora/ner/conll2003",
    "labels": "/home/corpora/ner/conll2003/labels.txt",
    "model_name_or_path": "albert-base-v2",
    "output_dir": "/home/models/albert/conll2003",
    "max_seq_length": 512,
    "num_train_epochs": 5,
    "per_gpu_train_batch_size": 16,
    "save_steps": 878,
    "seed": 1,
    "do_train": true,
    "do_eval": true,
    "do_predict": true,
    "overwrite_output_dir": true
}
```
2.2. For HAREM: 
```
{
    "data_dir": "/home/corpora/ner/harem/selective",
    "labels": "/home/corpora/ner/harem/selective/labels.txt",
    "model_name_or_path": "/home/models/albert/base_pt_brwac_en/albert_ckp_1176k",
    "output_dir": "/home/models/albert/base_pt_brwac_en/albert_ckp_1176k/harem",
    "max_seq_length": 512,
    "num_train_epochs": 5,
    "per_gpu_train_batch_size": 16,
    "save_steps": 878,
    "seed": 1,
    "do_train": true,
    "do_eval": true,
    "do_predict": true,
    "overwrite_output_dir": true
}
```
3. For English CoNLL-2003 used albert-base-v2, and for Portuguese HAREM used a checkpoint for a version we're training.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Following are the results I'm getting for each of the 4 runs. I think the results should be a lot closer to each other, considering the same hyperparameters and seeds are being used, right?

1. 2.7.0 / CoNLL-2003:

```
INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir='/home/corpora/ner/conll2003', device=device(type='cuda'), do_eval=True, do_lower_case=False, do_predict=True, do_train=True, eval_all_checkpoints=False, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, keep_accents=None, labels='/home/corpora/ner/conll2003/labels.txt', learning_rate=5e-05, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_seq_length=512, max_steps=-1, model_name_or_path='albert-base-v2', model_type='albert', n_gpu=1, no_cuda=False, num_train_epochs=5.0, output_dir='/home/models/albert/conll2003-2.7.0', overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=8, per_gpu_train_batch_size=16, save_steps=878, seed=1, server_ip='', server_port='', strip_accents=None, tokenizer_name='', use_fast=None, warmup_steps=0, weight_decay=0.0)

INFO - __main__ -     f1 = 0.8424940638466273
INFO - __main__ -     loss = 0.184873234364002
INFO - __main__ -     precision = 0.836973615236764
INFO - __main__ -     recall = 0.8480878186968839
```

2. 2.8.0 / CoNLL-2003:

```
INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/home/models/albert/conll2003', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=False, per_gpu_train_batch_size=16, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=878, save_total_limit=None, no_cuda=False, seed=1, fp16=False, fp16_opt_level='O1', local_rank=-1)

INFO - __main__ -     precision = 0.890553401480437
INFO - __main__ -     recall = 0.8946529745042493
INFO - __main__ -     f1 = 0.892598480833775
INFO - __main__ -     loss = 0.15480460636704224
```
3. 2.7.0 / HAREM

```
INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir='/home/corpora/ner/harem/selective', device=device(type='cuda'), do_eval=True, do_lower_case=False, do_predict=True, do_train=True, eval_all_checkpoints=False, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, keep_accents=None, labels='/home/corpora/ner/harem/selective/labels.txt', learning_rate=5e-05, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_seq_length=512, max_steps=-1, model_name_or_path='albert-base-v2', model_type='albert', n_gpu=1, no_cuda=False, num_train_epochs=5.0, output_dir='/home/models/albert/eduardo/base_pt_brwac_en/albert_ckp_1176k/harem-2.7.0', overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=8, per_gpu_train_batch_size=16, save_steps=878, seed=1, server_ip='', server_port='', strip_accents=None, tokenizer_name='', use_fast=None, warmup_steps=0, weight_decay=0.0)

INFO - __main__ -     f1 = 0.805560264451602
INFO - __main__ -     loss = 0.11168152010335869
INFO - __main__ -     precision = 0.8184636582845333
INFO - __main__ -     recall = 0.7930574098798397
```

4. 2.8.0 / HAREM

```
INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/home/models/albert/eduardo/base_pt_brwac_en/albert_ckp_1176k/harem', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=False, per_gpu_train_batch_size=16, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=878, save_total_limit=None, no_cuda=False, seed=1, fp16=False, fp16_opt_level='O1', local_rank=-1)

INFO - __main__ -     precision = 0.7228915662650602
INFO - __main__ -     recall = 0.7209612817089452
INFO - __main__ -     f1 = 0.7219251336898397
INFO - __main__ -     loss = 0.14575799524457683
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-99-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, Tesla V100-SXM2
- Using distributed or parallel set-up in script?: No


Comments:
FYI, ran in 2.9.0 as well and got the same results.

Hi @pvcastro , so the CoNLL results are looking good 🤔 Because there's an improviment with 2.8.

Regarding to HAREM: Could you verify that there's a `tokenizer_config.json` located in the `/home/models/albert/base_pt_brwac_en/albert_ckp_1176k` folder? Is it an uncased or cased model?

Hi @stefan-it !
There's an improvement for CoNLL, but does this improvement make sense? Shouldn't the results be closer to each other?

As for HAREM, there's a tokenizer_config.json, but it's empty...there are only the {}. The model is **cased**.

Could you use the following `tokenizer_config.json`:

```json
{"do_lower_case": false, "max_len": 512, "init_inputs": [], "strip_accents":false}
```

By default ALBERT models are lower-cased (if you don't specify it in that configuration):

https://github.com/huggingface/transformers/blob/26dad0a9fab0694ce1128d91838b9ca78e1554f6/src/transformers/tokenization_albert.py#L117

(Because the original ALBERT model is an uncased model).

Then please re-run the experiment (and also clear the `cached_*` files in your data folder/examples folder) - this should hopefully improve performance :)

Oh I see @stefan-it , it worked! Got 80.55 for HAREM doing this. So the reason this happened after the version change it's because before, the do_lower_case was passed as default from run_ner.py as False, and now this doesn't happen in 2.8.0 anymore. And for CoNLL the effect was the opposite...should be setting --do_lower_case for 2.7.0, since the default for this version is False and the model is uncased. 
Thanks!

Hi @stefan-it !
If you don't mind me asking this in this closed issue, now I'm facing the same problem concerning a portuguese electra model I'm benchmarking. 

I did what you suggested for ALBERT model, and for ELECTRA I'm doing the same thing...I create a tokenizer_config.json doing the same thing. However, for ELECTRA I'm getting a huge difference between the NER models trained using transformers 2.9.1 (happened with 2.7.0 and 2.8.0 too) and the ones trained with the chunk task in electra's original repository. 

This is what I'm running there as parameter:

`python /home/repositorios/electra/run_finetuning.py --data-dir /home/models/electra/electra_pt_small_pp_uncased_512/electra_ckp_100k/ner/harem/selective/electra_pt_small_pp_uncased_512 --model-name electra_pt_small_pp_uncased_512 --hparams {"task_names": ["chunk"], "model_size": "small", "max_seq_length": 512, "learning_rate": 3e-4, "num_train_epochs": 6, "train_batch_size": 32, "do_lower_case": true}`

And this is what I use for transformers:

`python /home/repositorios/transformers/examples/token-classification/run_ner.py --model_type electra --model_name_or_path /home/models/electra/electra_pt_small_pp_uncased_512/electra_ckp_100k --do_train --do_eval --do_predict --data_dir /home/corpora/ner/harem/selective --labels /home/corpora/ner/harem/selective/labels.txt --num_train_epochs 6 --per_gpu_train_batch_size 32 --per_gpu_eval_batch_size 32 --max_seq_length 512 --output_dir /home/models/electra/electra_pt_small_pp_uncased_512/electra_ckp_100k/ner/harem/selective --evaluate_during_training --save_steps 1000 --logging_steps 1000 --overwrite_output_dir --overwrite_cache`

This is the tokenizer_config.json used for this checkpoint:

`{"do_lower_case": true, "max_len": 512, "init_inputs": [], "strip_accents":false}`

This is what I do to convert the electra tf checkpoint to pytorch:

`python ${REPO_DIR}/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path ${CKP_DIR} --pytorch_dump_path ${CKP_DIR}/pytorch_model.bin --discriminator_or_generator discriminator --config_file ${CKP_DIR}/config.json`

I use the very same electra config.json for both.

For this same checkpoint and parameters above, I get 36,33% using transformers and 71.42% using the electra repository. Do you know anything I should be doing differently? 

Hi @pvcastro 

for an uncased model the convention is to use the "strip accents" feature, so please try to set `strip_accents": true` in your tokenizer config. Hopefully this solves the problem :)

Hi @stefan-it , thanks for the response!

I changed this single property in my tokenizer_config.json as you suggested, but made no difference. Got the exact same results :thinking: 

Just to be clear, I'm using overwrite-output-dir and overwrite-cache to make sure that I'm not using any old files, and the tokenizer_config.json in the output-dir is updated with strip_accents true.

That's really strange. Could you also set the same learning rate as in the ELECTRA example (Transformers uses 5e-5 as default).

I had a look at the ELECTRA fine-tuning example a while ago -> I did implement an own evaluation class that uses `seqeval` like in Transformers. The results were a bit different between `seqeval` and ELECTRA evaluation output, but definitely not ~35%!

Oh, that helped a lot. Now it went to 65.41%. Do you think the remaining difference is due to the evaluation differences?

Strange that for english CoNLL 2003 you're able to get good results using the standard 5e-5 learning rate, right?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.