Does anyone have the XLNet (and ALBERT) NER performance on CONLL-2003 
Author: bugface
Pull request: no

# ❓ Questions & Help
Most transformer models in the library can be fine-tuned for NER tasks.

In https://huggingface.co/transformers/v2.2.0/examples.html#named-entity-recognition, the performances of the roberta, bert, and distilbert have been reported. 

However, I did not find performances achieved by other models like XLNet.  

By any chance, does anyone experiment with other models and can report the performances for models like XLNet and albert on CONLL-2003? 


Comments:
With ALBERT v1:

https://github.com/huggingface/transformers/pull/1683#issuecomment-556001607

I got better results using the recent integrated ELECTRA model :)



@stefan-it j/w but do you know why bert-lg fine-tuned is listed as achieving 92.8 f1 on conll03 in [this paper?](https://paperswithcode.com/sota/named-entity-recognition-ner-on-conll-2003) Noticing it's over a pt higher f1 than Transformers's version. 

@stefan-it 

Your albert result is very close to what I got. 

By any chance, did you experiment with the XLNet?

My problem is:
In https://github.com/stevezheng23/xlnet_extension_tf, the author reported the ner performance as 0.9267. But I can only obtain a performance of 0.7626 with the same batch size and learning rate but longer training steps using the this library. I would like to confirm if the problem is my implementation but there is no baseline on XLNet.

@stefan-it Thank you for your contribution to the Electra model finetuned on CoNLL03. I see you shared the weights in this repository. Could you please share the license for these weights? I could not find a model card for it.

Hi @stefan-it !
I know that this issue is not about electra, but I have the same question regarding electra too :sweat_smile: 
I ran NER CoNLL-2003 training with electra small like this:

`python examples/ner/run_ner.py --model_type electra --model_name_or_path google/electra-small-discriminator --do_train --do_eval --do_predict --data_dir /home/corpora/ner/conll2003 --labels /home/corpora/ner/conll2003/labels.txt --num_train_epochs 6 --per_gpu_train_batch_size 256 --per_gpu_eval_batch_size 256 --max_seq_length 128 --output_dir /home/models/electra/conll2003 --evaluate_during_training --save_steps 1000 --logging_steps 1000 --overwrite_output_dir`

But got only 83.20% in F1. I know you ran electra small on the actual electra repository, but can you describe what you did in hugging face?

Hi @petulla , the problem with the BERT paper is, that they've used document context for each token during evaluation. See e.g. this [discussion](https://github.com/allenai/allennlp/pull/2067#issuecomment-443961816) in the AllenNLP repo :)

@guillaume-be I normally use MIT for all trained models (so should be fine for the fine-tuned ELECTRA model as well) :)

@pvcastro Try to use a smaller batch size, for example with the configuration:

```json
{
    "data_dir": "./data_en",
    "labels": "./data_en/labels.txt",
    "model_name_or_path": "google/electra-small-discriminator",
    "output_dir": "electra-small-en-1",
    "max_seq_length": 128,
    "num_train_epochs": 5,
    "per_gpu_train_batch_size": 16,
    "save_steps": 878,
    "seed": 1,
    "do_train": true,
    "do_eval": true,
    "do_predict": true,
    "--fp16": true
}
```

You should be able to reach ~88.35% on the test set and 92.13% on development set.

I just used the latest `master` version of Transformers + saved the JSON-based configuration as `config-electra-small.json`, then you can run training via `python3 run_ner.py config-electra-small.json` :)

Hi @stefan-it , thanks for the input! I ran this config and got a pretty similar result. I had no idea that a larger batch size had this impact. I get 85% for bs 128. Is this for all transformer models, or for electra only? Or for sequence labeling tasks, perhaps? Do you know why this happens? 

For Transformer-based model on sequence labeling tasks batch sizes of 8, 16 or 32 are a good choice for hyper-parameter search. So e.g. the [BERT paper](https://arxiv.org/abs/1810.04805) mentions [16. 32] for their experiments (see appendix A.3).

And there's an interesting paper from Reimers and Gurevych about hyper-parameters for lstm-based networks for sequence labeling, but in my opinion their recommendations for batch sizes are also valid for Transformer-based models: ["Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks"](https://arxiv.org/abs/1707.06799), see section 7.10 :)

Thanks @stefan-it , I'll dig into these references! 

Hi @stefan-it, it is quite strange that I got F1 0 for conll-03 dataset with **electra-large-discriminator**, while 0.88 and 0.91 with **small** and **base** models. The other settings are the same for the three models. Have you encountered this?

Hi @lkluo , I can't remember fine-tuning instabilities with the ELECTRA models... but could you paste the fine-tuning configuration that you've used 🤔



> Hi @lkluo , I can't remember fine-tuning instabilities with the ELECTRA models... but could you paste the fine-tuning configuration that you've used 🤔

Thanks @stefan-it. I think I may figure it out after I checked the loss, which converges slowly and which value remains 9.x after 5 epochs. Then I lower the learning rate from default **5e-5** to 10 times smaller, i.e. **5e-6**, then I can get 0.92 score. 

I also fine-tuned with **BERT-large** model using the default learning rate, and I am able to get a reasonable f1 score. Is there any special about **ELECTRA** large settings? Does batch size matter? It is limited to 12 due to GPU in my case. I saw somewhere people suggest larger batch size, smaller learning rate and longer training duration to reproduce good results. Could you share your configuration of **ELECTRA-LARGE**? Thanks a lot?

p.s., my configuration:

> {
    "data_dir": "",
    "labels": "",
    "model_name_or_path": "google/electra-large-discriminator",
    "output_dir": "",
    "max_seq_length": 128,
    "num_train_epochs": 5,
    "per_device_train_batch_size": 12,
    "save_steps": 750,
    "seed": 1,
    "do_train": true,
    "do_eval": true,
    "do_predict": true
}