Implement fine-tuning BERT on CoNLL-2003 named entity recognition task
Author: stecklin
Pull request: yes

I added a script for fine-tuning BERT on the CoNLL-2003 named entity recognition task, as an example for token classification. This was requested in #1216.
I followed the structure of the run_glue example, and implemented the data processing in a way suitable for all transformer models (although currently token classification is only implemented for BERT).
The training procedure is as described in the original BERT paper: Only the first sub-token of each CoNLL-tokenized word is classified and contributes to the loss, the remaining sub-tokens are ignored.

Comments:
Thanks for adding this :+1: 

I've one suggestion for some improvement (rfc): can we make the `get_labels()` function a  bit more configurable? E.g. reading the labels from a file `labels.txt` would be great, so I could use other datasets (e.g. GermEval, which has more labels) 🤔 What do you think 🤗

That would be useful for fine-tuning on other datasets, I agree. I tried to keep it very close to the run_glue example for the beginning (where the labels are also fixed), and wait for feedback from the maintainers to know if these kind of extensions are wanted or not ;-)
How about adding a new CLI argument where the user can specify a path to a labels file, and we use the default CoNLL labels when no path was specified?

@stecklin Would be great to have a cli argument for that (+ like your proposed suggestion) :heart: 

This looks awesome, thanks a lot @stecklin and @stefan-it!
Happy to review this when you folks think it's ready.
And ping me if I can help otherwise.

Results are pretty good. I wrote an additional prediction script that outputs a CoNLL compatible format, so that I could verify the results with the official CoNLL evaluation script. Here I fine-tuned a `bert-base-cased` model (5 epochs):

Development set:

```bash
processed 51362 tokens with 5942 phrases; found: 5997 phrases; correct: 5661.
accuracy:  99.10%; precision:  94.40%; recall:  95.27%; FB1:  94.83
              LOC: precision:  96.74%; recall:  96.90%; FB1:  96.82  1840
             MISC: precision:  89.54%; recall:  91.00%; FB1:  90.26  937
              ORG: precision:  92.24%; recall:  92.24%; FB1:  92.24  1341
              PER: precision:  96.06%; recall:  97.99%; FB1:  97.02  1879
```

Test set:

```bash
processed 46435 tokens with 5648 phrases; found: 5712 phrases; correct: 5185.
accuracy:  98.26%; precision:  90.77%; recall:  91.80%; FB1:  91.29
              LOC: precision:  92.12%; recall:  93.23%; FB1:  92.67  1688
             MISC: precision:  79.75%; recall:  82.48%; FB1:  81.09  726
              ORG: precision:  89.23%; recall:  90.31%; FB1:  89.77  1681
              PER: precision:  95.92%; recall:  95.92%; FB1:  95.92  1617
```

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1275?src=pr&el=h1) Report
> Merging [#1275](https://codecov.io/gh/huggingface/transformers/pull/1275?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/80889a0226b8f8022fd9ff65ed6bce71b60ba800?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1275/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1275?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1275   +/-   ##
=======================================
  Coverage   85.98%   85.98%           
=======================================
  Files          91       91           
  Lines       13579    13579           
=======================================
  Hits        11676    11676           
  Misses       1903     1903
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1275?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1275?src=pr&el=footer). Last update [80889a0...c55badc](https://codecov.io/gh/huggingface/transformers/pull/1275?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


The labels can be configured now :heavy_check_mark: 
@stefan-it Do we want to add your prediction script as well? I think that would be very useful, after all NER prediction is not as straightforward as e.g. sequence classification prediction.

@stecklin Thanks 😊 I'm currently using this script:

https://gist.github.com/stefan-it/c39b63eb0043182010f2f61138751e0f

It mainly copies parts from the `evaluate` function. 

But I think a more elegant way would be to fully re-use the evaluate function. The function currently returns the evaluation result, but maybe it could return a tuple of results and predicted tags?

@stefan-it I followed your suggestion, the evaluate function now returns the results and the predictions. I added the argument `--do_predict` to predict on a test set.
@thomwolf I think now would be a good moment for you to have a look. Let me know your feedback!

The mentioned script compatible with new "Transformers" source code ? 

This looks awesome, thank you for the script. I did something similar that worked but this code is totally better. Thanks @stecklin !!

Ok I've reviewed the PR and it looks great, thanks a lot @stecklin and @stefan-it.

I've rebased, switched from `pytorch-transformers` to `transformers` and added `seqeval` in the requirements.

The only missing element is to add a simple usage explanation in the examples readme file at `examples/README.md` which explain:
- how to download the training/testing data,
- an example of command-line to run the script, and
- an example of results with this command line.

@stefan-it do you want to share the command line you use for the above results?

 @thomwolf No problem, here's an example for GermEval 2014 (German NER):

# Data (Download and pre-processing steps)

Data can be obtained from the [GermEval 2014](https://sites.google.com/site/germeval2014ner/data) shared task page.

Here are the commands for downloading and pre-processing train, dev and test datasets. The original data format has four (tab-separated) columns, in a pre-processing step only the two relevant columns (token and outer span NER annotation) are extracted:

```bash
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-train.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > train.txt.tmp
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-dev.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > dev.txt.tmp
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-test.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > test.txt.tmp
```

The GermEval 2014 dataset contains some strange "control character" tokens like `'\x96', '\u200e', '\x95', '\xad' or '\x80'`. One problem with these tokens is, that `BertTokenizer` returns an empty token for them, resulting in misaligned `InputExample`s. I wrote a script that a) filters these tokens and b) splits longer sentences into smaller ones (once the max. subtoken length is reached).

```bash
wget "https://raw.githubusercontent.com/stefan-it/fine-tuned-berts-seq/master/scripts/preprocess.py"
```
Let's define some variables that we need for further pre-processing steps and training the model:

```bash
export MAX_LENGTH=128
export BERT_MODEL=bert-base-multilingual-cased
```

Run the pre-processing script on training, dev and test datasets:

```bash
python3 preprocess.py train.txt.tmp $BERT_MODEL $MAX_LENGTH > train.txt
python3 preprocess.py dev.txt.tmp $BERT_MODEL $MAX_LENGTH > dev.txt
python3 preprocess.py test.txt.tmp $BERT_MODEL $MAX_LENGTH > test.txt
```

The GermEval 2014 dataset has much more labels than CoNLL-2002/2003 datasets, so an own set of labels must be used:

```bash
cat train.txt dev.txt test.txt | cut -d " " -f 2 | grep -v "^$"| sort | uniq > labels.txt
```

# Training

Additional environment variables must be set:

```bash
export OUTPUT_DIR=germeval-model
export BATCH_SIZE=32
export NUM_EPOCHS=3
export SAVE_STEPS=750
export SEED=1
```

To start training, just run:

```bash
python3 run_ner.py --data_dir ./ \
--model_type bert \
--labels ./labels.txt \
--model_name_or_path $BERT_MODEL \
--output_dir $OUTPUT_DIR \
--max_seq_length  $MAX_LENGTH \
--num_train_epochs $NUM_EPOCHS \
--per_gpu_train_batch_size $BATCH_SIZE \
--save_steps $SAVE_STEPS \
--seed $SEED \
--do_train \
--do_eval \
--do_predict
```

If your GPU supports half-precision training, just add the `--fp16` flag. After training, the model will be both evaluated on development and test datasets.

# Evaluation

Evaluation on development dataset outputs the following for our example:

```bash
10/04/2019 00:42:06 - INFO - __main__ -   ***** Eval results  *****
10/04/2019 00:42:06 - INFO - __main__ -     f1 = 0.8623348017621146
10/04/2019 00:42:06 - INFO - __main__ -     loss = 0.07183869666975543
10/04/2019 00:42:06 - INFO - __main__ -     precision = 0.8467916366258111
10/04/2019 00:42:06 - INFO - __main__ -     recall = 0.8784592370979806
```

On the test dataset the following results could be achieved:

```bash
10/04/2019 00:42:42 - INFO - __main__ -   ***** Eval results  *****
10/04/2019 00:42:42 - INFO - __main__ -     f1 = 0.8614389652384803
10/04/2019 00:42:42 - INFO - __main__ -     loss = 0.07064602487454782
10/04/2019 00:42:42 - INFO - __main__ -     precision = 0.8604651162790697
10/04/2019 00:42:42 - INFO - __main__ -     recall = 0.8624150210424085
```

Please let me know if you have more questions 🤗

Hi, great work, thanks for sharing!

I think the argument `overwrite_cache` is not used in the code. I suspect there is a missing if check in the `load_and_cache_examples()` function.

There was something strange with git on this branch (32 files changed...) so I had to do a rebase and force push on your PR @stecklin. Please do a `git reset --hard` to be up-to-date with the new clean state on the remote repo.

Now it looks in order for merging with master.

@stefan-it  Not able to reproduce the above results. 

The best I can for dev dataset get is this  : 

11/27/2019 18:16:38 - INFO - __main__ -   ***** Eval results  *****
11/27/2019 18:16:38 - INFO - __main__ -     f1 = 0.12500000000000003
11/27/2019 18:16:38 - INFO - __main__ -     loss = 1.6597001552581787
11/27/2019 18:16:38 - INFO - __main__ -     precision = 0.2
11/27/2019 18:16:38 - INFO - __main__ -     recall = 0.09090909090909091

Any pointers on what I am missing ?


@oneraghavan What version/commit of `transformers` are you using? Do you use the GermEval dataset or another one? I'll check the example :)

@stefan-it  Thanks for quick response :) . I am using commit from oct 24. Has anything changed since then ? I am following the same steps said in examples/readme.md . Let me know if you want me to check with latest commit .

Could you try to use the latest `master` version? I re-do the experiment on GermEval, here are the results:

Evaluation on dev set:

```bash
f1 = 0.8702821546353977
loss = 0.07410008722260086
precision = 0.8530890804597702
recall = 0.8881824981301422
```

Evaluation on test set:

```bash
f1 = 0.860249697946033
loss = 0.07239935705435063
precision = 0.8561808561808562
recall = 0.8643573972159275
```

@stefan-it  I tried with the latest master. Not able to reproduce. I am exactly following the instructions given in readme.md. The following are the results I am getting .

11/28/2019 09:34:50 - INFO - __main__ -   ***** Eval results  *****
11/28/2019 09:34:50 - INFO - __main__ -     f1 = 0.12500000000000003
11/28/2019 09:34:50 - INFO - __main__ -     loss = 1.1732935905456543
11/28/2019 09:34:50 - INFO - __main__ -     precision = 0.2
11/28/2019 09:34:50 - INFO - __main__ -     recall = 0.09090909090909091

Can you check if there are any other parameters that is being given in run_ner.py parameters . How much epoch are you training ? 

In order to reproduce the conll score reported in BERT paper (92.4 bert-base and 92.8 bert-large) one trick is to apply a truecaser on article titles (all upper case sentences) as preprocessing step for conll train/dev/test. This can be simply done with the following method.

```
#https://github.com/daltonfury42/truecase
#pip install truecase
import truecase
import re




# original tokens
#['FULL', 'FEES', '1.875', 'REOFFER', '99.32', 'SPREAD', '+20', 'BP']

def truecase_sentence(tokens):
   word_lst = [(w, idx) for idx, w in enumerate(tokens) if all(c.isalpha() for c in w)]
   lst = [w for w, _ in word_lst if re.match(r'\b[A-Z\.\-]+\b', w)]

   if len(lst) and len(lst) == len(word_lst):
       parts = truecase.get_true_case(' '.join(lst)).split()

       # the trucaser have its own tokenization ...
       # skip if the number of word dosen't match
       if len(parts) != len(word_lst): return tokens

       for (w, idx), nw in zip(word_lst, parts):
           tokens[idx] = nw

# truecased tokens
#['Full', 'fees', '1.875', 'Reoffer', '99.32', 'spread', '+20', 'BP']

```

Also, i found useful to use : very small learning rate (5e-6) \ large batch size (128) \ high epoch num (>40).

With these configurations and preprocessing,  I was able to reach 92.8 with bert-large.