High F1 score. But poor accuracy during Inference due to tokenisation
Author: sudharsan2020
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

I am using **Bert-Base-cased** model to train my custom Named entity recognition(NER) model with a sequence length of **512**.

Language I am using the model on: **English**

The problem arises when using:
* [ ] the official example scripts: **token-classification/run_ner.py**

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: **Named entity recognition**
* [ ] my own task or dataset: **Custom Dataset**

## To reproduce

Steps to reproduce the behavior:

1.Use the default NER Pipeline to load the custom trained model
`        self.model_prediction_pipeline = pipeline(
            "ner",
            model=model_path,
            tokenizer= model_path,
            grouped_entities=True
        )`
2. I've attached the Evaluation results of the model.
`eval_loss = 0.021479165139844086`
`eval_precision = 0.8725970149253731`
`eval_recall = 0.8868932038834951`
`eval_f1 = 0.8796870297923562`
`epoch = 5.0`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
1. Model should produce a good accuracy corresponding to the F1 score.
2. However during Inference, I am not getting an accuracy over **30%**
3. Not sure if the inconsistent tokenisation leads to poor results.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.0
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.4.0 
- Tensorflow version (GPU?):NA
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No



Comments:
Hello! Why do you believe the tokenization to be the issue here?

@LysandreJik Thanks for reaching out.

Please find my observations with the inconsistency in the Tokenizer(possible issue), since I was using the HuggingFace provided script for training the custom NER Model.

**1. Expected name:**
AUDLEY THOMPSON

Predicted name:
{'entity_group': 'B-PER', 'score': 0.9993636608123779, 'word': 'AUDLE'}, 
{'entity_group': 'I-PER', 'score': 0.8126876294612885, 'word': '##Y THOMPS'}

**Issue:**
Last two letters got skipped

**2. Expected name:**
DANIEL, BROWN

Predicted name:
{'entity_group': 'B-PER', 'score': 0.9559168517589569, 'word': 'DAN'}, 
{'entity_group': 'I-PER', 'score': 0.9092316627502441, 'word': '##IE'}, 
{'entity_group': 'B-PER', 'score': 0.5071505904197693, 'word': '##L'}, 
{'entity_group': 'I-PER', 'score': 0.849787175655365, 'word': ', BROWN'}

**Issue:**
The wordpiece tokenizer splits the begin entity into smaller pieces. However model predicts that as an "I-PER" entity which makes it really difficult to merge continuous entities


**3. Expected name:**
VINEY, PAJTSHIA

Predicted name:
{'entity_group': 'B-PER', 'score': 0.9991838335990906, 'word': 'VI'}, 
{'entity_group': 'I-PER', 'score': 0.9591831763585409, 'word': '##Y , PA'} 
{'entity_group': 'I-PER', 'score': 0.7927274107933044, 'word': '##IA'}

**Issue:**
'NE' word is missed in the name: 'VINEY'
'JTSH' word is missed in the name: 'PAJTSHIA'

**4. Expected name:**
Pierson, Garcia 

Predicted name:
{'entity_group': 'B-PER', 'score': 0.9972472190856934, 'word': 'Pierson'}, 
{'entity_group': 'I-PER', 'score': 0.8200799822807312, 'word': 'GA'}, 
{'entity_group': 'I-PER', 'score': 0.8131067156791687, 'word': '##IA'}

**Issue:**
'RC' word is missed in the name: 'Garcia'

Please let me know if I am missing something.
**Missing characters** and **split tokens** are major reasons for the **accuracy drop** while merging the Begin(**B-PER**) and Info(**I-PER**) entities.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.