NER pipeline: Inconsistent entity grouping
Author: dav009
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

"mrm8488/bert-spanish-cased-finetuned-ner"

Language I am using the model on (English, Chinese ...):  Spanish

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. create a `ner` pipeline
2. pass flag `grouped_entities`
3. entities are not grouped as expected see sample below

```python
NER_MODEL = "mrm8488/bert-spanish-cased-finetuned-ner"
nlp_ner = pipeline("ner", model=NER_MODEL,
                   grouped_entities=True,
                   tokenizer=(NER_MODEL, {"use_fast": False}))

t = """Consuelo Araújo Noguera, ministra de cultura del presidente Andrés Pastrana (1998.2002) fue asesinada por las Farc luego de haber permanecido secuestrada por algunos meses."""
ner(t)
>>> 
[ {'entity_group': 'B-PER', 'score': 0.901019960641861, 'word': 'Consuelo'}, 
 {'entity_group': 'I-PER', 'score': 0.9990904808044434, 'word': 'Araújo Noguera'}, 
 {'entity_group': 'B-PER', 'score': 0.9998136162757874, 'word': 'Andrés'}, 
 {'entity_group': 'I-PER', 'score': 0.9996985991795858, 'word': 'Pastrana'}, 
 {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Far'}]
```

## Expected behavior

### Inconsistent grouping
I expect the first two items of the given sample( `B-PER`, and `I-PER`) to be grouped. As they are contiguous tokens and correspond to a single entity spot.   It seems the current code does not take into account `B` and `I` tokens.

expected output:
```
 {'entity_group': 'I-PER', 'score': 0.9990904808044434, 'word': ' Consuelo Araújo Noguera'}, 
 {'entity_group': 'I-PER', 'score': 0.9998136162757874, 'word': 'Andrés Pastrana'}, 
 {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Farc'}]
```

### Lost tokens?

for the same input, passing `grouped_entities=False` generates the following output:

```
[
{'word': 'Cons', 'score': 0.9994944930076599, 'entity': 'B-PER', 'index': 1},
{'word': '##uelo', 'score': 0.802545428276062, 'entity': 'B-PER', 'index': 2}, 
{'word': 'Ara', 'score': 0.9993102550506592, 'entity': 'I-PER', 'index': 3}, 
{'word': '##új', 'score': 0.9993743896484375, 'entity': 'I-PER', 'index': 4}, 
{'word': '##o', 'score': 0.9992871880531311, 'entity': 'I-PER', 'index': 5}, 
{'word': 'No', 'score': 0.9993029236793518, 'entity': 'I-PER', 'index': 6}, 
{'word': '##guera', 'score': 0.9981776475906372, 'entity': 'I-PER', 'index': 7}, 
{'word': 'Andrés', 'score': 0.9998136162757874, 'entity': 'B-PER', 'index': 15}, 
{'word': 'Pas', 'score': 0.999740719795227, 'entity': 'I-PER', 'index': 16}, 
{'word': '##tran', 'score': 0.9997414350509644, 'entity': 'I-PER', 'index': 17}, 
{'word': '##a', 'score': 0.9996136426925659, 'entity': 'I-PER', 'index': 18}, 
{'word': 'Far', 'score': 0.9989739060401917, 'entity': 'B-ORG', 'index': 28}, 
{'word': '##c', 'score': 0.7188423275947571, 'entity': 'I-ORG', 'index': 29}]
```

when using `grouped_entities`  the last entity `word` (`##c`) got lost, it is not even considered as a different group 

` {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Far'}]`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: OSX
- Python version: 3.7
- PyTorch version (GPU?):  1.5.0
- Tensorflow version (GPU?):
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
@dav009 Thanks for posting this issue!

1. **Inconsistent grouping** - correct that `B` and `I` tokens are not yet considered. Will have to include this in a new PR.
2. **Lost tokens** - the skipped tokens are those with an entity type found in the `ignore_labels` argument for `TokenClassificationPipeline`, which is set as  `["O"]` by default. If you don't want to skip any token, you can just set `ignore_labels=[]`.

I'm happy to work on `1` within the next week or so since I've already been planning to apply this fix. 

@enzoampil 👋   thanks for your prompt answer

> Lost tokens - the skipped tokens are those with an entity type found in the ignore_labels argument for TokenClassificationPipeline, which is set as ["O"] by default. If you don't want to skip any token, you can just set ignore_labels=[].

in the given sample, the missing entity is not tagged as `O` : 

- `##c` is tagged as `I-ORG`  in (`grouped_entities =False`)
 `{'word': '##c', 'score': 0.7188423275947571, 'entity': 'I-ORG', 'index': 29}]`

however it did not get included in the grouping results (`grouped_entities =True`)

@dav009  Understand now! Thanks for clarifying. Yes, it does seem to be related to the I and B issue. Think can handle this in the same PR.

@dav009 I handled a similar scenario for grouping the Begin and Info tags. 
The below code helps to **merge the tokens between Begin and Info tags**. Please adapt to your use

`def group_entities(self, prediction_results_list: List[Dict] ) -> List[RecordDataResponse]:
        final_prediction_list = []

        # Group the prediction list by the last 3 characters of the tag
        # and group the results appropriately
        # B-PER-TAG -> TAG
        # B-PER -> PER
        tmp_dict = defaultdict(list)
        added_index = 0
        prev_index = 0
        for index, entity in enumerate(prediction_results_list):
            try:
                if entity['entity_group'].startswith("B") and \
                        prediction_results_list[index + 1]['entity_group'].startswith("I"):
                    tmp_dict[index].append(entity)
                    added_index = index
                elif entity['entity_group'].startswith("I"):
                    if (1 == abs(added_index - index)) or (1 == abs(prev_index - index)):
                        tmp_dict[added_index].append(entity)
                        prev_index = index
                else:
                    tmp_dict[index].append(entity)
            except IndexError:
                tmp_dict[index].append(entity)

        # Flatten the sub-lists
        final_grouped_list = list(map(list, map(itertools.chain, tmp_dict.values())))

        for entity_group_list in final_grouped_list:

            # Get the unique number of entities per list
            _entity_count = len(
                set(
                    [
                        prediction_input["entity_group"]
                        for prediction_input in entity_group_list
                    ]
                )
            )

            if entity_group_list:
                if len(entity_group_list) > 1:

                    # Get the tag name
                    tag_name = str(entity_group_list[0]["entity_group"][-3:])

                    # Join the entities
                    entity_value = " ".join(
                        [
                            prediction_input["word"]
                            for prediction_input in entity_group_list
                        ]
                    )

                    # Remove duplicate names
                    _temp_entities = entity_value.split()
                    entity_value = " ".join(
                        sorted(set(_temp_entities), key=_temp_entities.index)
                    )

                    # Compute the average of confidence scores
                    mean_score = np.mean(
                        [
                            prediction_input["score"]
                            for prediction_input in entity_group_list
                        ]
                    )

                    # Frame the entities and ensure name is atleast has more than 1 character
                    if len(entity_value) > 1:
                        final_prediction_list.append(
                            RecordDataResponse(
                                entity_group=tag_name,
                                score=mean_score,
                                word=entity_value,
                            )
                        )
                else:
                    [
                        final_prediction_list.append(
                            RecordDataResponse(
                                entity_group=entity_group["entity_group"][-3:],
                                score=entity_group["score"],
                                word=entity_group["word"],
                            )
                        )
                        for entity_group in entity_group_list if len(re.sub(r"(?i)[^-0-9a-z\\s.,]+", "", entity_group["word"])) > 1
                    ]

        # Sort the by the list by confidence score and return in descending order
        return sorted(final_prediction_list, key=lambda x: x.score, reverse=True)
`

The code is invoked from the **pipeline**:

       `prediction_results_list = [
            prediction
            for prediction_input in prediction_input_list
            for prediction in self.model_prediction_pipeline(prediction_input)
            if prediction
            and prediction["word"] not in self.stop_list
        ]

        # Return the predictions
        return (
            self.group_entities(prediction_results_list)
            if prediction_results_list
            else []
        )`

@dav009 Opened a PR (above) that should resolve this :smile:

@enzoampil Just curious to know if your PR can handle the merging of multiple entities.

`entities_list = [
 {"word": "Patient", "score": 0.9977793097496033, "entity_group": "B-PER-TAG"},
 {"word": "Name", "score": 0.9968074560165405, "entity_group": "I-PER-TAG"},
 {"word": "Cecil", "score": 0.9995920658111572, "entity_group": "B-PER"},
 {"word": "D . Thomas", "score": 0.9938908666372299, "entity_group": "I-PER"},
 {"word": "Thomas", "score": 0.9993066191673279, "entity_group": "B-PER"}
]`

In this case, I would expect the below output after the entities are grouped:
`[
 {"word": "Patient Name", "score": 0.9977793097496033, "entity_group": "PER-TAG"},
 {"word": "Cecil D . Thomas", "score": 0.9995920658111572, "entity_group": "PER"},
 {"word": "Thomas", "score": 0.9993066191673279, "entity_group": "PER"}
]`

@enzoampil gonna check it out. 

maybe part of another issue but do you get `word` fields containing `##` is that expected?

@sudharsan2020 Setting `grouped_entities=True` should work for your example under the new PR, since similar entities w/ different prefixes are now grouped (e.g. "I-PER" and "B-PER") :smile:

@dav009 This is even after grouping correct? I suspect this is possible when word pieces have different core entity types (e.g. `ORG` vs `PER`).

Can you give an example?

Hello, I think @dav009 is refering to this : 

```Python
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
from transformers import TokenClassificationPipeline

model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

nlp = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=True
)

sequence = "In addition , the Blabla Group has completed the acquisition of ISO / TS16949 certification ."

res = nlp(sequence)
print(res)
```
I have this as a result : 

`[{'entity_group': 'I-ORG', 'score': 0.9988919496536255, 'word': 'Blabla Group'}, {'entity_group': 'I-MISC', 'score': 0.9711909890174866, 'word': 'ISO'}, {'entity_group': 'I-ORG', 'score': 0.6591967344284058, 'word': 'T'}, {'entity_group': 'I-MISC', 'score': 0.5822997689247131, 'word': '##S16'}, {'entity_group': 'I-MISC', 'score': 0.5067382454872131, 'word': '##9'}]`

Some word fields still have ## in it. I have just installed transformers right now (version 2.11.0) with a pip install command then paste the pipelines.py fixed in my transformers folder. 

@Nighthyst can you share the result when `grouped_entities=False`?

Yes, here is a comparison of the resultats with `grouped_entities=False` or when `grouped_entities=True` : 

```Python
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
from transformers import TokenClassificationPipeline

model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

nlp_not_grouped = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=False
)

nlp_grouped = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=True
)

seq1 = "In addition , the Blabla Group has completed the acquisition of ISO / TS16949 certification ."
seq2 = "Directors and certain categories of personnel , who are all included in a regularly updated list"\
", must disclose any trades they carry out in Faurecia"
seq3 = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very" \
           "close to the Manhattan Bridge."

sequences = [seq1, seq2, seq3]

for i, seq in enumerate(sequences):
    ngrouped, grouped = nlp_not_grouped(seq), nlp_grouped(seq)
    print(f"===================== sentence n°{i+1}")
    print("---Not grouped entities---")
    print(ngrouped)
    print("---Grouped entities---")
    print(grouped)
```
This is the results:

```
===================== sentence n°1
---Not grouped entities---
[{'word': 'B', 'score': 0.9997261762619019, 'entity': 'I-ORG', 'index': 5}, 
{'word': '##la', 'score': 0.997683048248291, 'entity': 'I-ORG', 'index': 6}, 
{'word': '##bla', 'score': 0.99888014793396, 'entity': 'I-ORG', 'index': 7}, 
{'word': 'Group', 'score': 0.9992784261703491, 'entity': 'I-ORG', 'index': 8}, 
{'word': 'ISO', 'score': 0.9711909890174866, 'entity': 'I-MISC', 'index': 14},
{'word': 'T', 'score': 0.6591967344284058, 'entity': 'I-ORG', 'index': 16}, 
{'word': '##S', 'score': 0.658642053604126, 'entity': 'I-MISC', 'index': 17}, 
{'word': '##16', 'score': 0.5059574842453003, 'entity': 'I-MISC', 'index': 18}, 
{'word': '##9', 'score': 0.5067382454872131, 'entity': 'I-MISC', 'index': 21}]
---Grouped entities---
[{'entity_group': 'I-ORG', 'score': 0.9988919496536255, 'word': 'Blabla Group'}, 
{'entity_group': 'I-MISC', 'score': 0.9711909890174866, 'word': 'ISO'}, 
{'entity_group': 'I-ORG', 'score': 0.6591967344284058, 'word': 'T'}, 
{'entity_group': 'I-MISC', 'score': 0.5822997689247131, 'word': '##S16'}, 
{'entity_group': 'I-MISC', 'score': 0.5067382454872131, 'word': '##9'}]
===================== sentence n°2
---Not grouped entities---
[{'word': 'F', 'score': 0.6292181611061096, 'entity': 'I-ORG', 'index': 27}, 
{'word': '##au', 'score': 0.7241453528404236, 'entity': 'I-LOC', 'index': 28}, 
{'word': '##re', 'score': 0.49484530091285706, 'entity': 'I-LOC', 'index': 29}, 
{'word': '##cia', 'score': 0.6472106575965881, 'entity': 'I-LOC', 'index': 30}]
---Grouped entities---
[{'entity_group': 'I-ORG', 'score': 0.6292181611061096, 'word': 'F'}, 
{'entity_group': 'I-LOC', 'score': 0.6220671037832896, 'word': '##aurecia'}]
===================== sentence n°3
---Not grouped entities---
[{'word': 'Hu', 'score': 0.9995108246803284, 'entity': 'I-ORG', 'index': 1}, 
{'word': '##gging', 'score': 0.989597499370575, 'entity': 'I-ORG', 'index': 2}, 
{'word': 'Face', 'score': 0.9979704022407532, 'entity': 'I-ORG', 'index': 3}, 
{'word': 'Inc', 'score': 0.9993758797645569, 'entity': 'I-ORG', 'index': 4}, 
{'word': 'New', 'score': 0.9993405938148499, 'entity': 'I-LOC', 'index': 11}, 
{'word': 'York', 'score': 0.9991927742958069, 'entity': 'I-LOC', 'index': 12}, 
{'word': 'City', 'score': 0.9993411302566528, 'entity': 'I-LOC', 'index': 13}, 
{'word': 'D', 'score': 0.986336350440979, 'entity': 'I-LOC', 'index': 19}, 
{'word': '##UM', 'score': 0.9396238923072815, 'entity': 'I-LOC', 'index': 20}, 
{'word': '##BO', 'score': 0.9121386408805847, 'entity': 'I-LOC', 'index': 21}, 
{'word': 'Manhattan', 'score': 0.9839190244674683, 'entity': 'I-LOC', 'index': 29}, 
{'word': 'Bridge', 'score': 0.9924242496490479, 'entity': 'I-LOC', 'index': 30}]
---Grouped entities---
[{'entity_group': 'I-ORG', 'score': 0.9966136515140533, 'word': 'Hugging Face Inc'}, 
{'entity_group': 'I-LOC', 'score': 0.9992914994557699, 'word': 'New York City'}, 
{'entity_group': 'I-LOC', 'score': 0.9460329612096151, 'word': 'DUMBO'}, 
{'entity_group': 'I-LOC', 'score': 0.9881716370582581, 'word': 'Manhattan Bridge'}]
```

Everything is fine for seq3 but seq1 and seq2 have the issue.


@Nighthyst I see, you're bringing up a different issue now. This is the case where the entity type of a word's word piece, is different from other word pieces.

A fix I can apply here is to automatically group word pieces together regardless of entity type. I can apply this to a new PR after merging this existing one.

@Nighthyst @enzoampil  indeed that's exactly the other issue  I came accross. Thanks for digging a sample for it.

Ok, I think we should open another issue for this problem : I've noticed other related issues

@Nighthyst sounds good, thanks! :) 

@enzoampil I was testing with your **ner_grouping** branch locally and these are the results **before** and **after grouping**. Do you think this is the expected behaviour?

**Without grouping:**

`[{'word': 'Peterson', 'score': 0.999268114566803, 'entity': 'B-PER', 'index': 17}, 
{'word': ',', 'score': 0.9992983937263489, 'entity': 'I-PER', 'index': 18}, 
{'word': '##David', 'score': 0.6536518931388855, 'entity': 'I-PER', 'index': 21}, 
{'word': 'David', 'score': 0.974104642868042, 'entity': 'B-PER', 'index': 37}, 
{'word': 'Peterson', 'score': 0.9984731078147888, 'entity': 'B-PER', 'index': 106}, 
{'word': 'David', 'score': 0.74308180809021, 'entity': 'B-PER', 'index': 393}, 
{'word': 'Peterson', 'score': 0.9972764253616333, 'entity': 'B-PER', 'index': 394}]`

**With grouping:**

`[{'entity_group': 'B-PER', 'score': 0.9992832541465759, 'word': 'Peterson ,'}, 
{'entity_group': 'I-PER', 'score': 0.6536518931388855, 'word': '##David'}, 
{'entity_group': 'B-PER', 'score': 0.974104642868042, 'word': 'David'}, 
{'entity_group': 'B-PER', 'score': 0.9984731078147888, 'word': 'Peterson'}, 
{'entity_group': 'B-PER', 'score': 0.8701791167259216, 'word': 'David Peterson'}]`

The **two I-PER entities weren't merged.**

Also observed few scenarios, in which the list **filtered_labels_idx** is **empty** which throws **IndexError**.
**src/transformers/pipelines.py**
`last_idx, _ = filtered_labels_idx[-1]`

Screenshot: https://ibb.co/JHxYgWn

Hi everyone, this PR was recently merged to resolve the original issue #4987.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.