Repository: huggingface/transformers

Issue:
Handling tag with no prefix for aggregation_strategy in TokenClassificationPipeline
# 🚀 Feature request

Previously the parameter grouped_entities would handle entity with no prefix (like "PER" instead of "B-PER") and would correctly group  similar entities next to each others. With the new parameter aggregation_strategy, this is not the case anymore.

## Motivation

In some simple models, the prefix add some complexity that is not always required. Because of this we are forced to add a prefix to make aggregation works even if not required by the model.

## Your contribution





Discussion:
cc @Narsil 
Hi @jbpolle what do you mean `correctly` ? We should not have changed behavior there, but indeed it's not part of the testing right now, so there might be some issues.

Could you provide a small script on an older transformers version that displays the intended behavior ?
Hello Nicolas,

Here is what it looks like now in the "hosted inference API » panel:


This is from my model here: 
https://huggingface.co/Jean-Baptiste/camembert-ner?text=Je+m%27appelle+jean-baptiste+et+je+vis+%C3%A0+montr%C3%A9al

In previous version, It would display « jean-baptiste PER » and « Montreal LOC ».

However I renamed my entities in the config.json file to I-PER, I-ORG,…which I believe should fix this issue. 
Before that the entities were just PER, LOC,…

I hope this help,
Thank you,
Jean-Baptiste

> Le 30 août 2021 à 09:15, Nicolas Patry ***@***.***> a écrit :
> 
> 
> Hi @jbpolle <https://github.com/jbpolle> what do you mean correctly ? We should not have changed behavior there, but indeed it's not part of the testing right now, so there might be some issues.
> 
> Could you provide a small script on an older transformers version that displays the intended behavior ?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/huggingface/transformers/issues/13325#issuecomment-908333781>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMIMGPPBHLPBSACPHW5PZWDT7OAAXANCNFSM5DAT4PAA>.
> Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. 
> 


Adding missing screenshot in previous message:

<img width="541" alt="PastedGraphic-1" src="https://user-images.githubusercontent.com/51430205/131381199-64fa35a0-05dd-4233-a3de-e53307bd6f71.png">

I went back to `4.3.3` and I can see that the splitting was exactly the same. (no grouping when tags didn't include B-, I- ).

The fact that the cache wasn't probably cleaned on the widget is still an issue, clearing it.
I was working on 4.3.2 and here is how this was working:
![image](https://user-images.githubusercontent.com/51430205/132059845-039b0ade-54f2-45b1-80b1-993641b520e6.png)

But now in 4.9:
![image](https://user-images.githubusercontent.com/51430205/132060946-dce28417-0080-43cb-93c3-95d0cc76e2cc.png)


And even when playing with new aggregation_strategy parameters, I can't get previous results.
Anyway it's fixed in my case by adding the prefix so don't hesitate to close the ticket.

Thank you, 
Ok, I must have tested it wrong before. I can confirm. This is indeed because the default for tags wasn't really explicited, but did behave as `I- `

Code was:

```python
entity["entity"].split("-")[0] != "B"
```
Which would resolve to `"PER" != "B"` whereas now the default tag was explicitely set as B-:
https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/token_classification.py#L413

The fix would be easy but I am unsure about reverting this now that was merged 6th June. 
Tagging  a core maintainer for advice for how to handle this. @LysandreJik 
 
We would need to run some numbers on the hub too, to get an idea of amount of affected repos.
I would fix it to behave the same as it was in v4.3.2 as this is the expected behavior when using `grouped_entities`

Resolving patch:
diff --git a/src/transformers/pipelines/token_classification.py b/src/transformers/pipelines/token_classification.py
--- a/src/transformers/pipelines/token_classification.py
+++ b/src/transformers/pipelines/token_classification.py
@@ -411,7 +411,8 @@ def get_tag(self, entity_name: str) -> Tuple[str, str]:
             tag = entity_name[2:]
         else:
             # It's not in B-, I- format
-            bi = "B"
+            # Default to I- for continuation.
+            bi = "I"
             tag = entity_name
         return bi, tag
 


Test patch:
diff --git a/tests/test_pipelines_token_classification.py b/tests/test_pipelines_token_classification.py
--- a/tests/test_pipelines_token_classification.py
+++ b/tests/test_pipelines_token_classification.py
@@ -318,6 +318,59 @@ def test_aggregation_strategy_byte_level_tokenizer(self):
             ],
         )
 
+    @require_torch
+    def test_aggregation_strategy_no_b_i_prefix(self):
+        model_name = "sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english"
+        tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
+        token_classifier = pipeline(task="ner", model=model_name, tokenizer=tokenizer, framework="pt")
+        # Just to understand scores indexes in this test
+        token_classifier.model.config.id2label = {0: "O", 1: "MISC", 2: "PER", 3: "ORG", 4: "LOC"}
+        example = [
+            {
+                # fmt : off
+                "scores": np.array([0, 0, 0, 0, 0.9968166351318359]),
+                "index": 1,
+                "is_subword": False,
+                "word": "En",
+                "start": 0,
+                "end": 2,
+            },
+            {
+                # fmt : off
+                "scores": np.array([0, 0, 0, 0, 0.9957635998725891]),
+                "index": 2,
+                "is_subword": True,
+                "word": "##zo",
+                "start": 2,
+                "end": 4,
+            },
+            {
+                # fmt: off
+                "scores": np.array([0, 0, 0, 0.9986497163772583, 0]),
+                # fmt: on
+                "index": 7,
+                "word": "UN",
+                "is_subword": False,
+                "start": 11,
+                "end": 13,
+            },
+        ]
+        self.assertEqual(
+            nested_simplify(token_classifier.aggregate(example, AggregationStrategy.NONE)),
+            [
+                {"end": 2, "entity": "LOC", "score": 0.997, "start": 0, "word": "En", "index": 1},
+                {"end": 4, "entity": "LOC", "score": 0.996, "start": 2, "word": "##zo", "index": 2},
+                {"end": 13, "entity": "ORG", "score": 0.999, "start": 11, "word": "UN", "index": 7},
+            ],
+        )
+        self.assertEqual(
+            nested_simplify(token_classifier.aggregate(example, AggregationStrategy.SIMPLE)),
+            [
+                {"entity_group": "LOC", "score": 0.996, "word": "Enzo", "start": 0, "end": 4},
+                {"entity_group": "ORG", "score": 0.999, "word": "UN", "start": 11, "end": 13},
+            ],
+        )
+
     @require_torch
     def test_aggregation_strategy(self):
         model_name = "sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english"