Repository: huggingface/transformers

Issue:
Entity level F-1 scores in run_ner.py
# 🚀 Feature request

The run_ner.py script in examples/token_classification reports evaluation metrics in terms of token F-1 scores (from what I can tell by examining the code). This issue is to request entity level F-1 scores as an evaluation metric. Token level scores will evaluate the F-1 score over an input such as ["I", "work", "for", "ABC", "com", "##pany", "in", "New", "York", "City"] with the label for each token considered separately. In this example, if the ground truth labels are ["O", "O", "O", "ORG", "O", "O", "O", "GPE", "GPE", "GPE"] the label for "New" , "York" and "City" from the predictions would be compared against ground truth separately.  Entity level scores would mark a true positive if for instance all 3 tokens in the span "New York City" are labelled correctly.

## Motivation

Token level F-1 scores can be a more lenient metric compared to entity level scores since labels on sub-words/entities are considered separately.

## Contribution

I can also help implement this but I am not certain how I should get started with this. Any help is appreciated.


Discussion:
Oh this makes me realize this script hasn't been switched to use the datasets metrics for seqeval. This would solve the issue as it computes all scores. Will do that tomorrow.

Resolving patch:
diff --git a/examples/token-classification/run_ner.py b/examples/token-classification/run_ner.py
--- a/examples/token-classification/run_ner.py
+++ b/examples/token-classification/run_ner.py
@@ -25,8 +25,7 @@
 from typing import Optional
 
 import numpy as np
-from datasets import ClassLabel, load_dataset
-from seqeval.metrics import accuracy_score, f1_score, precision_score, recall_score
+from datasets import ClassLabel, load_dataset, load_metric
 
 import transformers
 from transformers import (
@@ -124,6 +123,10 @@ class DataTrainingArguments:
             "one (in which case the other tokens will have a padding index)."
         },
     )
+    return_entity_level_metrics: bool = field(
+        default=False,
+        metadata={"help": "Whether to return all the entity levels during evaluation or just the overall ones."},
+    )
 
     def __post_init__(self):
         if self.dataset_name is None and self.train_file is None and self.validation_file is None:
@@ -323,6 +326,8 @@ def tokenize_and_align_labels(examples):
     data_collator = DataCollatorForTokenClassification(tokenizer)
 
     # Metrics
+    metric = load_metric("seqeval")
+
     def compute_metrics(p):
         predictions, labels = p
         predictions = np.argmax(predictions, axis=2)
@@ -337,12 +342,24 @@ def compute_metrics(p):
             for prediction, label in zip(predictions, labels)
         ]
 
-        return {
-            "accuracy_score": accuracy_score(true_labels, true_predictions),
-            "precision": precision_score(true_labels, true_predictions),
-            "recall": recall_score(true_labels, true_predictions),
-            "f1": f1_score(true_labels, true_predictions),
-        }
+        results = metric.compute(predictions=true_predictions, references=true_labels)
+        if data_args.return_entity_level_metrics:
+            # Unpack nested dictionaries
+            final_results = {}
+            for key, value in results.items():
+                if isinstance(value, dict):
+                    for n, v in value.items():
+                        final_results[f"{key}_{n}"] = v
+                else:
+                    final_results[key] = value
+            return final_results
+        else:
+            return {
+                "precision": results["overall_precision"],
+                "recall": results["overall_recall"],
+                "f1": results["overall_f1"],
+                "accuracy": results["overall_accuracy"],
+            }
 
     # Initialize our Trainer
     trainer = Trainer(


Test patch:
diff --git a/examples/test_examples.py b/examples/test_examples.py
--- a/examples/test_examples.py
+++ b/examples/test_examples.py
@@ -184,7 +184,7 @@ def test_run_ner(self):
 
         with patch.object(sys, "argv", testargs):
             result = run_ner.main()
-            self.assertGreaterEqual(result["eval_accuracy_score"], 0.75)
+            self.assertGreaterEqual(result["eval_accuracy"], 0.75)
             self.assertGreaterEqual(result["eval_precision"], 0.75)
             self.assertLess(result["eval_loss"], 0.5)