Repository: RasaHQ/rasa
Pull request: Exclude 'no-entity' from NER evaluation
**Proposed changes**:
- evaluation of NER does not consider 'no-enity' anymore
closes #4313 

**Status (please check what you already did)**:
- [x] made PR ready for code review
- [x] added some tests for the functionality
- [ ] updated the documentation
- [x] updated the changelog
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa_nlu#code-style) for instructions)


Problem statement:
Evaluation of NER: Don't include 'no-entity'
Currently, we take the weighted precision and f1 score as performance metric for our named entity recognition. Weighted means

> Calculate metrics for each label, and find their average weighted
> by support (the number of true instances for each label). This
> alters 'macro' to account for label imbalance; it can result in an
> F-score that is not between precision and recall. 

In most cases, ~90% of all words have the label 'no-entity'. If we include 'no-entity' as a label in the evaluation, this will influence the result a lot. The overall metrics are therefore always quite high. If you want to get a feeling on how good your NER is actually doing, you need to look at the metrics for the labels.

Solution:
Report the overall metrics without taking the label 'no-entity' into account. We can still list the precision, recall, f1 score for 'no-entity' separately in the report, but it should not influence the overall metrics anymore.

Hints from issue discussion:

Commit message: check if labels empty

Patch to review:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 4814e510338c..fb55b2873b25 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -47,6 +47,7 @@ Changed
   hold out validation set that is excluded from training data
 - defaults parameters and architectures for both ``EmbeddingPolicy`` and
   ``EmbeddingIntentClassifier`` are changed (this is a breaking change)
+- evaluation of NER does not include 'no-entity' anymore
 
 Fixed
 -----
diff --git a/docs/migration-guide.rst b/docs/migration-guide.rst
index 74779d531519..5f994dddd348 100644
--- a/docs/migration-guide.rst
+++ b/docs/migration-guide.rst
@@ -44,6 +44,11 @@ General
   name instead of policy name. Old naming style will not be read correctly when
   creating **compare** plots (``rasa test core``). Please remove old trained models
   in comparison folder and retrain. Normal core training is unaffected.
+- We updated the **evaluation metric** for our **NER**. We report the weighted precision and f1-score.
+  So far we included ``no-entity`` in this report. However, as most of the tokens actually don't have
+  an entity set, this will influence the weighted precision and f1-score quite a bit. From now on we
+  exclude ``no-entity`` from the evaluation. The overall metrics now only include proper entities. You
+  might see a drop in the performance scores when running the evaluation again.
 
 .. _migration-to-rasa-1.0:
 
diff --git a/rasa/nlu/test.py b/rasa/nlu/test.py
index 5817aedeb4f8..804dd092d136 100644
--- a/rasa/nlu/test.py
+++ b/rasa/nlu/test.py
@@ -37,6 +37,8 @@
 
 CVEvaluationResult = namedtuple("Results", "train test")
 
+NO_ENTITY = "no_entity"
+
 IntentEvaluationResult = namedtuple(
     "IntentEvaluationResult",
     "intent_target " "intent_prediction " "message " "confidence",
@@ -146,24 +148,42 @@ def log_evaluation_table(
 
 
 def get_evaluation_metrics(
-    targets: Iterable[Any], predictions: Iterable[Any], output_dict: bool = False
+    targets: Iterable[Any],
+    predictions: Iterable[Any],
+    output_dict: bool = False,
+    exclude_label: Text = None,
 ) -> Tuple[Union[Text, Dict[Text, Dict[Text, float]]], float, float, float]:
     """Compute the f1, precision, accuracy and summary report from sklearn."""
     from sklearn import metrics
 
-    targets = clean_intent_labels(targets)
-    predictions = clean_intent_labels(predictions)
+    targets = clean_labels(targets)
+    predictions = clean_labels(predictions)
+
+    labels = get_label_set(targets, exclude_label)
+    if not labels:
+        logger.warning("No labels to evaluate. Skip evaluation.")
+        return {}, 0.0, 0.0, 0.0
 
     report = metrics.classification_report(
-        targets, predictions, output_dict=output_dict
+        targets, predictions, labels=labels, output_dict=output_dict
+    )
+    precision = metrics.precision_score(
+        targets, predictions, labels=labels, average="weighted"
     )
-    precision = metrics.precision_score(targets, predictions, average="weighted")
-    f1 = metrics.f1_score(targets, predictions, average="weighted")
+    f1 = metrics.f1_score(targets, predictions, labels=labels, average="weighted")
     accuracy = metrics.accuracy_score(targets, predictions)
 
     return report, precision, f1, accuracy
 
 
+def get_label_set(targets: Iterable[Any], exclude_label: Optional[Text]) -> List[Text]:
+    """Get unique labels. Exclude 'exclude_label' if specified."""
+    labels = set(targets)
+    if exclude_label and exclude_label in labels:
+        labels.remove(exclude_label)
+    return list(labels)
+
+
 def remove_empty_intent_examples(
     intent_results: List[IntentEvaluationResult]
 ) -> List[IntentEvaluationResult]:
@@ -182,7 +202,7 @@ def remove_empty_intent_examples(
     return filtered
 
 
-def clean_intent_labels(labels: Iterable[Any]) -> List[Text]:
+def clean_labels(labels: Iterable[Any]) -> List[Text]:
     """Get rid of `None` intents. sklearn metrics do not support them."""
     return [l if l is not None else "" for l in labels]
 
@@ -411,23 +431,26 @@ def evaluate_entities(
 
     aligned_predictions = align_all_entity_predictions(entity_results, extractors)
     merged_targets = merge_labels(aligned_predictions)
-    merged_targets = substitute_labels(merged_targets, "O", "no_entity")
+    merged_targets = substitute_labels(merged_targets, "O", NO_ENTITY)
 
     result = {}
 
     for extractor in extractors:
         merged_predictions = merge_labels(aligned_predictions, extractor)
-        merged_predictions = substitute_labels(merged_predictions, "O", "no_entity")
+        merged_predictions = substitute_labels(merged_predictions, "O", NO_ENTITY)
         logger.info("Evaluation for entity extractor: {} ".format(extractor))
         if report_folder:
-            report, precision, f1, accuracy = get_evaluation_metrics(
-                merged_targets, merged_predictions, output_dict=True
-            )
-
             report_filename = extractor + "_report.json"
             extractor_report_filename = os.path.join(report_folder, report_filename)
 
+            report, precision, f1, accuracy = get_evaluation_metrics(
+                merged_targets,
+                merged_predictions,
+                output_dict=True,
+                exclude_label=NO_ENTITY,
+            )
             utils.write_json_to_file(extractor_report_filename, report)
+
             logger.info(
                 "Classification report for '{}' saved to '{}'."
                 "".format(extractor, extractor_report_filename)
@@ -435,7 +458,10 @@ def evaluate_entities(
 
         else:
             report, precision, f1, accuracy = get_evaluation_metrics(
-                merged_targets, merged_predictions
+                merged_targets,
+                merged_predictions,
+                output_dict=False,
+                exclude_label=NO_ENTITY,
             )
             if isinstance(report, str):
                 log_evaluation_table(report, precision, f1, accuracy)
@@ -1054,13 +1080,13 @@ def _compute_entity_metrics(
     aligned_predictions = align_all_entity_predictions(entity_results, extractors)
 
     merged_targets = merge_labels(aligned_predictions)
-    merged_targets = substitute_labels(merged_targets, "O", "no_entity")
+    merged_targets = substitute_labels(merged_targets, "O", NO_ENTITY)
 
     for extractor in extractors:
         merged_predictions = merge_labels(aligned_predictions, extractor)
-        merged_predictions = substitute_labels(merged_predictions, "O", "no_entity")
+        merged_predictions = substitute_labels(merged_predictions, "O", NO_ENTITY)
         _, precision, f1, accuracy = get_evaluation_metrics(
-            merged_targets, merged_predictions
+            merged_targets, merged_predictions, exclude_label=NO_ENTITY
         )
         entity_metric_results[extractor]["Accuracy"].append(accuracy)
         entity_metric_results[extractor]["F1-score"].append(f1)
diff --git a/tests/nlu/base/test_evaluation.py b/tests/nlu/base/test_evaluation.py
index 02b66c89cdd4..5fe44568e30e 100644
--- a/tests/nlu/base/test_evaluation.py
+++ b/tests/nlu/base/test_evaluation.py
@@ -25,6 +25,9 @@
     EntityEvaluationResult,
     evaluate_intents,
     evaluate_entities,
+    get_label_set,
+    get_evaluation_metrics,
+    NO_ENTITY,
 )
 from rasa.nlu.test import does_token_cross_borders
 from rasa.nlu.test import align_entity_predictions
@@ -444,6 +447,61 @@ def test_label_replacement():
     assert substitute_labels(original_labels, "O", "no_entity") == target_labels
 
 
+@pytest.mark.parametrize(
+    "targets,exclude_label,expected",
+    [
+        (
+            ["no_entity", "location", "location", "location", "person"],
+            NO_ENTITY,
+            ["location", "person"],
+        ),
+        (
+            ["no_entity", "location", "location", "location", "person"],
+            None,
+            ["no_entity", "location", "person"],
+        ),
+        (["no_entity"], NO_ENTITY, []),
+        (["location", "location", "location"], NO_ENTITY, ["location"]),
+        ([], None, []),
+    ],
+)
+def test_get_label_set(targets, exclude_label, expected):
+    actual = get_label_set(targets, exclude_label)
+    assert expected == actual
+
+
+@pytest.mark.parametrize(
+    "targets,predictions,expected_precision,expected_fscore,expected_accuracy",
+    [
+        (
+            ["no_entity", "location", "no_entity", "location", "no_entity"],
+            ["no_entity", "location", "no_entity", "no_entity", "person"],
+            1.0,
+            0.6666666666666666,
+            3 / 5,
+        ),
+        (
+            ["no_entity", "no_entity", "no_entity", "no_entity", "person"],
+            ["no_entity", "no_entity", "no_entity", "no_entity", "no_entity"],
+            0.0,
+            0.0,
+            4 / 5,
+        ),
+    ],
+)
+def test_get_evaluation_metrics(
+    targets, predictions, expected_precision, expected_fscore, expected_accuracy
+):
+    report, precision, f1, accuracy = get_evaluation_metrics(
+        targets, predictions, True, exclude_label=NO_ENTITY
+    )
+
+    assert f1 == expected_fscore
+    assert precision == expected_precision
+    assert accuracy == expected_accuracy
+    assert NO_ENTITY not in report
+
+
 def test_nlu_comparison(tmpdir):
     configs = [
         NLU_DEFAULT_CONFIG_PATH,


Code review comments:
diff_hunk: @@ -173,7 +176,7 @@ def get_evaluation_metrics(
     return report, precision, f1, accuracy
 
 
-def get_label_set(targets: Iterable[Any], exclude_label: Optional[Text]):
+def get_label_set(targets: Iterable[Any], exclude_label: Optional[Text]) -> List[Text]:
path: rasa/nlu/test.py
text: @user1:
`get_unique_labels`?