Repository: explosion/spaCy
Pull request: Determine labels by factory name in debug data


<!--- Provide a general summary of your changes in the title. -->

## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->

For all components, return labels for all components with the
corresponding factory name rather than for only the default name.

For `spancat`, return labels as a dict keyed by `spans_key`.

### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->

Bug fix.

## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I confirm that I have the right to submit this contribution under the project's MIT license.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.


Problem statement:
Depending on a model's name in config `spacy debug data` behaves differently
## How to reproduce the behavior
I have two identical config files with `transformer+ner` pipelines. If my NER component (not custom, absolutely tutorial rip-off) is called `entity_recognizer` (or anything else, I checked) and not `ner`, running `spacy debug data` with respective parameters produces different results on the same data:
+ For the case of `entity_recognizer` I get :
```
========================== Named Entity Recognition ==========================
ℹ 0 label(s)
0 missing value(s) (tokens with '-' label)
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
```
+ And when I rename the component to `ner`:
```
========================== Named Entity Recognition ==========================
ℹ 3 label(s)
0 missing value(s) (tokens with '-' label)
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
```
Goddamn, I had spent hours of debugging with this debug command until I realized, that my data is OK and I do not have 0 labels. 
## Your Environment
* Operating System: Ubuntu 20.04
* Python Version Used: 3.8
* spaCy Version Used: spacy 3.2

Hints from issue discussion:
Thanks for the report, that does look like a bug! The messages about having a good amount of examples in both outputs should be correct, since this is looking at `doc.ents` and not the pipeline config, but there's clearly a bug in code that looks up the labels by component name.
Hello, @adrianeboyd @tomateit  I want to work on this issue. Could you please assign me?
Hi, we don't usually formally assign people to issues here. You can open a PR (as a draft if it's not finalized) and link it to this issue by using the "fixes" keyword in the description, so a note will show up here automatically about the PR.

Commit message: Refactor for typing

Patch to review:
diff --git a/spacy/cli/debug_data.py b/spacy/cli/debug_data.py
index 688b07a9ba4..862edb1275b 100644
--- a/spacy/cli/debug_data.py
+++ b/spacy/cli/debug_data.py
@@ -14,7 +14,7 @@
 from ..schemas import ConfigSchemaTraining
 from ..pipeline._parser_internals import nonproj
 from ..pipeline._parser_internals.nonproj import DELIMITER
-from ..pipeline import Morphologizer
+from ..pipeline import Morphologizer, SpanCategorizer
 from ..morphology import Morphology
 from ..language import Language
 from ..util import registry, resolve_dot_names
@@ -699,8 +699,33 @@ def _get_examples_without_label(data: Sequence[Example], label: str) -> int:
     return count
 
 
-def _get_labels_from_model(nlp: Language, pipe_name: str) -> Set[str]:
-    if pipe_name not in nlp.pipe_names:
-        return set()
-    pipe = nlp.get_pipe(pipe_name)
-    return set(pipe.labels)
+def _get_labels_from_model(
+    nlp: Language, factory_name: str
+) -> Set[str]:
+    pipe_names = [
+        pipe_name
+        for pipe_name in nlp.pipe_names
+        if nlp.get_pipe_meta(pipe_name).factory == factory_name
+    ]
+    labels: Set[str] = set()
+    for pipe_name in pipe_names:
+        pipe = nlp.get_pipe(pipe_name)
+        labels.update(pipe.labels)
+    return labels
+
+
+def _get_labels_from_spancat(
+    nlp: Language, factory_name: str
+) -> Dict[str, Set[str]]:
+    pipe_names = [
+        pipe_name
+        for pipe_name in nlp.pipe_names
+        if nlp.get_pipe_meta(pipe_name).factory == factory_name
+    ]
+    labels: Dict[str, Set[str]] = {}
+    for pipe_name in pipe_names:
+        pipe = cast(SpanCategorizer, nlp.get_pipe(pipe_name))
+        if pipe.key not in labels:
+            labels[pipe.key] = set()
+        labels[pipe.key].update(pipe.labels)
+    return labels


Code review comments:
diff_hunk: @@ -699,8 +699,33 @@ def _get_examples_without_label(data: Sequence[Example], label: str) -> int:
     return count
 
 
-def _get_labels_from_model(nlp: Language, pipe_name: str) -> Set[str]:
-    if pipe_name not in nlp.pipe_names:
-        return set()
-    pipe = nlp.get_pipe(pipe_name)
-    return set(pipe.labels)
+def _get_labels_from_model(
+    nlp: Language, factory_name: str
+) -> Set[str]:
+    pipe_names = [
+        pipe_name
+        for pipe_name in nlp.pipe_names
+        if nlp.get_pipe_meta(pipe_name).factory == factory_name
+    ]
+    labels: Set[str] = set()
+    for pipe_name in pipe_names:
+        pipe = nlp.get_pipe(pipe_name)
+        labels.update(pipe.labels)
+    return labels
+
+
+def _get_labels_from_spancat(
+    nlp: Language, factory_name: str
+) -> Dict[str, Set[str]]:
+    pipe_names = [
+        pipe_name
+        for pipe_name in nlp.pipe_names
+        if nlp.get_pipe_meta(pipe_name).factory == factory_name
+    ]
+    labels: Dict[str, Set[str]] = {}
+    for pipe_name in pipe_names:
+        pipe = cast(SpanCategorizer, nlp.get_pipe(pipe_name))
path: spacy/cli/debug_data.py
text: @user1:
I would personally do this as an `assert`. If this depends on user input/settings I would even do an `if` check and raise a `ValueError`, but I don't think that will be the case? 

```suggestion
        pipe = nlp.get_pipe(pipe_name)
        assert isinstance(pipe, SpanCategorizer)
```

@author:
If mypy is fine with that, it's fine with me. Actually, there's no need for the `factory_name` arg, either...

@user1:
Yep, `mypy` parses the `assert` and will take that into account for the following lines :-)

@user1:
You could also have the `factory_name` set default to `"spancat"`?