Repository: explosion/spaCy
Pull request: Add ENT_IOB key to Matcher
<!--- Provide a general summary of your changes in the title. -->

## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->

We add `ENT_IOB` as a token pattern key, upon request by #3940 . I also added a tiny addition to the documentation for the new key.

### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->

Enhancement

## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I confirm that I have the right to submit this contribution under the project's MIT license.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.


Problem statement:
ENT_IOB as a token pattern key
## Feature description
Adding `ENT_IOB` attribute as one of the pattern key. Currently, it has not been integrated into entity ruler pipeline.

### Example
```
import spacy
from spacy.pipeline import EntityRuler

nlp = spacy.load('en')
ruler = EntityRuler(nlp, overwrite_ents=True)
pattern = [{'ENT_TYPE': 'GPE', 'OP': '+'}, {'ORTH': 'california'}]
ruler.add_patterns([{"label":'COMB', "pattern":pattern}])
```
```
print('Before Ruler')
print(nlp('Delhi New York california').ents)
Output: (Delhi, New York)
```
```
print('After Ruler')
nlp.add_pipe(ruler)
print(nlp('Delhi New York california').ents)
-- Output: (Delhi New York california,)
```
Correct Output should be **(Delhi, New York california)**

Logical answer would be to annotate only `New York california` as new entity, but here since `ENT_IOB` option is not available, it annotates all the `GPE` present in the given pattern

Suggested documentation improvement: ENT_IOB in DependencyMatcher
I have discovered that the following works as a pattern to add to a DependencyMatcher object:

pattern = [
    {'LEFT_ID': 'is_verb_lemma_pay',
     'REL_OP': '.*',
     'RIGHT_ID': 'is_cause',
     'RIGHT_ATTRS': {'ENT_TYPE': 'CAUSE', 'DEP': 'pobj', 'ENT_IOB': 3}
}]

You wouldn't know this from the documentation. The docs for DependencyMatcher refer to Matcher regarding attributes (albeit with a statement about form only not about which attributes are allowable), and Matcher itself doesn't allow ENT_IOB. Moreover, the format declaring other attributes to match on uses the non-underscore attribute name at the dict key but matches to the string value associated with the underscore version, e.g. {'POS': some_string} where the usage above matches the integer for ENT_IOB not the string associated with ENT_IOB_.

So add this? Would have saved me a bit of time... :-)

Hints from issue discussion:
Fair point, although the `ENT_IOB` values currently have pretty obscure semantics, which makes usability a bit difficult. The field takes a numeric value with the following interpretation:

0: Missing
1: I -- In
2: O -- Out
3: B -- Begin
@honnibal I guess we could also have it take the strings `I`, `O` and `B` and a `0` for missing?
@ines I've thought a couple of times about how to clean this up. It's pretty annoying: if we take `I`, then what about the hash code of `I`? At least 0 always corresponds to the empty string, so that's easy.

This is one of the only places in the library where we'd using a numeric encoding that's *not* the same as the string-store encoding. 
Any updates on getting `ENT_IOB` added to token patterns?

I'm trying to write a pattern that will pick up and overwrite an existing entity. But, without `ENT_IOB` information, it's hard to accurately pick up all the tokens for an entity.

Commit message: add IOB vals to schema

Patch to review:
diff --git a/spacy/matcher/matcher.pyx b/spacy/matcher/matcher.pyx
index 745d7cf437b..8c2cf665676 100644
--- a/spacy/matcher/matcher.pyx
+++ b/spacy/matcher/matcher.pyx
@@ -18,7 +18,7 @@ from ..tokens.doc cimport Doc, get_token_attr_for_matcher
 from ..tokens.span cimport Span
 from ..tokens.token cimport Token
 from ..tokens.morphanalysis cimport MorphAnalysis
-from ..attrs cimport ID, attr_id_t, NULL_ATTR, ORTH, POS, TAG, DEP, LEMMA, MORPH
+from ..attrs cimport ID, attr_id_t, NULL_ATTR, ORTH, POS, TAG, DEP, LEMMA, MORPH, ENT_IOB
 
 from ..schemas import validate_token_pattern
 from ..errors import Errors, MatchPatternError, Warnings
@@ -796,9 +796,13 @@ def _get_attr_values(spec, string_store):
                 attr = "ORTH"
             if attr == "IS_SENT_START":
                 attr = "SENT_START"
+                ent_iob = True
             attr = IDS.get(attr)
         if isinstance(value, str):
-            value = string_store.add(value)
+            if attr == ENT_IOB and value in Token.iob_strings():
+                value = Token.iob_strings().index(value)
+            else:
+                value = string_store.add(value)
         elif isinstance(value, bool):
             value = int(value)
         elif isinstance(value, int):
diff --git a/spacy/schemas.py b/spacy/schemas.py
index b3ea11d8bed..44429acf8e9 100644
--- a/spacy/schemas.py
+++ b/spacy/schemas.py
@@ -1,4 +1,4 @@
-from typing import Dict, List, Union, Optional, Any, Callable, Type, Tuple
+from typing import Dict, List, Union, Optional, Any, Callable, Type, Tuple, Literal
 from typing import Iterable, TypeVar, TYPE_CHECKING
 from enum import Enum
 from pydantic import BaseModel, Field, ValidationError, validator, create_model
@@ -12,6 +12,7 @@
 from .attrs import NAMES
 from .lookups import Lookups
 from .util import is_cython_func
+from .tokens import Token
 
 if TYPE_CHECKING:
     # This lets us add type hints for mypy etc. without causing circular imports
@@ -209,6 +210,8 @@ class TokenPatternOperator(str, Enum):
 UnderscoreValue = Union[
     TokenPatternString, TokenPatternNumber, str, int, float, list, bool
 ]
+IobString = Literal[Token.iob_strings()]
+IobStringValue = Union[TokenPatternString, IobString]
 
 
 class TokenPattern(BaseModel):
@@ -222,6 +225,7 @@ class TokenPattern(BaseModel):
     lemma: Optional[StringValue] = None
     shape: Optional[StringValue] = None
     ent_type: Optional[StringValue] = None
+    ent_iob: Optional[IobStringValue] = None
     norm: Optional[StringValue] = None
     length: Optional[NumberValue] = None
     spacy: Optional[StrictBool] = None
diff --git a/spacy/tests/matcher/test_matcher_api.py b/spacy/tests/matcher/test_matcher_api.py
index c02d65cdfe3..a27baf13099 100644
--- a/spacy/tests/matcher/test_matcher_api.py
+++ b/spacy/tests/matcher/test_matcher_api.py
@@ -642,3 +642,30 @@ def test_matcher_no_zero_length(en_vocab):
     matcher = Matcher(en_vocab)
     matcher.add("TEST", [[{"TAG": "C", "OP": "?"}]])
     assert len(matcher(doc)) == 0
+
+
+def test_matcher_ent_iob_key(en_vocab):
+    """Test that patterns with ent_iob works correctly."""
+    matcher = Matcher(en_vocab)
+    matcher.add("Rule", [[{"ENT_IOB": "I"}]])
+    doc1 = Doc(en_vocab, words=["I", "visited", "New", "York", "and", "California"])
+    doc1.ents = [Span(doc1, 2, 4, label="GPE"), Span(doc1, 5, 6, label="GPE")]
+    doc2 = Doc(en_vocab, words=["I", "visited", "my", "friend", "Alicia"])
+    doc2.ents = [Span(doc2, 4, 5, label="PERSON")]
+    matches1 = [doc1[start:end].text for _, start, end in matcher(doc1)]
+    matches2 = [doc2[start:end].text for _, start, end in matcher(doc2)]
+    assert len(matches1) == 1
+    assert matches1[0] == "York"
+    assert len(matches2) == 0
+
+    matcher = Matcher(en_vocab)  # Test iob pattern with operators
+    matcher.add("Rule", [[{"ENT_IOB": "I", "OP": "+"}]])
+    doc = Doc(
+        en_vocab, words=["I", "visited", "my", "friend", "Anna", "Maria", "Esperanza"]
+    )
+    doc.ents = [Span(doc, 4, 7, label="PERSON")]
+    matches = [doc[start:end].text for _, start, end in matcher(doc)]
+    assert len(matches) == 3
+    assert matches[0] == "Maria"
+    assert matches[1] == "Maria Esperanza"
+    assert matches[2] == "Esperanza"
diff --git a/spacy/tests/matcher/test_pattern_validation.py b/spacy/tests/matcher/test_pattern_validation.py
index 4d21aea818f..eaef8806acd 100644
--- a/spacy/tests/matcher/test_pattern_validation.py
+++ b/spacy/tests/matcher/test_pattern_validation.py
@@ -68,3 +68,20 @@ def test_pattern_errors(en_vocab):
     # error if subpattern attribute isn't recognized and processed
     with pytest.raises(MatchPatternError):
         matcher.add("TEST2", [[{"TEXT": {"XX": "xx"}}]])
+
+
+def test_iob_keys_invalid_value_no_validation(en_vocab):
+    matcher = Matcher(
+        en_vocab, validate=False
+    )  # Test iob pattern with invalid value, no validation
+    matcher.add("Rule", [[{"ENT_IOB": "XXX"}]])
+    doc = Doc(en_vocab, words=["I", "visited", "my", "friend", "Anna", "Maria"])
+    doc.ents = [Span(doc, 4, 6, label="PERSON")]
+    matches = [doc[start:end].text for _, start, end in matcher(doc)]
+    assert len(matches) == 0
+
+
+def test_iob_keys_invalid_value_validation(en_vocab):
+    matcher = Matcher(en_vocab)  # Test iob pattern with invalid value, error happens
+    with pytest.raises(MatchPatternError):
+        matcher.add("Rule", [[{"ENT_IOB": "XXX"}]])
diff --git a/website/docs/api/matcher.md b/website/docs/api/matcher.md
index c34560decb4..6c4a5bfabc3 100644
--- a/website/docs/api/matcher.md
+++ b/website/docs/api/matcher.md
@@ -44,6 +44,7 @@ rule-based matching are:
 | `SPACY`                                         | Token has a trailing space. ~~bool~~                                                                                      |
 |  `POS`, `TAG`, `MORPH`, `DEP`, `LEMMA`, `SHAPE` | The token's simple and extended part-of-speech tag, morphological analysis, dependency label, lemma, shape. ~~str~~       |
 | `ENT_TYPE`                                      | The token's entity label. ~~str~~                                                                                         |
+| `ENT_IOB`                                       | The IOB part of the token's entity tag. ~~str~~                                                                           |
 | `_` <Tag variant="new">2.1</Tag>                | Properties in [custom extension attributes](/usage/processing-pipelines#custom-components-attributes). ~~Dict[str, Any]~~ |
 | `OP`                                            | Operator or quantifier to determine how often to match a token pattern. ~~str~~                                           |
 


Code review comments:
diff_hunk: @@ -68,3 +68,20 @@ def test_pattern_errors(en_vocab):
     # error if subpattern attribute isn't recognized and processed
     with pytest.raises(MatchPatternError):
         matcher.add("TEST2", [[{"TEXT": {"XX": "xx"}}]])
+
+
+def test_iob_keys_invalid_value_no_validation(en_vocab):
+    matcher = Matcher(
+        en_vocab, validate=False
+    )  # Test iob pattern with invalid value, no validation
+    matcher.add("Rule", [[{"ENT_IOB": "XXX"}]])
+    doc = Doc(en_vocab, words=["I", "visited", "my", "friend", "Anna", "Maria"])
+    doc.ents = [Span(doc, 4, 6, label="PERSON")]
+    matches = [doc[start:end].text for _, start, end in matcher(doc)]
+    assert len(matches) == 0
+
+
+def test_iob_keys_invalid_value_validation(en_vocab):
+    matcher = Matcher(en_vocab)  # Test iob pattern with invalid value, error happens
+    with pytest.raises(MatchPatternError):
+        matcher.add("Rule", [[{"ENT_IOB": "XXX"}]])
path: spacy/tests/matcher/test_pattern_validation.py
text: @user1:
Can you incorporate this pattern into a parameter for the first test instead?