Repository: aphp/edsnlp (language: python)

Issue:
feat: add UMLS terminology 
<!--- Provide a general summary of your changes in the title. -->

## Description
For the moment, EDS-NLP only allows the extraction and normalisation of entities to ATC (via ROMEDI) and ICD10.
As UMLS is an international resource and brings together many terminologies (including SnomedCT) in many languages, integrating it would greatly benefit the library and its users to

- automatically categorise texts in a corpus according to different concept IDs
- perform entity searching
- create processing rules (if ent.concept_id is a child of CUIXXXXX then, ...)
- do pre-annotation of corpora

<!--- Describe the changes. -->
## What changes

- Add a method to download the UMLS data and create a CUI to synonym dictionary that is also saved locally using pystow.
- Add a `TerminologyMatcher` in the same fashion as CIM10 and its corresponding entrypoint.
- Add tests in tests/pipelines/ner/test_umls.py
- Edit documentation (docs/pipelines/index.md and docs/pipelines/ner/umls.md)
- Add new dependencies:`umls_downloader`, `tqdm`
- Edit changelog 

## Checklist

<!--- Every item must be checked before the PR is merged. [] -> [x] -->

- [ ] If this PR is a bug fix, the bug is documented in the test suite.
- [x] Changes were documented in the changelog (pending section).
- [x] If necessary, changes were made to the documentation (eg new pipeline).



Discussion / hints:
Hey @percevalw, are there some steps left to do before merging? 

Fix patch:
diff --git a/changelog.md b/changelog.md
index 80e740e41..bc6b77243 100644
--- a/changelog.md
+++ b/changelog.md
@@ -3,19 +3,20 @@
 ## Unreleased
 
 ### Added
-- Harmonize processing utils (distributed custom_pipe) to have the same API for Pandas and Pyspark
 - `eds.history` : Add the option to consider only the closest dates in the sentence (dates inside the boundaries and if there is not, it takes the closest date in the entire sentence).
 - `eds.negation` : It takes into account following past participates and preceding infinitives.
 - `eds.hypothesis`: It takes into account following past participates hypothesis verbs.
 - `eds.negation` & `eds.hypothesis` : Introduce new patterns and remove unnecessary patterns.
 - `eds.dates` : Add a pattern for preceding relative dates (ex: l'embolie qui est survenue **à 10 jours**).
 - Improve patterns in the `eds.pollution` component to account for multiline footers
+- Add UMLS terminology matcher `eds.umls`
 
 ### Fixed
 
 - `eds.hypothesis` : Remove too generic patterns.
-- ``EDSTokenizer`` : It now tokenizes ``"rechereche d'"`` as ``["recherche", "d'"]``, instead of ``["recherche", "d", "'"]``.
-- Typos : Fix small typose in the documentation and in the docstring.
+- `EDSTokenizer` : It now tokenizes `"rechereche d'"` as `["recherche", "d'"]`, instead of `["recherche", "d", "'"]`.
+- Fix small typos in the documentation and in the docstring.
+- Harmonize processing utils (distributed custom_pipe) to have the same API for Pandas and Pyspark
 
 ## v0.7.2 (2022-10-26)
 
diff --git a/docs/pipelines/index.md b/docs/pipelines/index.md
index c3daa7236..7774ffbe1 100644
--- a/docs/pipelines/index.md
+++ b/docs/pipelines/index.md
@@ -46,6 +46,7 @@ EDS-NLP provides easy-to-use spaCy components.
     | `eds.cim10`              | A CIM10 terminology matcher |
     | `eds.drugs`              | A Drug mentions extractor   |
     | `eds.adicap`             | A ADICAP codes extractor    |
+    | `eds.umls`               | A UMLS terminology matcher  |
 
 You can add them to your spaCy pipeline by simply calling `add_pipe`, for instance:
 
diff --git a/docs/pipelines/ner/umls.md b/docs/pipelines/ner/umls.md
new file mode 100644
index 000000000..e9e7a65a8
--- /dev/null
+++ b/docs/pipelines/ner/umls.md
@@ -0,0 +1,68 @@
+# UMLS
+
+The `eds.umls` pipeline component matches the UMLS (Unified Medical Language System from NIH) terminology.
+
+!!! warning "Very low recall"
+
+    When using the `exact' matching mode, this component has a very poor recall performance.
+    We can use the `simstring` mode to retrieve approximate matches, albeit at the cost of a significantly higher computation time.
+
+## Usage
+
+`eds.umls` is an additional module that needs to be setup by:
+
+1. `pip install -U umls_downloader`
+2. [Signing up for a UMLS Terminology Services Account](https://uts.nlm.nih.gov/uts/signup-login). After filling a short form, you will receive your token API within a few days.
+3. Set `UMLS_API_KEY` locally: `export UMLS_API_KEY=your_api_key`
+
+```python
+import spacy
+
+nlp = spacy.blank("fr")
+nlp.add_pipe("eds.umls")
+
+text = "Grosse toux: le malade a été mordu par des Amphibiens " "sous le genou"
+
+doc = nlp(text)
+
+doc.ents
+# Out: (toux, a, par, Amphibiens, genou)
+
+ent = doc.ents[0]
+
+ent.label_
+# Out: umls
+
+ent._.umls
+# Out: C0010200
+```
+
+You can easily change the default languages and sources with the `pattern_config` argument:
+
+```python
+import spacy
+
+# Enable the french and english languages, through the french MeSH and LOINC
+pattern_config = dict(languages=["FRE", "ENG"], sources=["MSHFRE", "LNC"])
+
+nlp = spacy.blank("fr")
+nlp.add_pipe("eds.umls", config=dict(pattern_config=pattern_config))
+```
+
+See more options of languages and sources [here](https://www.nlm.nih.gov/research/umls/sourcereleasedocs/index.html).
+
+## Configuration
+
+The pipeline can be configured using the following parameters :
+
+| Parameter             | Description                                                    | Default   |
+|-----------------------|----------------------------------------------------------------|-----------|
+| `term_matcher`        | Which algorithm should we use : `exact` or `simstring`         | `"exact"` |
+| `term_matcher_config` | Config of the algorithm (`SimstringMatcher`'s for `simstring`) | `{}` |
+| `pattern_config`      | Config of the terminology patterns loader                      | `{"languages"=["FRE"], sources=None}` (sources None means all available sources) |
+| `attr`                | spaCy attribute to match on (eg `NORM`, `TEXT`, `LOWER`)       | `"LOWER"` |
+| `ignore_excluded`     | Whether to ignore excluded tokens for matching                 | `False`   |
+
+## Authors and citation
+
+The `eds.umls` pipeline was developed by AP-HP's Data Science team and INRIA SODA's team.
diff --git a/edsnlp/matchers/phrase.pyx b/edsnlp/matchers/phrase.pyx
index 5f85cc43e..7b9d8b05d 100644
--- a/edsnlp/matchers/phrase.pyx
+++ b/edsnlp/matchers/phrase.pyx
@@ -4,6 +4,7 @@ import re
 from preshed.maps cimport map_clear, map_get, map_init, map_iter, map_set
 
 from spacy import Language
+from tqdm import tqdm
 
 from preshed.maps cimport MapStruct, key_t
 from spacy.matcher.phrasematcher cimport PhraseMatcher
@@ -94,7 +95,10 @@ cdef class EDSPhraseMatcher(PhraseMatcher):
             )
         ]
         with nlp.select_pipes(enable=token_pipelines):
-            for key, expressions in terms.items():
+            for key, expressions in tqdm(
+                terms.items(),
+                desc="Adding terms into the pipeline"
+            ):
                 if isinstance(expressions, dict):
                     attr = expressions.get("attr")
                     expressions = expressions.get("patterns")
@@ -106,15 +110,16 @@ cdef class EDSPhraseMatcher(PhraseMatcher):
                 self.add(key, patterns, attr)
 
     cdef void find_matches(self, Doc doc, int start_idx, int end_idx, vector[SpanC] *matches) nogil:
-        cdef MapStruct * current_node = self.c_map
-        cdef int start = 0
-        cdef int idx = start_idx
-        cdef int idy = start_idx
-        cdef key_t key
-        cdef void * value
-        cdef int i = 0
-        cdef SpanC ms
-        cdef void * result
+        cdef:
+            MapStruct * current_node = self.c_map
+            int start = 0
+            int idx = start_idx
+            int idy = start_idx
+            key_t key
+            void * value
+            int i = 0
+            SpanC ms
+            void * result
         while idx < end_idx:
             start = idx
             token = Token.get_struct_attr(&doc.c[idx], self.attr)
diff --git a/edsnlp/pipelines/core/terminology/terminology.py b/edsnlp/pipelines/core/terminology/terminology.py
index 031f989d8..31e4b5837 100644
--- a/edsnlp/pipelines/core/terminology/terminology.py
+++ b/edsnlp/pipelines/core/terminology/terminology.py
@@ -97,6 +97,11 @@ def __init__(
 
         self.set_extensions()
 
+    def set_extensions(self) -> None:
+        super().set_extensions()
+        if not Span.has_extension(self.label):
+            Span.set_extension(self.label, default=None)
+
     def process(self, doc: Doc) -> List[Span]:
         """
         Find matching spans in doc.
@@ -127,6 +132,7 @@ def process(self, doc: Doc) -> List[Span]:
                 label=self.label,
                 kb_id=match.label,
             )
+            span._.set(self.label, match.label_)
             spans.append(span)
 
         return spans
diff --git a/edsnlp/pipelines/factories.py b/edsnlp/pipelines/factories.py
index 3985789e0..c3c226112 100644
--- a/edsnlp/pipelines/factories.py
+++ b/edsnlp/pipelines/factories.py
@@ -26,6 +26,7 @@
 from .ner.scores.factory import create_component as score
 from .ner.scores.sofa.factory import create_component as sofa
 from .ner.scores.tnm.factory import create_component as tnm
+from .ner.umls.factory import create_component as umls
 from .qualifiers.family.factory import create_component as family
 from .qualifiers.history.factory import create_component as history
 from .qualifiers.hypothesis.factory import create_component as hypothesis
diff --git a/edsnlp/pipelines/ner/umls/__init__.py b/edsnlp/pipelines/ner/umls/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/edsnlp/pipelines/ner/umls/factory.py b/edsnlp/pipelines/ner/umls/factory.py
new file mode 100644
index 000000000..7e80e85c4
--- /dev/null
+++ b/edsnlp/pipelines/ner/umls/factory.py
@@ -0,0 +1,45 @@
+from typing import Any, Dict, Union
+
+from spacy.language import Language
+
+from edsnlp.pipelines.core.terminology import TerminologyMatcher, TerminologyTermMatcher
+
+from . import patterns
+
+PATTERN_CONFIG = dict(
+    languages=["FRE"],
+    sources=None,
+)
+
+DEFAULT_CONFIG = dict(
+    attr="NORM",
+    ignore_excluded=False,
+    term_matcher=TerminologyTermMatcher.exact,
+    term_matcher_config={},
+    pattern_config=PATTERN_CONFIG,
+)
+
+
+@Language.factory(
+    "eds.umls", default_config=DEFAULT_CONFIG, assigns=["doc.ents", "doc.spans"]
+)
+def create_component(
+    nlp: Language,
+    name: str,
+    attr: Union[str, Dict[str, str]],
+    ignore_excluded: bool,
+    term_matcher: TerminologyTermMatcher,
+    term_matcher_config: Dict[str, Any],
+    pattern_config: Dict[str, Any],
+):
+
+    return TerminologyMatcher(
+        nlp,
+        label="umls",
+        regex=None,
+        terms=patterns.get_patterns(pattern_config),
+        attr=attr,
+        ignore_excluded=ignore_excluded,
+        term_matcher=term_matcher,
+        term_matcher_config=term_matcher_config,
+    )
diff --git a/edsnlp/pipelines/ner/umls/patterns.py b/edsnlp/pipelines/ner/umls/patterns.py
new file mode 100644
index 000000000..c68b17c45
--- /dev/null
+++ b/edsnlp/pipelines/ner/umls/patterns.py
@@ -0,0 +1,160 @@
+import os
+import warnings
+import zipfile
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Dict, List, Tuple
+
+import pystow
+from tqdm import tqdm
+from umls_downloader import download_umls
+
+PATTERN_VERSION = "0.7.0"
+UMLS_VERSION = "2022AA"
+
+
+def get_patterns(config: Dict[str, Any]) -> Dict[str, List[str]]:
+    """Load the UMLS terminology patterns.
+
+    Parameters
+    ----------
+    config : dict[list]
+        Languages and sources to select from the whole terminology.
+        For both keys, None will select all values.
+
+    Return
+    ------
+    patterns : dict[list]
+        The mapping between CUI codes and their synonyms.
+
+    Notes
+    -----
+    When run for the first time, this method will download the
+    entire UMLS file and store it at ~/.data/bio/umls/2022AA/.
+    Therefore the second run will be significantly faster than
+    the first one.
+    """
+
+    path, module, filename = get_path(config)
+
+    if path.exists():
+        print(f"Loading {filename} from {module.base}")
+        return module.load_pickle(name=filename)
+    else:
+        patterns = download_and_agg_umls(config)
+        module.dump_pickle(name=filename, obj=patterns)
+        print(f"Saved patterns into {module.base / filename}")
+        return patterns
+
+
+def get_path(config: Dict[str, Any]) -> Tuple[Path, pystow.impl.Module, str]:
+    """Get the path, module and filename of the UMLS file.
+
+    Parameters
+    ----------
+    config : dict[list]
+        Languages and sources to select from the whole terminology.
+        For both keys, None will select all values.
+
+    Return
+    ------
+    path, module, filename : pathlib.Path, pystow.module, str
+
+    Notes
+    -----
+    `get_path` will convert the config dict into a pretty filename.
+
+    Examples
+    --------
+    >>> config = {"languages": ["FRE", "ENG"], "sources": None}
+    >>> print(get_path(config))
+    .data/bio/umls/2022AA/languagesFRE-ENG_sourcesNone.pkl"
+    """
+    config_txt = ""
+    for k, v in config.items():
+        if isinstance(v, (list, tuple)):
+            v = "-".join(v)
+        _config_txt = f"{k}{v}"
+        if config_txt:
+            _config_txt = f"_{_config_txt}"
+        config_txt += _config_txt
+
+    filename = f"{PATTERN_VERSION}_{config_txt}.pkl"
+    module = pystow.module("bio", "umls", UMLS_VERSION)
+    path = module.base / filename
+
+    return path, module, filename
+
+
+def download_and_agg_umls(config) -> Dict[str, List[str]]:
+    """Download the UMLS if not exist and create a mapping
+    between CUI code and synonyms.
+
+    Parameters
+    ----------
+    config : dict[list]
+        Languages and sources to select from the whole terminology.
+        For both keys, None will select all values.
+
+    Return
+    ------
+    patterns : dict[list]
+        The mapping between CUI codes and their synonyms.
+
+    Notes
+    -----
+    Performs filtering on the returned mapping only, not the downloaded
+    resource.
+    """
+
+    api_key = os.getenv("UMLS_API_KEY")
+    if not api_key:
+        warnings.warn(
+            "You need to define UMLS_API_KEY to download the UMLS. "
+            "Get a key by creating an account at "
+            "https://uts.nlm.nih.gov/uts/signup-login"
+        )
+
+    path = download_umls(version=UMLS_VERSION, api_key=api_key)
+
+    # https://www.ncbi.nlm.nih.gov/books/NBK9685/table/ch03.T.concept_names_and_sources_file_mr/ # noqa
+    header = [
+        "CUI",
+        "LAT",
+        "TS",
+        "LUI",
+        "STT",
+        "SUI",
+        "ISPREF",
+        "AUI",
+        "SAUI",
+        "SCUI",
+        "SDUI",
+        "SAB",
+        "TTY",
+        "CODE",
+        "STR",
+        "STRL",
+        "SUPPRESS",
+    ]
+    header_to_idx = dict(zip(header, range(len(header))))
+    patterns = defaultdict(list)
+
+    languages = config.get("languages")
+    sources = config.get("sources")
+
+    with zipfile.ZipFile(path) as zip_file:
+        with zip_file.open("MRCONSO.RRF", mode="r") as file:
+            for row in tqdm(
+                file.readlines(), desc="Loading 'MRCONSO.RRF' into a dictionnary"
+            ):
+                row = row.decode("utf-8").split("|")
+
+                if (languages is None or row[header_to_idx["LAT"]] in languages) and (
+                    sources is None or row[header_to_idx["SAB"]] in sources
+                ):
+                    cui = row[header_to_idx["CUI"]]
+                    synonym = row[header_to_idx["STR"]]
+                    patterns[cui].append(synonym)
+
+    return patterns
diff --git a/mkdocs.yml b/mkdocs.yml
index c2fafd4d2..7c8a09877 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -81,6 +81,7 @@ nav:
           - pipelines/ner/drugs.md
           - pipelines/ner/adicap.md
           - pipelines/ner/cim10.md
+          - pipelines/ner/umls.md
       - Trainable components:
           - pipelines/trainable/index.md
           - pipelines/trainable/ner.md
diff --git a/pyproject.toml b/pyproject.toml
index 6616f0b8b..e36a91af1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -2,7 +2,7 @@
 requires = [
     "setuptools",
     "cython>=0.25,<3.0",
-    "spacy>=3.2<4.0",
+    "spacy>=3.2,<4.0",
     "numpy==1.15.0; python_version<='3.7'",
     "numpy==1.17.3; python_version=='3.8'",
     "numpy==1.19.3; python_version=='3.9'",
diff --git a/requirements.txt b/requirements.txt
index 544d6beb8..350f6daca 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -9,3 +9,5 @@ regex<=2022.3.2  # due to dateparser bug issue 1045
 scikit-learn>=1.0.0
 spacy>=3.1,<3.4.2
 thinc>=8.0.14
+tqdm
+umls-downloader
diff --git a/setup.py b/setup.py
index 3da96ab0b..4dcc1b50b 100644
--- a/setup.py
+++ b/setup.py
@@ -84,6 +84,7 @@ def get_version(path):
     "drugs = edsnlp.components:drugs",
     "nested_ner = edsnlp.components:nested_ner",
     "adicap = edsnlp.components:adicap",
+    "umls = edsnlp.components:umls",
 ]
 
 scorers = [


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index f737e74b7..2a17736c1 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -26,12 +26,19 @@ jobs:
 
   Pytest:
     runs-on: ubuntu-latest
+    env:
+      UMLS_API_KEY: ${{ secrets.UMLS_API_KEY }}
     strategy:
       fail-fast: true
       matrix:
         python-version: ["3.7", "3.8", "3.9"]
     steps:
       - uses: actions/checkout@v2
+      - name: Cache downloaded resources
+        uses: actions/cache@v3
+        with:
+          path: ~/.data/
+          key: resources
       - name: Set up Java
         uses: actions/setup-java@v2
         with:
diff --git a/tests/pipelines/ner/test_umls.py b/tests/pipelines/ner/test_umls.py
new file mode 100644
index 000000000..2d4ca67ec
--- /dev/null
+++ b/tests/pipelines/ner/test_umls.py
@@ -0,0 +1,62 @@
+import pytest
+
+pytest.importorskip("umls_downloader", reason="umls_downloader package not found")
+
+import os  # noqa: E402
+from pathlib import Path  # noqa: E402
+
+from spacy.language import Language  # noqa: E402
+
+from edsnlp.pipelines.ner.umls.patterns import get_path, get_patterns  # noqa: E402
+from edsnlp.utils.examples import parse_example  # noqa: E402
+
+examples = [
+    "Grosse <ent umls=C0010200>toux</ent> : Le malade a été mordu "
+    "<ent umls=C1257901>par</ent> des <ent umls=C0002668>Amphibiens</ent>"
+    " sous le <ent umls=C0022742>genou</ent>"
+]
+pattern_config = {"lang": ["FRE"], "sources": ["MSHFRE"]}
+
+
+def test_get_patterns():
+
+    path, _, _ = get_path(pattern_config)
+    assert isinstance(path, Path)
+
+    api_key = os.getenv("UMLS_API_KEY")
+
+    try:
+        patterns = get_patterns(pattern_config)
+    except AttributeError:
+        if not api_key:
+            raise Exception(
+                "Missing UMLS_API_KEY env variable to download the UMLS. "
+                f"UMLS cannot be found at path: {path}.\nTo skip the test, "
+                f"uninstall the umls_downloader package."
+            )
+        raise
+
+    assert path.exists()
+
+    assert len(patterns) == 48587
+
+
+def test_add_pipe(blank_nlp: Language):
+    path, _, _ = get_path(pattern_config)
+    if not path.exists():
+        pytest.xfail(
+            "The umls_downloader package is installed but "
+            f"UMLS cannot be found at path: {path}"
+        )
+
+    blank_nlp.add_pipe("eds.umls", config=dict(pattern_config=pattern_config))
+
+    assert "eds.umls" in blank_nlp.pipe_names
+
+    for text, entities in map(parse_example, examples):
+        doc = blank_nlp(text)
+        assert len(doc.ents) == len(entities)
+
+        for ent, entity in zip(doc.ents, entities):
+            assert ent.text == text[entity.start_char : entity.end_char]
+            assert ent._.umls == ent.kb_id_ == entity.modifiers[0].value