Repository: aphp/edsnlp (language: python)

Issue:
identify tables

Suggestion for a new pipeline to detect tables of biological results. To discuss possible improvements to this first example. 

```python

import spacy
from edsnlp import components
from io import StringIO


nlp = spacy.blank("fr")

regex = dict(
    tables=[r"(\b.*[|¦].*\n)+",],
)

# Sentencizer component, needed for negation detection
nlp.add_pipe("sentences")
# Matcher component
nlp.add_pipe("matcher", config=dict(regex=regex))

text = """
Le patientqsfqfdf bla bla bla
Leucocytes ¦x10*9/L ¦4.97 ¦4.09-11
Hématies ¦x10*12/L¦4.68 ¦4.53-5.79
Hémoglobine ¦g/dL ¦14.8 ¦13.4-16.7
Hématocrite ¦% ¦44.2 ¦39.2-48.6
VGM ¦fL ¦94.4 + ¦79.6-94
TCMH ¦pg ¦31.6 ¦27.3-32.8
CCMH ¦g/dL ¦33.5 ¦32.4-36.3
Plaquettes ¦x10*9/L ¦191 ¦172-398
VMP ¦fL ¦11.5 + ¦7.4-10.8

qdfsdf

"""

doc = nlp(text)

table_str = doc.spans["tables"][0].text
print(table_str)


table_str = doc.spans["tables"][0].text
table_str


table_str_io = StringIO(table_str)

table_pandas = pd.read_csv(table_str_io, sep="¦", engine="python",header=None)

table_pandas
```

|    | 0           | 1        | 2      | 3         |
|---:|:------------|:---------|:-------|:----------|
|  0 | Leucocytes  | x10*9/L  | 4.97   | 4.09-11   |
|  1 | Hématies    | x10*12/L | 4.68   | 4.53-5.79 |
|  2 | Hémoglobine | g/dL     | 14.8   | 13.4-16.7 |
|  3 | Hématocrite | %        | 44.2   | 39.2-48.6 |
|  4 | VGM         | fL       | 94.4 + | 79.6-94   |
|  5 | TCMH        | pg       | 31.6   | 27.3-32.8 |
|  6 | CCMH        | g/dL     | 33.5   | 32.4-36.3 |
|  7 | Plaquettes  | x10*9/L  | 191    | 172-398   |
|  8 | VMP         | fL       | 11.5 + | 7.4-10.8  |


Discussion / hints:
Great idea @aricohen93.

Do you have time to work on it and propose an actual pipeline component? I figure `eds.tables`, within `edsnlp/pipelines/misc`?
@aricohen93 have you put more thought into this ? I suppose we could add this quite easily, perhaps on an "experimental" status?
Great idea Mr. Cohen !
I am eager to implement this pipeline ! 👍 
Very nice indeed ! 

I am just testing it, and it works well on example where I just have one table per file, but I have issue with my files that contains several tables, I just get the first one... any ideas on how I could handle them with your pipeline ? 


--> Sorry, finally, I just iterate on `doc.spans["tables"]` and it worked perfectly 

Many thanks for this tool !!


Fix patch:
diff --git a/changelog.md b/changelog.md
index 0e156fb04..4f948cb8f 100644
--- a/changelog.md
+++ b/changelog.md
@@ -1,4 +1,7 @@
 # Changelog
+## Unreleased
+### Added
+- `eds.tables`: new pipeline to identify formatted tables
 
 ## Unreleased
 
diff --git a/docs/index.md b/docs/index.md
index b4b11704c..562e26cfa 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -93,9 +93,11 @@ This example is complete, it should run as-is. Check out the [spaCy 101 page](tu
     | Pipeline           | Description                                 |
     | ------------------ | ------------------------------------------- |
     | `eds.dates`        | Date extraction and normalisation           |
+    | `eds.consultation_dates`| Identify consultation dates            |
     | `eds.measurements` | Measure extraction and normalisation        |
     | `eds.sections`     | Section detection                           |
     | `eds.reason`       | Rule-based hospitalisation reason detection |
+    | `eds.tables`       | Tables detection                            |
 
 === "NER"
 
@@ -111,6 +113,7 @@ This example is complete, it should run as-is. Check out the [spaCy 101 page](tu
     | `eds.cim10`              | A CIM10 terminology matcher |
     | `eds.drugs`              | A Drug mentions extractor   |
     | `eds.adicap`             | A ADICAP codes extractor    |
+    | `eds.umls`               | A UMLS terminology matcher  |
 
 === "Trainable"
 
diff --git a/docs/pipelines/index.md b/docs/pipelines/index.md
index 62855c087..25175b112 100644
--- a/docs/pipelines/index.md
+++ b/docs/pipelines/index.md
@@ -28,9 +28,11 @@ EDS-NLP provides easy-to-use spaCy components.
     | Pipeline           | Description                                 |
     | ------------------ | ------------------------------------------- |
     | `eds.dates`        | Date extraction and normalisation           |
+    | `eds.consultation_dates`| Identify consultation dates            |
     | `eds.measurements` | Measure extraction and normalisation        |
     | `eds.sections`     | Section detection                           |
     | `eds.reason`       | Rule-based hospitalisation reason detection |
+    | `eds.tables`       | Tables detection                            |
 
 === "NER"
 
diff --git a/docs/pipelines/misc/tables.md b/docs/pipelines/misc/tables.md
new file mode 100644
index 000000000..b4d6966b0
--- /dev/null
+++ b/docs/pipelines/misc/tables.md
@@ -0,0 +1,96 @@
+# Tables
+
+The `eds.tables` pipeline's role is to detect tables present in a medical document.
+We use simple regular expressions to extract tables like text.
+
+## Usage
+
+```python
+import spacy
+
+nlp = spacy.blank("fr")
+nlp.add_pipe("eds.normalizer")
+nlp.add_pipe("eds.tables")
+
+text = """
+SERVICE
+MEDECINE INTENSIVE –
+REANIMATION
+Réanimation / Surveillance Continue
+Médicale
+
+COMPTE RENDU D'HOSPITALISATION du 05/06/2020 au 10/06/2020
+Madame DUPONT Marie, née le 16/05/1900, âgée de 20 ans, a été hospitalisée en réanimation du
+05/06/1920 au 10/06/1920 pour intoxication médicamenteuse volontaire.
+
+
+Examens complémentaires
+Hématologie
+Numération
+Leucocytes ¦x10*9/L ¦4.97 ¦4.09-11
+Hématies ¦x10*12/L¦4.68 ¦4.53-5.79
+Hémoglobine ¦g/dL ¦14.8 ¦13.4-16.7
+Hématocrite ¦% ¦44.2 ¦39.2-48.6
+VGM ¦fL ¦94.4 + ¦79.6-94
+TCMH ¦pg ¦31.6 ¦27.3-32.8
+CCMH ¦g/dL ¦33.5 ¦32.4-36.3
+Plaquettes ¦x10*9/L ¦191 ¦172-398
+VMP ¦fL ¦11.5 + ¦7.4-10.8
+
+Sur le plan neurologique : Devant la persistance d'une confusion à distance de l'intoxication au
+...
+
+2/2Pat : <NOM> <Prenom>|F |<date> | <ipp> |Intitulé RCP
+
+"""
+
+doc = nlp(text)
+
+# A table span
+table = doc.spans["tables"][0]
+# Leucocytes ¦x10*9/L ¦4.97 ¦4.09-11
+# Hématies ¦x10*12/L¦4.68 ¦4.53-5.79
+# Hémoglobine ¦g/dL ¦14.8 ¦13.4-16.7
+# Hématocrite ¦% ¦44.2 ¦39.2-48.6
+# VGM ¦fL ¦94.4 + ¦79.6-94
+# TCMH ¦pg ¦31.6 ¦27.3-32.8
+# CCMH ¦g/dL ¦33.5 ¦32.4-36.3
+# Plaquettes ¦x10*9/L ¦191 ¦172-398
+# VMP ¦fL ¦11.5 + ¦7.4-10.8
+
+# Convert span to Pandas table
+df = table._.to_pd_table()
+type(df)
+# >> pandas.core.frame.DataFrame
+```
+The pd DataFrame:
+|      | 0           | 1        | 2      | 3         |
+| ---: | :---------- | :------- | :----- | :-------- |
+|    0 | Leucocytes  | x10*9/L  | 4.97   | 4.09-11   |
+|    1 | Hématies    | x10*12/L | 4.68   | 4.53-5.79 |
+|    2 | Hémoglobine | g/dL     | 14.8   | 13.4-16.7 |
+|    3 | Hématocrite | %        | 44.2   | 39.2-48.6 |
+|    4 | VGM         | fL       | 94.4 + | 79.6-94   |
+|    5 | TCMH        | pg       | 31.6   | 27.3-32.8 |
+|    6 | CCMH        | g/dL     | 33.5   | 32.4-36.3 |
+|    7 | Plaquettes  | x10*9/L  | 191    | 172-398   |
+|    8 | VMP         | fL       | 11.5 + | 7.4-10.8  |
+
+## Declared extensions
+
+The `eds.tables` pipeline declares one [spaCy extension](https://spacy.io/usage/processing-pipelines#custom-components-attributes) on the `Span` object: the `to_pd_table()` method returns a parsed pandas version of the table.
+
+## Configuration
+
+The pipeline can be configured using the following parameters :
+
+| Parameter         | Explanation                                      | Default                |
+| ----------------- | ------------------------------------------------ | ---------------------- |
+| `tables_pattern`  | Pattern to identify table spans                  | `rf"(\b.*{sep}.*\n)+"` |
+| `sep_pattern`     | Pattern to identify column separation            | `r"¦"`                 |
+| `ignore_excluded` | Ignore excluded tokens                           | `True`                 |
+| `attr`            | spaCy attribute to match on, eg `NORM` or `TEXT` | `"TEXT"`               |
+
+## Authors and citation
+
+The `eds.tables` pipeline was developed by AP-HP's Data Science team.
diff --git a/edsnlp/pipelines/factories.py b/edsnlp/pipelines/factories.py
index c3c226112..113716331 100644
--- a/edsnlp/pipelines/factories.py
+++ b/edsnlp/pipelines/factories.py
@@ -15,6 +15,7 @@
 from .misc.measurements.factory import create_component as measurements
 from .misc.reason.factory import create_component as reason
 from .misc.sections.factory import create_component as sections
+from .misc.tables.factory import create_component as tables
 from .ner.adicap.factory import create_component as adicap
 from .ner.cim10.factory import create_component as cim10
 from .ner.covid.factory import create_component as covid
diff --git a/edsnlp/pipelines/misc/tables/__init__.py b/edsnlp/pipelines/misc/tables/__init__.py
new file mode 100644
index 000000000..861ef7628
--- /dev/null
+++ b/edsnlp/pipelines/misc/tables/__init__.py
@@ -0,0 +1,2 @@
+from .patterns import regex, sep
+from .tables import TablesMatcher
diff --git a/edsnlp/pipelines/misc/tables/factory.py b/edsnlp/pipelines/misc/tables/factory.py
new file mode 100644
index 000000000..047c2a55f
--- /dev/null
+++ b/edsnlp/pipelines/misc/tables/factory.py
@@ -0,0 +1,32 @@
+from typing import Dict, List, Optional, Union
+
+from spacy.language import Language
+
+from edsnlp.pipelines.misc.tables import TablesMatcher
+from edsnlp.utils.deprecation import deprecated_factory
+
+DEFAULT_CONFIG = dict(
+    tables_pattern=None,
+    sep_pattern=None,
+    attr="TEXT",
+    ignore_excluded=True,
+)
+
+
+@deprecated_factory("tables", "eds.tables", default_config=DEFAULT_CONFIG)
+@Language.factory("eds.tables", default_config=DEFAULT_CONFIG)
+def create_component(
+    nlp: Language,
+    name: str,
+    tables_pattern: Optional[Dict[str, Union[List[str], str]]],
+    sep_pattern: Optional[str],
+    attr: str,
+    ignore_excluded: bool,
+):
+    return TablesMatcher(
+        nlp,
+        tables_pattern=tables_pattern,
+        sep_pattern=sep_pattern,
+        attr=attr,
+        ignore_excluded=ignore_excluded,
+    )
diff --git a/edsnlp/pipelines/misc/tables/patterns.py b/edsnlp/pipelines/misc/tables/patterns.py
new file mode 100644
index 000000000..e4c25f019
--- /dev/null
+++ b/edsnlp/pipelines/misc/tables/patterns.py
@@ -0,0 +1,4 @@
+sep = r"¦"
+regex = dict(
+    tables=rf"(\b.*{sep}.*\n)+",
+)
diff --git a/edsnlp/pipelines/misc/tables/tables.py b/edsnlp/pipelines/misc/tables/tables.py
new file mode 100644
index 000000000..106f63216
--- /dev/null
+++ b/edsnlp/pipelines/misc/tables/tables.py
@@ -0,0 +1,93 @@
+from io import StringIO
+from typing import Dict, Optional, Union
+
+import pandas as pd
+from spacy.language import Language
+from spacy.tokens import Doc, Span
+
+from edsnlp.pipelines.core.matcher import GenericMatcher
+from edsnlp.pipelines.misc.tables import patterns
+from edsnlp.utils.filter import get_spans
+
+
+class TablesMatcher(GenericMatcher):
+    """Pipeline to identify the Tables.
+
+    It adds the key `tables` to doc.spans.
+
+    Parameters
+    ----------
+    nlp : Language
+        spaCy nlp pipeline to use for matching.
+    tables_pattern : Optional[Dict[str, str]]
+        The regex pattern to identify tables.
+        The key of dictionary should be `tables`
+    attr : str
+        spaCy's attribute to use:
+        a string with the value "TEXT" or "NORM", or a dict with
+        the key 'term_attr'. We can also add a key for each regex.
+    ignore_excluded : bool
+        Whether to skip excluded tokens.
+    """
+
+    def __init__(
+        self,
+        nlp: Language,
+        tables_pattern: Optional[Dict[str, str]],
+        sep_pattern: Optional[str],
+        attr: Union[Dict[str, str], str],
+        ignore_excluded: bool,
+    ):
+
+        if tables_pattern is None:
+            self.tables_pattern = patterns.regex
+        else:
+            self.tables_pattern = tables_pattern
+
+        if sep_pattern is None:
+            self.sep = patterns.sep
+        else:
+            self.sep = sep_pattern
+
+        super().__init__(
+            nlp,
+            terms=None,
+            regex=self.tables_pattern,
+            attr=attr,
+            ignore_excluded=ignore_excluded,
+        )
+
+        if not Span.has_extension("to_pd_table"):
+            Span.set_extension("to_pd_table", method=self.to_pd_table)
+
+        self.set_extensions()
+
+    def __call__(self, doc: Doc) -> Doc:
+        """Find spans that contain tables
+
+        Parameters
+        ----------
+        doc : Doc
+
+        Returns
+        -------
+        Doc
+        """
+        matches = self.process(doc)
+        tables = get_spans(matches, "tables")
+        # parsed = self.parse(tables=tables)
+
+        doc.spans["tables"] = tables
+
+        return doc
+
+    def to_pd_table(self, span) -> pd.DataFrame:
+        table_str_io = StringIO(span.text)
+        parsed = pd.read_csv(
+            table_str_io,
+            sep=self.sep,
+            engine="python",
+            header=None,
+            on_bad_lines="skip",
+        )
+        return parsed
diff --git a/mkdocs.yml b/mkdocs.yml
index b1a559935..46070fc9a 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -76,6 +76,7 @@ nav:
           - pipelines/misc/consultation-dates.md
           - pipelines/misc/sections.md
           - pipelines/misc/reason.md
+          - pipelines/misc/tables.md
       - Named Entity Recognition:
           - pipelines/ner/index.md
           - pipelines/ner/score.md
@@ -97,6 +98,7 @@ nav:
       - utilities/tests/blocs.md
       - utilities/tests/examples.md
       - utilities/matchers.md
+      - utilities/processing/spark.md
   - Code Reference: reference/
   - contributing.md
   - changelog.md
diff --git a/pyproject.toml b/pyproject.toml
index 5f65bfd6c..2cdebf506 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -23,8 +23,8 @@ dependencies = [
     "umls-downloader>=0.1.1",
     "numpy>=1.15.0,<1.23.2; python_version<'3.8'",
     "numpy>=1.15.0; python_version>='3.8'",
-    "pandas>=1.1.0; python_version<'3.8'",
-    "pandas>=1.4.0; python_version>='3.8'",
+    "pandas>=1.1.0,<2.0.0; python_version<'3.8'",
+    "pandas>=1.4.0,<2.0.0; python_version>='3.8'",
 ]
 [project.optional-dependencies]
 dev = [


Test patch:
diff --git a/tests/pipelines/misc/test_tables.py b/tests/pipelines/misc/test_tables.py
new file mode 100644
index 000000000..69ff76f3d
--- /dev/null
+++ b/tests/pipelines/misc/test_tables.py
@@ -0,0 +1,31 @@
+TEXT = """
+Le patientqsfqfdf bla bla bla
+Leucocytes ¦x10*9/L ¦4.97 ¦4.09-11
+Hématies ¦x10*12/L¦4.68 ¦4.53-5.79
+Hémoglobine ¦g/dL ¦14.8 ¦13.4-16.7
+Hématocrite ¦% ¦44.2 ¦39.2-48.6
+VGM ¦fL ¦94.4 + ¦79.6-94
+TCMH ¦pg ¦31.6 ¦27.3-32.8
+CCMH ¦g/dL ¦33.5 ¦32.4-36.3
+Plaquettes ¦x10*9/L ¦191 ¦172-398
+VMP ¦fL ¦11.5 + ¦7.4-10.8
+
+qdfsdf
+
+2/2Pat : <NOM> <Prenom> |<date> | <ipp> |Intitulé RCP
+
+
+"""
+
+
+def test_tables(blank_nlp):
+    blank_nlp.add_pipe("eds.normalizer")
+    blank_nlp.add_pipe("eds.tables")
+
+    doc = blank_nlp(TEXT)
+
+    assert len(doc.spans["tables"]) == 1
+
+    span = doc.spans["tables"][0]
+    df = span._.to_pd_table()
+    assert df.iloc[5, 0] == "TCMH "