Package: clinlp
clinlp:hospital:clinical+ :netherlands:nl+ :clipboard:NLP= :sparkles:clinlp:star: Performant and production-ready NLP pipelines for clinical text written in Dutch:rocket: Open source, created and maintained by the Dutch Clinical NLP community:triangular_ruler: Useful out of the box, but customization highly recommendedRead theprinciples and goals, futher down :arrow_down:Contact and contributingclinlpis very much still being shaped, so if you are enthusiastic about using or contributing toclinlp, please don't hesitate to get in touch (email|issue). We would be very happy to discuss your ideas and needs, whether its from the perspective of an (end) user, engineer or clinician, and formulate a roadmap with next steps together.Getting startedInstallationpipinstallclinlpExampleimportspacyfromclinlpimportTermnlp=spacy.blank("clinlp")# Normalizationnlp.add_pipe("clinlp_normalizer")# Sentencesnlp.add_pipe("clinlp_sentencizer")# Entitiesconcepts={"prematuriteit":["preterm","<p3","prematuriteit","partus praematurus"],"hypotensie":["hypotensie",Term("bd verlaagd",proximity=1)],"veneus_infarct":["veneus infarct",Term("VI",attr="TEXT")]}entity_matcher=nlp.add_pipe("clinlp_entity_matcher",config={"attr":"NORM","fuzzy":1})entity_matcher.load_concepts(concepts)# Qualifiersnlp.add_pipe("clinlp_context_algorithm",config={"phrase_matcher_attr":"NORM"})text=("Preterme neonaat (<p3), bd enigszins verlaagd, familieanamnese vermeldt eveneens hypotensie ""bij moeder. Thans geen aanwijzingen voor veneus infarkt wat ook geen ""verklaring voor de partus prematurus is. Risico op VI blijft aanwezig.")doc=nlp(text)Find information in the doc object:fromspacyimportdisplacydisplacy.render(doc,style="ent")With relevant qualifiers (defaults omitted for readability):forentindoc.ents:print(ent,ent._.qualifiers_str)Pretermeset()<p3set()bd enigszins verlaagdset()hypotensie{'Experiencer.OTHER'}veneus infarkt{'Negation.NEGATED'}partus prematurusset()VI{'Plausibility.HYPOTHETICAL'}CitingIf you useclinlp, please find the appropriate citation by clicking the Zenodo button below. This should always point you to the current latest release:DocumentationIntroductionclinlpis built on top of spaCy, a widely used library for Natural Language Processing. Before getting started withclinlp, it may be useful to readspaCy 101: Everything you need to know (~10 mins). Main things to know are that spaCy consists of a tokenizer (breaks a text up into small pieces, i.e. words), and various components that further process the text.Currently,clinlpoffers the following components, tailored to Dutch Clinical text, further discussed below:TokenizerNormalizerSentence splitterEntity matcherQualifier detection (negation, historical, etc.)Context AlgorithmTransformer based negation detectionTokenizerTheclinlptokenizer is built into the blank model:nlp=spacy.blank("clinlp")It employs some custom rule based logic, including:Clinical text-specific logic for splitting punctuation, units, dosages (e.g.20mg/dag:arrow_right:20mg/dag)Custom lists of abbreviations, units (e.g.pt.,zn.,mmHg)Custom tokenizing rules (e.g.xdd:arrow_right:xdd)RegardingDEDUCEtags as a single token (e.g.[DATUM-1]).Deidentification is not builtinclinlpand should be done as a preprocessing step.NormalizerThe normalizer sets theToken.normattribute, which can be used by further components (entity matching, qualification). It currently has two options (enabled by default):LowercasingRemoving diacritics, where possible. For instance, it will mapë:arrow_right:e, but keeps most other non-ascii characters intact (e.g.µ,²).Note that this component only has effect when explicitly configuring successor components to match on theToken.normattribute.Sentence splitterThe sentence splitter can be added as follows:nlp.add_pipe("clinlp_sentencizer")It is designed to detect sentence boundaries in clinical text, whenever a character that demarks a sentence ending is matched (e.g. newline, period, question mark). It also correctly detects items in an enumerations (e.g. starting with-or*).Entity matcherclinlpincludes aclinlp_entity_matchercomponent that can be used for matching entities in text, based on a dictionary of known concepts and their terms/synonyms. It includes options for matching on different token attributes, proximity matching, fuzzy matching and matching pseudo/negative terms.The most basic example would be the following, with further options described below:concepts={"sepsis":["sepsis","lijnsepsis","systemische infectie","bacteriemie",],"veneus_infarct":["veneus infarct","VI",]}entity_matcher=nlp.add_pipe("clinlp_entity_matcher")entity_matcher.load_concepts(concepts):bulb: Theclinlp_entity_matchercomponent wraps the spaCyMatcherandPhraseMatchercomponents, adding some convenience and configurability. However, theMatcher,PhraseMatcherorEntityRulercan also be used directly withclinlpfor those who prefer it.AttributeSpecify the token attribute the entity matcher should use as follows (by defaultTEXT):entity_matcher=nlp.add_pipe("clinlp_entity_matcher",config={"attr":"NORM"})AnyToken attributecan be used, but in the above example theclinlp_normalizershould be added before the entity matcher, or theNORMattribute is simply the literal text.clinlpdoes not include Part of Speech tags and dependency trees, at least not until a reliable model for Dutch clinical text is created, though it's always possible to add a relevant component from a trained (general) Dutch model if needed.Proximity matchingThe proxmity setting defines how many tokens can optionally be skipped between the tokens of a pattern. Withproxmityset to1, the patternslaapt slechtwill also matchslaapt vaak slecht, but notslaapt al weken slecht.entity_matcher=nlp.add_pipe("clinlp_entity_matcher",config={"proximity":1})Fuzzy matchingFuzzy matching enables finding misspelled variants of terms. For instance, withfuzzyset to1, the patterndiabeteswill also matchdiabets,ddiabetes, ordiabetis, but notdiabetseorddiabetess. The threshold is based on Levenshtein distance with insertions, deletions and replacements (but not swaps).entity_matcher=nlp.add_pipe("clinlp_entity_matcher",config={"fuzzy":1})Additionally, thefuzzy_min_lenargument can be used to specify the minimum length of a phrase for fuzzy matching. This also works for multi-token phrases. For example, withfuzzyset to1andfuzzy_min_lenset to5, the patternbloeding graad iiwould also matchbloedin graad ii, but notbloeding graad iii.entity_matcher=nlp.add_pipe("clinlp_entity_matcher",config={"fuzzy":1,"fuzzy_min_len":5})TermsThe settings above are described at the matcher level, but can all be overridden at the term level by adding aTermto a concept, rather than a literal phrase:fromclinlpimportTermconcepts={"sepsis":["sepsis","lijnsepsis",Term("early onset",proximity=1),Term("late onset",proximity=1),Term("EOS",attr="TEXT",fuzzy=0),Term("LOS",attr="TEXT",fuzzy=0)]}entity_matcher=nlp.add_pipe("clinlp_entity_matcher",config={"attr":"NORM","fuzzy":1})entity_matcher.load_concepts(concepts)In the above example, by default theNORMattribute is used, andfuzzyis set to1. In addition, for the termsearly onsetandlate onsetproximity matching is set to1, in addition to matcher-level config of matching theNORMattribute and fuzzy matching. For theEOSandLOSabbreviations theTEXTattribute is used (so the matching is case sensitive), and fuzzy matching is disabled.Pseudo/negative phrasesOn the term level, it is possible to add pseudo or negative patterns, for those phrases that need to be excluded. For example:concepts={"prematuriteit":["prematuur",Term("prematuur ademhalingspatroon",pseudo=True),]}In this caseprematuurwill be matched, but not in the context ofprematuur ademhalingspatroon(which may indicate prematurity, but is not a definitive diagnosis).Spacy patternsFinally, if you need more control than literal phrases and terms as explained above, the entity matcher also acceptsspaCy patterns. These patterns do not respect any other configurations (like attribute, fuzzy, proximity, etc.):concepts={"delier":[Term("delier",attr="NORM"),Term("DOS",attr="TEXT"),[{"NORM":{"IN":["zag","ziet","hoort","hoorde","ruikt","rook"]}},{"OP":"?"},{"OP":"?"},{"OP":"?"},{"NORM":{"FUZZY1":"dingen"}},{"OP":"?"},{"NORM":"die"},{"NORM":"er"},{"OP":"?"},{"NORM":"niet"},{"OP":"?"},{"NORM":{"IN":["zijn","waren"]}}],]}Concept dictionary from external sourceWhen matching entities, it is possible to load external lists of concepts (e.g. from a medical thesaurus such as UMLS) fromcsvthrough thecreate_concept_dictfunction. Yourcsvshould contain a combination of concept and phrase on each line, with optional columns to configure theTerm-options described above (e.g.attribute,proximity,fuzzy). You may present the columns in any order, but make sure the names match theTermattributes. Any other columns are ignored. For example:conceptphraseattrproximityfuzzyfuzzy_min_lenpseudocommentprematuriteitprematuriteitsome commentprematuriteit<p3112hypotensiehypotensiehypotensiebd verlaagd1veneus_infarctveneus infarctveneus_infarctVITEXTWill result in the following concept dictionary:{"prematuriteit":["prematuriteit",Term("<p3",proximity=1,fuzzy=1,fuzzy_min_len=2),],"hypotensie":["hypotensie",Term("bd verlaagd",proximity=1)],"veneus_infarct":["veneus infarct",Term("VI",attr="TEXT")]}Qualifier detectionAfter finding entities, it"s often useful to qualify these entities, e.g.: are they negated or affirmed, historical or current?clinlpcurrently implements two options: the rule-based Context Algorithm, and a transformer-based negation detector. In both cases, the result can be found in theentity._.qualifiers,entity._.qualifiers_dictandentity._.qualifiers_strattributes (including all defaults, e.g.AffirmedforNegation).Context AlgorithmThe rule-basedContext Algorithmis fairly accurate, and quite transparent and fast. A set of rules, that checks for negation, temporality, plausibility and experiencer, is loaded by default:nlp.add_pipe("clinlp_context_algorithm",config={"phrase_matcher_attr":"NORM"})A custom set of rules, including different types of qualifiers, can easily be defined. Seeclinlp/resources/context_rules.jsonfor an example, and load it as follows:cm=nlp.add_pipe("clinlp_context_algorithm",config={"rules":"/path/to/my_own_ruleset.json"})Transformer based negation detectionclinlpalso includes a wrapper around the transformer based negation detector, as described invan Es et al, 2022. The underlying transformer can be found onhuggingface. It is reported as more accurate than the rule-based version (see paper for details), at the cost of less transparency and additional computational cost.First, install the additional dependencies:pipinstall"clinlp[transformers]"Then add it using:tn=nlp.add_pipe("clinlp_negation_transformer")Some configuration options, like the number of tokens to consider, can be specified in theconfigargument.Where to go from hereWe hope to extendclinlpwith new functionality and more complete documentation in the near future. In the meantime, if any questions or problems arise, we recommend:Checking the source codeGetting in touch (email|issue)Principles and goalsFunctional:Provides NLP pipelines optimized for Dutch clinical textPerformant and production-readyUseful out-of-the-box, but highly configurableA single place to visit for your Dutch clinical NLP needs(Re-)uses existing components where possible, implements new components where neededNot intended for annotating, training, and analysis — already covered by existing packagesDevelopment:Free and open sourceTargeted towards the technical userCurated and maintained by the Dutch Clinical NLP communityBuilt using thespaCyframework (>3.0.0)Therefore non-destructiveWork towards some level of standardization of components (abstraction, protocols)Follows industry best practices (system design, code, documentation, testing, CI/CD)Overarching goals:Improve the quality of Dutch Clinical NLP pipelinesEnable easier (re)use/valorization of effortsHelp mature the field of Dutch Clinical NLPHelp develop the Dutch Clinical NLP community