Package: nerwhal
NERwhalA multi-lingual suite for named-entity recognition in Python.⚠️ Disclaimer ⚠️:This is a prototype. Do not use for anything critical.DescriptionNERwhal's mission is to make defining custom recognizers for different NER approaches as easy as possible.
To achieve this, different NER backends are implemented behind a unified API.
Each recognizer is based on one of the backends.
Users can detect named entities by implementing custom recognizers for one or more of the backends.Check out our blog post about NERwhal onMedium.Powerful NER backendsNERwhal makes use of some of the most powerful NER platforms out there:Regular expressions: Usingregular expressionsyou can define a named entity as a set of strings.Entity Ruler:spaCy’sEntity Rulerlets you define patterns for sequences of tokens. (spaCyis also used for tokenization)FlashText: TheFlashText Algorithmcan search texts very efficiently for long lists of keywords.Deep Learning: TheStanzalibrary and models (which providestate-of-the-art results for NERin many languages) power NERwhal's statistical recognition. Currently, Stanza supports NER for 8 languages.Smart combination of the resultsThe suite can combine the results of these methods in a smart way to get best results.
E.g. a match with a higher score can overwrite a lower scored one, or, if one entity was identified several times, its confidence score can be increased.Context wordsEach recognizer can define a list of context words that may occur in the context of named entities.
If a context word is found in the same sentence as the entity, the confidence score is increased.Integrated recognizersNERwhal follows the philosophy that recognizers are specific to the language, use case, and requirements.
The recommended way to use is to define your own custom recognizers.
Yet to exemplify its usage and to help you bootstrap your own recognition suite, some example recognizers are implemented innerwhal/integrated_recognizers.
Please refer to each recognizers' PyDoc for more information, and keep in mind that none of these recognizers will catch all occurrences of their category, and that they may produce false positives results.InstallationNERwhal can be installed from PyPi and has to be installed in a virtual environment (venv or conda for instance).pipinstallnerwhalUsageTo recognize named entities, pass a text and config object to therecognizemethod.
Select the recognizers to be used in the config object.>>>fromnerwhalimportrecognize,Config>>>>>>config=Config(language="de",use_statistical_ner=True,recognizer_paths=["nerwhal/integrated_recognizers/email_recognizer.py"])>>>>>>recognize("Ich heiße Luke und meine E-Mail ist luke@skywalker.com.",config=config,return_tokens=True){'tokens':[Token(text='Ich',has_ws=True,br_count=0,start_char=0,end_char=3),Token(text='heiße',has_ws=True,br_count=0,start_char=4,end_char=9),...Token(text='.',has_ws=False,br_count=0,start_char=54,end_char=55)],'ents':[NamedEntity(start_char=10,end_char=14,tag='PER',text='Luke',score=0.8,recognizer='StanzaNerBackend',start_tok=2,end_tok=3),NamedEntity(start_char=36,end_char=54,tag='EMAIL',text='luke@skywalker.com',score=0.95,recognizer='EmailRecognizer',start_tok=7,end_tok=8)]}Implementing custom recognizersTo implement a custom recognizer, you have to implement one of the interfaces inrecognizer_bases.
For examples see one of theintegrated_recognizers.DevelopmentInstall requirementsYou can install all (production and development) requirements using:pip install -r requirements.txtInstall the pre-commit hooksThis repository uses git hooks to validate code quality and formatting.pre-commit install
git config --bool flake8.strict true  # Makes the commit fail if flake8 reports an errorTo run the hooks:pre-commit run --all-filesTestingRun all tests with:pytest --cov-report term --cov=nerwhalTo skip tests that require the download of Stanza models run:pytest -m "not stanza"How to contact usFor usage questions, bugs, or suggestions please file a Github issue.
If you would like to contribute or have other questions please emailhello@openredact.org.LicenseMIT License