Package: nlpannotator
Automated annotation of natural languages using selected toolchainsThis project just had its first version release and is still under development.DescriptionThenlpannotatorpackage serves as modular toolchain to combine different natural language processing (nlp) tools to annotate texts (sentencizing, tokenization, part-of-speech (POS) and lemma).Tools that can be combined are:spaCy(sentencize, tokenize, POS, lemma)stanza(sentencize, tokenize, POS, lemma)SoMaJo(sentencize, tokenize)Flair(POS)Treetagger(tokenize, POS, lemma)
These tools can be combined in any desired fashion, to target either maximum efficiency or accuracy.InstallationInstall the project and its dependencies fromPyPi:pip install nlpannotatorThe language models need to be installed separately. You can make use of the convenience scriptherewhich installs all language models for all languages that have been implemented for spaCy and stanza.The package also makes use ofTreetagger. You can use thetreetagger_installcommand in your shell to install the dependency, or call it within Pythonimport nlpannotator.install_treetagger
nlpannotator.install_treetagger.install_tagger()OptionsAll input options are provided in an input dictionary. Two pre-set toolchains can be used:fastusingspaCyfor all annotations;accurateusingSoMaJofor sentencizing and tokenization, andstanzafor POS and lemma; andmanualwhere any combination of spaCy, stanza, SoMaJo,Flair,Treetaggercan be used, given the tool supports the selected annotation and language.KeywordDefault settingPossible optionsDescriptioninputexample_en.txtName of the text file containing the raw text for annotationcorpus_nametestName of the corpus that is generatedlanguageensee belowLanguage of the text to annotateprocessing_optionmanualfast, accurate, manualSelect the tool pipeline -fastandaccurateprovide you with good default options for Englishprocessing_typesentencize, tokenize, pos, lemmasee belowtoolspacy, spacy, spacy, spacysee belowTool to use for each of the four annotation typesoutput_formatxmlxml, vrtFormat of the generated annotated text fileencodingyesyes, noDirectly encode the annotated text file intocwbToolsThe available annotation tools are listed below, and can be set using the following keywords:spaCy:spacystanza:stanzaSoMaJo:somajoFlair:flairTreetagger:treetaggerProcessorsThe available processors depend on the selected tool. This is a summary of the possible options:ToolAvailable processorsspacysentencize, tokenize, pos, lemmastanzasentencize, tokenize, pos, lemmasomajosentencize, tokenizeflairpostreetaggertokenize, pos, lemmaSome of the processors depend on each other. For example,posandlemmaare only possible aftersentencizeandtokenize.tokenizedepends onsentencize.LanguagesThe availabe languages depend on the selected tool. So far, the following languages have been added to the pipeline (there may be additional language models available for the respective tool, but they have not been added to this package - for stanza, the pipeline will still run and load the model on demand).ToolAvailable languagesspacyen, de, fr, it, ja, pt, ru, esstanzaload on demand fromavailable stanza modelssomajoen, deflairen, detreetaggeren, de, fr, es(both tokenization and pos/lemma)treetaggerbg, nl, et, fi, gl, it, kr, la, mn, pl, ru, sk, sw(only pos/lemma)Input/Outputnlpannotatorexpects a raw text file as an input, together with an input dictionary that specifies the selected options. The input dictionary is also printed out when a run is initiated, so that the selected options are stored and can be looked up at a later time.
Both of these can be provided through aJupyterinterface as in theDemo Notebook.The output that is generated is either ofvrtformat (for cwb) orxml. Both output formats can directly be encoded into cwb.Demo notebookTake a look at theDemoNotebookor run it onBinder.Questions and bug reportsPlease ask questions / submit bug reports using ourissue tracker.ContributeContributions are wellcome. Please fork the nlpannotator repo and open a Pull Request for any changes to the code. These will be reviewed and merged by our team.
Make sure that your contributions areclean,properly formattedand for any new modulesfollow the general design principle.Take a look at thesource code documentation.The additions must have at least have 80% test coverage.ReleasesA summary of the releases and release notes are availablehere.