Package: taxonerd
Looking for taxon mentions in text? Ask TaxoNERDFeaturesModelsInstallationUsageExtensionsI would be happy to hear about your use of TaxoNERD : what is your use case? How did TaxoNERD help you? What could make TaxoNERD even more helpful? Please feel free to drop me an email (nicolas[dot]leguillarme[at]univ-grenoble-alpes[dot]fr) or to open an issue.Cite TaxoNERDLe Guillarme, N., & Thuiller, W. (2022).TaxoNERD: deep neural models for the recognition of taxonomic entities in the ecological and evolutionary literature. Methods in Ecology and Evolution, 13(3), 625-641.FeaturesTaxoNERD is a domain-specific tool for recognizing taxon mentions in the biodiversity literature.:tada:New models are out ! Now including additional NLP components (tagger, attribut ruler, lemmatizer, parser) and more accurate common names detection.TaxoNERD is available as a command-line tool, a Python module, a spaCy pipeline,and a R package thanks to reticulate.TaxoNERD provides two architectures : themdarchitecture uses spaCy's standard Tok2Vec layer with word vectors for speed, while thebiobertarchitecture uses a Transformer-based pretrained language model (dmis-lab/biobert-v1.1) for accuracy.TaxoNERD finds scientific names, common names, abbreviated species names and user-defined abbreviations.TaxoNERD can link taxon mentions to entities in a reference taxonomy (NCBI Taxonomy, GBIF Backbone and TAXREF at the moment, more to come).TaxoNERD is fast (once the model is loaded), and can run on CPU or GPU.Entity linking does not need an internet connection, but may require a lot of RAM depending on the size of the taxonomy (e.g. GBIF Backbone -> ~12.5Gb).Thanks totextract,TaxoNERD can extract taxon mentions from (almost) any document(including txt, pdf, csv, xls, jpg, png, and many other formats). With TaxoNERD, the detection of taxonomic entities in a JPG file is as simple as that:taxonerd ask -m en_core_eco_weak_biobert -f ./tests/test_data/test_jpg/test.jpgT0	LIVB 180 192	Harbour sealT1	LIVB 194 208	Phoca vitulinaT2	LIVB 361 375	Pacific salmonT3	LIVB 377 394	Oncorhynchus spp.T4	LIVB 455 467	harbour sealT5	LIVB 714 718	sealT6	LIVB 793 805	harbour sealT7	LIVB 1127 1133	fishesT8	LIVB 1137 1148	cephalopodsModelsModelDescriptionInstall URLen_core_eco_mdA full spaCy pipeline for ecological data with 50k word vectors (taken fromen_core_sci_md) fine-tuned on a gold standard corpus.Downloaden_core_eco_biobertA full spaCy pipeline for ecological data with dmis-lab/biobert-v1.1 as the transformer model, fine-tuned on a gold standard corpus.Downloaden_core_eco_weak_mdA full spaCy pipeline for ecological data with 50k word vectors (taken fromen_core_sci_md) fine-tuned on a silver standard corpus.Downloaden_core_eco_weak_biobertA full spaCy pipeline for ecological data with dmis-lab/biobert-v1.1 as the transformer model, fine-tuned on a silver standard corpus.DownloadWhat model should I choose ?If you have access to a GPU, we recommend using one of the biobert models as they tend to be more accurate than the md models.The en_core_eco_weak_md and en_core_eco_weak_biobert have been fine-tuned on a silver standard corpus generated using weak supervision. Therefore, they have been trained on a much larger amount of (noisy) data than their gold standard counterparts. As a result, they tend to have better recall, especially with respect to common names detection. They also have high precision. Nevertheless, their performance has not been accurately evaluated.If you do not trust weakly-supervised data and you are not really interested in detecting common names, en_core_eco_md and en_core_eco_biobert are for you. These models have been fine-tuned on a gold standard corpus (a combination ofCOPIOUSandBacteria Biotope 2019) and their performance has been benchmarked in our paper.InstallationTaxoNERD for PythonInstalling the package from pip will automatically install all dependencies, including pandas, spaCy, scispaCy and textract. Make sure you install this package before you install the models. Also note that this package requires Python 3.8+ and spaCy v3.4+.$ pip install taxonerdFor GPU support, find your CUDA version usingnvcc --versionand add the version in brackets, e.g.pip install taxonerd[cuda113]for CUDA 11.3. Seesetup.cfgfor supported CUDA versions.To download the models:$ pip install https://github.com/nleguillarme/taxonerd/releases/download/v1.5.0/en_core_eco_md-1.0.2.tar.gz
$ pip install https://github.com/nleguillarme/taxonerd/releases/download/v1.5.0/en_core_eco_biobert-1.0.2.tar.gz
$ pip install https://github.com/nleguillarme/taxonerd/releases/download/v1.5.0/en_core_eco_weak_md-1.0.0.tar.gz
$ pip install https://github.com/nleguillarme/taxonerd/releases/download/v1.5.0/en_core_eco_weak_biobert-1.0.0.tar.gzEntity linker files are downloaded and cached the first time the linker is used. This may take some time, but it should only be done once. Currently (v1.5.1), there are 3 supported linkers:gbif_backbone: Links toGBIF Backbone Taxonomy (2019-09-06)(~9.5M names for ~3.5M taxa).taxref: Links toTAXREF (v13)(~1.2M names for ~267k taxa).ncbi_taxonomy: Links toThe NCBI Taxonomy(~3.4M names).TaxoNERD for R> install.packages("https://github.com/nleguillarme/taxonerd/releases/download/v1.5.1/taxonerd_for_R_1.5.1.tar.gz", repos=NULL)
> vignette("taxonerd") # See vignette for more information on how to install and use TaxoNERD for RUsageTaxoNERD can be used as:a command-line toola Python modulea spaCy pipelineUse as command-line tool$taxonerdask--helpUsage: taxonerd ask [OPTIONS] [INPUT_TEXT]Options:-m, --model TEXT       A TaxoNERD model [default = en_ner_eco_md]-i, --input-dir TEXT   Input directory-o, --output-dir TEXT  Output directory-f, --filename TEXT    Input text file-a, --with-abbrev      Add abbreviation detector to the pipeline-s, --with-sentence    Add sentence segmenter to the pipeline-l, --link-to TEXT     Add entity linker to the pipeline-t, --thresh FLOAT     Similarity threshold for entity linking [default =0.7]--prefer-gpu           Use GPU if available-v, --verbose          Verbose mode--help                 Show this message and exit.ExamplesTaxonomic NER from the terminal$taxonerdask-men_core_eco_biobert"Brown bears (Ursus arctos), which are widely distributed throughout the northern hemisphere, are recognised as opportunistic omnivores"T0	LIVB 0 11	Brown bearsT1	LIVB 13 25	Ursus arctosTaxonomic NER with entity linking from the terminal$taxonerdask-men_core_eco_biobert-lgbif_backbone"Brown bears (Ursus arctos), which are widely distributed throughout the northern hemisphere, are recognised as opportunistic omnivores"T0	LIVB 0 11	Brown bears	[('GBIF:2433433', 'Brown Bear', 0.8313919901847839)]T1	LIVB 13 25	Ursus arctos	[('GBIF:2433433', 'Ursus arctos', 1.0)]$taxonerdask-men_core_eco_biobert-lgbif_backbone-t0.85"Brown bears (Ursus arctos), which are widely distributed throughout the northern hemisphere, are recognised as opportunistic omnivores"T0	LIVB 13 25	Ursus arctos	[('GBIF:2433433', 'Ursus arctos', 1.0)]Taxonomic NER from a text file (with abbreviation detection)$taxonerdask-men_core_eco_biobert--with-abbrev-f./tests/test_data/test_txt/test1.txtT0	LIVB 4 21	pinewood nematodeT1	LIVB 23 26	PWNT2	LIVB 29 55	Bursaphelenchus xylophilusT3	LIVB 57 70	B. xylophilusT4	LIVB 99 108	pine wiltT5	LIVB 196 204	SerratiaT6	LIVB 257 260	PWNT7	LIVB 342 364	Serratia grimesii BXF1T8	LIVB 387 390	PWNT9	LIVB 440 444	BXF1Taxonomic NER from a directory containing text files, with results written in the output directory$taxonerdask--focus-onaccuracy-i./tests/test_data/test_txt-otest_ann$lstest_ann/test1.ann  test2.ann$cattest_ann/test2.annT0	LIVB 700 711	Brown bearsT1	LIVB 713 725	Ursus arctosT2	LIVB 1062 1073	brown bearsT3	LIVB 1161 1172	brown bearsT4	LIVB 1339 1350	brown bearsT5	LIVB 1555 1565	brown bearT6	LIVB 1782 1793	brown bearsT7	LIVB 1863 1874	brown bearsT8	LIVB 1958 1969	brown bearsT9	LIVB 1974 1980	salmonT10	LIVB 2026 2037	brown bearsT11	LIVB 2219 2230	brown bearsT12	LIVB 2392 2401	Sika deerT13	LIVB 2403 2416	Cervus nipponT14	LIVB 2555 2559	deerT15	LIVB 2594 2604	brown bearT16	LIVB 2798 2808	brown bearT17	LIVB 3146 3150	deerT18	LIVB 3188 3199	chum salmonT19	LIVB 3201 3218	Oncorhynchus ketaT20	LIVB 3280 3289	Sika deerT21	LIVB 3350 3361	pink salmonT22	LIVB 3363 3375	O. gorbuschaT23	LIVB 3381 3392	chum salmonT24	LIVB 3518 3528	Brown bearT25	LIVB 4001 4012	brown bearsT26	LIVB 4071 4082	brown bearsUse as python module>>>fromtaxonerdimportTaxoNERD>>>taxonerd=TaxoNERD(prefer_gpu=False)>>>nlp=taxonerd.load(model="en_core_eco_md",exclude=[],linker="taxref",threshold=0.7)>>>nlp.pipe_names['tok2vec','tagger','attribute_ruler','lemmatizer','pysbd_sentencizer','parser','ner','taxo_abbrev_detector','taxref_linker']N.B.By default, all components are included in the pipeline. Use theexcludeargument to specify the components to exclude. Excluded components won’t be loaded. This may speed up the detection process. The minimal pipeline for taxonomic NER is['tok2vec', 'ner'].ExamplesFind taxonomic entities in an input string>>>taxonerd.find_in_text("Brown bears (Ursus arctos), which are widely distributed throughout the northern hemisphere, are recognised as opportunistic omnivore")offsetstextentitysentT0LIVB1325Ursusarctos[(TAXREF:60826,Ursusarctos,1.0)]0Find taxonomic entities in an input file>>>taxonerd.find_in_file("./tests/test_data/test_txt/test2.txt",output_dir=None)offsetstextentitysentT0LIVB713725Ursusarctos[(TAXREF:60826,Ursusarctos,1.0)]4T1LIVB19741980salmon[(TAXREF:730671,Salmonia,0.85158771276474)]12T2LIVB23922401Sikadeer[(TAXREF:61025,SikaDeer,1.0)]14T3LIVB24032416Cervusnippon[(TAXREF:61025,Cervusnippon,1.0)]14T4LIVB31353141salmon[(TAXREF:730671,Salmonia,0.85158771276474)]18T5LIVB31463150deer[(TAXREF:186210,deer,1.0)]18T6LIVB31883199chumsalmon[(TAXREF:730671,Salmonia,0.7018352746963501)]19T7LIVB32013218Oncorhynchusketa[(TAXREF:195439,Oncorhynchus,0.8319146037101...19T8LIVB32803289Sikadeer[(TAXREF:61025,SikaDeer,1.0)]19T9LIVB33503361pinksalmon[(TAXREF:67798,PinkSalmon,1.0)]20T10LIVB33813392chumsalmon[(TAXREF:730671,Salmonia,0.7018352746963501)]20T11LIVB34813485deer[(TAXREF:186210,deer,1.0)]20Find taxonomic entities in all the files in the input directory, and write the results in the output directory>>>taxonerd.find_in_corpus("./tests/test_data/test_txt","./test_ann"){'test1.txt':'./test_ann/test1.ann','test2.txt':'./test_ann/test2.ann'}Use as spaCy pipeline>>>fromtaxonerdimportTaxoNERD>>>taxonerd=TaxoNERD(prefer_gpu=True)>>>nlp=taxonerd.load(model="en_core_eco_biobert")>>>doc=nlp("Brown bears (Ursus arctos), which are widely distributed throughout the northern hemisphere, are recognised as opportunistic omnivore")>>>doc.ents(Brownbears,Ursusarctos)>>>[tok.lemma_fortokindoc]['Brown','bear','(','ursus','arcto',')',',','which','be','widely','distribute','throughout','the','northern','hemisphere',',','be','recognise','as','opportunistic','omnivore']More examples in ourdemo Notebook.ExtensionsCombining TaxoNERD with gazetteer-based NER for improved taxonomic entities recognitionLicenseLicense: MITAuthorsTaxoNERD was written bynleguillarme.