Package: scispacy
This repository contains custom pipes and models related to using spaCy for scientific documents.In particular, there is a custom tokenizer that adds tokenization rules on top of spaCy's
rule-based tokenizer, a POS tagger and syntactic parser trained on biomedical data and
an entity span detection model. Separately, there are also NER models for more specific tasks.Just looking to test out the models on your data? Check out ourdemo(Note: this demo is running an older version of scispaCy and may produce different results than the latest version).InstallationInstalling scispacy requires two steps: installing the library and intalling the models. To install the library, run:pipinstallscispacyto install a model (see our full selection of available models below), run a command like the following:pipinstallhttps://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.3/en_core_sci_sm-0.5.3.tar.gzNote: We strongly recommend that you use an isolated Python environment (such as virtualenv or conda) to install scispacy.
Take a look below in the "Setting up a virtual environment" section if you need some help with this.
Additionally, scispacy uses modern features of Python and as such is only available forPython 3.6 or greater.Setting up a virtual environmentCondacan be used set up a virtual environment with the
version of Python required for scispaCy.  If you already have a Python
environment you want to use, you can skip to the 'installing via pip' section.Follow the installation instructions for Conda.Create a Conda environment called "scispacy" with Python 3.9 (any version >= 3.6 should work):condacreate-nscispacypython=3.9Activate the Conda environment. You will need to activate the Conda environment in each terminal in which you want to use scispaCy.sourceactivatescispacyNow you can installscispacyand one of the models using the steps above.Once you have completed the above steps and downloaded one of the models below, you can load a scispaCy model as you would any other spaCy model. For example:importspacynlp=spacy.load("en_core_sci_sm")doc=nlp("Alterations in the hypocretin receptor 2 and preprohypocretin genes produce narcolepsy in some animals.")Note on upgradingIf you are upgradingscispacy, you will need to download the models again, to get the model versions compatible with the version ofscispacythat you have. The link to the model that you download should contain the version number ofscispacythat you have.Available ModelsTo install a model, click on the link below to download the model, and then runpipinstall</path/to/download>Alternatively, you can install directly from the URL by right-clicking on the link, selecting "Copy Link Address" and runningpipinstallCMD-V(topastethecopiedURL)ModelDescriptionInstall URLen_core_sci_smA full spaCy pipeline for biomedical data with a ~100k vocabulary.Downloaden_core_sci_mdA full spaCy pipeline for biomedical data with a ~360k vocabulary and 50k word vectors.Downloaden_core_sci_lgA full spaCy pipeline for biomedical data with a ~785k vocabulary and 600k word vectors.Downloaden_core_sci_scibertA full spaCy pipeline for biomedical data with a ~785k vocabulary andallenai/scibert-baseas the transformer model. You may want touse a GPUwith this model.Downloaden_ner_craft_mdA spaCy NER model trained on the CRAFT corpus.Downloaden_ner_jnlpba_mdA spaCy NER model trained on the JNLPBA corpus.Downloaden_ner_bc5cdr_mdA spaCy NER model trained on the BC5CDR corpus.Downloaden_ner_bionlp13cg_mdA spaCy NER model trained on the BIONLP13CG corpus.DownloadAdditional Pipeline ComponentsAbbreviationDetectorThe AbbreviationDetector is a Spacy component which implements the abbreviation detection algorithm in "A simple algorithm
for identifying abbreviation definitions in biomedical text.", (Schwartz & Hearst, 2003).You can access the list of abbreviations via thedoc._.abbreviationsattribute and for a given abbreviation,
you can access it's long form (which is aspacy.tokens.Span) usingspan._.long_form, which will point to
another span in the document.Example Usageimportspacyfromscispacy.abbreviationimportAbbreviationDetectornlp=spacy.load("en_core_sci_sm")# Add the abbreviation pipe to the spacy pipeline.nlp.add_pipe("abbreviation_detector")doc=nlp("Spinal and bulbar muscular atrophy (SBMA) is an\inherited motor neuron disease caused by the expansion\of a polyglutamine tract within the androgen receptor (AR).\SBMA can be caused by this easily.")print("Abbreviation","\t","Definition")forabrvindoc._.abbreviations:print(f"{abrv}\t({abrv.start},{abrv.end}){abrv._.long_form}")>>>AbbreviationSpanDefinition>>>SBMA(33,34)Spinalandbulbarmuscularatrophy>>>SBMA(6,7)Spinalandbulbarmuscularatrophy>>>AR(29,30)androgenreceptorNoteIf you want to be able toserialize yourdocobjects, load the abbreviation detector withmake_serializable=True, e.g.nlp.add_pipe("abbreviation_detector", config={"make_serializable": True})EntityLinkerTheEntityLinkeris a SpaCy component which performs linking to a knowledge base. The linker simply performs
a string overlap - based search (char-3grams) on named entities, comparing them with the concepts in a knowledge base
using an approximate nearest neighbours search.Currently (v2.5.0), there are 5 supported linkers:umls: Links to theUnified Medical Language System, levels 0,1,2 and 9. This has ~3M concepts.mesh: Links to theMedical Subject Headings. This contains a smaller set of higher quality entities, which are used for indexing in Pubmed. MeSH contains ~30k entities. NOTE: The MeSH KB is derived directly from MeSH itself, and as such uses different unique identifiers than the other KBs.rxnorm: Links to theRxNormontology. RxNorm contains ~100k concepts focused on normalized names for clinical drugs. It is comprised of several other drug vocabularies commonly used in pharmacy management and drug interaction, including First Databank, Micromedex, and the Gold Standard Drug Database.go: Links to theGene Ontology. The Gene Ontology contains ~67k concepts focused on the functions of genes.hpo: Links to theHuman Phenotype Ontology. The Human Phenotype Ontology contains 16k concepts focused on phenotypic abnormalities encountered in human disease.You may want to play around with some of the parameters
below to adapt to your use case (higher precision, higher recall etc).resolve_abbreviations : bool = True, optional (default = False)Whether to resolve abbreviations identified in the Doc before performing linking.
This parameter has no effect if there is noAbbreviationDetectorin the spacy
pipeline.k : int, optional, (default = 30)The number of nearest neighbours to look up from the candidate generator per mention.threshold : float, optional, (default = 0.7)The threshold that a mention candidate must reach to be added to the mention in the Doc
as a mention candidate.no_definition_threshold : float, optional, (default = 0.95)The threshold that a entity candidate must reach to be added to the mention in the Doc
as a mention candidate if the entity candidate does not have a definition.filter_for_definitions: bool, default = TrueWhether to filter entities that can be returned to only include those with definitions
in the knowledge base.max_entities_per_mention : int, optional, default = 5The maximum number of entities which will be returned for a given mention, regardless of
how many are nearest neighbours are found.This class sets the._.kb_entsattribute on spacy Spans, which consists of a
List[Tuple[str, float]] corresponding to the KB concept_id and the associated score
for a list ofmax_entities_per_mentionnumber of entities.You can look up more information for a given id using the kb attribute of this class:print(linker.kb.cui_to_entity[concept_id])Example Usageimportspacyimportscispacyfromscispacy.linkingimportEntityLinkernlp=spacy.load("en_core_sci_sm")# This line takes a while, because we have to download ~1GB of data# and load a large JSON file (the knowledge base). Be patient!# Thankfully it should be faster after the first time you use it, because# the downloads are cached.# NOTE: The resolve_abbreviations parameter is optional, and requires that# the AbbreviationDetector pipe has already been added to the pipeline. Adding# the AbbreviationDetector pipe and setting resolve_abbreviations to True means# that linking will only be performed on the long form of abbreviations.nlp.add_pipe("scispacy_linker",config={"resolve_abbreviations":True,"linker_name":"umls"})doc=nlp("Spinal and bulbar muscular atrophy (SBMA) is an\inherited motor neuron disease caused by the expansion\of a polyglutamine tract within the androgen receptor (AR).\SBMA can be caused by this easily.")# Let's look at a random entity!entity=doc.ents[1]print("Name: ",entity)>>>Name:bulbarmuscularatrophy# Each entity is linked to UMLS with a score# (currently just char-3gram matching).linker=nlp.get_pipe("scispacy_linker")forumls_entinentity._.kb_ents:print(linker.kb.cui_to_entity[umls_ent[0]])>>>CUI:C1839259,Name:Bulbo-SpinalAtrophy,X-Linked>>>Definition:AnX-linkedrecessiveformofspinalmuscularatrophy.ItisduetoamutationofthegeneencodingtheANDROGENRECEPTOR.>>>TUI(s):T047>>>Aliases(abbreviated,total:50):Bulbo-SpinalAtrophy,X-Linked,Bulbo-SpinalAtrophy,X-Linked,....>>>CUI:C0541794,Name:Skeletalmuscleatrophy>>>Definition:Aprocess,occurringinskeletalmuscle,thatischaracterizedbyadecreaseinproteincontent,fiberdiameter,forceproductionandfatigueresistanceinresponseto...>>>TUI(s):T046>>>Aliases:(total:9):Skeletalmuscleatrophy,ATROPHYSKELETALMUSCLE,skeletalmuscleatrophy,....>>>CUI:C1447749,Name:ARprotein,human>>>Definition:Androgenreceptor(919aa,~99kDa)isencodedbythehumanARgene.Thisproteinplaysaroleinthemodulationofsteroid-dependentgenetranscription.>>>TUI(s):T116,T192>>>Aliases(abbreviated,total:16):ARprotein,human,AndrogenReceptor,DihydrotestosteroneReceptor,AR,DHTR,NR3C4,...Hearst Patterns (v0.3.0 and up)This component implementsAutomatic Aquisition of Hyponyms from Large Text Corporausing the SpaCy Matcher component.Passingextended=Trueto theHyponymDetectorwill use the extended set of hearst patterns, which include higher recall but lower precision hyponymy relations (e.g X compared to Y, X similar to Y, etc).This component produces a doc level attribute on the spacy doc:doc._.hearst_patterns, which is a list containing tuples of extracted hyponym pairs. The tuples contain:The relation rule used to extract the hyponym (type:str)The more general concept  (type:spacy.Span)The more specific concept (type:spacy.Span)Usage:importspacyfromscispacy.hyponym_detectorimportHyponymDetectornlp=spacy.load("en_core_sci_sm")nlp.add_pipe("hyponym_detector",last=True,config={"extended":False})doc=nlp("Keystone plant species such as fig trees are good for the soil.")print(doc._.hearst_patterns)>>>[('such_as',Keystoneplantspecies,figtrees)]CitingIf you use ScispaCy in your research, please citeScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. Additionally, please indicate which version and model of ScispaCy you used so that your research can be reproduced.@inproceedings{neumann-etal-2019-scispacy,
    title = "{S}cispa{C}y: {F}ast and {R}obust {M}odels for {B}iomedical {N}atural {L}anguage {P}rocessing",
    author = "Neumann, Mark  and
      King, Daniel  and
      Beltagy, Iz  and
      Ammar, Waleed",
    booktitle = "Proceedings of the 18th BioNLP Workshop and Shared Task",
    month = aug,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/W19-5034",
    doi = "10.18653/v1/W19-5034",
    pages = "319--327",
    eprint = {arXiv:1902.07669},
    abstract = "Despite recent advances in natural language processing, many statistical models for processing text perform extremely poorly under domain shift. Processing biomedical and clinical text is a critically important application area of natural language processing, for which there are few robust, practical, publicly available models. This paper describes scispaCy, a new Python library and models for practical biomedical/scientific text processing, which heavily leverages the spaCy library. We detail the performance of two packages of models released in scispaCy and demonstrate their robustness on several tasks and datasets. Models and code are available at https://allenai.github.io/scispacy/.",
}ScispaCy is an open-source project developed bythe Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.