Package: concise-concepts
Concise ConceptsWhen wanting to apply NER to concise concepts, it is really easy to come up with examples, but pretty difficult to train an entire pipeline. Concise Concepts uses few-shot NER based on word embedding similarity to get you going
with easy! Now with entity scoring!UsageThis library defines matching patterns based on the most similar words found in each group, which are used to fill aspaCy EntityRuler. To better understand the rule definition, I recommend playing around with thespaCy Rule-based Matcher Explorer.TutorialsTechVizTheDataScienceGuycreated anice tutorialon how to use it.Icreated atutorialin collaboration with Rubrix.The sectionMatching Pattern Rulesexpands on the construction, analysis and customization of these matching patterns.Installpip install concise-conceptsQuickstartTake a look at theconfiguration sectionfor more info.Spacy Pipeline ComponentNote that,custom embedding modelsare passed viamodel_path.importspacyfromspacyimportdisplacyimportconcise_conceptsdata={"fruit":["apple","pear","orange"],"vegetable":["broccoli","spinach","tomato"],"meat":['beef','pork','turkey','duck']}text="""Heat the oil in a large pan and add the Onion, celery and carrots.Then, cook over a medium–low heat for 10 minutes, or until softened.Add the courgette, garlic, red peppers and oregano and cook for 2–3 minutes.Later, add some oranges and chickens. """nlp=spacy.load("en_core_web_md",disable=["ner"])nlp.add_pipe("concise_concepts",config={"data":data,"ent_score":True,# Entity Scoring section"verbose":True,"exclude_pos":["VERB","AUX"],"exclude_dep":["DOBJ","PCOMP"],"include_compound_words":False,"json_path":"./fruitful_patterns.json","topn":(100,500,300)},)doc=nlp(text)options={"colors":{"fruit":"darkorange","vegetable":"limegreen","meat":"salmon"},"ents":["fruit","vegetable","meat"],}ents=doc.entsforentinents:new_label=f"{ent.label_}({ent._.ent_score:.0%})"options["colors"][new_label]=options["colors"].get(ent.label_.lower(),None)options["ents"].append(new_label)ent.label_=new_labeldoc.ents=entsdisplacy.render(doc,style="ent",options=options)StandaloneThis might be useful when iterating over few_shot training data when not wanting to reload larger models continuously.
Note that,custom embedding modelsare passed viamodel.importgensimimportspacyfromconcise_conceptsimportConceptualizermodel=gensim.downloader.load("fasttext-wiki-news-subwords-300")nlp=spacy.load("en_core_web_sm")data={"disease":["cancer","diabetes","heart disease","influenza","pneumonia"],"symptom":["headache","fever","cough","nausea","vomiting","diarrhea"],}conceptualizer=Conceptualizer(nlp,data,model)conceptualizer.nlp("I have a headache and a fever.").entsdata={"disease":["cancer","diabetes"],"symptom":["headache","fever"],}conceptualizer=Conceptualizer(nlp,data,model)conceptualizer.nlp("I have a headache and a fever.").entsConfigurationMatching Pattern RulesA general introduction about the usage of matching patterns in theusage section.Customizing Matching Pattern RulesEven though the baseline parameters provide a decent result, the construction of these matching rules can be customized via the config passed to the spaCy pipeline.exclude_pos: A list of POS tags to be excluded from the rule-based match.exclude_dep: A list of dependencies to be excluded from the rule-based match.include_compound_words:  If True, it will include compound words in the entity. For example, if the entity is "New York", it will also include "New York City" as an entity.case_sensitive: Whether to match the case of the words in the text.Analyze Matching Pattern RulesTo motivate actually looking at the data and support interpretability, the matching patterns that have been generated are stored as./main_patterns.json. This behavior can be changed by using thejson_pathvariable via the config passed to the spaCy pipeline.Fuzzy matching usingspaczzfuzzy: A boolean value that determines whether to use fuzzy matchingdata={"fruit":["apple","pear","orange"],"vegetable":["broccoli","spinach","tomato"],"meat":["beef","pork","fish","lamb"]}nlp.add_pipe("concise_concepts",config={"data":data,"fuzzy":True})Most Similar Word Expansiontopn: Use a specific number of words to expand over.data={"fruit":["apple","pear","orange"],"vegetable":["broccoli","spinach","tomato"],"meat":["beef","pork","fish","lamb"]}topn=[50,50,150]assertlen(topn)==lennlp.add_pipe("concise_concepts",config={"data":data,"topn":topn})Entity Scoringent_score: Use embedding based word similarity to score entities against their groupsimportspacyimportconcise_conceptsdata={"ORG":["Google","Apple","Amazon"],"GPE":["Netherlands","France","China"],}text="""Sony was founded in Japan."""nlp=spacy.load("en_core_web_lg")nlp.add_pipe("concise_concepts",config={"data":data,"ent_score":True,"case_sensitive":True})doc=nlp(text)print([(ent.text,ent.label_,ent._.ent_score)forentindoc.ents])# output## [('Sony', 'ORG', 0.5207586), ('Japan', 'GPE', 0.7371268)]Custom Embedding Modelsmodel_path: Use customsense2vec.Sense2Vec,gensim.Word2vecgensim.FastText, orgensim.KeyedVectors, or a pretrained model fromgensimlibrary or a custom model path. For using asense2vec.Sense2Vectake a lookhere.model: withinstandalone usage, it is possible to pass these models directly.data={"fruit":["apple","pear","orange"],"vegetable":["broccoli","spinach","tomato"],"meat":["beef","pork","fish","lamb"]}# model from https://radimrehurek.com/gensim/downloader.html or path to local filemodel_path="glove-wiki-gigaword-300"nlp.add_pipe("concise_concepts",config={"data":data,"model_path":model_path})