Package: semantic-compare
python-semantic-сompareExtracts, compares, transforms and sorts with buckets phrases.InstallationInstall a library$pipinstallsemantic-compareThe library requires a spacy model for natural language processing. If you want to use English, please run this command$python-mspacydownloaden_core_web_lgUsageExtract phrasesSimple Usagefromsemantic_compareimportSemanticComparatorassccomparator=sc(sentencizer=True)phrases=comparator.extract_phrases("Create, promote and develop a business.")Output:['Create a business','promote a business','develop a business']sentencizer- a splitter of sentences by punctuation(dot, question mark, exclamation mark).Advanced Usagefromsemantic_compareimportSemanticComparatorassc# Sentence splitterdefour_sentencizer(doc):"""Sentence splitter function that allows splitting document on sentencesby different punctuations and new line"""fori,tokeninenumerate(doc[:-2]):iftoken.text=="•"or"•"intoken.text:doc[i].is_sent_start=Trueelif(token.text=="."ortoken.text=='...'\ortoken.text=='?'ortoken.text=='!'ortoken.text=='\n')\anddoc[i+1].is_title:doc[i+1].is_sent_start=Trueelse:doc[i+1].is_sent_start=Falsereturndoc# load small english spacy model(can be any spacy model)comparator=sc(spacy_model='en_core_web_sm')# Add a custom pipe for text preprocessingcomparator.add_custom_pipe(our_sentencizer,before='parser')phrases=comparator.extract_phrases('''Must Have:* Experience shaping the BI strategy from C-Level to Technical developers.* Extensive delivery of platform within a Business Intelligence and Analytics function.* Communication with stakeholders on all levels.''')print('\n'.join(phrases))Usingadd_custom_pipeyou can add your custom pipe for text processing in spacy.Compare phrases (Semantic similarity)Get the similarity of phrases against each other.Example 1:phrase1='Understand customer needs'phrase2='Capture business requirements'similarity=comparator.compare_phrases(phrase1,phrase2)print(similarity)Output:0.38569751Example 2:Get a two-dimensional matrix that clusters the similarity of phrases against each other.phrases_1=['Communication with stakeholders','Understand customer needs','Experience shaping the BI strategy','shaping the BI strategy','Delivery of platform Analytics function',]phrases_2=['Extensive delivery of platform within a Business Intelligence and Analytics function','shaping the BI strategy','Experience shaping the BI strategy from C-Level to Technical developers','Communication with stakeholders on all levels','Capture business requirements','Play computer games',]similarity=comparator.build_similarity_matrix(phrases_1,phrases_2)print(similarity)Output:[[-0.03689054  0.0372301   0.17840812  0.09079809  0.65748763]
[ 0.18079719  0.12055688  0.77624094  1.          0.22749564]
[ 0.08472343  0.11505745  0.7030021   0.48876476  0.13252231]
[ 0.7132235   0.07449755  0.178031    0.15712512  0.0676512 ]
[ 0.11637229  0.38569745  0.23005028  0.25646406  0.26493344]
[ 0.17955953  0.15243992  0.11233422  0.16087453  0.03144675]]Bucket sortingWhen you compare two documents you can see which phrases present in both or only in a specific document.phrases_1=['Communication with stakeholders','Understand customer needs','Experience shaping the BI strategy','shaping the BI strategy','Delivery of platform Analytics function',]phrases_2=['Extensive delivery of platform within a Business Intelligence and Analytics function','shaping the BI strategy','Experience shaping the BI strategy from C-Level to Technical developers','Communication with stakeholders on all levels','Capture business requirements','Play computer games',]# cut_off - a percentage of similarity should be bigger than it so that we consider that phrases are similar(default=0.3)in_both,in_doc1,in_doc2=comparator.bucket_sorting(phrases_1,phrases_2,similarity,cut_off=0.5)Transfrom phrasesGet all steps of transformation from one phrase to another.Example:print(comparator.transform_phrase('Understand customer needs','Capture business requirements',))Output["Understand customer needs","Capture customer needs","Capture business requirements"]