Package: polyfuzz
PolyFuzzperforms fuzzy string matching, string grouping, and contains extensive evaluation functions.
PolyFuzz is meant to bring fuzzy string matching techniques together within a single framework.Currently, methods include a variety of edit distance measures, a character-based n-gram TF-IDF, word embedding
techniques such as FastText and GloVe, and ðŸ¤— transformers embeddings.Corresponding medium post can be foundhere.InstallationYou can installPolyFuzzvia pip:pipinstallpolyfuzzYou may want to install more depending on the transformers and language backends that you will be using. The possible installations are:pipinstallpolyfuzz[sbert]pipinstallpolyfuzz[flair]pipinstallpolyfuzz[gensim]pipinstallpolyfuzz[spacy]pipinstallpolyfuzz[use]If you want to speed up the cosine similarity comparison and decrease memory usage when using embedding models,
you can usesparse_dot_topnwhich is installed via:pipinstallpolyfuzz[fast]Installation IssuesYou might run into installation issues withsparse_dot_topn. If so, one solution that has worked for many
is by installing it via conda first before installing PolyFuzz:condainstall-cconda-forgesparse_dot_topnIf that does not work, I would advise you to look through their
issues](https://github.com/ing-bank/sparse_dot_topn/issues) page or continue to use PolyFuzz withoutsparse_dot_topn.Getting StartedFor an in-depth overview of the possibilities ofPolyFuzzyou can check the full documentationhereor you can follow along
with the notebookhere.Quick StartThe main goal ofPolyFuzzis to allow the user to perform different methods for matching strings.
We start by defining two lists, one to map from and one to map to. We are going to be usingTF-IDFto create
n-grams on a character level in order to compare similarity between strings. Then, we calculate the similarity
between strings by calculating the cosine similarity between vector representations.We only have to instantiatePolyFuzzwithTF-IDFand match the lists:frompolyfuzzimportPolyFuzzfrom_list=["apple","apples","appl","recal","house","similarity"]to_list=["apple","apples","mouse"]model=PolyFuzz("TF-IDF")model.match(from_list,to_list)The resulting matches can be accessed throughmodel.get_matches():>>>model.get_matches()FromToSimilarity0appleapple1.0000001applesapples1.0000002applapple0.7837513recalNone0.0000004housemouse0.5879275similarityNone0.000000NOTE 1: If you want to compare distances within a single list, you can simply pass that list as such:model.match(from_list)NOTE 2: When instantiatingPolyFuzzwe also could have used "EditDistance" or "Embeddings" to quickly
access Levenshtein and FastText (English) respectively.ProductionThe.matchfunction allows you to quickly extract similar strings. However, after selecting the right models to be used, you may want to use PolyFuzz
in production to match incoming strings. To do so, we can make use of the familiarfit,transform, andfit_transformfunctions.Let's say that we have a list of words that we know to be correct calledtrain_words. We want to any incoming word to mapped to one of the words intrain_words.
In other words, wefitontrain_wordsand we usetransformon any incoming words:fromsklearn.datasetsimportfetch_20newsgroupsfromsklearn.feature_extraction.textimportCountVectorizerfrompolyfuzzimportPolyFuzztrain_words=["apple","apples","appl","recal","house","similarity"]unseen_words=["apple","apples","mouse"]# Fitmodel=PolyFuzz("TF-IDF")model.fit(train_words)# Transformresults=model.transform(unseen_words)In the above example, we are usingfitontrain_wordsto calculate the TF-IDF representation of those words which are saved to be used again intransform.
This speeds uptransformquite a bit since all TF-IDF representations are stored when applyingfit.Then, we apply save and load the model as follows to be used in production:# Save the modelmodel.save("my_model")# Load the modelloaded_model=PolyFuzz.load("my_model")Group MatchesWe can group the matchesToas there might be significant overlap in strings in our to_list.
To do this, we calculate the similarity within strings in to_list and usesingle linkageto then
group the strings with a high similarity.When we extract the new matches, we can see an additional columnGroupin which all theTomatches were grouped to:>>>model.group(link_min_similarity=0.75)>>>model.get_matches()FromToSimilarityGroup0appleapple1.000000apples1applesapples1.000000apples2applapple0.783751apples3recalNone0.000000None4housemouse0.587927mouse5similarityNone0.000000NoneAs can be seen above, we grouped apple and apples together toapplesuch that when a string is mapped toappleit
will fall in the cluster of[apples, apple]and will be mapped to the first instance in the cluster which isapples.Precision-Recall CurveNext, we would like to see how well our model is doing on our data. We express our results asprecisionandrecallwhere precision is defined as the minimum similarity score before a match is correct and
recall the percentage of matches found at a certain minimum similarity score.Creating the visualizations is as simple as:model.visualize_precision_recall()ModelsCurrently, the following models are implemented in PolyFuzz:TF-IDFEditDistance (you can use any distance measure, seedocumentation)FastText and GloVeðŸ¤— TransformersWithFlair, we can use all ðŸ¤— Transformersmodels.
We simply have to instantiate any Flair WordEmbedding method and pass it through PolyFuzzy.All models listed above can be found inpolyfuzz.modelsand can be used to create and compare different models:frompolyfuzz.modelsimportEditDistance,TFIDF,Embeddingsfromflair.embeddingsimportTransformerWordEmbeddingsembeddings=TransformerWordEmbeddings('bert-base-multilingual-cased')bert=Embeddings(embeddings,min_similarity=0,model_id="BERT")tfidf=TFIDF(min_similarity=0)edit=EditDistance()string_models=[bert,tfidf,edit]model=PolyFuzz(string_models)model.match(from_list,to_list)To access the results, we again can callget_matchesbut since we have multiple models we get back a dictionary
of dataframes back.In order to access the results of a specific model, callget_matcheswith the correct id:>>>model.get_matches("BERT")FromToSimilarity0appleapple1.0000001applesapples1.0000002applapple0.9280453recalapples0.8252684housemouse0.8875245similaritymouse0.791548Finally, visualize the results to compare the models:model.visualize_precision_recall(kde=True)Custom GrouperWe can even use one of thepolyfuzz.modelsto be used as the grouper in case you would like to use
something else than the standard TF-IDF model:model=PolyFuzz("TF-IDF")model.match(from_list,to_list)edit_grouper=EditDistance(n_jobs=1)model.group(edit_grouper)Custom ModelsAlthough the options above are a great solution for comparing different models, what if you have developed your own?
If you follow the structure of PolyFuzz'sBaseMatcheryou can quickly implement any model you would like.Below, we are implementing the ratio similarity measure from RapidFuzz.importnumpyasnpimportpandasaspdfromrapidfuzzimportfuzzfrompolyfuzz.modelsimportBaseMatcherclassMyModel(BaseMatcher):defmatch(self,from_list,to_list,**kwargs):# Calculate distancesmatches=[[fuzz.ratio(from_string,to_string)/100forto_stringinto_list]forfrom_stringinfrom_list]# Get best matchesmappings=[to_list[index]forindexinnp.argmax(matches,axis=1)]scores=np.max(matches,axis=1)# Prepare dataframematches=pd.DataFrame({'From':from_list,'To':mappings,'Similarity':scores})returnmatchesThen, we can simply create an instance of MyModel and pass it through PolyFuzz:custom_model=MyModel()model=PolyFuzz(custom_model)CitationTo cite PolyFuzz in your work, please use the following bibtex reference:@misc{grootendorst2020polyfuzz,author={Maarten Grootendorst},title={PolyFuzz: Fuzzy string matching, grouping, and evaluation.},year=2020,publisher={Zenodo},version={v0.2.2},doi={10.5281/zenodo.4461050},url={https://doi.org/10.5281/zenodo.4461050}}ReferencesBelow, you can find several resources that were used for or inspired by when developing PolyFuzz:Edit distance algorithms:These algorithms focus primarily on edit distance measures and can be used inpolyfuzz.models.EditDistance:https://github.com/jamesturk/jellyfishhttps://github.com/ztane/python-Levenshteinhttps://github.com/seatgeek/fuzzywuzzyhttps://github.com/maxbachmann/rapidfuzzhttps://github.com/roy-ht/editdistanceOther interesting repos:https://github.com/ing-bank/sparse_dot_topnUsed in PolyFuzz for fast cosine similarity between sparse matrices