Fuzzy String Match With Python on Large Datasets and Why You Should Not Use FuzzyWuzzy | by Alina Zhang | Towards Data Science

Languages are ambiguous. The text referring to the same thing could be written slightly differently, or even misspelled. Assuming that you are trying to join two tables by the column of addresses, the same location shows in table A as “520 Xavier Ave, California City” while “520 Xavier Avenue, CA” in table B. How would you handle this issue?
There are a few more examples demonstrating the same content written in different ways:
The Queen’s Gambit vs Netflix The Queen’s Gambit (miniseries)
Toronto Raptors vs Raptors
Los Angeles Lakers vs Lakers
helloworld@gmail.com vs helloworld@gmail.con
Tesla, Inc. vs TSLA
We want to treat them as the same thing before we feed the data to machine learning models or apply any other data analysis methods.
When it comes to fuzzy string match, the first solution data scientists typically take is FuzzyWuzzy. FuzzyWuzzy package is a Levenshtein distance based method which widely used in computing similarity scores of strings. But why we should not use it? The answer is simple: it is way too slow.
The estimated time of computing similarity scores for a 406,000-entity dataset of addresses is 337 hours. It only took 21 minutes on the same dataset with another solution FuzzyCouple which we will introduce soon. FYI, my box is MacBook Pro 2019 with an Intel Core i7 processor.
337 hours vs 21 minutes on large datasets — is it a strong enough argument that convinces you to try FuzzyCouple?
If you say that it is the first time you heard about FuzzyCouple, you are absolutely right. Because I created the term that this method can be introduced as “I applied FuzzyCouple” instead of “I used TF-IDF as vectorizer then computed similarity scores with cosine similarity.”
The ARXIV dataset used in this project contains about 41,000 research papers related to machine learning, NER, and computer vision published between 1992 and 2018.
replace symbols: \n “ ‘ / ( ) { } | @ , ; # with space
remove multiple spaces
to lower case
remove stop words
import retext = 'Supporting "Temporal \'Reasoning by Mapping #@{ Calendar Expressions to Minimal\n Periodic Sets'REPLACE_BY_SPACE_RE = re.compile('[\n\"\'/(){}\[\]\|@,;#]')text = re.sub(REPLACE_BY_SPACE_RE, ' ', text)text = re.sub(' +', ' ', text)text = text.lower()print(text)
As illustrated above, the column TITLE has removed stopwords, symbols, extra spaces, and been converted to lower cases.
from wordcloud import WordCloudwordcloud = WordCloud().generate(' '.join(df['TITLE']))# plot the WordCloud imageplt.figure(figsize = (10, 8), facecolor = None)plt.imshow(wordcloud)plt.axis("off")plt.tight_layout(pad = 0)plt.show()
As shown in the plot, the most common words used in research paper titles are neural, network, learning, analysis, classification, model, using, based, algorithm.
from sklearn.feature_extraction.text import TfidfVectorizertfidf_vectorizer = TfidfVectorizer(ngram_range=(1,2), max_df=0.9, min_df=5, token_pattern='(\S+)')tf_idf_matrix = tfidf_vectorizer.fit_transform(df['TITLE'])
Column TITLE is the strings we remove stopwords and symbols from original paper titles. Column SIMILAR_TITLE is similar to TITLE. similarity_score indicates how similar they are which ranges from 0 to 1.
As shown in the above example, the TITLE ‘unsupervised learning disentangled representations video’ is similar to ‘inferencing based unsupervised learning disentangled representations’ with a high score of 0.94.
Let’s check the corresponding titles in the original papers:
As we have seen, they are both research papers about disentangled representations with unsupervised learning algorithms.
In this article, we introduced FuzzyCouple which is a super-fast solution for fuzzy string matching at scale. As mentioned at the beginning, text or languages can be ambiguous. FuzzyCouple is an efficient and practical method for identifying the “same thing” in unstructured data.
Reference:
Super Fast String Matching in Python by Ven Dan