Surprisingly Effective Way To Name Matching In Python | by Mala Deep | Towards Data Science

Recently I came across this dataset, where I needed to analyze the sales recording of digital products. I got the dataset of having almost 572000 rows and 12 columns. I was so excited to work on such big data. With great enthusiasm, I gave a quick view of data, and I found the same name repeatedly taking different rows. (Ah! Much time for data cleaning!).
On some product column, it contains iphone while on other it was written Iphone, or iphone 7 + and iphone 7 Plus, and for some regular customer list, some have Pushpa Yadav while on other Puspa Yadav (the name is Pseudo).
These are the same product name and customer name but were taken as different forms, i.e., deal with different versions of the same name. These sorts of problems are common scenarios for data scientists to tackle during data analysis. This scenario has a name called data matching or fuzzy matching (probabilistic data matching) or simply data deduplication or string/ name matching.
A common reason might be:
Typing error during data entry.
Abbreviations usages.
System data entry is not well validated to check for such errors.
Others.
Whatsoever, as a data scientist or analyst, it is our responsibility to match those data to create a master record for further analysis.
So, I jotted out the action to solve the problem:
1. Manually check and solve.
2. Look for useful libraries/ resources that the great mind of the community has shared.
The first choice was truly cumbersome for such an extended dataset (572000* 12 ), so I started looking into different libraries and tried thee approaches with a fuzzy matcher (not available on conda Now), fuzzywuzzy which on core uses Levenshtein distance and difflib.
However, on using them, I found that for such big data, it is more time-consuming. Thus, I need to look for a faster and effective method.
After many days of frustration, I finally came to know about the solution shared by Chris van den Berg.
ngram
Vectorization
TF-IDF
Cosine similarity with sparse_dot_topn: The leading juice
Working demo ( Code-work)
As our goal here is not just to match the strings but also match it in a faster way. Thus the concept of ngram, TF-IDF with cosine similarity, comes into play.
Before going into the working demo (code-work), let’s understand the basics.
N-grams are extensively used in text mining and natural language processing, which are a set of co-occurring words within a given sentence or (word file).To find n-gram, we move one word forward( can move any step as per need).
For example, for the room type “Standard Room Ocean View Waikiki Tower”
If N=3 (known as Trigrams), then the n-grams (3-grams) would be:
Standard Room Ocean
Room Ocean View
Ocean View Waikiki
View Waikiki Tower
Explore more about ngrams from the research paper
Why n=3?
Question might arises why n = 3, can be taken as n= 1(unigrams ) or n=2 (bi-grams).
The intuition here is that bi-grams and tri-grams can capture contextual information compared to just unigrams. For example, “Room Ocean View” carries more meaning than only “Room,” “Ocean,” and “View” when observed independently.
TF-IDF stands for term frequency-inverse document frequency, and the TF-IDF weight is a weight often used in information retrieval and text mining two prime concerns.
1. Used to evaluate how important a word is to a document in a collection
2. The importance increases proportionally to the number of times a word appears
Two ideas behind TF-IDF
Term Frequency (TF), aka. The number of times a word appears in a document, divided by the total number of words in that document, i.e., it measures how frequently a term occurs in a document.
Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears, i.e., it measures how important a term is.
While computing TF, all terms are considered equally important. However, it is known that specific words, such as “is,” “of,” and “that,” may appear a lot of times but have little importance.
After having each word split, there is need to convert each of those token(lemmas) into a vector that SciKit Learns algorithm models can work with, as these algorithms only understand the concept of numerical features irrespective of its underlying type (text, image, and numbers, etc.) allowing us to perform complex machine learning tasks on different types of data.
Scikit-learn’s Tfidfvectorizer aims to convert a collection of raw documents to a matrix of TF-IDF features, and it can compute the word counts, IDF, and TF-IDF values all at once.
With this, we generate tf_idf_matrix, which is a sparse matrix as with Tfidfvectorizer, we are converting raw text to a numerical vector representation of words and n-grams. This makes it easy to directly use as already said; algorithms only understand the concept of numerical features irrespective of its underlying type (text, image, numbers, etc.).
How the two text documents are close to each other in terms of their context(surface closeness) and meaning, i.e., lexical similarity and semantic similarity respectively is called Text Similarity and there is a various method to calculate text similarities such as Cosine similarity, Euclidean distance, Jaccard coefficient, and Dice.
Cosine similarity measures the text-similarity between two documents irrespective of their size. Mathematically, the Cosine similarity metric measures the cosine of the angle between two n-dimensional vectors projected in a multi-dimensional space, and value ranges from 0 to 1,
where,
1 means more similarity
0 means less similarity.
Mathematically:
However, the Data Scientists at ING Wholesale Banking Advanced Analytics team found out Cosine Similarity has some disadvantages:
The sklearn version does a lot of type checking and error handling.
The sklearn version calculates and stores all similarities in one go, while we are only interested in the most similar ones.
Therefore both results in the use of more memory consumption and time.
To optimize for these disadvantages, they created their library, which stores only the top N highest matches in each row, and only the similarities above the threshold.
Get more about library: sparse_dot_topn
They proudly claim that this approach improves the speed by about 40% and reduces memory consumption.
With these basic foundational knowledge, we start our code-work.
I assume you are familiar with Jupyter notebook and downloading libraries. If not, please check my previous post, where I have shared the beginners’ guide for exploratory data analysis.
Working environment: Jupyter Notebook from Anaconda
Dataset: Room Types
Snapshot of Dataset
Now let’s import the needed libraries/methods and read our dataset and save it to “df,” you can name “df” as anything with a quick look to the dataset.
#  Importing libraries and module and some setting for notebookimport pandas as pd import refrom sklearn.feature_extraction.text import TfidfVectorizerimport numpy as npfrom scipy.sparse import csr_matriximport sparse_dot_topn.sparse_dot_topn as ct  # Leading Juice for usimport timepd.set_option('display.max_colwidth', -1)# reading dataset as dfdf =  pd.read_csv('room_type.csv')# printing first five rowsdf.head(5)
Here, we are taking n = 3 thus 3-gram or trigrams as most room types contain two or three words, and we split sentences into tokens with removing all special characters, punctuation and single characters (,-./). And collect those 3-grams.
def ngrams(string, n=3):string = re.sub(r'[,-./]|\sBD',r'', string)    ngrams = zip(*[string[i:] for i in range(n)])    return [''.join(ngram) for ngram in ngrams]# Testing ngrams work for verificationprint('All 3-grams in "Deluxroom":')ngrams('Deluxroom')
For verification, we test the 3-gram result for “Deluxroom” and we have output as:
Looks good!
Now, After having each word split (token or lemmas (n-gram generated items) ), we need to count the word occurs in each document for which we use Tfidfvectorizer, provided by Scikit-Learn library and convert a collection of raw materials to a matrix of TF-IDF features.
room_types = df['RoomTypes']vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)tf_idf_matrix = vectorizer.fit_transform(room_types)
Let’s view the generated sparse CSR matrix.
print(tf_idf_matrix[0])
Looks good!
We are dealing with a CSR matrix with sparse_dot_topn library.
def awesome_cossim_top(A, B, ntop, lower_bound=0):    # force A and B as a CSR matrix.    # If they have already been CSR, there is no overhead    A = A.tocsr()    B = B.tocsr()    M, _ = A.shape    _, N = B.shape     idx_dtype = np.int32     nnz_max = M*ntop     indptr = np.zeros(M+1, dtype=idx_dtype)    indices = np.zeros(nnz_max, dtype=idx_dtype)    data = np.zeros(nnz_max, dtype=A.dtype)ct.sparse_dot_topn(        M, N, np.asarray(A.indptr, dtype=idx_dtype),        np.asarray(A.indices, dtype=idx_dtype),        A.data,        np.asarray(B.indptr, dtype=idx_dtype),        np.asarray(B.indices, dtype=idx_dtype),        B.data,        ntop,        lower_bound,        indptr, indices, data)return csr_matrix((data,indices,indptr),shape=(M,N))
We store the top 10 most similar items and only items with a similarity above 0.8 and show the time taken by the model.
#  Top 10 with similarity above 0.8t1 = time.time()matches = awesome_cossim_top(tf_idf_matrix, tf_idf_matrix.transpose(), 10, 0.8)t = time.time()-t1print("SELFTIMED:", t)
Then, we unpack the resulting sparse matrix;
# unpacks the resulting sparse matrixdef get_matches_df(sparse_matrix, name_vector, top=100):    non_zeros = sparse_matrix.nonzero()        sparserows = non_zeros[0]    sparsecols = non_zeros[1]        if top:        nr_matches = top    else:        nr_matches = sparsecols.size        left_side = np.empty([nr_matches], dtype=object)    right_side = np.empty([nr_matches], dtype=object)    similairity = np.zeros(nr_matches)        for index in range(0, nr_matches):        left_side[index] = name_vector[sparserows[index]]        right_side[index] = name_vector[sparsecols[index]]        similairity[index] = sparse_matrix.data[index]        return pd.DataFrame({'left_side': left_side,                          'right_side': right_side,                           'similairity': similairity})
We view our matches.
# store the  matches into new dataframe called matched_df and # printing 10 samplesmatches_df = get_matches_df(matches, room_types, top=200)matches_df = matches_df[matches_df['similairity'] < 0.99999] # For removing all exact matchesmatches_df.sample(10)
Suite, 1 Bedroom and Deluxe Suite, 1 Bedroom is probably not the same room type, and we got the similarity measure of 0.81. Looks good!
In sorted order, we view our matches.
# printing the matches in sorted ordermatches_df.sort_values(['similairity'], ascending=False).head(10)
The matches look pretty satisfying! The cosine similarity gives a good indication of the similarity between the two room types.Grand Corner King Room 1 King Bed and Grand Corner King Room are probably the same room type, and we got the similarity measure of 0.93.
Thus, the correct visual assessment and the matches made with this strategy are very fulfilling.
So, by using the ngram for tokenization, TF-IDF for vector matrix, and TfidfVectorizer to count the word occurs in each document and using cosine similarity with sparse_dot_topn, we matched the strings most quickly even for the large dataset (Got good result with 572000*12 ).
Get the entire working Jupyter notebook in my GitHub.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.
If you have any queries regarding the article or want to work together on your next data science project, ping me on LinkedIn.
Stay tuned for next Data Science related post.
Using TF-IDF to identify the signal from the noise
Boosting the selection of the most similar entities in large scale datasets
Cosine Similarity — Text Similarity Metric
An Introduction to N-grams: What Are They and Why Do We Need Them?
Quick Exploratory Data Analysis: Pandas Profiling