Group thousands of similar spreadsheet text cells in seconds | by Luke Whyte | Towards Data Science

Here’s a common spreadsheet or database problem:
+-----+-------------------+| row |     fullname      |+-----+-------------------+|   1 | John F. Doe       ||   2 | Esquivel, Mara    ||   3 | Doe, John F       ||   4 | Whyte, Luke       ||   5 | Doe, John Francis |+-----+-------------------+
Rows 1, 3 and 5 likely refer to the same person with slight deviations in spelling and formatting. In small datasets, the cells can be cleaned by hand. But what about in huge datasets? How do we comb thousands of text entries and group similar entities?
Ideally, there’d be an easy way to add a third column like this:
+-----+-------------------+---------------+| row |     fullname      |  name_groups  |+-----+-------------------+---------------+|   1 | John F. Doe       | Doe John F    ||   2 | Esquivel, Mara    | Esquivel Mara ||   3 | Doe, John F       | Doe John F    ||   4 | Whyte, Luke       | Whyte Luke    ||   5 | Doe, John Francis | Doe John F    |+-----+-------------------+---------------+
Well, there is and thats what we’re going to do.
TLDR: I built a tool for this. You can install the Python module here. But if you want to learn about the concepts behind the tool — or if you just distrust me vehemently (Hi, Honey) — read on.
Topics we’ll cover:
Building a Document Term Matrix with TF-IDF and N-GramsUsing cosine similarity to calculate proximity between stringsUsing a hash table to convert our findings to a “groups” column in our spreadsheet
Building a Document Term Matrix with TF-IDF and N-Grams
Using cosine similarity to calculate proximity between strings
Using a hash table to convert our findings to a “groups” column in our spreadsheet
For this tutorial, I’m going to use this dataset of U.S. Department of Labor wage theft investigations. It contains every DOL investigation of employers due to minimum wage or overtime violations from 1984 through 2018.
The data includes a legal_name column, which lists the name of the company being investigated. However, entry formats vary wildly:
+-----+----------------------+| row |      legal_name      |+-----+----------------------+|   1 | Wal-mart Inc         ||   2 | Walmart Stores Inc.  ||   3 | Wal-mart stores Inc  ||   4 | Wal-Mart stores Inc. |+-----+----------------------+
We’ll normalize and group the entries under legal_name then use our groups to do some quick analysis.
Our biggest challenge here is that every entry in our column needs to be compared with every other entry. Thus, a sheet with 400,000 rows necessitates 400,0002 calculations, and my laptop overheats when I open Netflix before remembering to close Photoshop.
It would be much faster if we could use matrix multiplication to make simultaneous calculations, which we can do with a Document Term Matrix, TF-IDF and N-Grams.
Lets define those terms:
Document Term Matrix
A Document Term Matrix is essentially an extension of the Bag of Words (BOW) concept, which I love because it sounds like something a masked man would steal on Sesame Street.
BOW involves counting the frequency of words in a string. So, given the sentence:
“Rhode Island is neither a road nor is it an island. Discuss.”
We can produce a BOW representation like this:
+---------+-------+|  term   | count |+---------+-------+| rhode   |     1 || island  |     2 || is      |     2 || neither |     1 || a       |     1 || road    |     1 || nor     |     1 || it      |     1 || an      |     1 || discuss |     1 |+---------+-------+
A Document Term Matrix (DTM) extends BOW to multiple strings (or, in the nomenclature, “multiple documents”). Imagine we have the following three strings:
“Even my brother has needs”“My brother needs a lift”“Bro, do you even lift?”
“Even my brother has needs”
“My brother needs a lift”
“Bro, do you even lift?”
The DTM could look like this:
The value of each entry is determined by counting how many times each word appears in each string.
The problem with the above approach is that insignificant words like ‘the’, ‘is’ and ‘if’ tend to appear more frequently than important words, which could skew our analysis.
Thus, instead of counting words, we can assign them a TF-IDF score, which evaluates the importance of each word to the DTM.
TF-IDF
To calculate TF-IDF scores, we multiply the amount of times a term appears in a single document (Term Frequency or TF) by the significance of the term to the whole corpus (Inverse Document Frequency or IDF) — the more documents a word appears in, the less valuable that word is thought to be in differentiating documents from one another.
Take a gander here if you are interested in the math behind calculating TF-IDF scores.
The important takeaway is that, for each word in our Document Term Matrix, if we replace the word count with a TF-IDF score, we can weigh words more effectively when checking for string similarity.
N-Grams
Finally we’ll tackle this problem:
Burger Kingis two words. BurgerKing should be two words, but a computer will see it as one. Thus, when we calculate our Document Term Matrix, these terms won’t match.
N-grams are a way of breaking strings into smaller chunks where N is the size of the chunk. So, if we set N to 3, we get:
['Bur', 'urg', 'rge', 'ger', 'er ', 'r K', ' Ki', 'Kin', 'ing']
and:
['Bur', 'urg', 'rge', 'ger', 'erK', 'rKi', 'Kin', 'ing']
which have significantly more overlap than the original strings.
Thus when we construct our Document Term Matrix, lets calculate the TF-IDF score for N-Grams instead of words.
Finally, some code:
Here’s the code for building a Document Term Matrix with N-Grams as column headers and TF-IDF scores for values:
On line 6, we convert our CSV to a Pandas DataFrame.
Line 10 extracts unique values from the legal_name column of the dataset and places them in a one-dimensional NumPy array.
On line 14, we write the function for building our 5 character N-Grams (I pulled the function from here). A few characters are filtered out using a regular expression.
Line 20 passes the ngrams_analyzer to the TF-IDF vectorizer we’ll use to build out matrix.
Finally, on line 23, we build our Document Term Matrix.
Sparse vs dense matrices and how to crash your computer
The result of the above code, tfidf_matrix, is a Compressed Sparse Row (CSR) matrix.
If you are unfamiliar with sparse matrices, this is a great introduction. For our purposes, know that any matrix of mostly zero values is a sparse matrix. This is distinct from a dense matrix of mostly non-zero values.
Our matrix of N-Grams has 237,573 rows and 389,905 columns. The first 10 rows and columns look like this:
That’s pretty sparse. There’s no reason to store all those zeros in memory. If we do, there’s a chance we’ll run out of RAM and trigger a MemoryError.
Enter the CSR matrix, which stores only the matrix’s nonzero values and references to their original location.
This is an oversimplification and you can learn the nitty gritty here. The important takeaway is that the CSR format saves memory while still allowing for fast row access and matrix multiplication.
Cosine similarity is a metric between 0 and 1 used to determine how similar strings are irrespective of their length.
It measures the cosine of the angle between strings in a multidimensional space. The closer that value is to 1 (cosine of 0°), the higher the string similarity.
Take the following three strings:
I love dogsI love love love love love love love dogsI hate hate hate cats
I love dogs
I love love love love love love love dogs
I hate hate hate cats
And put them in a Document Term Matrix:
Then plot this matrix on a multidimensional space where each dimension corresponds to one of our four terms. That could look something like this:
If we look at the distance between our points, “I love dogs” and “I hate cats” are physically closer to one another than “I love dogs” and “I love ... love dogs”.
However, if we look at the angle between our points’ lines — the cosine distance — we can see that the angle between “I love dogs” and “I love ... love dogs” is much smaller than the angle between “I love dogs” and “I hate cats”.
Thus, the cosine similarity between String 1 and String 2 will be a higher (closer to 1) than the cosine similarity between String 1 and String 3.
Here’s a deeper explanation.
Calculating cosine similarity in Python
We could use scikit-learn to calculate cosine similarity. This would return a pairwise matrix with cosine similarity values like:
We would then filter this matrix by a similarity threshold — something like 0.75 or 0.8 — in order to group strings we believe represent the same entity.
However, if instead we use this module built by the data scientists at ING Bank, we can filter by our similarity threshold as we build the matrix. The approach is faster than scikit-learn and returns a less memory intensive CSR matrix for us to work with.
ING wrote a blog post explaining why, if you’re interested.
So, let’s add the following to our script:
Now we have a CSR matrix representing the cosine similarity between all our strings. Time to bring it home.
We’re now going to build a Python dictionary with a key for each unique string in our legal_name column.
The fastest way to do this is to convert our CSR matrix to a Coordinate (COO) matrix. A COO matrix is another representation of a sparse matrix.
By example, if we have this sparse matrix:
+------------+| 0, 0, 0, 4 || 0, 1, 0, 0 || 0, 0, 0, 0 || 3, 0, 0, 7 |+------------+
And we convert it to a COO matrix, it will become an object with three properties — row, col, data — that containing the following three arrays, respectively:
[0, 1, 3, 3]: The row index for each non-zero value (0-indexed)[3, 1, 0, 3]: The column index for each non-zero value (0-indexed)[4, 1, 3, 7]: The non-zero values from our matrix
[0, 1, 3, 3]: The row index for each non-zero value (0-indexed)
[3, 1, 0, 3]: The column index for each non-zero value (0-indexed)
[4, 1, 3, 7]: The non-zero values from our matrix
Thus, we can say that the coordinates for the value 4(stored in matrix.data[0]) are (0,3)(stored in (matrix.row[0],matrix.col[0]).
Let’s build our COO matrix and use it to populate our dictionary:
On line 2 we convert our cosine matrix to a coordinate matrix.
On lines 39–43 we iterate through our coordinate matrix, pull out the row and column indices for our non-zero values — which, remember, all have a cosine similarity of over 0.8 — and then convert them to their string values.
To clarify, let’s unpack lines 39–43 further with a stripped down example. Again, take this cosine matrix:
If we’d built it using awesome_cossim_topn with the threshold set to 0.8 and then converted it to a COO matrix, we could represent it like this:
  (row, col) | data   ------------|------   (0,0)      |    1   (0,2)      | 0.84   (1,1)      |    1   (2,0)      | 0.84   (2,2)      |    1
vals would equal ['Walmart', 'Target', 'Wal-mart stores'].
Thus, inside the loop, our first (row, col) pair to pass the row != col conditional would be (0, 2) which we then pass to add_pair_to_lookup as (vals[0], vals[2) or ('Walmart', 'Wal-mart stores').
Continuing with this example, after all our strings pass through add_pair_to_lookup, we’d end up with:
>>> group_lookup{    'Walmart': 'Walmart',    'Wal-mart stores': 'Walmart'}
There are no strings similar to ‘Target’, thus it is not assigned a group.
Vectorize that Panda
Finally, we can use the power of vectorization in Pandas, to map each legal_name value to a new Group column in our DataFrame and export our new CSV.
Since Pandas functions can operate on an entire array simultaneously — instead of sequentially on individual values — this process is very fast:
The fillna method allows us to substitute the legal_name value for Group when no key exists in group_lookup.
Putting it all together:
All that is left to do is throw this data into a Pivot Table and see which employers owe(d) the most in back wages to employees.
Spoiler alert: it’s Wal-Mart. 183 investigations have resulted in almost $41 million in back wages that they’ve agreed to pay.
One last note
If you wanted to group by two or more columns instead of one, one approach could be to create a temporary column for grouping in your DataFrame of just the entries for each column concatenated into a single string:
columns_to_group = ['legal_name', 'address']df['grouper'] = df[   columns_to_group.pop(0)].astype(str).str.cat(   df[columns_to_group].astype(str))
You would then set vals as:
vals = df['grouper'].unique().astype('U')
And, when exporting at the end, drop that column:
df.drop(columns=['grouper']).to_csv('./dol-data-grouped.csv')
Again, I created a Python module to do all this. Check it out!!