A Layman’s Guide to Fuzzy Document Deduplication | by Ryan Basques | Towards Data Science

In our era of exponentially growing data, complex projects, large teams, and a desire to move on to the next deadline, small things often fall through the cracks.
Our team of data analysts saw this at a large law firm, where important documents were often shared, edited, shared again, and stored away on a dedicated document server. Twenty years and a million documents later, the server was a labyrinth of nested folders, unintelligible filenames, and corrupted files.
Our goal was to sift through the mass of storage for important documents, but we had a problem to solve first.
We proposed a new, clean location for very important documents. However this required documents to be searchable, relevant, and notably, unique. From a quick peek at the documents, we could see that there were many duplicates in the data.
We couldn’t rely on file names, dates or any other unique identifier. All that we had was the content in the text documents themselves.
Why is this problem more complicated than writing a simple “exactly equal” script that looks for documents where the texts are completely identical? The wrinkle is that in practice, searching for 100% similarity misses many duplicate documents. The reasons are varied, but include:
Differences during OCR (Optical Character Recognition — used when converting PDFs/images to text)
Minor revisions, drafts
Variability in the versioning/methodology of the software used to save or load the document
By allowing for a “fuzzy” deduplication where the script identifies mostly similar documents, we can sidestep any issues that might cause two otherwise duplicate documents to fall out of step with each other like stray punctuation, capitalization, whitespace, etc. As you will see, we will have a lever to adjust just how similar a document has to be to be classified as a duplicate.
While this use-case is quite specific, there are many situations when mass deduplication may be desirable, including:
Save storage space
Check for plagiarism/copyrights
Ensure only new questions on a question/answer website
Check new entries into a database to ensure uniqueness
Eliminate noise from a bloated dataset
First, let’s look at the solution from a theoretical perspective. Afterward, we can show a brief implementation in Python for those interested in the code behind the theory.
For illustration, we’ll consider a real-life use case. We’ll use a dataset from a Kaggle competition where the goal is to detect duplicate advertisements in an online marketplace. For this guide, we’ll focus on the texts of the advertisements.
Folks submitting ads often make tweaks to their existing ad and re-submit it to have a higher chance of people finding their product.
A quick trip to Craigslist shows us an example of this right away:
See how some of these are worded slightly differently? An exact match system would have never caught these.
We want to reduce the clutter for users by flagging and removing duplicate ads (even if they are not the same word-for-word).
Some basic information about the dataset:
We are using a sample of 100,000 ads. This solution can scale to millions of documents depending on how much time and computing power/memory is available to you.
Executing a simple “exactly equal” script like we described above, we see that about 96,000 of these ads are originals (or at least, not an exact duplicate of any other ad).
There are about 1,000 ads that are exact duplicates. Some are duplicated dozens of times, others just once.
Although the duplicate ads should represent just 1% of the dataset, the many copies bloat them to 3.7% of the data.
The text itself is in Russian — fortunately this doesn’t matter, as the solution is ambivalent to language. I’ll use Google Translate where necessary.
Let’s take a look at the top most duplicated advertisements in our dataset, using the “exactly equal” script:
We can see right away some of the downside of strict duplicate detection. Notice how “in a good condition” is broken out into three separate categories because of capitalization or punctuation. Our simple script erroneously classifies these as completely independent, original texts.
Another frequent pattern in the data is more suspicious — much longer, specific ads with slightly different wording:
Below we will develop a more sophisticated solution to detect near duplicates in addition to exact duplicates.
Before we dive further into this case, let’s talk about the concepts that will support our endeavor.
To deduplicate these redundant ads in a “fuzzier” way, we’ll need to find a way to turn text into numbers to analyze similarity quantitatively.
There are a few different methods, but one of the most popular and effective ways to measure “string similarity”, or the sameness of text, is called cosine similarity.
The logic behind it is quite straightforward. If we can convert text into a vector (line on a graph), we can compare it numerically to other vectors (more lines on the same graph). The placement of the line depends on the contents of the document.
For example, consider the following three phrases:
· “buttery popcorn”
· “salty popcorn”
· “buttery croissant”
Intuitively, we can see some similarity among these phrases, or documents. Two of them are buttery, two of them are popcorn. However, in order to scale to comparing millions of documents, we need to make the problem understandable by a machine.
Let’s take a simple count of each time a word occurs in each of the phrases. The result will look something like below:
Each row is what we call a document vector, representing each respective document quantitatively.
We’ve only included two features (columns) here, although you could have as many features as you have unique words in your corpus (entire collection of documents). We could add a third dimension for croissant, for example.
Because we only have buttery and popcorn, we can graph this dataset in 2D to illustrate how cosine similarity works:
The y-axis measures how often we see “buttery”, while the x-axis measures how often we see “popcorn”. We can see our three documents plotted with blue dots.
buttery croissant is high in the buttery dimension
salty popcorn is high in the popcorn dimension
buttery popcorn has both
The green text indicates our cosine similarity calculation. We can see that buttery croissant and salty popcorn have a very large angle sitting between them of 90 degrees. Buttery popcorn and salty popcorn have a smaller angle of only 45 degrees.
We could stop here, but taking a cosine of these angles will normalize them for us, to 0.0 and 0.7 respectively. Handily, a cosine calculation will always have a minimum of -1 (for 180 degrees), and a maximum of 1 (for 0 degrees).
We’ve successfully quantified how different these pieces of text are with two features. Considering that our documents contain hundreds or thousands of unique words, we will have just as many features. It would be impossible to visualize this many document vectors on a graph like above, but rest assured that the logic is exactly the same in thousands of dimensions. The computer will handle it for us.
However, there’s still one wrinkle. If we’re just counting the words in a document, our method doesn’t take into account word order. For example, how would our cosine similarity method above account for the difference between:
“good condition, not negotiable”
“not good condition, negotiable”
Not very well, because they contain the exact same words, just in a different order. We can see that these phrases have very different meanings, but our cosine similarity would simply see that they contain all the same words, for a score of 100% similar!
An n-gram simply describes a sequence of n items from a given sample. In text, this may be a sequence of n words from a given sentence. So, let’s say you want all 2-grams (or bigrams) from the document “good condition, not negotiable” — the following are the three bigrams you’d receive:
· “good_condition”
· “condition_not”
· “not_negotiable”
Similarly, if you want 3-grams (or trigrams), you’d receive:
· “good_condition_not”
· “condition_not_negotiable”
Can you see why having these additional features might be useful?
Instead of creating a document vector on only single words, we create it on all bigrams and trigrams, too! Now “good condition, not negotiable” and “not good condition, negotiable” will justifiably receive a lower similarity score since their bigrams and trigrams do not line up.
One question you may ask is — why go through the effort of measuring the angles between document vectors? We can see from the graph above that you can plot a point for each document — why not just look at the distance between the dots (Euclidean distance)?
It’s a great question. To answer it, let’s check out the below graph:
Now we’ve added magnitude to the documents, where some words occur multiple times. We have one document that is very buttery (x4), one that is buttery and popcorn (x2, x1) and one that is just popcorn (x1).
While intuition would tell us that the second document is closer to the first since they both overload on butter, the Euclidean distance A (2.2) is actually larger than Euclidean distance B (2.0). Using these distances, we would say the documents sharing one “popcorn” term are more similar than the documents sharing two “buttery” terms!
However, angles A and B (measured with cosine similarity) support our intuition that the buttery documents are more similar than the popcorn documents.
In fact, when magnitude is normalized (all dots are equal length from the origin of the graph), Euclidean distance and cosine similarity will return the same results.
One additional benefit of cosine similarity is that it is less expensive for a computer to calculate, which can result in some serious time savings when we’re talking about millions or billions of comparisons.
The field of natural language processing is deep and wide, and there are many additional ways to increase the quality of your data through preprocessing, depending on your use-case.
For example, stemming will normalize all words to their most basic form (running becomes run, delivered becomes deliver).
Another transformation takes the document vectors we’ve created, called bag-of-words vectors (because we’re simply counting how many words are in each document), and converting it into term frequency — inverse document frequency vectors (TF-IDF). This is a very powerful transformation that can help you in tasks like document classification. You discount words that occur frequently across your entire corpus (the, he, her, it, etc.) and focus on rarer words that occur in specific documents but not across the entire corpus, because these latter words are much more likely to be meaningful.
These transformations are worth knowing about, but we won’t be using them here because duplicates will generally use the same word forms (no need for stemming) and term frequency (no need to specially weight different terms/documents).
Finally, if you’re on the cutting edge of the field, you’ll look into using document embeddings, which is another method of mapping textual meaning into a vector of numbers. This is outside the scope of this article (and may be overkill for our deduplication application).
Nicely done! You now have a grasp on a very useful tool. Let’s see how we can apply it to our earlier duplicate-ad-detection example.
The first step is to create a document-term matrix. You’ve already seen one of these earlier with the popcorn example — it’s simply a structure that keeps track of documents on the rows, and specific terms (note: no longer “words” since we could also have n-grams thrown in) on the columns. Each time a term occurs in a document, we’ll increment the appropriate cell in the matrix by one.
Let’s see what this matrix looks like for a few examples in our data. I’ve translated the Russian examples to English for better illustration, although the method applies in any language.
Orange cells represent an intersection between our documents and our terms. We can see that multiple documents contain the same term — like condition belonging to four documents. This will contribute to a higher cosine similarity score between these documents.
Now that we have transformed our textual data into document vectors, we can apply a pairwise cosine calculation to evaluate the similarity between all of the documents. Pairwise just means that we’ll receive a number for every two examples in our dataset — every document needs to be compared with every other document to see if there are any similarities.
Note that this can become computationally expensive — in the technical section we’ll discuss a great package that saves on resources while performing this calculation.
Let’s see what a cosine similarity calculation on the above examples provides us:
Each document is placed along the rows and the columns, with the pairwise cosine similarities filling in the cells.
We see 100% similarity along the diagonal as we would expect — all documents are identical to themselves. For example, “14.5 cm in excellent condition” is in the first row and the first column, resulting in an orange 100% in the first cell. Of course, when we perform our analysis, we will not count these self-similar cells as “duplicates” in our dataset.
Right away, we see some interesting results:
“14.5 cm in excellent condition” and “In great condition” share enough terms that they have a 71% cosine similarity.
“Shirt in good condition” and “shirt in good condition” receive a 100% similarity because our process is case insensitive.
We might argue that the first match is not a case of duplication, whereas the second is a case of duplication. We can start to see an intuition for forming a duplication cutoff between 71% and 100% — but more on choosing this threshold later.
Note that I included the original Russian text in the graphic as well. This helps explain why “14.5 cm in excellent condition” and “in good condition” did not receive any similarity. In Russian, the two condition terms are spelled slightly differently (one meaning “condition”, the other technically meaning “state”), resulting in completely different/independent meanings behind the term.
These results are from a small sample of 6 documents to provide us with some intuition. Let’s see what the results look like on our dataset of 100,000 documents.
For comparison, let’s see the findings side-by-side with the results from doing an exact/strict deduplication criteria earlier in the article:
Our fuzzy deduplication found 2,244 duplicate documents, or about 2% of the total dataset. When accounting for the bloating effect of multiple copies of these duplicate ads, these duplicates account for 7.5% of our data!
By allowing fuzzy deduplication, we’ve found twice as many duplicate documents as before.
We’ll looks at three examples of the fuzzy findings that we’re now picking up.
In this first example, we pick up variations on “great condition”. While interesting, this doesn’t strike as an example of duplicitous advertising — it’s just a common phrase for folks selling items!
This second example represents a pattern of many ads seen throughout the dataset. While the word orders are different, the contents are the same! We can be confident that this is a duplicitous advertisement selling the same pair of sneakers.
As our final example, we see dozens of advertisements for trampolines, where the only variation is the dimensions of the trampoline itself. A user of this website who is not interested in trampolines certainly doesn’t need to see such variety. As the website owner you may choose to take action on this type of redundancy.
Other examples fitting this pattern are duplicate ads where the only difference is in the address, phone number, or product color.
Now that we’ve fully developed a cosine similarity deduplication method and have seen some results, we’re almost finished with the theory part of this article. There is one very important outstanding item to discuss.
One important area for consideration is the cosine similarity threshold we choose. Remember, when we compare two document vectors, we’ll get a cosine similarity between 0% and 100%. It’s up to us to decide whether the result passes or fails a predefined set-point.
A 0% cosine similarity cutoff would flag everything as duplicative of everything else, and a 100% cutoff would require exact duplicates (non-fuzzy) for a duplicate flag to be thrown.
There is no universally “correct” answer, since the cutoff we set will depend on our specific use-case.
On one extreme, let’s say you are deduplicating old manuscripts that are rare and very valuable. You’d be in trouble for deleting even one original manuscript because you thought it was a duplicate of a very similar but different manuscript. If some duplicates do slip by and stay in the dataset, it’s not as big of a deal. Here, you’d err toward a higher deduplication cutoff that is stricter on detecting duplicates.
On the other extreme, you are deduplicating millions of tweets as a preprocessing step for a larger project. Each tweet in itself doesn’t matter much, and your main goal is shrinking the data. You want to get rid of bulky redundant tweets by spambots, for example. It’s perfectly fine if you accidentally remove original tweets — you’re just trying to shrink the size of your corpus by removing low-hanging fruit.
Here is where the concepts of precision and recall become helpful — these are the opposing forces that will help you set a good deduplication cutoff.
Precision asks — of the “positive” documents we’ve flagged (in our case, positive is synonymous with duplicate), how many were true positive/duplicate documents?
Keep in mind that if we flag a document as “positive”/duplicate, but it is not a true “positive”/duplicate, we just erroneously flagged an original document as a duplicate! Depending on our use-case, this may mean that we’ve cued an original document for deletion.
In the former rare manuscript example, we’re optimizing for precision. We want to make sure that all of the documents we’ve flagged as duplicates are truly duplicates. Nobody would blame us for having a 99% deduplication cutoff to be safe.
On the other hand, recall asks — of all the true positive/duplicate documents out there, how many did we correctly identify as duplicates?
How is this different than precision? In precision, we care about how many true positives we got vs false positives. In recall, we care about how many true positives we got vs false negatives.
In the tweet deduplication example, we care more about deduplicating the entire universe of tweets much more than we do making sure that each tweet we flag is truly a duplicate. We’re willing to sacrifice precision for recall. Because of this trade-off, we’re OK with a lower cosine similarity cutoff of say, 70%.
Let’s say we have a dataset of 100 documents. We’ve manually looked at the data, and know that there are 20 duplicate documents that should be deleted.
At a cosine similarity cutoff of 99%, we delete 15 duplicate documents. However, we’re 100% sure that each of the 15 we deleted were truly duplicates. We are very precise (15/15–100%), but have poor recall (15/20–75%).
At a cosine similarity cutoff of 70%, we delete 30 documents. We certainly deleted the 20 true duplicates, for a very good recall score (20/20–100%). However, we also deleted 10 documents that were not truly duplicates. This lends to a poor precision score (20/30–66%).
See how precision and recall oppose each other? In practice, it makes sense to break out a sample of your dataset, label originals and duplicates, and try to optimize the cutoff threshold for your specific situation.
Additionally, cosine similarity could be just one feature of a larger statistical model. In the advertisement deduplication example, it could make sense to include information about the product images, usernames, posting timestamp, and so on, ultimately feeding into a binary classification model for flagging duplicate ads.
For illustration purposes, we used a 95% similarity cutoff. Relatively strict, but as we saw above, we still gain significant performance in fuzzy deduplication.
That’s it for the theory! Thank you if you’ve stuck around this long — you now have a practical familiarity with a valuable method to help you tame duplicative data.
If you have a technical bent and want to know how to code something like the above examples, we’ll continue on with some Python code-snippets that can perform the methodology we just discussed.
Gensim is a Python library popularly used for topic modeling. However it also has very valuable utilities for deduplication.
While there are several efficient ways to calculate cosine similarity in Python, including use of the popular SKLearn library, Gensim’s major advantage comes when your dataset grows very large.
When your corpus grows beyond your computer’s ability to store data in-memory, Gensim automatically shards (splits) your corpus into computer-readable chunks and saves it to disk. By placing your data on disk, you can work with much larger datasets before running into memory issues. You can also load and re-use the shards stored on your disk.
Relatedly, once you generate a similarity index (which we’ll see below), you can save it to disk, and incrementally add more documents to it instead of recalculating from scratch. This allows for a one-time time-intensive lift to load your existing corpus, with very fast updates whenever you add new data to your corpus and want to recalculate similarities.
First, we import the modules we’ll need from Gensim. Note that to run the code yourself, you’ll need to have downloaded Gensim with pip or conda.
#Import necessary gensim packagesfrom gensim.utils import simple_preprocessfrom gensim.models.phrases import Phrases, Phraserfrom gensim import corporafrom gensim.similarities import Similarity
Let’s define some test data to work with.
#Test datasetdocuments = [    “Used SpaceX rocket as-is, buyer must transport.”,    “Used SpaceX rocket as-is, buyer must transport.”,    “For sale: bulk 100lbs pack of spaghetti noodles”,    “Spaghetti noodles for sale — 100lbs bulk pack”,    “Pale blue tuxedo, used, good condition. Call 555–555–5555”,    “Brand new yellow tuxedo in great condition!”]
In general, we work with the documents at word-level. Here’s a simple way to break documents into the correct format:
#Convert documents to collection of wordstexts = [[text for text in simple_preprocess(doc, deacc=True)] for doc in documents]
simple_preprocess is a helpful Gensim function for parsing a document into individual words. You can pass it additional arguments depending on what you need. Here, we pass “deacc=True” to strip our punctuations, accents and numbers.
Here is “Used SpaceX rocket as-is, buyer must transport.” after simple_preprocess:
[‘used’, ‘spacex’, ‘rocket’, ‘as’, ‘is’, ‘buyer’, ‘must’, ‘transport’]
We’ll use the word-level structures to train a Gensim n-gram phraser.
Whereas a normal bigram generator would produce a bigram for each word pair (used_spacex, spacex_rocket, rocket_as), the Gensim phraser is pretrained to select collocations, or words that would commonly be found together. For example, only spacex_rocket would be kept because the phraser deems it a likely collocation of words.
This selectivity is a great way to save on program runtime by not storing a feature for every possible permutation of words.
#Build a bigram model to capture every pair of words in the textsbigram = Phrases(texts, min_count=1)bigram_phraser = Phraser(bigram)
The min_count argument specifies how many times the phrase must be found in the corpus to be kept. We’re keeping it at 1 since our dataset is so small.
Now that we have a trained Gensim Phraser, we’ll use it during conversion of sentences into terms. This looks the same as before, except we’re including the bigram_phraser variable.
#Reconvert documents to collection of words/bigramstexts_bigrams = [[text for text in bigram_phraser[ simple_preprocess(doc, deacc=True)]] for doc in documents]
For example, “Spaghetti noodles for sale — 100lbs bulk pack” now becomes:
['spaghetti_noodles', 'for_sale', 'lbs', 'bulk', 'pack']
Gensim uses package-specific structures to run more efficiently. Specifically, we need to create a Gensim dictionary and Gensim corpus.
#Create dictionarydictionary = corpora.Dictionary(texts_bigrams)
The Gensim dictionary simply maps each specific term (key) to a unique ID (value). A normal Python dictionary representation looks like:
In: {k: v for k, v in dictionary.items()}Out:     {0: 'as_is',    1: 'buyer_must',    2: 'spacex_rocket',    3: 'transport',    4: 'used',    ...    18: 'great',    19: 'in',    20: 'new',    21: 'yellow'}
Next is the corpus:
#Create corpuscorpus = [dictionary.doc2bow(docString) for docString in texts_bigrams]
The Gensim corpus uses the dictionary we just created to convert our original text documents to numeric representation. By replacing text with numbers, the computation becomes much faster.
For example, the first document in our data — “Used SpaceX rocket as-is, buyer must transport.” — now looks like:
[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1)]
Each set of parenthesis (tuples) represents a word. The first number is where the word lies in our dictionary, and the second number is how many times it occurs in the document.
For (0, 1), 0 represents our bigram “as_is”, which occurs once in the document.
These numbers really just represent a Bag of Words — a term we used earlier meaning the deconstructed, independent collection of terms from a document.
Each one of these tuples represents one dimension in our document-term vector.
We’re ready to calculate cosine similarity for our collection of document-term vectors!
Gensim does all the heavy lifting for us, all we have to use is the right syntax:
#Build similarity indexindex = Similarity(corpus=corpus,                   num_features=len(dictionary),                   output_prefix='on_disk_output')
This instantiates the similarity index, which computes and tracks the cosine similarity of each document against all the others — a pairwise calculation. Gensim is optimized to perform this calculation relatively quickly. The output_prefix argument is simply the name of the file that will be saved to your hard disk — this can be loaded back later if needed.
Let’s break open the index and see what’s inside:
#Parse similarities from indexdoc_id = 0similar_docs = {}for similarities in index:    similar_docs[doc_id] = list(enumerate(similarities))    doc_id += 1
similar_docs is now a Python dictionary that tracks each document, and the similarity scores between it and all other documents in the corpus.
For example, the first entry looks like:
{0: [(0, 0.9999), #“Used SpaceX rocket as-is, buyer must transport.”    (1, 0.9999),  #“Used SpaceX rocket as-is, buyer must transport.”    (2, 0.0),    (3, 0.0),    (4, 0.1690), #“Pale blue tuxedo, used, good condition...    (5, 0.0)], ...}
So the first document (“0” — Python is zero-indexed) in our collection (“Used SpaceX rocket as-is, buyer must transport.”) is exactly the same as itself, and exactly the same as the second document.
It shares 17% similarity with the fifth document (“Pale blue tuxedo, used, good condition. Call 555–555–5555”). We can see that “used” is a shared token.
We’ll use a threshold of 90% similar. Depending on the use-case, we could move this up or down, as discussed above.
sim_threshold = 0.9
The below helper code will return to us the pairs of documents that exceed our threshold:
for doc_id, sim_doc_tuples in similar_docs.items():    for sim_doc_tuple in sim_doc_tuples:        sim_doc_id = sim_doc_tuple[0]        sim_score = sim_doc_tuple[1]if sim_score >= sim_threshold and doc_id != sim_doc_id:            print(f"Found similar documents, score of {sim_score:.2f}:")            print('\t', documents[doc_id])            print('\t', documents[sim_doc_id], '\n')
The results printed include:
There we have it. We were able to validate that the first pair of documents are exact duplicates, and the second are fuzzy duplicates. Feel free to use this test code to play with some of your own data.
That’s it for this guide! You now are familiar with valuable text analytics techniques, and are ready to implement a sophisticated method for fuzzy deduplication on your next project.