Correcting your spelling error via 2 distance | by Edward Ma | Towards Data Science

When dealing with text, we may need to deal with incorrect text. Although we can still use character embeddings and word embeddings to compute a similar vectors. It is good for unseen data and out-of-vocabulary (OOV). However, it will be better if we can correct typo.
Typo can be generated in several scenarios. If you work on optical character recognition (OCR), post-processing step for OCR output is very critical part as some error will be introduced from OCR engine and it can be caused by bad quality of image and OCR engine error. Another typo source comes from human. When you work on chatbot project, the input comes from human and it must include typos.
To achieve a better result, it will be a better to correct typo as earlier as we can. After reading this post, you will understand:
Spelling Corrector
Implementation
Take Away
Norvig implemented a very simple but amazing library to correct the spelling error in 2007. Possible candidate corrections are computed by different ways and finding the most likelihood word from there. There are 2 phases to find the possible candidate words.
First of all, it uses 4 different ways to generate new word while the edit distance between original word and candidate word is 1. Difference from Levenshtein Distance, it considers:
Deletion: Remove one letter
Transposition: Swap two adjacent letters
Replacement: Change one letter to another
Insertion: Add one letter
Taking “edward” as an example, “edwar”, “edwadr”, “edwadd”, “edwward” are examples of “Deletion”, “Transposition”, “Replacement” and “Insertion” respectively. Obviously, tons of invalid words will be generated. Therefore, it will filter out by a given vocabulary (it call “known word” in library). To expand potential candidates, algorithm repeat this step again but the edit distance is 2.
Second part is selecting candidate from possible candidates based on probability. For example, the occurrences of “Edward” in the given dictionary is 2%, the probability is 0.02. The highest probability word will be chosen from potential candidates.
To facility the spell check, corpus is needed. For sake of easier for demonstration, I simply use dataset from sklearn library without pre-processing. You should use your domain specific dataset to build a better corpus for your data.
Build corpus
from collections import Counterfrom sklearn.datasets import fetch_20newsgroupsimport recorpus = []for line in fetch_20newsgroups().data:    line = line.replace('\n', ' ').replace('\t', ' ').lower()    line = re.sub('[^a-z ]', ' ', line)    tokens = line.split(' ')    tokens = [token for token in tokens if len(token) > 0]    corpus.extend(tokens)corpus = Counter(corpus)
Correction
spell_corrector = SpellCorrector(dictionary=corpus)spell_corrector.correction('edwar')
Output is
edward
To access all code, you can visit my github repo.
Spell corrector does not consider the context but just the spelling purely. However, given that it is introduced in 11 years ago (2007). It is an amazing tool.
From author coding, the preprocessing result should only keep English character and lower case. In other word, special character and number should be removed.
Performance (in terms of speed) is very fast.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. You can reach me from Medium Blog, LinkedIn or Github.