Natural Language Processing: Beginner to Advanced
By: Ishan Singh
Tags: Machine Learning, Artificial Intelligence, Data Science, NLP, Naturallanguageprocessing

The NLP Project

Natural Language Processing: Beginner to Advanced

Part-3: Advanced Lexical Processing — how to handle miscellaneous noise in the textual data and how to build your own spelling corrector.

Image by Lorenzo Cafaro from Pixabay

In the previous part of the series ‘The NLP Project’, we learned all the basic lexical processing techniques such as removing stop words, tokenization, stemming, and lemmatization.

Even after going through all those preprocessing steps, a lot of noise is still present in the textual data. For example, spelling mistakes that happen by mistake as well as by choice (informal words such as ‘lol’, ‘u’, ‘gud’ etc.). Also, the problem of spelling variations of a word that occurs due to different pronunciations (e.g. Mumbai, Bombay).

The two methods that we talked about in the last part — stemming and lemmatization — are both parts of a technique called canonicalization. Basically, canonicalization means reducing a word to its base form.

There are scenarios where we can not canonicalize a word just by using stemming and lemmatization. Thus, we will need another technique to canonicalize the words correctly. For example, if the word ‘allowing’ is misspelled as ‘alowing’, then we will have a redundant token as ‘alow’ after stemming the misspelled word, and lemmatization wouldn’t even work as it only works on correct dictionary spelling.

The same problem is with the pronunciation of same words in different patois. For example, the word ‘humour’ used in British English, is spelled as ‘humor’ in American English. Both are correct spellings, but they will give two different base forms, after stemming, when used in the form ‘humouring’ and ‘humoring’.

There are some words that have different pronunciation in different languages. As a result, they are ultimately written differently. For example a common Indian surname like ‘Srivastava’ is also spelled as ‘Shrivastava’, ‘Shrivastav’, or ‘Srivastav’. Therefore, we must reduce all forms of a particular word to one single and common word.

To attain this, we will get to know about a technique is called phonetic hashing. The phonetic hashing method combines the same phonemes (smallest unit of sound) into one bucket and gives them the same hash code for all the variations. So, the words ‘color’ and ‘colour’ have the same code.

Phonetic hashing is a technique used to canonicalize words that have different variants but the same phonetic characteristics, that is, the same pronunciation.

Now, Soundexes are algorithms that can be used to calculate the hash code of a given word. Phonetic Hashing uses these Soundex algorithms. The algorithms differ from language to language. American Soundex is the most popular Soundex algorithm and we will be using that. Also, it doesn’t matter which language the input word comes from — as long as the words sound similar, they will get the same hash code.

Phonetic hashing is a four-character code. So, let us calculate the hash code(also known as Soundex) of the word ‘Chennai’ by applying the following steps:-

Retain the first letter of the word as is — Soundex is based on the rationale that the English pronunciation depends on the first letter and pattern of the consonants. So, C becomes the first letter. Remove all the vowels. So we have ‘CHNN’ now. Replace consonants with digits as follows (after the first letter)

After encoding the code becomes C55.

4. If two or more letters with the same number are adjacent in the original name (before step 1), only retain the first letter; also two letters with the same number separated by ‘h’ or ‘w’ are coded as a single number, whereas such letters separated by a vowel are coded twice. This rule also applies to the first letter. Now, we have two 5’s, so we will merge them into one and get C5.

5. If there are too few letters in our word that we can’t assign three numbers, append with zeros until there are three numbers. If we have four or more numbers, retain only the first three. Since we have one letter and only one number, we will append two zeroes at the end of the encoding to make it a four-character code. Hence, we get C500 as the final code for ‘Chennai’.

Thus, we see that the Soundex code for city names like ‘Bengaluru’ and ‘Bangalore’ is the same, B524, which is what we were aiming for.

So, we saw how to deal with different pronunciations of a particular word. Next, let us see how to deal with misspellings.

Simply put, an edit distance is the minimum number of edits required to convert one string to another, that is, given string to the target string.

Now, you must be wondering what is an edit. So, the possible edits are as follows:-

Insertion of a character into a given string. To convert the misspelled word ‘Aquire’ to its correct spelling ‘Acquire’, we need to insert the character ‘c’ in the given misspelled string.

of a character into a given string. To convert the misspelled word ‘Aquire’ to its correct spelling ‘Acquire’, we need to insert the character ‘c’ in the given misspelled string. Deletion of a character from a given string. To convert ‘neccessary’ to ‘necessary’, we need to delete the extra ‘c’ from the given string.

of a character from a given string. To convert ‘neccessary’ to ‘necessary’, we need to delete the extra ‘c’ from the given string. Substitution of a character in a given string. To convert ‘Absense’ to its correct form ‘Absence’, we need to substitute the character ‘s’ with the character ‘c’.

The most common edit distance algorithm is the Levenshtein edit distance which you can learn about, in detail, here. There is another edit option that you can do called the ‘transpose operation’, where you can swap the two adjacent characters of a given string. But, this operation is only available in the Damerau-Levenshtein edit distance.