Stemming Text with NLTK. Stemming is one of the most used... | by Ivo Bernardo | Towards Data Science

The ability to work with text data is a huge technological achievement that enables companies and organizations to deal with unstructured information and support decisions based on information that was extremely hard to collect and store in the past.
Naturally, as computers don’t really understand characters and words as humans do, this brought a lot of new challenges that are solved using specific techniques that are exclusive to work with text data such as stemming or lemmatization.
In this post, we are going to explore the concept of stemming and how stemming may help in your text mining pipelines.
Stemming, also called suffix stripping, is a technique used to reduce text dimensionality. Stemming is also a type of text normalization that enables you to standardize some words into specific expressions also called stems.
Let’s start with an example of a sentence:
‘This is definitely a controversy as the attorney labeled the case “extremely controversial”’
If we want to represent this text as an array (check out another post from this series) the words controversy and controversial would be considered different tokens and would have different positions in any array you might build.
Stemming helps us to standardize words that share the same suffix and that normally are derivations of gramatically similar words.
But, as usual, in several text mining pipelines, there are a lot of choices involved that may cause loss of information. Keep in mind that it’s an ongoing discussion in the NLP field if stemming words is really a good idea given the computing power and more advanced models that we are able to apply in our NLP pipeline today (Deep Neural Networks, as example).
Stemming may be a good idea for algorithms that may deal poorly with high number of dimensions. Recall that stemming is a normalization technique but as most normalization problems (principal component analysis, for instance) it causes you to lose some of the original information that you have on your text. If it is a good or bad application for your NLP pipeline it really depends on the model you are using, the amount of space you have to save your data (although nowadays space should not be a problem) and the characteristics of the algorithm that you want to use.
Given that, let’s explore the most common stemmers available in the NLTK (Natural Language Toolkit) library.
Porter’s Stemmer is actually one of the oldest stemmer applications applied in computer science. It first mention was in 1980 in the paper An algorithm for suffix stripping by Martin Porter and it is one of the widely used stemmers available in nltk.
Porter’s Stemmer applies a set of five sequential rules (also called phases) to determine common suffixes from sentences. As an example, phase one starts by removing plural and past participles from sentences such as the word cats that is transformed into cat:
cats -> cat
From phase two until phase five, Porter’s algorithm stems common word endings that are applied according to different combinations of preceding consonants and vowels.
A detailed description of each phase is available here.
One of the most important concepts for porter stemmer is the concept of m in the following formula:
Each C is considered a string with one or more consonants. Each V is considered a string with one or more vowels. Vowels are considered the letters a, e, i, o ,u and y with certain restrictions.
The m is one of the most important concepts in Porter’s stemmer because it defines the threshold for apllying certain rules to a specific word. Let’s see some examples of words:
Tree -> This word can be represented by the representation CV because TR is considered a C representation and EE is considered a V representation. The m in this case is zero because this word only fits into the format [C][V]
Creating -> This word can be represented by CVCVC. To fit the Porter formula you can represent this by [C][VC]2, hence m is equal to 2.
Oats -> This word can be represented by VC that fits into Porter Formula as[VC]1. In this case m is equal to 1.
Keep this concept of m in your mind because it will be important for some explanations below.
We have Porter’s Stemmer available right of the bat in the nltk library:
from nltk.stem import PorterStemmer
To use it, one has to declare an instance of PorterStemmer, after importing it:
porter = PorterStemmer()
With the instance of PorterStemmer defined we can stem words based on the rules defined on the algorithm:
porter.stem('cats')
This command results in the word: ‘cat’ — the stemmed version of the word cats. Let’s see more common examples:
porter.stem('amazing') returns ‘amaz’
This stem happens because ing is such a common termination in english words that the word amazing gets stemmed into the word amaz. The amaz stem is also produced by the following words, Amazement, Amaze and Amazed:
porter.stem('amazement') returns ‘amaz’porter.stem('amaze') returns ‘amaz’porter.stem('amazed') returns ‘amaz’
All these 4 words, amazement, amaze, amazed and amazing are considered the same word in our text after the stemming process — that’s why we call this text normalization!
As a counter example, the word amazon is not stemmed to amaz:
porter.stem('amazon') returns ‘amazon‘
Why was the on in amazon not removed? This is because certain word endings such as ON are not considered in Porter’s algorithm — only when they are preceeded by an i for words such as nation, premonition, etc. Let’s test the stemming for these two examples:
porter.stem('nation') returns ‘nation‘porter.stem('premonition') returns 'premonit'
And now a new question unfolds — Why was premonition stemmed and nation was not?
Certain word endings are only considered for a certain threshold that resembles word size (our m!). If you check the Porter rules you will see that the suffix ION or TION are only applied on the second and fourth phase within certain conditions based on m.
Let us go step by step through each phase to really understand the rules behind Porter’s Stemmer. As the stemmer is applied sequentially, more than one rule may be applied, for example, the word conditional (m is equals to 4) follows the following flow through the Stemmer phases:
Phase 1: It doesn’t match any of the patterns that this step considers such as ending with IES or SSES and other rules present in phase 1b and 1c.
Phase 2: As m is greater than 0 and the word ends with tional the stemmer applies a cropping of the suffix al and our word becomes condition.
Phase 3: Our word condition (notice that now we treat our word as condition and not as the original conditional as we’ve applied something in phase 2).
Phase 4: The word ends with ion and has m>1 so the suffix ion is stripped and we get the stem condit.
Phase 5: Our word condit does not match any pattern that can be stripped in phase 5 so our final stem is condit.
As you can see in the process above, our word may be stemmed several times throughout the phases of the stemming process.
Return to this link if you need to check the list of rules applied in each phase.
Porter’s Stemmer is one of the most used stemming techniques that one can use in Natural Language Processing but as it’s been almost 30 years since it’s first implementation and development, Martin Porter developed an updated version called Porter2 that is also commonly called Snowball Stemmer due to it’s nltk implementation.
Snowball stemmer (formally called Porter2) is an updated version of Porter’s Stemmer with new rules that were introduced modifying some of the existing ones already existing in Porter’s Stemmer.
The detailed rules of Snowball / Porter2 are found here.
The logic and process is exactly the same as Porter’s Stemmer, the word is stemmed sequentially throughout the five phases of the stemmer.
Another relevant difference between Snowball and Porter is that Snowball also details rules for other languages — as an example, you can access the rules for Portuguese language here.
You can access Snowball in nltk pretty easily, similarly to how we’ve accessed Porter’s Stemmer — here we also define the instance at the same time, with the catch that we have to define the language as an argument:
from nltk.stem import SnowballStemmersnowball = SnowballStemmer(language='english')
Other than reviewing extensively Snowball’s rules, let’s check some important differences between Porter and Snowball using some examples:
porter.stem('fairly') -> returns fairlisnowball.stem('fairly') -> returns fair
In the example below, Snowball is much better on normalizing the adverb fairly, having it produce the stem fair, while Porter’s produce the stem fairli. Doing so makes the stem of the word fairly the same as the adjective fair, which seems to make sense from a normalization perspective.
This rule also happens with other adverbs:
porter.stem('loudly') -> returns loudlisnowball.stem('loudly') -> returns loud
Another improvement in Snowball is that its rules prevent some overstemming (cropping too much suffix from the word). Let’s see the examples of the words generically and generous —starting with Porter’s first:
porter.stem('generically') -> returns generporter.stem('generous') -> returns gener
With Porter’s Stemmer, these words that convey completely different meanings, would be stemmed into the same expression gener.
With Snowball, that does not happen as both words are stemmed into different expressions:
snowball.stem('generically') -> returns genericalsnowball.stem('generous') -> returns generous
In conclusion, SnowballStemmer is widely accepted as an improved version from Porter’s Stemmer.
The nltk implementation is based on Porter2, an improvement of the original algorithm developed by Martin Porter.
Lancaster Stemmer is a stemmer developed and presented in the paper Another Stemmer by Chris Paice from Lancaster University.
Its rules are more agressive than Porter and Snowball and it is one of the most agressive stemmers as it tends to overstem a lot of words.
As a general rule of thumb, think that the rules of Lancaster’s Stemmer try to reduce the word to the shortest stem possible. You can check some of Lancaster’s rules here.
Here is nltk’s implementation of Lancaster’s Stemmer:
from nltk.stem import LancasterStemmerlanc = LancasterStemmer()
Let’s see some examples of how words are stemmed with Lancaster’s Stemmer, comparing the results with Snowball Stemmer approach — beginning with the word salty:
snowball.stem('salty') returns 'salti'lanc.stem('salty') returns 'sal'
Now comparing with the word sales:
snowball.stem('sales') returns 'sale'lanc.stem('sales') returns 'sal'
As you can see, Lancaster stems the words salty and sales with the same stem , something that does not make much sense in terms of the meaning of both words. This happens because Lancaster tends to overtstem some words and this is the main reason why we say that Lancaster Stemming is one of the most aggressive stemmers.
Now that we’ve understood how stemming normalizes text by cropping suffixes, how can we understand the impact of this normalization in our corpus?
As stemming is a normalization technique that causes loss of information one of the most straightforward ways is to understand the ratio between the length of the text after and before stemming. This will give you, as an approximation, the percentage of information you retain after stemming.
For this example, let’s use the first and second paragraph of the European Union definition article in Wikipedia to evaluate our stemmers:
eu_definition = 'The European Union (EU) is a political and economic union of 27 member states that are located primarily in Europe. Its members have a combined area of 4,233,255.3 km2 (1,634,469.0 sq mi) and an estimated total population of about 447 million. The EU has developed an internal single market through a standardised system of laws that apply in all member states in those matters, and only those matters, where members have agreed to act as one. EU policies aim to ensure the free movement of people, goods, services and capital within the internal market; enact legislation in justice and home affairs; and maintain common policies on trade, agriculture, fisheries and regional development. Passport controls have been abolished for travel within the Schengen Area. A monetary union was established in 1999, coming into full force in 2002, and is composed of 19 EU member states which use the euro currency. The EU has often been described as a sui generis political entity (without precedent or comparison).The EU and European citizenship were established when the Maastricht Treaty came into force in 1993. The EU traces its origins to the European Coal and Steel Community (ECSC) and the European Economic Community (EEC), established, respectively, by the 1951 Treaty of Paris and 1957 Treaty of Rome. The original members of what came to be known as the European Communities were the Inner Six: Belgium, France, Italy, Luxembourg, the Netherlands, and West Germany. The Communities and their successors have grown in size by the accession of new member states and in power by the addition of policy areas to their remit. The United Kingdom became the first member state to leave the EU on 31 January 2020. Before this, three territories of member states had left the EU or its forerunners. The latest major amendment to the constitutional basis of the EU, the Treaty of Lisbon, came into force in 2009.'
Let’s pass this on our Porter’s Stemmer and see the outcome. To pass the entire text on nltk’s implementation we have to tokenize our text using the word_tokenize function:
from nltk.stem import PorterStemmerfrom nltk.tokenize import word_tokenizetokenized_eu = word_tokenize(eu_definition)porter_eu = [porter.stem(word) for word in tokenized_eu]
Let’s check the resulting stemmed version:
'the european union ( EU ) is a polit and econom union of 27 member state that are locat primarili in europ . it member have a combin area of 4,233,255.3 km2 ( 1,634,469.0 sq mi ) and an estim total popul of about 447 million . the EU ha develop an intern singl market through a standardis system of law that appli in all member state in those matter , and onli those matter , where member have agre to act as one . EU polici aim to ensur the free movement of peopl , good , servic and capit within the intern market ; enact legisl in justic and home affair ; and maintain common polici on trade , agricultur , fisheri and region develop . passport control have been abolish for travel within the schengen area . A monetari union wa establish in 1999 , come into full forc in 2002 , and is compos of 19 EU member state which use the euro currenc . the EU ha often been describ as a sui generi polit entiti ( without preced or comparison ) .the EU and european citizenship were establish when the maastricht treati came into forc in 1993 . the EU trace it origin to the european coal and steel commun ( ecsc ) and the european econom commun ( eec ) , establish , respect , by the 1951 treati of pari and 1957 treati of rome . the origin member of what came to be known as the european commun were the inner six : belgium , franc , itali , luxembourg , the netherland , and west germani . the commun and their successor have grown in size by the access of new member state and in power by the addit of polici area to their remit . the unit kingdom becam the first member state to leav the EU on 31 januari 2020 . befor thi , three territori of member state had left the EU or it forerunn . the latest major amend to the constitut basi of the EU , the treati of lisbon , came into forc in 2009 .'
This is the stemmed version using Porter’s stemmer for the eu_definition text. It’s ininteligible for a human but remember that the main goal of stemming is to normalize the text to improve the generalization of machine learning algorithms. Our stemmed version contains 1430 characters, excluding spaces. We can obtain this using:
len(''.join(porter_eu))
Our original sentence contained 1591 characters, and we can obtain this using:
len(''.join(word_tokenize(eu_definition)))
The percentage of information retained is the ratio between these numbers (1430/1591), around 89.88%.
If we apply the same logic for the same text with Snowball and Lancaster, we have the following values:
Snowball: 90.88%
Lancaster: 78.44%
As expected, the stemmer that retains less information is the Lancaster Stemmer.
Here’s a small gist with code if you want to use stemming in your projects:
And that’s it! Stemming is one of the most commonly used text normalization techniques that may fit your Natural Language Pipeline when you want to reduce your vocabulary (distinct words) that will be used throughout your code. But remember, as with any normalization, everytime you apply stemming you will lose some of the original information from your corpus.
Even so, stemming is a good tool for data scientists to have in their text mining toolkit, as it is still widely used throughout NLP applications and may benefit both supervised and unsupervised approaches.
Thank you for taking the time to read this post! Feel free to add me on LinkedIn (https://www.linkedin.com/in/ivobernardo/) and check my company’s website(https://daredata.engineering/home).
If you are interested in getting training on Analytics you can also visit my page on Udemy (https://www.udemy.com/user/ivo-bernardo/)
This example is taken from my NLP course for absolute beginners available on the Udemy platform — the course is suitable for beginners and people that want to learn the fundamentals of Natural Language Processing. The course also contains more than 50 coding exercises that enable you to practice as you learn new concepts.