Implementation of Basic NLP Techniques with spaCy | Towards Data Science

Natural Language Processing (NLP) is a very interesting technique because, with this technique, the computer can recognize our natural language and respond like an intelligent person. At first, when I came to know about the magic of NLP, it was amazing to me.
[If you want to know about my journey to NLP as well as the basics of NLP, I will strongly recommend you to read out the article — A Complete Guideline to Natural Language Processing (NLP).]
I believe practical experience is the best way to learn a technique, and only theoretical knowledge doesn’t help. Apple’s Siri, Microsoft’s Cortana, Google’s Voice Assistance, Amazon’s Alexa, etc., all of them are using NLP. How interesting their works are! And hopefully, we will also create some interesting projects.
So, it’s time to make our hands dirty by having a hands-on experience of implementing tokenization, lemmatization, stemming, stop word, POS tagging, etc. We also learn which python library is best for practical use and why.
Let’s get started.........
Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data[1].
Both spaCy and NLTK are open-source Natural Language Processing libraries.
◉ spaCy
spaCy first came into the spotlight in 2015 as an open source NLP library. It can perform most of the NLP tasks efficiently. spaCy mainly implements the most common efficient NLP techniques for performing the task. There is no option to select a specific algorithm to perform a task. For example, there is more than one technique for stemming, and you want to choose one of them. Unfortunately, spaCy does not provide us the flexibility; rather, it automatically selects the best commonly used algorithm to perform the task.
If you want to know more about spaCy, please visit the official website of spaCy [2].
◉ NLTK
NLTK stands for Natural Language Toolkit, one of the oldest and popular Natural Language Processing Library. It was first released in 2001, which means it is much older than spaCy. It facilitates many NLP functionalities, but it is less efficient. You can select different algorithms in NLTK.
You may have the official documentation of NLTK from here[3].
➤ Comparative Analysis of NLTK and spaCy
Both NLTK and spaCy are important libraries for NLP. Some tasks are efficient in spaCy, and some are efficient in NLTK. We need to use both libraries in different circumstances, and let’s see some comparisons.
The above figure clearly indicates that spaCy is 20 times and 443 times faster than NLTK in the case of tokenization and tagging.
We can not classify the libraries based on superiority; instead, both are needed in different cases. In general, spaCy works better than NLTK in comparison to the speed and implementation, but NLTK is also required.
Throughout the article I will show you the basic implementation of NLP tasks like tokenization, stemming, lemmatization, POS tagging, text matching, etc. Let’s make our hands dirty with some code. First of all, I will show how to create the environment for spaCy.
If you use Anaconda, you must write the following command to the Anaconda command prompt.
conda install -c conda-forge spacyor pip install -U spacy
If you locally use libraries other than Anaconda, you may use the following command.
pip install -U spacy
As we use the English language for the demonstration purpose, we need to download the necessary elements of the English language. Run the command to the command prompt.
N.B. — You must have to run this command as admin or use sudo.
python -m spacy download en
◉ Working with spaCy
import spacynlp = spacy.load('en_core_web_sm')
When we run the following piece of code, it will import the spaCy library and load the model to nlp . If we call the object nlp on a text, spaCy will tokenize the text and save it to a Doc object. After that, the Doc object is again passed through different steps, and the process is known as a pipeline. Generally, the pipeline contains tagger, lemmatizer, parser, and named entity recognizer. The following image clearly describes the pipeline process[5].
By running the code nlp.pipepline, we may find out the components of the pipeline. There are some token attributes for finding the original token text, lemmatization, pos tagging, etc. In the following table, there are some other attributes.
The tokenization technique splits the text into its small units. Let’s have an example[6].
doc = nlp('"Let\'s go to N.Y.!"')for token in doc:    print(token.text)
If we run the code, we will find the output as below.
"Let'sgotoN.Y.!"
The doc object contains tokens of the text, and the attribute .text holds the original text. spaCy is very smart as it doesn’t separate N.Y. as well as consider ‘s as a token. The following image clearly describes the process of tokenization.
In a sentence, a word or a group of words indicates names (e.g., Name of location, person, area, country, state, monetary values, and so on). The primary purpose of a named entity is to identify it. The named entities can be accessible with ents attributes of Doc object.
Output of the code
Google | 's | monthly | revenue | is | $ | 6 | million | ______________________________________________________Google - ORG - Companies, agencies, institutions, etc.monthly - DATE - Absolute or relative dates or periods$6 million - MONEY - Monetary values, including unit
The first line of the output shows the tokens of the sentence, and the rest of the lines indicate the named entities along with the proper documentation. By using the .noun_chunks attribute, we will get the base noun phrase.
spaCy has a built-in visualizer to visualize the named entity. For doing so, we have to import displacy module from spaCy library. The following code shows the output as follows.
from spacy import displacydoc = nlp(u'Over last few years USA generates $6 million revenue.')displacy.render(doc, style='ent', jupyter=True)
Output
With spaCy, we can also visualize the dependencies of the token of a sentence. For demonstration purposes, follow the piece of the following code.
from spacy import displacydocument = nlp(u'Bangladesh is a beautiful country')displacy.render(document, style='dep', jupyter=True, options={'distance': 110})
Output
Stemming means chopping the words/ tokens to find out their root words. But it doesn’t always provide meaningful strings. The spaCy library doesn’t have any stemmer; instead, it entirely relies on lemmatization.
◉Porter Stemmer
Porter stemmer has its own mapping rules by which words are chopped in different stages. The algorithm uses five phases for reducing the word. Firstly, the suffixes are removed with some predefined rules, which are given below.
Here, S1 denotes the suffixes of longer text, and S2 denotes the reduced form after the reduction of the word. At a time, only one rule can be applied.
The above rules are more complex than the first rules. The m denotes the measure of the length. If you want to know more about the porter stemmer algorithm, please visit the link[7]. Let’s have a coding example.
import nltkfrom nltk.stem.porter import *p_stemmer = PorterStemmer()words = ['run','runner','running','ran','runs','easily','fairly']for word in words:    print(word+' --> '+p_stemmer.stem(word))
Output
run --> runrunner --> runnerrunning --> runran --> ranruns --> runeasily --> easilifairly --> fairli
◉Snowball Stemmer
The stemmer is an improved version of the porter stemmer, and it gives better results than the previous stemming algorithm in case of logic and speed. Let’s implement with NLTK.
from nltk.stem.snowball import SnowballStemmers_stemmer = SnowballStemmer(language='english')words = ['run','runner','running','ran','runs','easily','fairly']for word in words:    print(word+' --> '+s_stemmer.stem(word))
Output
run --> runrunner --> runnerrunning --> runran --> ranruns --> runeasily --> easilifairly --> fair
It seems the result is slightly improved.
Lemmatization is not just reducing or chopping the words like stemming. The main idea of lemmatization is to find out the main root word. Please read out my previous article from here to learn more about lemmatization.
The above code produces the following output
He           PRON   561228191312463089     -PRON-is           VERB   10382539506755952630   bea            DET    11901859001352538922   arunner       NOUN   12640964157389618806   runnerrunning      VERB   12767647472892411841   runin           ADP    3002984154512732771    ina            DET    11901859001352538922   acompetition  NOUN   4661638505416061516    competitionbecause      ADP    16950148841647037698   becausehe           PRON   561228191312463089     -PRON-loves        VERB   3702023516439754181    loveto           PART   3791531372978436496    torun          VERB   12767647472892411841   runsince        ADP    10066841407251338481   sincehe           PRON   561228191312463089     -PRON-ran          VERB   12767647472892411841   runtoday        NOUN   11042482332948150395   today
The .pos_ attribute returns the parts of speech for a token, .lemma provides the hash value for a particular token, and .lemma_ gives the lemma for a token. If we observe the token running, run, ran, we will find it comes from the same lemma run. It’s really an interesting and amazing job.
Stop words come so frequently in the sentence but don’t important to change the meaning of a sentence. Like a, an, the, then, etc. In the spaCy library, there are 305 stop words by default. We can modify the list of the stop words according to our needs. Let’s have a codified example.
import spacynlp = spacy.load('en_core_web_sm')print(nlp.Defaults.stop_words)
Output
{'hers', 'show', 'though', 'various', 'sixty', 'say', 'quite', 'ten', 'anything', 'although', 'hereby', 'in', 'ours', 'herself', 'among', 'unless', 'and', 'whole', 'anywhere', 'latter', 'therein', 'whereafter', 'that', 'one', 'whose', 'either', 'within', 'eight', 'three', 'latterly', 'anyone', 'a', 'less', 'former', 'been', 'same', 'anyway', 'else', 'cannot', 'five', 'i', 'until', 'last', 'thus', 'give', 'move', 'thereafter', 'via', 'than', 'empty', 'off', 'neither', 'too', 'please', 'over', 'just', 'otherwise', 'has', 'her', 'put', 'its', 'whether', 'herein', 'myself', 'me', 'nevertheless', 'whatever', 'someone', 'towards', 'whereby', 'onto', 'sometimes', 'thence', 'them', 'done', 'at', 'back', 'nor', 'another', 'behind', 'together', 'take', 'amongst', 'being', 'seemed', 'seeming', 'fifteen', 'do', 'further', 'something', 'again', 'this', 'were', 'wherein', 'how', 'up', 'must', 'get', 'whereas', 'much', 'upon', 'yet', 'both', 'many', 'very', 'may', 'after', 'regarding', 'full', 'through', 'below', 'his', 'well', 'everything', 'so', 'our', 'should', 'seem', 'while', 'for', 'might', 'mine', 'when', 'with', 'you', 'few', 'never', 'because', 'own', 'also', 'due', 'hence', 'it', 'more', 'their', 'such', 'becomes', 'first', 'hereupon', 'since', 'third', 'twenty', 'who', 'she', 'nobody', 'name', 'really', 'enough', 'least', 'two', 'whoever', 'which', 'yours', 'moreover', 'seems', 'before', 'therefore', 'then', 'used', 'even', 'nowhere', 'without', 'other', 'around', 'made', 'hundred', 'no', 'twelve', 'several', 'your', 'meanwhile', 'per', 'except', 'yourselves', 'why', 'some', 'not', 'yourself', 'sometime', 'somehow', 'become', 'beyond', 'almost', 'will', 'somewhere', 'the', 'everyone', 'about', 'everywhere', 'anyhow', 'side', 'next', 'fifty', 'they', 'most', 'perhaps', 'across', 'themselves', 'besides', 'against', 'can', 'him', 'there', 'noone', 'under', 'formerly', 'already', 'all', 'if', 'my', 'or', 'serious', 'four', 'thereupon', 'whence', 'here', 'whither', 'beside', 'wherever', 'to', 'himself', 'between', 'ourselves', 'none', 'on', 'became', 'an', 'have', 'part', 'did', 'had', 'each', 'six', 'those', 'from', 'whenever', 'any', 'am', 'would', 'make', 'could', 'does', 'go', 'call', 'indeed', 'these', 'often', 'above', 'during', 'by', 'nine', 'thereby', 'others', 'afterwards', 'throughout', 'whom', 'amount', 'as', 'hereafter', 'top', 'mostly', 'us', 'whereupon', 'once', 'only', 'still', 'namely', 'forty', 'ca', 'along', 'be', 'itself', 'where', 'see', 'into', 'toward', 'but', 'is', 'keep', 'bottom', 'ever', 'becoming', 'every', 'always', 'front', 'nothing', 'we', 'of', 'out', 'eleven', 'alone', 'he', 'however', 'rather', 'down', 'thru', 'now', 'using', 'are', 'doing', 'what', 'beforehand', 're', 'was', 'elsewhere'}
◉Checking a specific token whether it is a stop word or not.
nlp.vocab['myself'].is_stop
It returns True as the default list of stop words contains the word ‘myself’.
nlp.vocab['mystery'].is_stop
The code returns False as the default list of stop words doesn’t contain the word ‘mystery’ .
◉Adding and removing a stop word to the default spaCy stop words list
The code nlp.Defaults.stop_words.add(‘btw’) adds the string btw to the default list and nlp.Defaults.stop_words.remove(‘hers’) removes the word hers from the default stop word list.
spaCy offers rule-based matching tools named Matcher, it allows us to set rules or regular expressions to match with a Doc object, and it returns a list containing the found matches. To learn more, visit the link[7].
◉Rule-based Matcher
➤Importing Matcher library and creating a matcher object.
from spacy.matcher import Matchermatcher = Matcher(nlp.vocab)
➤Creating a pattern and add it to the matcher
pattern_1 = [{'LOWER': 'solarpower'}]pattern_2 = [{'LOWER': 'solar'}, {'LOWER': 'power'}]pattern_3 = [{'LOWER': 'solar'}, {'IS_PUNCT': True}, {'LOWER': 'power'}]matcher.add('SolarPower', None, pattern1, pattern2, pattern3)
The above piece of code contains three patterns.
pattern_1 denotes any match with the lowercase solarpower .
pattern_2 denotes any match with adjacent lowercase solar and power.
pattern_3 denotes any match with lowercase solar and power along with any punctuation mark between them.
➤Applying the matcher to a doc object.
document = nlp(u'The Solar Power industry continues to grow as demand \for solarpower increases. Solar-power cars are gaining popularity.')found_matches = matcher(document)print(found_matches)
matcher returns a list of tuples. Each tuple contains an ID for the match, with start & end tokens that map to the span doc[start:end].
➤Finding out the text for the matches.
for match_id, start, end in found_matches:    # get string representation    string_id = nlp.vocab.strings[match_id]    # get the matched span    span = doc[start:end]           print(match_id, string_id, start, end, span.text)
Output
8656102463236116519 SolarPower 1 3 Solar Power8656102463236116519 SolarPower 10 11 solarpower8656102463236116519 SolarPower 13 16 Solar-power
The match_id is simply the hash value of the string_ID 'SolarPower'.
◉Phrase Matcher
If you want to learn about PhraseMatcher, please visit the link [8].
NLP is one of the amazing techniques in modern computing by which computers can recognize our Natural Languages. There are some techniques by which computer processes the natural language because the human and machine learning process is different. Different libraries like spaCy, NLTK have made the process easy for us. It is impossible to accommodate all necessary techniques in a single article. So, keep your eyes on the upcoming articles for more amazing implementation techniques and projects of NLP. You may also read out the following articles on NLP written by me.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
[1]. https://en.wikipedia.org/wiki/Natural_language_processing
[2]. spaCy · Industrial-strength Natural Language Processing in Python
[3]. NLTK :: Natural Language Toolkit