An Introduction to Natural Language Processing (NLP) Terms — part 2
By: Andrew Jamieson
Tags: Naturallanguageprocessing, Deep Learning, Natural Language Process, NLP

Introducing more NLP terms

In an earlier blog I gave an introduction to NLP, how it works, and some beginning terms. In this blog, I’ll add more terms.

Tokenize

This means splitting a document into parts of language. Usually, this means splitting into words, but we can also tokenize sentences, or even individual letters or characters. We tokenize our text to create word embeddings.

“This” “sentence” “is” “tokenized” “by” “words”

Stop Words

Stop words are frequently occurring words that don’t add to the meaning of our text. Words like “the”, “of”, “and”, “a”, “to”. We often want to remove these words to reduce the size of the text document and make sure that meaningful words have more impact. The top 25 words in the English language make up almost a third of all written material.* Removing stop words is an easy way to make a dramatic size reduction in our text document.

* Source: The Reading Teacher’s Book of Lists

Stop Words

However, we don’t always want to remove all stop words. Words that are negations often reverse the meaning of the sentence, so it may not make sense to remove them. The most common negations are: “not”, “no”, “don’t”, “never”, “didn’t”. Even words like “hardly”, “seldom” “a little” can change the meaning of a sentence.

N-grams

N-grams are word combinations that make more sense when they are grouped than when they are separated. For example, the words “Los” and “Angeles” have a much more specific meaning when joined together as “Los Angeles”. Los Angeles is two words — so is known as a bi-gram. In the same way, New York City is a tri-gram.

Stemming

Stemming involves removing the prefix or suffix of words so that only the base of the word remains. (Note: in Python, most common English language stemmers focus on suffixes and not prefixes.) So, playing and play would be treated as the same word. Sometimes you get some weird spelling when the stemmer chops off the end of the word — ponies become poni for example.

A Stemmed Poni … where’s my “y”

Lemmatization

Lemmatization is similar to stemming but more sophisticated. It reduces a word to its root meaning or lemma. This may be a different word. For example the lemma for “am”, “are”, “is”, “being”, “was”, were” is “be”. Also, some libraries like spaCy will convert all pronouns into “PRON”. Lemmatization uses a dictionary like WordNet to retrieve the correct lemma.

Which method should you use? Lemmatization is generally more accurate, but it takes longer to run as it uses a dictionary, rather than an algorithm. Lemmatization doesn’t work with unknown words that are not in the dictionary.

Part of Speech

POS is used to identify the meaning of a word in a sentence. It involves assigning each word its related part of speech. It is useful for determining different meanings of words. For example, in the sentences below, “run” is used twice, but it has different meanings. POS tag can identify that it is a noun in the first sentence and a verb in the second.

The movie had a record-breaking run (noun) at the box office. I had to run (verb) to catch the bus.

Noun Phrase Extraction

This is a way to identify noun phrases in the text and extract them. This can give you some sense of the overall topic of the text. For example, noun phrases for Los Angeles could include: Pacific Ocean, palm trees, entertainment industry, and theme parks

Los Angeles

Keyphrase Extraction

Extracting keywords or phrases from the text that represents the main idea of the content. This provides a concise summary of the text.

Name Entity Recognition

This is a tool to identify important entities in the text. These entities include person, organization, location, time, and date. Here’s an example using the Stanford CoreNLP library, with text from Wikipedia.

Source: Stanford CoreNLP Demo

https://corenlp.run/

TF-IDF

TF-IDF stands for Term Frequency — Inverse Document Frequency. TF-IDF is a method to determine the most important words in each document when compared with a group of documents. The idea is that if a word is common to every document, then it doesn’t add any extra meaning to a specific document, so it can be down-weighted. However, if a word occurs frequently in a specific document, but seldom occurs in other documents, then it does contribute to the meaning of that document.

Sentiment Analysis

Sentiment Analysis is evaluating the feeling or attitude of text, as to whether it is positive, neutral, or negative. This is especially popular in analyzing social media, customer reviews, and brand monitoring. It can even be applied to emojis. There are some drawbacks to sentiment analysis. Context is important — consider “That was a close shave” — great for a razor, terrible for an airport incident. People often use negative words for a positive meaning “That was sick”, and positive words for a negative meaning … especially with irony and sarcasm.

A close shave

Happy NLP’ing.

- Andrew

References

The Reading Teacher’s Book of Lists, 6th Ed, by Jacqueline Kress, Edward Fry

Stanford CoreNLP Library

CoreNLP Demo