5 Lesser-Known Python Libraries for Your Next NLP Project | by Michael Markin | Towards Data Science

When I first began reading on Medium, my favorite types of articles were ones that introduced me to new Python libraries. Several of the libraries I’ve learned about are now part of my everyday projects.
To pass it forward, I wanted to share a collection of 5 awesome Python libraries that I’ve discovered over the past couple of years of doing various Natural Language Processing (NLP) work.
Sure, you can write a long list of regular expressions to expand contractions in your text data (i.e. don’t → do not; can’t → cannot; haven’t → have not). But why not save your efforts and leverage a Python library to do the heavy lifting for you?
Contractions is an easy-to-use library that will expand both common English contractions and slang. It is fast, efficient, and handles most edge cases, such as missing apostrophes.
pip install contractions
import contractionss = "ive gotta go! i'll see yall later."text = contractions.fix(s, slang=True)print(text)
ORIGINAL: ive gotta go! i’ll see yall later.OUTPUT: I have got to go! I will see you all later.
An important part of text preprocessing is creating uniformity and whittling down the list of unique words without losing too much meaning. For instance, bag-of-words models and TF-IDF create large sparse matrixes, in which each variable is a distinct vocabulary word in the corpus. Expanding contractions can further reduce dimensionality, or even help filter out stopwords.
Documentation
Restoring missing punctuation to plain English text... sounds easy right? Well, it’s definitely a lot trickier for a computer to do this.
Distilbert-punctuator is the only working Python library I could find that performs this task. And it’s super accurate as well! That’s because it uses a slimmed-down variation of BERT, which is a state-of-the-art, pretrained language model by Google. It was further fine-tuned on a combination of over 20,000 news articles and 4,000 TED Talk transcripts to detect sentence boundaries. When inserting sentence-ending punctuation, such as periods, the model will also appropriately capitalize the next starting letter.
pip install distilbert-punctuator
Pro tip: This library requires quite a few dependencies. If you have trouble with installation, then try it out on Google Colab, in which case you will need to run !pip install distilbert-punctuator instead.
from dbpunctuator.inference import Inference, InferenceArgumentsfrom dbpunctuator.utils import DEFAULT_ENGLISH_TAG_PUNCTUATOR_MAPargs = InferenceArguments(        model_name_or_path="Qishuai/distilbert_punctuator_en",        tokenizer_name="Qishuai/distilbert_punctuator_en",        tag2punctuator=DEFAULT_ENGLISH_TAG_PUNCTUATOR_MAP    )punctuator_model = Inference(inference_args=args,                              verbose=False)text = [    """however when I am elected I vow to protect our American workforceunlike my opponent I have faith in our perseverance our sense of trust and our democratic principles will you support me    """]print(punctuator_model.punctuation(text)[0])
ORIGINAL: however when I am elected I vow to protect our American workforceunlike my opponent I have faith in our perseverance our sense of trust and our democratic principles will you support meOUTPUT:However, when I am elected, I vow to protect our American workforce. Unlike my opponent, I have faith in our perseverance, our sense of trust and our democratic principles. Will you support me?
Sometimes, you simply want your text data to be more grammatically correct and presentable. Whether the task is fixing messy Twitter posts or chatbot messages, this library is for you.
Documentation
Textstat is an easy-to-use, lightweight library that provides various metrics on your text data, such as reading level, reading time, and word count.
pip install textstat
import textstattext = """Love this dress! it's sooo pretty. i happened to find it in a store, and i'm glad i did bc i never would have ordered it online bc it's petite. """# Flesch reading ease scoreprint(textstat.flesch_reading_ease(text))  # 90-100 | Very Easy  # 80-89  | Easy  # 70-79  | Fairly Easy  # 60-69  | Standard  # 50-59  | Fairly Difficult  # 30-49  | Difficult  # <30    | Very Confusing# Reading time (output in seconds)# Assuming 70 milliseconds/characterprint(textstat.reading_time(text, ms_per_char=70))# Word count print(textstat.lexicon_count(text, removepunct=True))
ORIGINAL:Love this dress! it's sooo pretty. i happened to find it in a store, and i'm glad i did bc i never would have ordered it online bc it's petite.OUTPUTS:74.87 # reading score is considered 'Fairly Easy'7.98  # 7.98 seconds to read30    # 30 words
These metrics add an additional layer of analysis. Say, for example, you are looking into a dataset of celebrity news articles from a gossip magazine. Using textstat, you may find that written pieces that are quicker and easier reads tend to be more popular and have longer retention rates.
Documentation
The primary purpose of this low-code library is to detect gibberish (or unintelligible words). It uses a model that is trained on a large corpus of English words.
pip install gibberish-detector
Next, you will also need to train the model on your end, but it’s very simple and only takes a minute. Just follow these steps:
Download the training corpus called big.txt from hereOpen your CLI and cd over to the directory in which big.txt is locatedRun the following: gibberish-detector train .\big.txt > gibberish-detector.model
Download the training corpus called big.txt from here
Open your CLI and cd over to the directory in which big.txt is located
Run the following: gibberish-detector train .\big.txt > gibberish-detector.model
A file called gibberish-detector.model will be created in your current directory.
from gibberish_detector import detector# load the gibberish detection modelDetector = detector.create_from_model('.\gibberish-detector.model')text1 = "xdnfklskasqd"print(Detector.is_gibberish(text1))text2 = "apples"print(Detector.is_gibberish(text2))
True  # xdnfklskasqd (this is gibberish)False # apples (this is not)
I’ve used gibberish-detector in the past to help me remove bad observations from datasets.
It can also be implemented for error handling on user inputs. For instance, you may want to return an error message if a user enters meaningless, gibberish text on your web app.
Documentation
I’ve saved the best for last. This versatile library is truly a hidden gem.
First off, what is data augmentation? It is any technique that expands the size of a training set by adding slightly modified copies of the existing data. Data augmentation is commonly used when the existing data is either limited in diversity or imbalanced. For computer vision problems, augmentation is used to create new samples by cropping, rotating, and changing the brightness of images. With numerical data, synthesized instances can be created by using clustering techniques.
But what if we are working with text data?
This is where NLPAug comes in. The library can augment text by either substituting or inserting words that are semantically associated.
It does this by employing pretrained language models like BERT, which is a powerful approach because that takes into account the context of words. Based on the parameters you set, the top n number of similar words will be used to modify the text.
Pretrained word embeddings, such as Word2Vec and GloVe, can also be used to replace words with synonyms. Click here for a great article that demonstrates the library’s full list of capabilities.
pip install nlpaug
import nlpaug.augmenter.word as naw# main parameters to adjustACTION = 'substitute' # or use 'insert'TOP_K = 15 # randomly draw from top 15 suggested wordsAUG_P = 0.40 # augment 40% of words within textaug_bert = naw.ContextualWordEmbsAug(    model_path='bert-base-uncased',     action=ACTION,     top_k=TOP_K,    aug_p=AUG_P    )text = """Come into town with me today to buy food!"""augmented_text = aug_bert.augment(text, n=3) # n: num. of outputsprint(augmented_text)
ORIGINAL:Come into town with me today to buy food!OUTPUTS:• drove into denver with me today to purchase groceries!• head off town with dad today to buy coffee!• come up shop with mom today to buy lunch!
Let’s say you are training a supervised classification model on a dataset that has 15k positive reviews, and only 4k negative reviews. A heavily imbalanced dataset such as this will create model bias towards the majority class (positive reviews) during training.
Simply duplicating examples of the minority class (negative reviews) will not add any new information to the model. Instead, utilize the advanced text augmentation features of NLPAug to increase the minority class with variety. This technique has been shown to improve AUC and F1-Score.
Documentation
As data scientists, Kaggle competitors, or programmers in general, it is important that we keep as many tools as possible in our back pocket. We can leverage these libraries to solve problems, enhance our datasets, and spend more time thinking about solutions instead of writing code.
If you are interested in NLP, check out my recent article on Named-Entity-Recognition (NER). I show how to use spaCy — another powerful NLP library — to build a model that automatically identifies words related to home improvement services within unstructured text.
levelup.gitconnected.com
Thanks for reading!