Popular Python Libraries in NLP: Dealing with Language Detection, Translation & Beyond!
By: Pallavi Ahuja
Tags: Named Entity Recognition, Naturallanguageprocessing, Python Libraries, Language Translation, Machine Learning

The internet is flooded with articles and posts for detecting the language of texts and then translating it to any other language using Machine Learning or Deep Learning LSTM Models, transfer learning transformer Models like BERT, GPT-2, etc. and building a deep neural network for developing your own translation model.

But if you don’t want to write long codes and develop your own algorithms then there are certain python libraries which come handy to do these tasks on huge data and they work quite well mostly.

Following are few libraries which make your life easy for such tasks and are multi functional.

Google Translate:

This library comes with the name Googletrans. Googletrans is a free and unlimited library which implemented Google Translate API to make calls to such methods like detect and translate. It is fast and reliable and uses same servers that translate.google.com uses.

Easy to install: “pip install googletrans”

It supports multiple languages including some of the indian languages which can be listed by below python code.

It detects multiple texts in different languages with the confidence of detection result which lies between 0.0 and 1.0 .

Translating from one to other language is also quite simple as shown by below code.

If we don’t give source language of the text, this library automatically tries to detect it and translates it to given destination language. If no destination language is given as shown in above sample code, then it tries to translate the given text to set language of the machine on which this runs. In this case it translated Korean to English, as my system’s default set language is English.

If you want to use translated text for your text pre-processing then use below sample code:

So here, I have tried to translate German to Hindi. Isn’t it cool? Let’s move to next library.

TextBlob:

Textblob is a python library for text processing and mainly used for natural language processing tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more that can be used to detect and translate a language.

Easy to install: “pip install textblob”

We also need to download necessary NLTK corpora by using below python statement:

python -m textblob.download_corpora

Language Detection:

Translation:

Sentiment analysis:

The sentiment function returns two properties-polarity and subjectivity. the polarity of the sentence is 0.8, indicating that the sentiment is positive. Polarity is of ‘float’ type and lies in the range of -1,1, where 1 means a high positive sentiment, and -1 means a high negative sentiment.

The output also prints subjectivity of the text which is 0.75 in our example. Subjectivity is also of ‘float’ type and lies in the range of 0,1. The value closer to 1 indicates that the sentence is mostly a public opinion and not a factual piece of information and vice versa.

Parts-of-Speech tagging:

Noun Phrase Extraction:

Spelling Correction:

StanfordNERTagger:

NLTK is a leading platform in python for building variety of programs to work with human language data and has already worked wonders in the field of Natural Language Processing. This has gamut of in-built functions for text pre-processing including tokenization, stemming, lemmatization, stopwords removal, punctuation removal, part-of-speech tagging, text parsing, etc. Here we will not go into those details but learn about a special package named as StanfordNERTagger, which again plays an important role in text pre-processing where you want to extract named entities like person’s name, location, etc.

StanfordNERtagger is a python package available in NLTK library which is an alternative to NLTK’s named entity recognition (NER) classifier. This tagger is largely seen as the standard in named entity recognition, but since it uses an advanced statistical learning algorithm it’s more computationally expensive than the option provided by NLTK.

A big benefit of the Stanford NER tagger is that it provides us with different models for pulling out named entities. We can use any of the following:

3 class model for recognizing locations, persons, and organizations

4 class model for recognizing locations, persons, organizations, and miscellaneous entities

7 class model for recognizing locations, persons, organizations, times, money, percents, and dates

NER classifier is written in JAVA. These are available for free from the Stanford Natural Language Processing Group. NTLK also provides a wrapper to the Stanford tagger so we can use it in python. For different languages you need to download jar files. For this post, we will show a sample code function only for English language.

For this tagger to work, you need to have java installed on your system. Above code shows that the path to java application is defined. From the page link given above you can download jar files and then need to initialize the path to model .gz file and path to jar file used to execute this model. In the sample code we are using class 3 model.

The output of the code produces a list of tuples. Each token is tagged (using 3 class model) with either ‘PERSON’, ‘LOCATION’, ‘ORGANIZATION’, or ‘O’. The ‘O’ simply stands for other, i.e., non-named entities.

P.S.: In case you like the post, please share your feedback in the comments or any misleading information in the blog and you can get in touch with me via Linkedin.