4 Python libraries to detect English and Non-English language
By: Manmohan Singh
Tags: Spacy, Naturallanguageprocessing, Deep Learning, Language Learning, Python

Are you sure that the input text data for your model is in English? Well, no one can be sure about this, as no one will read around 20k records of text data.

So, how non-English text will affect your English text trained model?

Pick any non-English text and pass it through as input to your English text trained classification model. You will come to know that the category is assigned to non-English text by the model.

If your model is dependent on one language then, other languages in your textual data should be considered as noise.

But why?

The job of the text classification model is to classify. And, it will do its job despite its input text will be in English or not.

What can we do to avoid such a situation?

Your model will not stop classifying the non-English text. So, you have to detect the non-English text and remove it from trained data and prediction data.

This process comes under the data cleaning part. Inconsistency in your data will result in a decrease in the accuracy of the model. Sometimes, multiple languages present in text data could be one of the reasons your model behaves strangely.

So, in this article, we will discuss the different python libraries which detect the language(s) of the text data.

Let’s start with the spaCy library.

1. SpaCy

You need to install the spacy-langdetect and spacy python libraries for the below code to work.

#1. Download the best-matching default model and create a shortcut link.

#2. Add LanguageDetector() function and model to NLP pipeline.

#3. Pass the text data into the pipeline for language detection.

#4 Store the detected language and accuracy in the detect_language variable.

We have tested the above library with only single language text data. What will happen when the text has multiple language sentences?

Let’s find out.

Predicted language is en, that is, English. Prediction accuracy is also low. But, this text also has a German-language sentence. And, this library did not predict that.

Reason?

This library returns the detected language of longer sentences.

The spaCy is a python library used in different Natural Language Processing (NLP) tasks. It’s deploying a spacy_langdetect library model in the spaCy NLP pipeline.

Know about spaCy here. Know about spacy_langdetect here.

2. Pycld2

What if we want to know the detected language of all sentences?

In that case, use the below code. You need to install the pycld2 python library for the below code to work.

Pycld2 python library is a python binding for the Compact Language Detect 2 (CLD2). You can explore the different functionality of Pycld2. Know about the Pycld2 here.

3. TextBlob

Let’s say you are using TextBlob for the NLP task. Then will you use spaCy or TextBlob for the language detection? I will use TextBlob and also should you.

And we also love choices.

You need to install the textblob python library for the below code to work.

TextBlob also returns the detected language of longer sentences. Use pycld2 library for multiple language sentences.

Do you know which language is represented by the “ru”?

If you don’t then, then visit this Wikipedia link. Or you can check the below list. It is the standard short-form (ISO 639–1 code) of the language used in data science.

TextBlob provides an API to perform different NLP tasks. Some applications of TextBlob are text processing, sentiment analysis, classification, spelling correction, keyword extraction, part-of-speech tagging, etc. know about TextBlob here.

4. Googletrans

Googletrans python library uses the google translate API to detect the language of text data. But this library is not reliable. So, be careful before using this library. You can consider this library as another choice for language detection.

You need to install the googletrans python library for the below code to work.

Googletrans also has the functionality for language translation. Know about Googletrans here.

Application of language detection

Find out bias in text data based on the languages. You can classify the article based on the different languages. Language is generally associated with the region. This method helps you to classify the article based on languages. You can use this method in the language translation model. You can use it in data cleaning and data manipulation processes.

Conclusion

We should consider language detection as one of the data cleaning processes, for textual data. Internet text data are not always present in the English language.

In this article, I have explains different python libraries to detect the languages.

These libraries will help you to remove the noise from your data. It’s better to know that your data is noise-free. And you have eliminated one reason if your model doesn’t perform on new data.

Hopefully, this article will help you to reduce the noise in your data.