Benchmarking Language Detection for NLP
By: Jenny Lee
Tags: NLP, Editors Pick, Getting Started, Data Science, Python

Benchmarking Language Detection for NLP

Photo by JACQUELINE BRANDWAYN on Unsplash

Most NLP applications tend to be language-specific and therefore require monolingual data. In order to build an application in your target language, you may need to apply a preprocessing technique that filters out text written in non-target languages. This requires proper identification of the language of each input example. Below I list some tools you can use as Python modules for this preprocessing requirement, and provide a performance benchmark assessing the speed and accuracy of each one.

1) langdetect

langdetect is a re-implementation of Google’s language-detection library from Java to Python. Simply pass your text to the imported detect function and it will output the two-letter ISO 693 code of the language for which the model gave the highest confidence score. (Refer to this page for a full list of 693 codes and their respective languages.) If you use detect_langs instead, it will output a list of the top languages that the model has predicted, along with their probabilities.

from langdetect import DetectorFactory, detect, detect_langs text = "My lubimy mleko i chleb." detect(text) # 'cs'

detect_langs(text) # [cs:0.7142840957132709, pl:0.14285810606233737, sk:0.14285779665739756]

A few embellishing points:

The library makers recommend that you set the DetectorFactory seed to some number. This is because langdetect’s algorithm is non-deterministic, which means if you try to run it on a text that’s too short or too ambiguous, you might get different results each time you run it. Setting the seed enforces consistent results during development/evaluation. You may also want to surround the detect call in a try/except block with LanguageDetectException , otherwise you will likely get a “No features in text” error, which occurs when there the language of the given input cannot be evaluated as when it contains strings like URLs, numbers, formulas, etc.

from langdetect import DetectorFactory, detect

from langdetect.lang_detect_exception import LangDetectException DetectorFactory.seed = 0 def is_english(text):

try:

if detect(text) != "en":

return False

except LangDetectException:

return False

return True

2) spaCy language detector

If you use spaCy for your NLP needs, you can add a custom language detection component to your existing spaCy pipeline, which will enable you to set an extension attribute called .language on the Doc object. This attribute can then be accessed via Doc._.language , which will return the predicted language along with its probability.

import spacy

from spacy_langdetect import LanguageDetector text2 = 'In 1793, Alexander Hamilton recruited Webster to move to New York City and become an editor for a Federalist Party newspaper.' nlp = spacy.load('en_core_web_sm')

nlp.add_pipe(LanguageDetector(), name='language_detector', last=True) doc = nlp(text)

doc._.language # {'language': 'en', 'score': 0.9999978351575265}

3) langid

langid boasts of its speed in particular (more on this below). It works similarly to the above tools, but it can additionally be used as a command-line tool by running python langid.py . Check out their repo for more details and other options.

To use langid as a Python library, use the classify function:

import langid langid.classify(text2) # ('en', -127.75649309158325)

You can calibrate the probability prediction, originally computed in the log-probability space, to what can be interpreted as confidence scores in the range of 0 to 1:

from langid.langid import LanguageIdentifier, model lang_identifier = LanguageIdentifier.from_modelstring(model, norm_probs=True)

lang_identifier.classify(text2) # ('en', 0.999999999999998)

4) FastText

fasttext notes that its pre-trained language identification model takes less than 1MB of memory while being able to classify thousands of documents per second.

Download a model of your choice:

lid.176.bin: faster and slightly more accurate (file size=126MB).

lid.176.ftz: the compressed version of the model (file size=917kB).

import fasttext



path_to_pretrained_model = '/tmp/lid.176.bin'

fmodel = fasttext.load_model(path_to_pretrained_model)

fmodel.predict([text2]) # ([['__label__en']], [array([0.9331119], dtype=float32)]

Speed 🚅

If you plan to use your application requiring language identification in production, speed is likely an important consideration for you. Below is a quick benchmark of the four tools described above.

I downloaded a dataset of 10,502 tweets from Kaggle, which was randomly sampled from publicly available geotagged Twitter messages originating from 130 different countries. They were annotated for being in English or non-English, among other features.

Data source: https://www.kaggle.com/rtatman/the-umass-global-english-on-twitter-dataset.