Benchmarking Language Detection for NLP | by Jenny Lee | Towards Data Science

Most NLP applications tend to be language-specific and therefore require monolingual data. In order to build an application in your target language, you may need to apply a preprocessing technique that filters out text written in non-target languages. This requires proper identification of the language of each input example. Below I list some tools you can use as Python modules for this preprocessing requirement, and provide a performance benchmark assessing the speed and accuracy of each one.
pypi.org
langdetect is a re-implementation of Google’s language-detection library from Java to Python. Simply pass your text to the imported detect function and it will output the two-letter ISO 693 code of the language for which the model gave the highest confidence score. (Refer to this page for a full list of 693 codes and their respective languages.) If you use detect_langs instead, it will output a list of the top languages that the model has predicted, along with their probabilities.
from langdetect import DetectorFactory, detect, detect_langstext = "My lubimy mleko i chleb."detect(text) #  'cs'detect_langs(text)  # [cs:0.7142840957132709, pl:0.14285810606233737, sk:0.14285779665739756]
A few embellishing points:
The library makers recommend that you set the DetectorFactory seed to some number. This is because langdetect’s algorithm is non-deterministic, which means if you try to run it on a text that’s too short or too ambiguous, you might get different results each time you run it. Setting the seed enforces consistent results during development/evaluation.You may also want to surround the detect call in a try/except block with LanguageDetectException, otherwise you will likely get a “No features in text” error, which occurs when there the language of the given input cannot be evaluated as when it contains strings like URLs, numbers, formulas, etc.
The library makers recommend that you set the DetectorFactory seed to some number. This is because langdetect’s algorithm is non-deterministic, which means if you try to run it on a text that’s too short or too ambiguous, you might get different results each time you run it. Setting the seed enforces consistent results during development/evaluation.
You may also want to surround the detect call in a try/except block with LanguageDetectException, otherwise you will likely get a “No features in text” error, which occurs when there the language of the given input cannot be evaluated as when it contains strings like URLs, numbers, formulas, etc.
from langdetect import DetectorFactory, detectfrom langdetect.lang_detect_exception import LangDetectExceptionDetectorFactory.seed = 0def is_english(text):    try:        if detect(text) != "en":            return False    except LangDetectException:        return False    return True
spacy.io
If you use spaCy for your NLP needs, you can add a custom language detection component to your existing spaCy pipeline, which will enable you to set an extension attribute called .language on the Doc object. This attribute can then be accessed via Doc._.language, which will return the predicted language along with its probability.
import spacyfrom spacy_langdetect import LanguageDetectortext2 = 'In 1793, Alexander Hamilton recruited Webster to move to New York City and become an editor for a Federalist Party newspaper.'nlp = spacy.load('en_core_web_sm')nlp.add_pipe(LanguageDetector(), name='language_detector', last=True)doc = nlp(text)doc._.language  # {'language': 'en', 'score': 0.9999978351575265}
langid boasts of its speed in particular (more on this below). It works similarly to the above tools, but it can additionally be used as a command-line tool by running python langid.py. Check out their repo for more details and other options.
github.com
To use langid as a Python library, use the classify function:
import langidlangid.classify(text2)  # ('en', -127.75649309158325)
You can calibrate the probability prediction, originally computed in the log-probability space, to what can be interpreted as confidence scores in the range of 0 to 1:
from langid.langid import LanguageIdentifier, modellang_identifier = LanguageIdentifier.from_modelstring(model, norm_probs=True)lang_identifier.classify(text2) # ('en', 0.999999999999998)
fasttext notes that its pre-trained language identification model takes less than 1MB of memory while being able to classify thousands of documents per second.
Download a model of your choice:
lid.176.bin: faster and slightly more accurate (file size=126MB).
lid.176.ftz: the compressed version of the model (file size=917kB).
import fasttextpath_to_pretrained_model = '/tmp/lid.176.bin'fmodel = fasttext.load_model(path_to_pretrained_model)fmodel.predict([text2])  # ([['__label__en']], [array([0.9331119], dtype=float32)]
If you plan to use your application requiring language identification in production, speed is likely an important consideration for you. Below is a quick benchmark of the four tools described above.
I downloaded a dataset of 10,502 tweets from Kaggle, which was randomly sampled from publicly available geotagged Twitter messages originating from 130 different countries. They were annotated for being in English or non-English, among other features.
Data source: https://www.kaggle.com/rtatman/the-umass-global-english-on-twitter-dataset.
import pandas as pddf = pd.read_csv('all_annotated.tsv', sep='\t')
The results of the speed test!
fasttext took only 129 ms to predict on 10,000+ datapoints. langid came in second and the other contenders were many orders of magnitude slower.
from sklearn.metrics import accuracy_scoreytrue = df['Definitely English'].to_list()tweets = df['Tweet'].to_list()# get the predictions of each detectorlangdetect_preds = [lang_detect(t) for t in tweets]spacy_preds = [nlp(t)._.language['language'] for t in tweets]langid_preds = [lang_identifier.classify(text)[0] for t in tweets]fasttext_preds = [p[0].replace('__label__', '') for p in fmodel.predict(tweets)[0]]# binarize the labelslangdetect_preds_binary = [1 if p == 'en' else 0 for p in langdetect_preds]spacy_preds_binary = [1 if p == 'en' else 0 for p in spacy_preds]langid_preds_binary = [1 if p == 'en' else 0 for p in langid_preds]fasttext_preds_binary = [1 if p == 'en' else 0 for p in fasttext_preds]# evaluate accuracy against the true labels (1 for English, 0 otherwise)accuracy_score(ytrue, langdetect_preds_binary)  # 0.8448866882498571 accuracy_score(ytrue, spacy_preds_binary)  # 0.8448866882498571accuracy_score(ytrue, langid_preds_binary)  # 0.8268901161683488accuracy_score(ytrue, fasttext_preds_binary)  # 0.8598362216720624
fasttext had the highest accuracy score, followed by langdetect and spacy-langdetect. Something tells me that spacy-langdetect is just langdetect under the hood. ;) (They had exactly the same accuracy score...and that would also explain the similar library names.)
And for good measure, here are the precision, recall and f1-score for each model.
from sklearn.metrics import classification_reportprint(classification_report(ytrue, langdetect_preds_binary))print(classification_report(ytrue, spacy_preds_binary))print(classification_report(ytrue, langid_preds_binary))print(classification_report(ytrue, fasttext_preds_binary))# langdetect              precision    recall  f1-score   support           0       0.80      0.94      0.86      5416           1       0.92      0.75      0.82      5086    accuracy                           0.84     10502   macro avg       0.86      0.84      0.84     10502weighted avg       0.86      0.84      0.84     10502# spacy-langdetect              precision    recall  f1-score   support           0       0.80      0.94      0.86      5416           1       0.92      0.75      0.82      5086    accuracy                           0.84     10502   macro avg       0.86      0.84      0.84     10502weighted avg       0.86      0.84      0.84     10502# langid              precision    recall  f1-score   support           0       0.79      0.90      0.84      5416           1       0.88      0.74      0.81      5086    accuracy                           0.83     10502   macro avg       0.83      0.82      0.82     10502weighted avg       0.83      0.83      0.83     10502# fasttext              precision    recall  f1-score   support           0       0.91      0.80      0.86      5416           1       0.82      0.92      0.86      5086    accuracy                           0.86     10502   macro avg       0.86      0.86      0.86     10502weighted avg       0.87      0.86      0.86     10502
I would go with fasttext if you’re dealing with a very large dataset.