Introduction to Google’s Compact Language Detector v3 in Python | by Ng Wai Foong | Towards Data Science

By reading this piece, you will learn to use the Python-bindings of Google’s CLD3 to detect and identify the languages used in a text. For your information, Compact Language Detector v3 (CLD3) is a language identification package released by Google. At the time of this writing, it has enormous support for most of the common languages. You can find the complete list at the following link.
It contains a trained model and inference code based on C++. Based on the official documentation, the
“... inference code extracts character ngrams from the input text and computes the fraction of times each of them appears.
The model averages the embeddings corresponding to each ngram type according to the fractions, and the averaged embeddings are concatenated to produce the embedding layer. The remaining components of the network are a hidden (Rectified linear) layer and a softmax layer.”
The basic architecture for CLD3 is as follows which uses banana as the input text:
Since the end of Aug 2020, they have released an official Python package which comes with inference code in Python. Check out the following code from the official repository for the actual implementation which is based on pybind package.
Let’s proceed to the next section and start installing the necessary modules.
In order to install CLD3 from PyPI, you need to install the following additional packages in your local machine:
the Protobuf compiler which is the protoc executable
the Protobuf development headers and libprotoc library
a compiler, preferably g++
If you are using Debian/Ubuntu operating system, run the following command:
sudo apt-get install -y --no-install-recommends g++ protobuf-compiler libprotobuf-dev
Check the following README for more information on the installation process for other operating system.
It is highly recommended to create a virtual environment before you continue with the installation. Activate it and run the following command in your terminal to install the official Python-bindings for CLD3:
pip install gcld3
Once you have installed all the required packages, you can use it directly inside your Python script. Create a new Python script called test.py.
Add the following import statement at the top of the file.
import gcld3
Next, initialize a new NNetLanguageIdentifier object.
detector = gcld3.NNetLanguageIdentifier(min_num_bytes=0, max_num_bytes=1000)
It accepts the following parameters:
min_num_bytes : minimum number of bytes to consider during inference. Setting this to a value higher than 0 will return und for empty string.
max_num_bytes : maximum number of bytes to consider during inference. It will truncate excess bytes should the input text exceeds the value set.
If you intent to filter out empty string and short strings, you initialize the instance as follows:
detector = gcld3.NNetLanguageIdentifier(min_num_bytes=10, max_num_bytes=1000)
CDL3 comes with two inference functions. The first one is the FindLanguage function which is meant for single language inference. It accepts the a single text argument which represents the sample text.
sample = "Welcome to Medium."result = detector.FindLanguage(text=sample)
It returns a Result object which contains the following fields:
language : language code represented in BCP-47-style.
is_reliable : a boolean which reflects the reliability of the detection.
proportion : a float which represents the proportion and weights of the predicted language in the sample text. Range from 0 to 1. This field is useful during multi-language detection.
probability : the confidence level of the prediction. Range from 0 to 1.
Simply call each of the fields as follows to get the corresponding values:
result.languageresult.is_reliableresult.proportionresult.probability
You should use FindTopNMostFreqLangs for detecting multiple languages. In addition to text argument, it accepts another integer argument called num_langs.
sample = ("Welcome to Medium. Bienvenido a Medium.")results = detector.FindTopNMostFreqLangs(text=sample, num_langs=2)
FindTopNMostFreqLangs returns a list of Result object. Simply call the respective fields in a loop to get the desired output.
for i in results:    print(i.language)
Please be noted that the length of the list is based on num_langs. If you set it to a value of 5, it will return 5 Result objects in a list.
You can find the complete code at the following gist.
Let’s recap what you have learned today.
This article started off with a brief introduction on the basic architecture and concept behind Google’s Compact Language Detector v3.
Next, it covered the installation process based on the operating system that you have. Python package for CLD3 is installed via pip install.
In the implementation section, it explained on the basis of FindLanguage and FindTopNMostFreqLangs functions. Both functions returns either a single or a list of Result object which contains a few useful parameters such as language and probability.
Thanks for reading this piece. Hope you will come back and read my other articles!
Github — CLD3PyPI — gcld3
Github — CLD3
PyPI — gcld3