Multilingual Sentence Models in NLP | by Daulet Nurmanbetov | Towards Data Science

Overview of two major multilingual sentence embedding models
Multilingual Models are a type of Machine Learning model that can understand different languages. One example would be to classify whether a piece of text is a toxic comment. Using a regular Machine learning model we would be able to detect only English language toxic comments but not toxic comments made in Spanish. But if we used a multilingual model we would be able to detect toxic comments in English, Spanish and multiple other languages.
So how do the multilingual models work? One common approach is to turn any incoming language into a language-agnostic vector in a space, where all languages for the same input would point to the same area. That is to say, any incoming phrases with the same meaning would map to the same area in latent space.
There are several multilingual embeddings available today, they allow you to swap any words with vectors. The main two are LASER embeddings by Facebook (Language-Agnostic SEntence Representations) and Multilingual USE embeddings by Google (Universal Sentence Encoder). LASER embeddings cover 93 major languages while USE covers only 16 languages. Let’s take a deeper look at LASER.
LASER works by swapping (embedding) any input sentence into a vector in 1024-dimensional space. Each sentence gets once such vector,
LASER vector representations of sentences are generic with respect to both the input language and the NLP task. Here is an example —
On the left are some sentences embedded by LASER. On the right are sentences Embedded from the German language. As you can see vectors with the same meaning are located very close.
This allows us to use LASER as input features to our NLP model downstream for a specific task we have and don’t worry about the input language. Perhaps something as simple as logistic regression on the LASER output embeddings and suddenly we have a language-agnostic classifier that works on any language.
The best part is Facebook openly shared code and the model with the world. Let’s take a look at how to set up and run this awesome Multilingual embedding model.
Here is the Github to LASER. We will install LASER in docker on our machine by running the following code, the following bash commands also stand up docker listening to port 8050 —
#!/bin/bashgit clone https://github.com/facebookresearch/LASERcd LASERdocker build --tag=laser dockerdocker run -p 8050:80 -it laser python app.py
Now we are ready to cook with fire. In python terminal, define the following function, to embed sentences with LASER —
#!/bin/pythonimport requestsimport numpy as npdef get_vect(query_in, lang = 'en', address = '127.0.0.1:8050'):    url = "http://" + address + "/vectorize"    params = {"q": query_in, "lang": lang}    resp = requests.get(url=url, params=params).json()    return resp["embedding"]
Let’s define several sentences in various languages. Define the following dictionary —
input_dict = {    "en":"Machine learning isn't as hard as people think.",    "de":"Maschinelles Lernen ist nicht so schwer wie die Leute denken.",    "fr":"L'apprentissage automatique n'est pas aussi difficile qu'on le pense.",    "it":"L'apprendimento automatico non è così difficile come la gente pensa.",    "es":"El aprendizaje automático no es tan difícil como la gente piensa.",    "ru":"Машинное обучение не так сложно, как думают люди.",    "ar":"التعلم الآلي ليس صعبا كما يظن الناس.",    "pt":"O aprendizado de máquina não é tão difícil quanto as pessoas pensam.",}
Now let’s send the dictionary we’ve defined to docker running our LASER embedding service —
embedded_dict = {}for key in input_dict:    embedded_dict[key] = np.array(get_vect(input_dict[key], lang = key))
And here how our embeddings look like —
While each embedding is a 1024 dimensional vector. One thing to note though, LASER service averages the embeddings across sentences within a given paragraph. So sending 5 paragraphs of text would yield 5 vectors of 1024 dim each. Vectors internally are imputed for each sentence, however, the design they chose averages said vectors —
If you notice, the sentence we sent to LASER is “Machine learning isn’t as hard as people think.” in 8 different languages. Let’s verify the claim that the vectors of the same meaning are in the same vector space. For that, we’ll use cosine similarity of two vectors, where similarity of 1 means vectors are the same —
from sklearn.metrics.pairwise import cosine_similaritycos_lib = cosine_similarity(embedded_dict['it'], embedded_dict['es'])
And indeed, comparing the Italian sentence against a Spanish sentence yields a similarity of 99+% showing us that those vectors are pointing to the same region in our 1024 dimensional space.
As we’ve seen in most cases same sentences have almost identical vector representations. Meaning even when languages are different in the examples, the meaning behind them is the same and the LASER embedding system can convey that to the computer.
LASER embeddings allow us to abstract away the language of the document and work directly with vectors for the task that is needed, such as document classification or toxic language detection.