Getting Started with NLP for Indic Languages | by Sundar V | Towards Data Science

Want to go beyond English and use the real power of Natural Language Processing (NLP) to serve the second-most populous country in the world? Everyone knows India is a very diverse country and a home for lots of languages but do you know India speaks 780 languages [1]. It’s time to go beyond English when it comes to NLP. This article is for those who somewhat know about NLP and want to start using it for Indian languages.
Before going into the topic, we will glance at some essential concepts and recent accomplishments in NLP. NLP helps computers in understanding human language. Text Classification, Information Extraction, Semantic Parsing, Question Answering, Text Summarization, Machine Translation, and Conversational Agents are some applications of NLP.
For computers to understand human language, we initially need to represent the words in numeric form. Numerically represented words can then be utilized by machine learning models to do any NLP task. Traditionally, methods like One Hot Encoding, TF-IDF Representation were used to describe the text as numbers. But the traditional methods resulted in sparse representation by failing to capture the word meaning.
Neural Word Embeddings then came to rescue by solving the problems in traditional methods. Word2Vec and GloVe are the two most commonly used word embedding. These methods came up with dense representations where words having similar meaning will have similar representations. A significant weakness with this method is words are considered to have a single meaning. But we know that a word can have multiple meaning depending on the context where it is used.
NLP took a major leap by the modern family of language models (AI’s Muppet Empire). Word embedding is no longer independent of context. The same word can have multiple numeric representations depending on the context where it is used. BERT, ELMo, ULMFit, GPT-2 are some currently popular language models. The latest generation is so good, and some people consider it dangerous [3]. The news written by these language models were even rated as credible as the New York Times by the readers [4].
To know more about language models, I would highly recommend reading [2].
Will now go into the topic by getting word embedding for Indic languages. Numerically representing words plays a role in any NLP task. We are going to use the Natural Language Toolkit for Indic Languages (iNLTK) library. iNLTK is an open-source Deep Learning library built on top of PyTorch aiming to provide out of the box support for various NLP tasks that an application developer might need for Indic languages. ULMFiT language models were used to build iNLTK [6]. iNLTK was trending on GitHub in May 2019 and has had 23,000+ Downloads on PyPi [5].
We are using iNLTK here because of its simplicity and support for many Indic languages. iNLTK currently supports the below 12 Indic languages. Please follow this link to install iNLTK.
Using iNLTK, we can quickly get the embedding vectors for sentences written in Indic languages. Below is an example that shows how we can get the embedding vectors for a sentence written in Hindi. Given sentence will be broken into tokens, and each token will be represented using a (400 x 1) vector. A token can be a word, or it can be a sub-word. Since tokens can be sub words, we can get a meaningful vector representation for rare words too.
# Example 1 - Get Embedding Vectorsfrom inltk.inltk import setupfrom inltk.inltk import tokenizefrom inltk.inltk import get_embedding_vectors'''Note: You need to run setup('<code-of-language>') when you use a language for the FIRST TIME ONLY.This will download all the necessary models required to do inference for that language.'''setup('hi')example_sent = "बहुत समय से मिले नहीं"# Tokenize the sentenceexample_sent_tokens = tokenize(example_sent,'hi')# Get the embedding vector for each tokenexample_sent_vectors = get_embedding_vectors(example_sent, 'hi')print("Tokens:", example_sent_tokens)print("Number of vectors:", len(example_sent_vectors))print("Shape of each vector:", len(example_sent_vectors[0]))
Output:
Tokens: ['▁बहुत', '▁समय', '▁से', '▁मिले', '▁नहीं ']Number of vectors: 5Shape of each vector: 400
Now that we got the word embeddings, many might wonder how good these numerical representations are? Do similar tokens have similar descriptions? We can get answers to all those questions using the visualizations provided for every language. For example, as shown below, all neighboring tokens to ‘प्रेम’ (love) are closely related to love, affection, wedding, etc.
As I said earlier, numerically represented natural language can be used by machine learning models to do many NLP tasks. Apart from that, we can immediately use iNLTK for numerous NLP tasks. For instance, we can use iNLTK to predict next ’n’ words, get similar sentences, get sentence encoding, identify language, etc.
In below example, We will be using iNLTK to predict next ’n’ words, and to get similar sentences. For an input like “It’s been a long time since we met” in Tamil, we get a prediction for next ’n’ words as “And, due to this”. And the results for similar sentences task is also impressive.
# Example 2 - Predict Next 'n' Words, Get Similar Sentencesfrom inltk.inltk import setupfrom inltk.inltk import predict_next_wordsfrom inltk.inltk import get_similar_sentences'''Note: You need to run setup('<code-of-language>') when you use a language for the FIRST TIME ONLY.This will download all the necessary models required to do inference for that language.'''setup('ta')example_sent = "உங்களைப் பார்த்து நிறைய நாட்கள் ஆகிவிட்டது"# Predict next 'n' tokensn = 5pred_sent = predict_next_words(example_sent, n, 'ta')# Get 'n' similar sentencen = 2simi_sent = get_similar_sentences(example_sent, n, 'ta')print("Predicted Words:", pred_sent)print("Similar Sentences:", simi_sent)
Output:
Predicted Words: உங்களைப் பார்த்து நிறைய நாட்கள் ஆகிவிட்டது. மேலும், இதற்கு காரணமாகSimilar Sentences: ['உங்களைத் பார்த்து நாட்கள் ஆகிவிட்டது ', 'உங்களைப் பார்த்து ஏராளமான நாட்கள் ஆகிவிட்டது ']
As I said, in the beginning, it’s time to go beyond English and use the real power of NLP to serve everyone. Lots of research in recent terms are focused on multilingual NLP. iNLTK is one such library that concentrates on Indic languages. Today you can contribute to iNLTK by adding support to a new language, improving existing models, and adding new functionalities. Apart from iNLTK, I will recommend looking into multilingual models [7], Indic NLP [8]. Don’t forget that everything you just came across is just the tip of the iceberg. Before concluding this article, I want to add in a few words expressing iNLTK’s vision from its creator.
I always wished for something like it to exist, which will make NLP more accessible and democratize its benefits to non-English speakers. iNLTK kind of grew organically and now, since it is being appreciated a lot, I see there’s a lot still left to be done. My vision for iNLTK is that it should be the go-to library for anyone working with low-resource languages. — Gaurav Arora, Creator of iNLTK.
[1] http://blogs.reuters.com/india/2013/09/07/india-speaks-780-languages-220-lost-in-last-50-years-survey/[2] https://towardsdatascience.com/from-word-embeddings-to-pretrained-language-models-a-new-age-in-nlp-part-1-7ed0c7f3dfc5[3] https://blog.deeplearning.ai/blog/the-batch-biggest-ai-stories-of-2019-driverless-cars-stall-deepfakes-go-mainstream-face-recognition-gets-banned[4] https://www.foreignaffairs.com/articles/2019-08-02/not-your-fathers-bots[5] https://github.com/goru001/inltk[6] https://in.pycon.org/cfp/2019/proposals/natural-language-toolkit-for-indic-languages-inltk~eZQ6d/[7] https://huggingface.co/transformers/multilingual.html[8] https://anoopkunchukuttan.github.io/indic_nlp_library/