NLP for Indian Languages. Join me in exploring the need, the... | by Navkiran Singh | Towards Data Science

Chances are if you’re reading this article, you have a decent grasp of English. If you’re from India, it’s also likely that English is not the only language you know. Most Indians are multilingual 💬 and study more than one language in school.
However, can you guess the percentage of English speakers in the country? The answer may surprise you. Give it a go!🏃
The percentage of English speakers in India is...just 10% [1]. That’s 10% of a one billion-plus population!
Most Indians have Hindi as their first language, followed by Marathi, Telugu, Punjabi, etc. For a lot of people living in rural communities, English is not even a language they understand or speak.
As a side note, for an insight into the demographic profile of English speakers in India, refer [2].
Natural Language Processing, usually shortened as NLP, is an interdisciplinary field that deals with the interaction between computers and humans using the natural language.
NLP is a way for computers to analyze, understand and derive meaning from human language in a smart and useful way.
Now that the background of languages in India and purpose of NLP is clear, let’s dig deeper into why this is important.
The dream of an all-inclusive Digital India cannot be realized without bringing NLP research and application in India at par with that of languages like English.
When engaging with smartphones, the language barrier can be a huge obstacle to many.
Take the case of farmers and agriculture which has long been considered the backbone of India. Farmers play an obviously important role in feeding the country. Helping such farmers improve their methods (through precision agriculture, farmer helplines, chatbots, etc) has been an aim of development projects and an important part of the fight against global hunger. But many small farmers are not knowledgeable in English, meaning it is difficult for them to share and learn about new farming practices since most of the information is in English.
Can you imagine a mobile application like Google assistant but tailor-made for Indian farmers? It’d allow them to ask their questions in their native tongue, the system would understand their query and suggest relevant information from around the globe 🌏.
Do you think this is possible to do without NLP for Indian regional languages?
And, this is just one possible use-case. From making information more accessible to understanding farmer suicides [4], NLP has a huge role to play.
Thus, there is a clear need to bolster NLP research for Indian languages so that such people who don’t know English can get “online” in the true sense of the word, ask questions, in their mother tongue and get answers.
In fact, people at AI4Bharat, a platform to accelerate AI innovation in India, summarized the scenario quite aptly:
Though India is rich in languages, it is poor in resources on these languages.
The need also becomes clear when we look at some of the applications of NLP in India.
They are:
Smartphone users in India crossed 500 million in 2019. Businesses are feeling a need to increase user engagement at the local level. NLP can go a long way in achieving that — by improving search accuracy(Google Assistant now supports multiple Indian Languages), chatbots and virtual agents, etc.NLP has huge application in helping people with disabilities — interpretation of sign languages, text to speech, speech to text, etc.Digitisation of Indian Manuscripts to preserve knowledge contained in them.Signboard Translation from Vernacular Languages to make travel more accessible.Fonts for Indian Scripts for improving the impact/readability of advertisements, signboards, presentations, reports, etc.
Smartphone users in India crossed 500 million in 2019. Businesses are feeling a need to increase user engagement at the local level. NLP can go a long way in achieving that — by improving search accuracy(Google Assistant now supports multiple Indian Languages), chatbots and virtual agents, etc.
NLP has huge application in helping people with disabilities — interpretation of sign languages, text to speech, speech to text, etc.
Digitisation of Indian Manuscripts to preserve knowledge contained in them.
Signboard Translation from Vernacular Languages to make travel more accessible.
Fonts for Indian Scripts for improving the impact/readability of advertisements, signboards, presentations, reports, etc.
There are many more. The ideal scenario would be to have corpora and tools available in as good quality as they are for English to support work in these areas.
There are certain problems and issues affecting NLP in general like,
Ambiguity at different levels — syntactic, semantic, phonological ambiguity, etc.
Dealing with idioms and metaphors.
Dealing with emotions.
Finding referents of anaphora and cataphora.
Understanding discourse and challenges in pragmatics.
But there are some issues which arise due to differences between English and Indian languages. Thus, they are specific to Indian languages.
Below an attempt has been made to gather information about work in NLP for specific regional languages. If you feel something is amiss or some work has not been properly credited, contact me via email and I'll fix it.
Some of these differences are:
1. In sentence structuring, English uses subject-verb-object word order whereas Hindi uses subject-object-verb.
2. Presence of post-position case markers instead of prepositions.
3. Words can be moved freely in many Indian languages without changing the meaning.
Issues in dealing with Hindi:
1. Scarcity of annotated corpora and tools.
2. Lack of education and training institutes.
3. Large set of morphological variants.
Online resources to learn NLP for Indian languages are limited as compared to English.
1. Large number of morphological variants and word root complexity.
2. Punjab has different dialects in different regions of Punjab with minor variations. This increases preprocessing work.
Maajhi — spoken in Amritsar, Gurdaspur and Tarn-Tarn districts
Malwai — Bathinda, Patiala, Firozpur and Moga
Doabi — Jalandhar, Hoshiarpur and Nakodar
3. Hard to look for datasets related to a specific region.
4. Understanding sarcasm in Punjabi.
5. Understanding poetry.
A lot of datasets available at authentic sources like Kaggle, UCI machine learning repository, etc are in English.
Limited datasets, corpora and other resources are a huge hurdle. There is only one list of stopwords having 184 words in total, released for the public back in 2016. Refer [5] for more details.
It’s interesting to note that when I searched for resources for telugu NLP, the top GitHub result I found was this and it’s 4 years old at the time of writing with no updates.
Telugu does have a dataset on Kaggle, however. There is also a dataset for Tamil. Nonetheless, the discussion and kernels are very few and seemingly haven’t drawn much attention from Indian data science community.
Lack of datasets/corpora.
Lack of consolidated efforts.
Large number of morphological variations.
Lack of standards.
Refer [6] for more details.
Some tools I was able to find are listed below with their official descriptions and links to their documentation.
My purpose is to include these for completeness. Their documentation is pretty informative and detailed with relevant usage examples. For api usage examples, it's best to refer to the source documentation, since it will always be up-to-date.Nonetheless, if more examples are needed, I’ll add them in a future update or a future post. Let me know if there's any other tool that should be mentioned here.
Indic NLP Library
Indic NLP Library
From the official description:
The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text.The library provides the following functionalities:1. Text Normalization2. Script Information3. Tokenization4. Word Segmentation5. Script Conversion6. Romanization7. Indicization8. Transliteration9. Translation
Please refer to the official documentation for api usage and examples.
2. iNLTK
iNLTK aims to provide out of the box support for various NLP tasks that an application developer might need.Supported languages at the time of writing:1. Hindi  2. Punjabi 3. Sanskrit   4. Gujarati   5. Kannada  6. Malayalam   7. Nepali   8. Odia   9. Marathi   10. Bengali   11. Tamil  12. Urdu   13. English 
Please refer to the official documentation for api usage and examples.
3. StanfordNLP — Even though I was not able to find a language pack for Indian Languages on Stanford CoreNLP (website link), I did find a possible approach to use it for Indian Languages, kindly refer here . I’ve not tried it personally.
The journey of NLP in India seems long and daunting but as the famous proverb goes:
journeys of a thousand miles begin with a single step!
Good news is that due to the efforts of hundreds of researchers, industry specialists and students, we have made significant progress in a very short time. Future of NLP in India seems bright and applications limitless.
But again as Robert Frost famously wrote in his poem, ‘Stopping by Woods on a Snowy Evening’, we have miles to go before we sleep.
[1] Multilingualism in India, statistics compiled on Wikipedia from Indian Census data (2011) among other sources. (Accessed 20 March 2020)
[2] Rukmini S (2019), In India, who speaks in English, and where? , Livemint. (Accessed 20 March 2020)
[3] Madalli, Devika & Patel, Dimple. (2009). Issues in Indian languages computing in particular reference to search and retrieval in Telugu language. Library Hi Tech. 27. 450–459. 10.1108/07378830910988568.
[4] Singh, Jaspreet & Singh, Gurvinder & Singh, Rajinder & Singh, Prithvipal. (2018). Morphological Evaluation and Sentiment Analysis of Punjabi Text using Deep Learning Classification. Journal of King Saud University — Computer and Information Sciences. 10.1016/j.jksuci.2018.04.003.
[5] Kaur, Jasleen & Saini, Jatinderkumar. (2016). Punjabi Stop Words: A Gurmukhi, Shahmukhi and Roman Scripted Chronicle. 32–37. 10.1145/2909067.2909073.
[6] Kurian, C. and Kannan Balakrishnan, K., 2008. Natural Language Processing in India Prospects and Challenges. In Proceedings of the International Conference on “Recent Trends in Computational Science.
You reached the end! Thank you for reading and have a great rest of your day. 😄 I hope you found the content useful.
If you have any feedback 💭 or noticed a mistake, please leave a comment. You may also reach me at my email or my personal site here. I’d ❤️to hear your thoughts.