Package: inltk
Natural Language Toolkit for Indic Languages (iNLTK)iNLTK aims to provide out of the box support for various NLP tasks
that an application developer might need for Indic languages. Paper for iNLTK library has been accepted at EMNLP-2020's NLP-OSS workshop. Here's thepreprint for the paperDocumentationCheckout detailed docs along with Installation instructions
athttps://inltk.readthedocs.ioSupported languagesNative languagesLanguageCodeHindihiPunjabipaGujaratiguKannadaknMalayalammlOriyaorMarathimrBengalibnTamiltaUrduurNepalineSanskritsaEnglishenTeluguteCode Mixed languagesLanguageScriptCodeHinglish (Hindi+English)Latinhi-enTanglish (Tamil+English)Latinta-enManglish (Malayalam+English)Latinml-enRepositories containing models used in iNLTKLanguageRepositoryDataset used for Language modelingPerplexity of ULMFiT LM(on validation set)Perplexity of TransformerXL LM(on validation set)Dataset used for ClassificationClassification:Test set AccuracyClassification:Test set MCCClassification: Notebookfor ReproducibilityULMFiT Embeddings visualizationTransformerXL Embeddings visualizationHindiNLP for HindiHindi Wikipedia Articles - 172kHindi Wikipedia Articles - 55k34.0635.8726.0934.78BBC News ArticlesIIT Patna Movie ReviewsIIT Patna Product Reviews78.7557.7475.710.710.370.59NotebookNotebookNotebookHindi Embeddings projectionHindi Embeddings projectionBengaliNLP for BengaliBengali Wikipedia Articles41.239.3Bengali News Articles (Soham Articles)90.710.87NotebookBengali Embeddings projectionBengali Embeddings projectionGujaratiNLP for GujaratiGujarati Wikipedia Articles34.1228.12iNLTK Headlines Corpus - Gujarati91.050.86NotebookGujarati Embeddings projectionGujarati Embeddings projectionMalayalamNLP for MalayalamMalayalam Wikipedia Articles26.3925.79iNLTK Headlines Corpus - Malayalam95.560.93NotebookMalayalam Embeddings projectionMalayalam Embeddings projectionMarathiNLP for MarathiMarathi Wikipedia Articles1817.42iNLTK Headlines Corpus - Marathi92.400.85NotebookMarathi Embeddings projectionMarathi Embeddings projectionTamilNLP for TamilTamil Wikipedia Articles19.8017.22iNLTK Headlines Corpus - Tamil95.220.92NotebookTamil Embeddings projectionTamil Embeddings projectionPunjabiNLP for PunjabiPunjabi Wikipedia Articles24.4014.03IndicNLP News Article Classification Dataset - Punjabi97.120.96NotebookPunjabi Embeddings projectionPunjabi Embeddings projectionKannadaNLP for KannadaKannada Wikipedia Articles70.1061.97IndicNLP News Article Classification Dataset - Kannada98.870.98NotebookKannada Embeddings projectionKannada Embeddings projectionOriyaNLP for OriyaOriya Wikipedia Articles26.5726.81IndicNLP News Article Classification Dataset - Oriya98.830.98NotebookOriya Embeddings ProjectionOriya Embeddings ProjectionSanskritNLP for SanskritSanskrit Wikipedia Articles~6~3Sanskrit Shlokas Dataset84.3 (valid set)Sanskrit Embeddings projectionSanskrit Embeddings projectionNepaliNLP for NepaliNepali Wikipedia Articles31.529.3Nepali News Dataset98.5 (valid set)Nepali Embeddings projectionNepali Embeddings projectionUrduNLP for UrduUrdu Wikipedia Articles13.1912.55Urdu News Dataset95.28 (valid set)Urdu Embeddings projectionUrdu Embeddings projectionTeluguNLP for TeluguTelugu Wikipedia Articles27.4729.44Telugu News DatasetTelugu News Andhra Jyoti95.492.09NotebookNotebookTelugu Embeddings projectionTelugu Embeddings projectionTanglishNLP for TanglishSynthetic Tanglish Dataset37.50-Dravidian Codemix HASOC @ FIRE 2020Dravidian Codemix Sentiment Analysis @ FIRE 2020F1 Score: 0.88F1 Score: 0.62-NotebookNotebookTanglish Embeddings Projection-ManglishNLP for ManglishSynthetic Manglish Dataset45.84-Dravidian Codemix HASOC @ FIRE 2020Dravidian Codemix Sentiment Analysis @ FIRE 2020F1 Score: 0.74F1 Score: 0.69-NotebookNotebookManglish Embeddings Projection-HinglishNLP for HinglishSynthetic Hinglish Dataset86.48-----Hinglish Embeddings Projection-Note: English model has been directly taken fromfast.aiEffect of using Transfer Learning + Paraphrases from iNLTKLanguageRepositoryDataset used for ClassificationResults on usingcomplete training setPercentage Decreasein Training set sizeResults on usingreduced training setwithout ParaphrasesResults on usingreduced training setwith ParaphrasesHindiNLP for HindiIIT Patna Movie ReviewsAccuracy: 57.74MCC: 37.2380% (2480 -> 496)Accuracy: 47.74MCC: 20.50Accuracy: 56.13MCC: 34.39BengaliNLP for BengaliBengali News Articles (Soham Articles)Accuracy: 90.71MCC: 87.9299% (11284 -> 112)Accuracy: 69.88MCC: 61.56Accuracy: 74.06MCC: 65.08GujaratiNLP for GujaratiiNLTK Headlines Corpus - GujaratiAccuracy: 91.05MCC: 86.0990% (5269 -> 526)Accuracy: 80.88MCC: 70.18Accuracy: 81.03MCC: 70.44MalayalamNLP for MalayalamiNLTK Headlines Corpus - MalayalamAccuracy: 95.56MCC: 93.2990% (5036 -> 503)Accuracy: 82.38MCC: 73.47Accuracy: 84.29MCC: 76.36MarathiNLP for MarathiiNLTK Headlines Corpus - MarathiAccuracy: 92.40MCC: 85.2395% (9672 -> 483)Accuracy: 84.13MCC: 68.59Accuracy: 84.55MCC: 69.11TamilNLP for TamiliNLTK Headlines Corpus - TamilAccuracy: 95.22MCC: 92.7095% (5346 -> 267)Accuracy: 86.25MCC: 79.42Accuracy: 89.84MCC: 84.63For more details around implementation or to reproduce results, checkout respective repositories.ContributingAdd a new language supportIf you would like to add support for language of your own choice to iNLTK,
please start with checking/raising a issueherePlease checkout the steps I'dmentioned here for Teluguto begin with. They should be almost similar for other languages as well.Improving models/using models for your own researchIf you would like to take iNLTK's models and refine them with your own
dataset or build your own custom models on top of it, please check out the
repositories in the above table for the language of your choice. The repositories above
contain links to datasets, pretrained models, classifiers and all of the code for that.Add new functionalityIf you wish for a particular functionality in iNLTK - Start by checking/raising a issuehereWhat's next..and being worked uponShout out if you want to help :)AddMaithilisupport..and NOT being worked uponShout out if you want to lead :)Add NER support for all languagesAdd Textual Entailment support for all languagesWork on aunified model for all the languagesPOS supportin iNLTKAdd translations - to and from languages in iNLTK + EnglishiNLTK's AppreciationBy Jeremy Howard on TwitterBy Sebastian Ruder on TwitterBy Vincent Boucher,By Philip Vollet,By Steve NourionLinkedInBy Kanimozhi,By Soham,By ImaadonLinkedIniNLTK wastrending on GitHubin May 2019CitationIf you use this library in your research, please consider citing:@misc{arora2020inltk,
      title={iNLTK: Natural Language Toolkit for Indic Languages}, 
      author={Gaurav Arora},
      year={2020},
      eprint={2009.12534},
      archivePrefix={arXiv},
      primaryClass={cs.CL}}