Package: indic-nlp-library
Indic NLP LibraryThe goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text.The library provides the following functionalities:Text NormalizationScript InformationWord Tokenization and DetokenizationSentence SplittingWord SegmentationSyllabificationScript ConversionRomanizationIndicizationTransliterationTranslationThe data resources required by the Indic NLP Library are hosted in a different repository. These resources are required for some modules. You can download from theIndic NLP Resourcesproject.If you are interested in Indian language NLP resources, you should check theIndic NLP Catalogfor pointers.Pre-requisitesPython 3.x(For Python 2.x version check the tagPYTHON_2.7_FINAL_JAN_2019. Not actively supporting Python 2.x anymore, but will try to maintain as much compatibility as possible)Indic NLP ResourcesUrduhack: Needed only if Urdu normalization is required. It has other dependencies like Tensorflow.Other dependencies are listed in setup.pyConfigurationInstallation from pip:pip install indic-nlp-libraryIf you want to use the project from the github repo, add the project to the Python Path:Clone this repositoryInstall dependencies:pip install -r requirements.txtRun:export PYTHONPATH=$PYTHONPATH:<project base directory>In either case, export the path to theIndic NLP ResourcesdirectoryRun:export INDIC_RESOURCES_PATH=<path to Indic NLP resources>UsageYou can use the Python API to access all the features of the library. Many of the most common operations are also accessible via a unified commandline API.Getting StartedCheckthis IPython Notebookfor examples to use the Python API.You can find the Python 2.x NotebookhereDocumentationYou can find detailed documentationHEREThis documents the Python API as well as the commandline reference.CitingIf you use this library, please include the following citation:@misc{kunchukuttan2020indicnlp,
author = "Anoop Kunchukuttan",
title = "{The IndicNLP Library}",
year = "2020",
howpublished={\url{https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf}}
}You can find the documentHEREWebsitehttp://anoopkunchukuttan.github.io/indic_nlp_libraryAuthorAnoop Kunchukuttan (anoop.kunchukuttan@gmail.com)Companies, Organizations, Projects using IndicNLP LibraryAI4Bharat-IndicNLPSuiteThe Classical Language ToolkitMicrosoft NLP RecipesFacebook M2M-100Revision Log0.81 : 26 May 2021- Bug fix in version number extraction0.80 : 24 May 2021- Improved sentence splitting
- Bug fixes
- Support for Urdu Normalizer0.71 : 03 Sep 2020- Improved documentation
- Bug fixes0.7 : 02 Apr 2020:- Unified commandline 
- Improved documentation
- Added setup.py0.6 : 16 Dec 2019:- New romanizer and indicizer
- Script Unifiers
- Improved script normalizers
- Added contrib directory for sample uses
- changed to MIT license0.5 : 03 Jun 2019:- Improved word tokenizer to handle dates and numbers. 
- Added sentence splitter that can handle common prefixes/honorofics and uses some heuristics.
- Added detokenizer
- Added acronym transliterator that can convert English acronyms to Brahmi-derived scripts0.4 : 28 Jan 2019: Ported to Python 3, and lots of feature additions since last release; primarily around script information, script similarity and syllabification.0.3 : 21 Oct 2014: Supports morph-analysis between Indian languages0.2 : 13 Jun 2014: Supports transliteration between Indian languages and tokenization of Indian languages0.1 : 12 Mar 2014: Initial version. Supports text normalization.LICENSEIndic NLP Library is released under the MIT license