Package: klpt
Kurdish Language Processing ToolkitWelcome /Hûn bi xêr hatin/ بە خێر بێن! 🙂Kurdish Language Processing Toolkit--KLPT is anatural language processing(NLP) toolkit in Python for theKurdish language. The current version comes with four core modules, namelypreprocess,stem,transliterateandtokenizeand addresses basic language processing tasks such as text preprocessing, stemming, tokenization, spell-checking and morphological analysis for theSoraniand theKurmanjidialects of Kurdish.Latest update on April 29th, 2022 🎉In the latest version, I focused on Kurmanji for which a morphological analyzer, a stemmer and a lemmatizer are now added to the toolkit. These tasks were partially addressed previously using theApertium project. Now, that is fully replaced by the Kurmanji implementation ofKurdish Hunspell.Install KLPTOperating system: macOS / OS X · Linux · Windows (Cygwin, MinGW, Visual
Studio)Python version: Python 3.5+Package managers:pippipUsing pip, KLPT releases are available as source packages and binary wheels. Please make sure that a compatible Python version is installed:pipinstallklptAll the data files including lexicons and morphological rules are also installed with the package.Although KLPT is not dependent on any NLP toolkit, there is one important requirement, particularly for thestemmodule. That iscyhunspellwhich should be installed with a version >= 2.0.1.About this versionPlease note that KLPT is under development and some of the functionalities will appear in the future versions. You can find out regarding the progress of each task at theProjectssection. In the current version, the following tasks are included:ModulesTasksSorani (ckb)Kurmanji (kmr)preprocessnormalization✓ (v0.1.0)✓ (v0.1.0)standardization✓ (v0.1.0)✓ (v0.1.0)unification of numerals✓ (v0.1.0)✓ (v0.1.0)stopwords✓ (v0.1.4)✓ (v0.1.4)tokenizeword tokenization✓ (v0.1.0)✓ (v0.1.0)MWE tokenization✓ (v0.1.0)✓ (v0.1.0)sentence tokenization✓ (v0.1.0)✓ (v0.1.0)part-of-speech tagging✗✗transliterateArabic to Latin✓ (v0.1.0)✓ (v0.1.0)Latin to Arabic✓ (v0.1.0)✓ (v0.1.0)Detection of u/w and î/y✓ (v0.1.0)✓ (v0.1.0)Detection of Bizroke (i)✗✗stemmorphological analysis✓ (v0.1.0)✓ (v0.1.1)stemming✓ (v.0.1.5)✓ (v.0.1.6) 🆕lemmatization✓ (v.0.1.5)✓ (v.0.1.6) 🆕spell error detection and correction✓ (v0.1.0)✓ (v.0.1.6) 🆕Basic usageOnce the package is installed, you can import the toolkit as follows:importklptIn the following, a few examples are provided to work with various modules. Almost all the classes have three arguments in common:dialect: the name of the dialect asSoraniorKurmanji(ISO 639-3 code will be also added)script: the script of your input text as "Arabic" or "Latin"numeral: the type of the numerals asArabic [١٢٣٤٥٦٧٨٩٠]Farsi [۱۲۳۴۵۶۷۸۹۰]Latin [1234567890]PreprocessThis module deals with normalizing scripts and orthographies by using writing conventions based on dialects and scripts. The goal is not to correct the orthography but to normalize the text in terms of the encoding and common writing rules. The input encoding should be in UTF-8 only. To this end, three functions are provided as follows:normalize: deals with different encodings and unifies characters based on dialects and scriptsstandardize: given a normalized text, it returns standardized text based on the Kurdish orthographies following recommendations forKurmanjiandSoraniunify_numerals: conversion of the various types of numerals used in Kurdish textspreprocess: one single function for normalization, standardization and unification of numeralsIt is recommended that the output of this module be used as the input of subsequent tasks in an NLP pipeline.>>>fromklpt.preprocessimportPreprocess>>>preprocessor_ckb=Preprocess("Sorani","Arabic",numeral="Latin")>>>preprocessor_ckb.normalize("لە ســـاڵەکانی ١٩٥٠دا")'لە ساڵەکانی 1950دا'>>>preprocessor_ckb.standardize("راستە لەو ووڵاتەدا")'ڕاستە لەو وڵاتەدا'>>>preprocessor_ckb.unify_numerals("٢٠٢٠")'2020'>>>preprocessor_ckb.preprocess("راستە لە ووڵاتەی ٢٣هەمدا")'ڕاستە لە وڵاتەی 23هەمدا'>>>preprocessor_kmr=Preprocess("Kurmanji","Latin")>>>preprocessor_kmr.standardize("di sala 2018-an")'di sala 2018an'>>>preprocessor_kmr.standardize("hêviya")'hêvîya'In addition, it is possible to remove Kurdishstopwordsusing thestopwordsvariable. You can define a function like the following to do so:fromklpt.preprocessimportPreprocessdefremove_stopwords(text,dialect,script):p=Preprocess(dialect,script)return[tokenfortokenintext.split()iftokennotinp.stopwords]TokenizationThis module focuses on the tokenization of both Kurmanji and Sorani dialects of Kurdish with the following functions:word_tokenize: tokenization of texts into tokens (bothmulti-word expressionsand single-word tokens).mwe_tokenize: tokenization of texts by only taking compound forms into accountsent_tokenize: tokenization of texts into sentencesThis module is based on theKurdish tokenization project. It is recommended that the output of this module be used as the input of theStemmodule.>>>fromklpt.tokenizeimportTokenize>>>tokenizer=Tokenize("Kurmanji","Latin")>>>tokenizer.word_tokenize("ji bo fortê xwe avêtin")['▁ji▁','bo','▁▁fortê‒xwe‒avêtin▁▁']>>>tokenizer.mwe_tokenize("bi serokê hukûmeta herêma Kurdistanê Prof. Salih re saz kir.")'bi serokê hukûmeta herêma Kurdistanê Prof . Salih re saz kir .'>>>tokenizer_ckb=Tokenize("Sorani","Arabic")>>>tokenizer_ckb.word_tokenize("بە هەموو هەمووانەوە ڕێک کەوتن")['▁بە▁','▁هەموو▁','هەمووانەوە','▁▁ڕێک‒کەوتن▁▁']TransliterationThis module aims at transliterating one script of Kurdish into another one. Currently, only the Latin-based and the Arabic-based scripts of Sorani and Kurmanji are supported. The main function in this module istransliterate()which also takes care of detecting the correct form of double-usage graphemes, namely و ↔ w/u and ی ↔ î/y. In some specific occasions, it can also predict the placement of the missingi(also known asBizroke/بزرۆکە).The module is based on theKurdish transliteration project.>>>fromklpt.transliterateimportTransliterate>>>transliterate=Transliterate("Kurmanji","Latin",target_script="Arabic")>>>transliterate.transliterate("rojhilata navîn")'رۆژهلاتا ناڤین'>>>transliterate_ckb=Transliterate("Sorani","Arabic",target_script="Latin")>>>transliterate_ckb.transliterate("لە وڵاتەکانی دیکەدا")'le wiłatekanî dîkeda'StemThe Stem module deals with various tasks, mainly through the following functions:check_spelling: spell error detectioncorrect_spelling: spell error correctionanalyze: morphological analysisstem: stemming, e.g. "بڕ" → "بڕاوە" or "dixwî" → "xw"lemmatize: lemmatization, e.g. "بردن" → "بردمنەوە" or "jimartibûye" → "hejmartin"The module is based on theKurdish Hunspell projectfor Sorani and theApertium projectfor Kurmanji. Please note that this module is currently getting further completed and we are aware of its current shortcomings.>>>fromklpt.stemimportStem>>>stemmer=Stem("Sorani","Arabic")>>>stemmer.check_spelling("سوتاندبووت")False>>>stemmer.correct_spelling("سوتاندبووت")(False,['ستاندبووت','سووتاندبووت','سووڕاندبووت','ڕووتاندبووت','فەوتاندبووت','بووژاندبووت'])>>>stemmer.analyze("دیتبامن")[{'pos':['verb'],'description':'past_stem_transitive_active','stem':'دی','lemma':['دیتن'],'base':'دیت','prefixes':'','suffixes':'بامن'}]>>>stemmer.stem("دەچینەوە")['چ']>>>stemmer.stem("گورەکە",mark_unknown=True)# گوڵەکە in Hewlêrî dialect['_گور_']>>>stemmer.lemmatize("گوڵەکانم")['گوڵ','گوڵە']>>>stemmer=Stem("Kurmanji","Latin")>>>stemmer.analyze("dibêjim")[{'base':'bêj','prefixes':'di','suffixes':'im','pos':['verb'],'description':'present_stem_transitive_active','stem':'bêj','lemma':['gotin']}]>>>stemmer.stem("dixwî")['xw']>>>stemmer.lemmatize("jimartibûye")['hejmartin']📖Please note that a more complete documentation of the toolkit is available athttps://sinaahmadi.github.io/klpt/.Become a sponsorPlease consider donating to the project. Data annotation and resource creation requires tremendous amount of time and linguistic expertise. Even a trivial donation will make a difference. You can do so bybecoming a sponsorto accompany me in this journey and help the Kurdish language have a better place within other natural languages on the Web. Depending on your support,You can be an official sponsorYou will get a GitHub sponsor badge on your profileIf you have any questions, I will focus on itIf you want, I will add your name or company logo on the front page of your preferred projectYour contribution will be acknowledged in one of my future papers in a field of your choiceAnd, thanks for those who have already sponsored this project. More significant achievements will be made thanks to you!ContributeAre you interested in this project? Each task is addressed individually. Please check the following repositories to find which one you are more interested in:Kurdish tokenizationKurdish HunspellKurdish transliterationIn addition, our main objective is to extend the current toolkit to include more tasks, particularly part-of-speech tagging, named-entity recognition and syntactic analysis. Further instructions are provided athttps://sinaahmadi.github.io/klpt/about/contributing/. You can also join us onGitter.Don't forget,open-source is fun!😊RequirementsPython >=3.6cyhunspell>= 2.0.1Cite this projectPlease consider citingthis paper, if you use any part of the data or the tool (bibfile):@inproceedings{ahmadi2020klpt,
    title = "{KLPT--Kurdish Language Processing Toolkit}",
    author = "Ahmadi, Sina",
    booktitle = "Proceedings of the second Workshop for {NLP} Open Source Software ({NLP}-{OSS})",
    month = nov,
    year = "2020",
    publisher = "Association for Computational Linguistics"
}LicenseKurdish Language Processing ToolkitbySina Ahmadiis licensed under aCreative Commons Attribution-ShareAlike 4.0 International Licensewhich means:You are free to share, copy and redistribute the material in any medium or format and also adapt, remix, transform, and build upon the material
for any purpose,even commercially.You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.If you remix, transform, or build upon the material,you must distribute your contributions under the same license as the original.