Package: dadmatools-light
DadmaTools: A Python NLP Library for PersianNamed Entity Recognition|Part of Speech Tagging|Dependency ParsingConstituency Parsing|ChunkingTokenizer|LemmatizerDadmaToolsDadmaTools is a repository for Natural Language Processing resources for the Persian Language. The aim is to make it easier and more applicable to practitioners in the industry to use Persian NLP, and hence this project is licensed to allow commercial use. The project features code examples on how to use the models in popular NLP frameworks such as spaCy and Transformers, as well as Deep Learning frameworks such as PyTorch. Furthermore, DadmaTools support common Persian embedding and Persian datasets.
for more details about how to use this tool read the instruction below.NLP ModelsNatural Language Processing is an active area of research, and it consists of many different tasks.
The DadmaTools repository provides an overview of Persian models for some of the most basic NLP tasks (and is continuously evolving).Here is the list of NLP tasks we currently cover in the repository. These NLP tasks are defined as pipelines. Therefore, a pipeline list must be created and passed through the model. This will allow the user to choose the only task needed without loading others.
Each task has its abbreviation as follows:Named Entity Recognition:nerPart of speech tagging:posDependency parsing:depConstituency parsing:consChunking:chunkLemmatizing:lemTokenizing:tokNormalizingNotethat the normalizer can be used outside of the pipeline as there are several configs (the default config is in the pipeline with the name of def-norm).Notethat if no pipeline is passed to the model, the tokenizer will be loaded as default.Use CaseNormalizercleaning text and unify characters.Note: None means no action!fromdadmatools.models.normalizerimportNormalizernormalizer=Normalizer(full_cleaning=False,unify_chars=True,refine_punc_spacing=True,remove_extra_space=True,remove_puncs=False,remove_html=False,remove_stop_word=False,replace_email_with="<EMAIL>",replace_number_with=None,replace_url_with="",replace_mobile_number_with=None,replace_emoji_with=None,replace_home_number_with=None)text="""<p>دادماتولز اولین نسخش سال ۱۴۰۰ منتشر شده.امیدواریم که این تولز بتونه کار با متن رو براتون شیرین‌تر و راحت‌تر کنهلطفا با ایمیل dadmatools@dadmatech.ir با ما در ارتباط باشیدآدرس گیت‌هاب هم که خب معرف حضور مبارک هست:https://github.com/Dadmatech/DadmaTools</p>"""normalized_text=normalizer.normalize(text)#<p> دادماتولز اولین نسخش سال 1400 منتشر شده. امیدواریم که این تولز بتونه کار با متن رو براتون شیرین‌تر و راحت‌تر کنه لطفا با ایمیل <EMAIL> با ما در ارتباط باشید آدرس گیت‌هاب هم که خب معرف حضور مبارک هست: </p>#full cleaningnormalizer=Normalizer(full_cleaning=True)normalized_text=normalizer.normalize(text)#دادماتولز نسخش سال منتشر تولز بتونه کار متن براتون شیرین‌تر راحت‌تر کنه ایمیل ارتباط آدرس گیت‌هاب معرف حضور مبارکPipeline - Tokenizer, Lemmatizer, POS Tagger, Dependancy Parser, Constituency Parserimportdadmatools.pipeline.languageaslanguage# here lemmatizer and pos tagger will be loaded# as tokenizer is the default tool, it will be loaded as well even without callingpips='tok,lem,pos,dep,chunk,cons'nlp=language.Pipeline(pips)# you can see the pipeline with this codeprint(nlp.analyze_pipes(pretty=True))# doc is an SpaCy objectdoc=nlp('از قصهٔ کودکیشان که می‌گفت، گاهی حرص می‌خورد!')docobject has different extensions. First, there aresentencesindocwhich is the list of the list ofToken. EachTokenalso has its own extensions. Note that we defined our own extension as well in DadmaTools. If any pipeline related to the specific extensions is not called, that extension will have no value.To better see the results which you can use this code:dictionary=language.to_json(pips,doc)print(dictionary)[[{'id':1,'text':'از','lemma':'از','pos':'ADP','rel':'case','root':2},{'id':2,'text':'قصهٔ','lemma':'قصه','pos':'NOUN','rel':'obl','root':10},{'id':3,'text':'کودکی','lemma':'کودکی','pos':'NOUN','rel':'nmod','root':2},{'id':4,'text':'شان','lemma':'آنها','pos':'PRON','rel':'nmod','root':3},{'id':5,'text':'که','lemma':'که','pos':'SCONJ','rel':'mark','root':6},{'id':6,'text':'می\u200cگفت','lemma':'گفت#گو','pos':'VERB','rel':'acl','root':2},{'id':7,'text':'،','lemma':'،','pos':'PUNCT','rel':'punct','root':6},{'id':8,'text':'گاهی','lemma':'گاه','pos':'NOUN','rel':'obl','root':10},{'id':9,'text':'حرص','lemma':'حرص','pos':'NOUN','rel':'compound:lvc','root':10},{'id':10,'text':'می\u200cخورد','lemma':'خورد#خور','pos':'VERB','rel':'root','root':0},{'id':11,'text':'!','lemma':'!','pos':'PUNCT','rel':'punct','root':10}]]sentences=doc._.sentencesforsentenceinsentences:text=sentence.textfortokeninsentences:token_text=token.textlemma=token.lemma_## this has value only if lem is calledpos_tag=token.pos_## this has value only if pos is calleddep=token.dep_## this has value only if dep is calleddep_arc=token._.dep_arc## this has value only if dep is calledsent_constituency=doc._.constituency## this has value only if cons is calledsent_chunks=doc._.chunks## this has value only if cons is calledners=doc._.ners## this has value only if ner is calledNotethat_.constituencyand_.chunksare the object ofSuParclass.Loading Persian NLP DatasetsWe provide an easy-to-use way to load some popular Persian NLP datasetsHere is the list of supported datasets.DatasetTaskPersianNERNamed Entity RecognitionARMANNamed Entity RecognitionPeymaNamed Entity RecognitionFarsTailTextual EntailmentFaSpellSpell CheckingPersianNewsText ClassificationPerUDTUniversal DependencyPnSummaryText SummarizationSnappfoodSentimentSentiment ClassificationTEPText Translation(eng-fa)WikipediaCorpusCorpusPersianTweetsCorpusall datasets are iterator and can be used like below:fromdadmatools.datasetsimportFarsTailfromdadmatools.datasetsimportSnappfoodSentimentfromdadmatools.datasetsimportPeymafromdadmatools.datasetsimportPerUDTfromdadmatools.datasetsimportPersianTweetsfromdadmatools.datasetsimportPnSummaryfarstail=FarsTail()#len of datasetprint(len(farstail.train))#like a generatorprint(next(farstail.train))#dataset detailspn_summary=PnSummary()print('PnSummary dataset information: ',pn_summary.info)#loop over datasetsnpfood_sa=SnappfoodSentiment()fori,iteminenumerate(snpfood_sa.test):print(item['comment'],item['label'])#get first tokens' lemma of all dev itemsperudt=PerUDT()fortoken_listinperudt.dev:print(token_list[0]['lemma'])#get NER tag of first Peyma's datapeyma=Peyma()print(next(peyma.data)[0]['tag'])#corpustweets=PersianTweets()print('tweets count : ',len(tweets.data))print('sample tweet: ',next(tweets.data))get dataset info:fromdadmatools.datasetsimportget_all_datasets_infoget_all_datasets_info().keys()#dict_keys(['Persian-NEWS', 'fa-wiki', 'faspell', 'PnSummary', 'TEP', 'PerUDT', 'FarsTail', 'Peyma', 'snappfoodSentiment', 'Persian-NER', 'Arman', 'PerSent'])#specify taskget_all_datasets_info(tasks=['NER','Sentiment-Analysis'])the output will be:{"ARMAN":{"description":"ARMAN dataset holds 7,682 sentences with 250,015 sentences tagged over six different classes.\n\nOrganization\nLocation\nFacility\nEvent\nProduct\nPerson","filenames":["train_fold1.txt","train_fold2.txt","train_fold3.txt","test_fold1.txt","test_fold2.txt","test_fold3.txt"],"name":"ARMAN","size":{"test":7680,"train":15361},"splits":["train","test"],"task":"NER","version":"1.0.0"},"PersianNer":{"description":"source: https://github.com/Text-Mining/Persian-NER","filenames":["Persian-NER-part1.txt","Persian-NER-part2.txt","Persian-NER-part3.txt","Persian-NER-part4.txt","Persian-NER-part5.txt"],"name":"PersianNer","size":976599,"splits":[],"task":"NER","version":"1.0.0"},"Peyma":{"description":"source: http://nsurl.org/2019-2/tasks/task-7-named-entity-recognition-ner-for-farsi/","filenames":["peyma/600K","peyma/300K"],"name":"Peyma","size":10016,"splits":[],"task":"NER","version":"1.0.0"},"snappfoodSentiment":{"description":"source: https://huggingface.co/HooshvareLab/bert-fa-base-uncased-sentiment-snappfood","filenames":["snappfood/train.csv","snappfood/test.csv","snappfood/dev.csv"],"name":"snappfoodSentiment","size":{"dev":6274,"test":6972,"train":56516},"splits":["train","test","dev"],"task":"Sentiment-Analysis","version":"1.0.0"}}Loading Persian Word Embeddingsdownload, load and use some pre-trained Persian word embeddings.dadmatools supports all glove, fasttext, and word2vec formats.fromdadmatools.embeddingsimportget_embedding,get_all_embeddings_info,get_embedding_infofrompprintimportpprintpprint(get_all_embeddings_info())#get embedding information of specific embeddingembedding_info=get_embedding_info('glove-wiki')#### load embedding ####word_embedding=get_embedding('glove-wiki')#get vector of the wordprint(word_embedding['سلام'])#vocabvocab=word_embedding.get_vocab()### some useful functions ###print(word_embedding.top_nearest("زمستان",10))print(word_embedding.similarity('کتب','کتاب'))print(word_embedding.embedding_text('امروز هوای خوبی بود'))The following word embeddings are currently supported:NameEmbedding AlgorithmCorpusglove-wikigloveWikipediafasttext-commoncrawl-binfasttextCommonCrawlfasttext-commoncrawl-vecfasttextCommonCrawlword2vec-conllword2vecPersian CoNLL17 corpusEvaluationWe have compared our pos tagging, dependancy parsing, and lemmatization models tostanzaandhazm.PerDT (F1 score)ToolkitPOS Tagger (UPOS)Dependancy Parser (UAS/LAS)LemmatizerDadmaTools97.52%95.36%/92.54%99.14%stanza97.35%93.34%  /  91.05%98.97%hazm--89.01%Seraji (F1 score)ToolkitPOS Tagger (UPOS)Dependancy Parser (UAS/LAS)LemmatizerDadmaTools97.83%92.5%/89.23%-stanza97.43%87.20% /  83.89%-hazm--86.93%Tehran university tree bank (F1 score)ToolkitConstituency ParserDadmaTools (without preprocess))82.88%Stanford (with some preprocess on POS tags)80.28InstallationTo get started using DadmaTools in your python project, simply install via the pip package. Note that installing the default pip package
will not install all NLP libraries because we want you to have the freedom to limit the dependency on what you use. Instead, we provide you with an installation option if you want to install all the required dependencies.Install with pipTo get started using DadmaTools, simply install the project with pip:pipinstalldadmatoolsNote that the default installation of DadmaToolsdoesinstall other NLP libraries such as SpaCy and supar.You can check therequirements.txtfile to see what version the packages has been tested with.Install from githubAlternatively you can install the latest version from github using:pipinstallgit+https://github.com/Dadmatech/dadmatools.gitHow to use (Colab)You can see the codes and the output here.CiteWill be added in future.