Package: nepalitokenizers
Nepali TokenizersThis package provides access to pre-trainedWordPieceandSentencePiece(Unigram) tokenizers for Nepali language, trained using HuggingFace'stokenizerslibrary. It is a simple and short Python package tailored specifically for Nepali language with a default set of configurations for the normalizer, pre-tokenizer, post-processor, and decoder.It delegates further customization by providing an interface to HuggingFace'sTokenizerpipeline, allowing users to adapt the tokenizers according to their requirements.InstallationYou can installnepalitokenizersusing pip:pipinstallnepalitokenizersUsageAfter installing the package, you can use the tokenizers in your Python code:WordPiece TokenizerfromnepalitokenizersimportWordPiecetext="हाम्रा सबै क्रियाकलापहरु भोलिवादी छन् । मेरो पानीजहाज वाम माछाले भरिपूर्ण छ । इन्जिनियरहरुले गएको हप्ता राजधानीमा त्यस्तै बहस गरे ।"tokenizer_wp=WordPiece()tokens=tokenizer_wp.encode(text)print(tokens.ids)print(tokens.tokens)print(tokenizer_wp.decode(tokens.ids))Output[1, 11366, 8625, 14157, 8423, 13344, 9143, 8425, 1496, 9505, 22406, 11693, 12679, 8340, 27445, 1430, 1496, 13890, 9008, 9605, 13591, 14547, 9957, 12507, 8700, 1496, 2]
['[CLS]', 'हाम्रा', 'सबै', 'क्रियाकलाप', '##हरु', 'भोलि', '##वादी', 'छन्', '।', 'मेरो', 'पानीजहाज', 'वाम', 'माछा', '##ले', 'भरिपूर्ण', 'छ', '।', 'इन्जिनियर', '##हरुले', 'गएको', 'हप्ता', 'राजधानीमा', 'त्यस्तै', 'बहस', 'गरे', '।', '[SEP]']
हाम्रा सबै क्रियाकलापहरु भोलिवादी छन् । मेरो पानीजहाज वाम माछाले भरिपूर्ण छ । इन्जिनियरहरुले गएको हप्ता राजधानीमा त्यस्तै बहस गरे ।SentencePiece (Unigram) TokenizerfromnepalitokenizersimportSentencePiecetext="कोभिड महामारीको पिडाबाट मुक्त नहुँदै मानव समाजलाई यतिबेला युद्धको विध्वंसकारी क्षतिको चिन्ताले चिन्तित बनाएको छ ।"tokenizer_sp=SentencePiece()tokens=tokenizer_sp.encode(text)print(tokens.ids)print(tokens.tokens)print(tokenizer_wp.decode(tokens.ids))Output[7, 9, 3241, 483, 12081, 9, 11079, 23, 2567, 11254, 1002, 789, 20, 3334, 2161, 9, 23517, 2711, 1115, 9, 1718, 12, 5941, 781, 19, 8, 1, 0]
['▁', 'को', 'भि', 'ड', '▁महामारी', 'को', '▁पिडा', 'बाट', '▁मुक्त', '▁नहुँदै', '▁मानव', '▁समाज', 'लाई', '▁यतिबेला', '▁युद्ध', 'को', '▁विध्वंस', 'कारी', '▁क्षति', 'को', '▁चिन्ता', 'ले', '▁चिन्तित', '▁बनाएको', '▁छ', '▁।', '<sep>', '<cls>']
कोभिड महामारीको पिडाबाट मुक्त नहुँदै मानव समाजलाई यतिबेला युद्धको विध्वंसकारी क्षतिको चिन्ताले चिन्तित बनाएको छ ।Configuration & CustomizationEach tokenizer class has a default and standard set of configurations for the normalizer, pre-tokenizer, post-processor, and decoder. For more information, look at the training files available in thetrain/directory.The package delegates further customization by providing an interface to directly access to HuggingFace's tokenizer pipeline. Therefore, you can treatnepalitokenizers's tokenizer instances as HuggingFace'sTokenizerobjects. For example:fromnepalitokenizersimportWordPiece# importing from the HuggingFace tokenizers packagefromtokenizers.processorsimportTemplateProcessingtext="हाम्रो मातृभूमि नेपाल हो"tokenizer_sp=WordPiece()# using default post processortokens=tokenizer_sp.encode(text)print(tokens.tokens)# change the post processor to not add any special tokens# treat tokenizer_sp as HuggingFace's Tokenizer objecttokenizer_sp.post_processor=TemplateProcessing()tokens=tokenizer_sp.encode(text)print(tokens.tokens)Output['[CLS]', 'हाम्रो', 'मातृ', '##भूमि', 'नेपाल', 'हो', '[SEP]']
['हाम्रो', 'मातृ', '##भूमि', 'नेपाल', 'हो']To learn more about further customizations that can be performed, visitHuggingFace's Tokenizer Documentation.Note: The delegation to HuggingFace's Tokenizer pipeline was done with the following generic wrapper class becausetokenizers.Tokenizeris not an acceptable base type for inheritance.
It is a useful trick I use for solving similar issues:classDelegate:"""A generic wrapper class that delegates attributes and method callsto the specified self.delegate instance."""@propertydef_items(self):returndir(self.delegate)def__getattr__(self,name):ifnameinself._items:returngetattr(self.delegate,name)raiseAttributeError(f"'{self.__class__.__name__}' object has no attribute '{name}'")def__setattr__(self,name,value):ifname=="delegate"ornamenotinself._items:super().__setattr__(name,value)else:setattr(self.delegate,name,value)def__dir__(self):returndir(type(self))+list(self.__dict__.keys())+self._itemsTrainingThe python files used to train the tokenizers are available in thetrain/directory. You can also use these files to train your own tokenizers on a custom text corpus.These tokenizers were trained on two datasets:1. The Nepali Subset of theOSCARdatasetYou can download it using the following code:importdatasetsfromtqdm.autoimporttqdmimportosdataset=datasets.load_dataset('oscar','unshuffled_deduplicated_ne',split='train')os.mkdir('data')batch=[]counter=0forsampleintqdm(dataset):sample=sample['text'].replace('\n',' ')batch.append(sample)iflen(batch)==10_000:withopen(f'data/ne_{counter}.txt','w',encoding='utf-8')asf:f.write('\n'.join(batch))batch=[]counter+=12. A Large Scale Nepali Text Corpus by Rabindra Lamsal (2020)To download the dataset, follow the instructions provided in this link:A Large Scale Nepali Text Corpus.LicenseThis package is licensed under the Apache 2.0 License, which is consistent with the license used by HuggingFace'stokenizerslibrary. Please see theLICENSEfile for more details.