Package: banglanlptoolkit
Bangla NLP ToolkitCreated byA F M Mahfuzul Kabirmahfuzulkabir.comhttps://www.linkedin.com/in/mahfuzulkabirInstallationInstall the requirements first with:pip install -r requirements.txtinstall the package withpip install banglanlptoolkitIntroductionThis package contains several toolkits for Bangla NLP text processing and augmentation. The available tools are listed below.Bangla Text NormalizerBangla Punctuation GeneratorBangla Text AugmentationDocumentations:For detailed use of Bangla Text Normalizer, followthis documentation.For detailed use of Bangla Punctuation Generation, followthis documentation.For detailed use of Bangla Text Augmentation (both online and offline), followthis documentation.Thank you very much for using my package. I handle this package all on my own, so if there's any issue with it, I might not always be available to fix it. But if you do encounter such event, feel free to let me know and I'll fix them as soon as I can.Bangla Text NormalizerBangla text normalization is a known problem in language processing for normalizing Bangla text data in computer readable format. The unicode normalization normalizes all characters of a text string in the same unicode format and removes unwanted characters present. The csebuetnlp normalizer is used for models such as BanglaBERT, BanglaT5 etc.The package uses two normalization toolkits for Bangla text processing. The unicode normalizer is used fromhere. The other normalizer is specifically used for BanglaT5 translation module and taken fromhere.Bangla Punctuation GeneratorThe scarcity of good punctuation generator model for Bangla language was very dominant even a few months ago. However, with development of Bangla AI models, we now have very good punctuation generation models for our language as well.The package uses an open-source punctuation generation model fromthisKaggle dataset. I currently have this model in my huggingface for ease of use without any token. You can replace with any model of your like if you want.Bangla Text AugmentationThe package uses three kind of text augmentation techniques.Bangla Token ReplacementBack TranslationBangla ParaphrasingThe token replacement method uses fill-mask method to replace random tokens from a sentence and then replace them. The package uses BanglishBERT Generator model by CSEBUETNLP for this task. The model can be found inhere.The back translation method translates the sentences from Bangla to English and then to Bangla again. The package uses bn-en and en-bn models of BanglaT5 by CSEBUETNLP for this task. The models can be found here:bn2en,en2bn.The paraphrasing toolkit uses Bangla paraphrase model of BanglaT5 by CSEBUETNLP. The model can be found inhere.The package supports both online and offline augmentations. Offline augmentation can be used to generate new dataframe of augmented texts from original dataframe. This can be saved in a variable or to a file for later use. While offline augmentation can be faster for utilizing processing power (GPU parallelism), it can get a bit annoying because of saving the augmented data every once in a while. People also love to use online augmentation, meaning, augmenting the data 'on the fly' in predefined custom dataset class. This improves performance by augmentation of sentences during training or inference, with no hassle of saving the data separately.Fromversion 1.1.5, I'm happy to introduce online augmentation techniques in this package. This technique was inspired from the exact same technique oftorchvision.transpose, meaning, you can stack several augmentation techniques with acomposeclass. You can also write your own custom class of augmentation or transform techniques and use them withcompose.Inspired fromA Visual Survey of Data Augmentation in NLPCSE BUET NLPBangla Unicode Normalizer by Bengali AiBangla Text AugmentationIf you use this package, please don't forget to cite the links and papers mentioned.