Package: sbnltk
pypi-download-statsPlease use colab for getting no problem. For transformer model, please install simpleTransformer first or usebn_nlpfor static models. I uploaded dataset and training details in my github.There is a problem in sentiment analyzer. I Will fix it soon.SBNLTKSUST-Bangla Natural Language toolkit. A python module for Bangla NLP tasks.Demo Version : 1.0NEED python 3.6+ vesrion!! Use virtual Environment for not getting unessessary Issues!!INSTALLATIONPYPI INSTALLATIONpip3 install sbnltk
pip3 install simpletransformersMANUAL INSTALLATION FROM GITHUBClone this projectInstall all the requirementsCall the setup.py from terminalWhat will you get here?Bangla Text PreprocessorBangla word dust,punctuation,stop word removalBangla word sorting according to Bangla or English alphabetBangla word normalizationBangla word stemmerBangla Sentiment analysis(logisticRegression,LinearSVC,Multilnomial_naive_bayes,Random_Forst)Bangla Sentiment analysis with BertBangla sentence pos tagger (static, sklearn)Bangla sentence pos tagger with BERT(Multilingual-cased,Multilingual uncased)Bangla sentence NER(Static,sklearn)Bangla sentence NER with BERT(Bert-Cased, Multilingual Cased/Uncased)Bangla word word2vec(gensim,glove,fasttext)Bangla sentence embedding(Contexual,Transformer/Bert)Bangla Document Summarization(Feature based, Contexual, sementic Based)Bangla Bi-lingual project(Bangla to english google translator without blocking IP)Bangla document information ExtractionSEE THE CODE DOCS FOR USES!TASKS, MODELS, ACCURACY, DATASET AND DOCSTASKMODELACCURACYDATASETAboutCode DOCSPreprocessorPunctuation, Stop Word, DUST removal Word normalization, others..-----------docsWord tokenizersbasic tokenizers Customized tokenizers--------docsSentence tokenizersBasic tokenizers Customized tokenizers Sentence Cluster----------docsStemmerStemmerOP85.5%----docsSentiment AnalysislogisticRegression88.5%20,000+docsLinearSVC82.3%20,000+docsMultilnomial_naive_bayes84.1%20,000+docsRandom Forest86.9%20,000+docsBERT93.2%20,000+docsPOS taggerStatic method55.5%1,40,973  wordsdocsSK-LEARN classification81.2%6,000+ sentencesdocsBERT-Multilingual-Cased69.2%6,000+docsBERT-Multilingual-Uncased78.7%6,000+docsNER taggerStatic method65.3%4,08,837 EntitydocsSK-LEARN classification81.2%65,000+docsBERT-Cased79.2%65,000+docsBERT-Mutilingual-Cased75.5%65,000+docsBERT-Multilingual-Uncased90.5%65,000+docsWord EmbeddingGensim-word2vec-100D- 1,00,00,000+ tokens-2,00,00,000+  sentencesdocsGlove-word2vec-100D- 2,30,000+ tokens-5,00,000 sentencesdocsfastext-word2vec-200D 3,00,000+-5,00,000 sentencesdocsSentence EmbeddingContextual sentence embedding------docsTransformer embedding_hd-3,00,000+ human datadocsTransformer embedding_gd-3,00,000+ google datadocsExtractive  SummarizationFeature-based based70.0% f1 score------docsTransformer sentence sentiment Based67.0%------docsWord2vec--sentences contextual Based60.0%-----docsBi-lingual projectsgoogle translator with large data detector--------docsInformation ExtractionStatic word features-docsSemantic and contextual-docsBangla Coreference Resolution-Next releases after testing this demoTaskVersionCoreference Resolutionv1.1Language translationV1.1Masked Language modelV1.1Information retrieval ProjectsV1.1Entity Segmentationv1.3Factoid Question Answeringv1.2Question Classificationv1.2sentiment Word embeddingv1.3So many others features---Package InstallationYou have to install these packages manually, if you get any module error.simpletransformersfasttextModelsEverything is automated here. when you call a model for the first time, it will be downloaded automatically.With GPU or Without GPUWith GPU, you can run any models without getting any warnings.Without GPU, You will get some warnings. But this will not affect in result.MotivationWith approximately 228 million native speakers and another 37 million as second language speakers,Bengali is the fifth most-spoken native
language and the seventh most spoken language by total number of speakers in the world. But still it is a low resource language. Why?DatasetFor all sbnltk dataset and existing Dataset, see this linkBangla NLP DatasetTrainerFor training, You can see thisColab Trainer. In future i will make a Trainer module!When will full version come?Very soon. We are working on paper and improvement our modules. It will be released sequentially.About accuracyAccuracy can be varied for the different datasets. We measure our model with random datasets but small scale. As human resources for this project are not so large.Contribute HereIf you found any issue, please create an issue or contact with me.