Package: test-mark
Bangla Feature Extractor(BFE)BFE is a Bangla Natural Language Processing based feature extractor.Current FeaturesCountVectorizerTfIdfWord EmbeddingWord2VecFastTextInstallationpip install bfeExample1. CountVectorizerFit n TransformTransformGet WordsetFit n TransformfrombfeimportCountVectorizerct=CountVectorizer()X=ct.fit_transform(X)# X is the word features#Output: the countVectorized matrix form of given featuresTransformfrombfeimportCountVectorizerct=CountVectorizer()get_mat=ct.transform("রাহাত")#Output: the countVectorized matrix form of given wordGet WordsetfrombfeimportCountVectorizerct=CountVectorizer()ct.get_wordSet()#Output: get the raw wordset used in training model2. TfIdfFit n TransformTransformCoefficientsFit n TransformfrombfeimportTfIdfVectorizerk=TfIdfVectorizer()doc=["কাওছার আহমেদ","শুভ হাইদার"]matrix1=k.fit_transform(doc)print(matrix1)'''Output:[[0.150515 0.150515 0.       0.      ][0.       0.       0.150515 0.150515]]'''TransformfrombfeimportTfIdfVectorizerk=TfIdfVectorizer()doc=["আহমেদ সুমন","কাওছার করিম"]matrix2=k.transform(doc)print(matrix2)'''Output:[[0.150515 0.       0.       0.      ][0.       0.150515 0.       0.      ]]'''CoefficientsfrombfeimportTfIdfVectorizerk=TfIdfVectorizer()doc=["কাওছার আহমেদ","শুভ হাইদার"]k.fit_transform(doc)wordset,idf=k.coefficients()print(wordset)#Output: ['আহমেদ', 'কাওছার', 'হাইদার', 'শুভ']print(idf)'''Output:{'আহমেদ': 0.3010299956639812, 'কাওছার': 0.3010299956639812, 'হাইদার': 0.3010299956639812, 'শুভ': 0.3010299956639812}'''3. Word EmbeddingWord2VecTrainingGet Word VectorGet SimilarityGet n Similar WordsGet Middle WordGet Odd WordsGet Similarity PlotTrainingfrombfeimportBN_Word2Vec#Training Against Sentencesw2v=BN_Word2Vec(sentences=[['আমার','প্রিয়','জন্মভূমি'],['বাংলা','আমার','মাতৃভাষা']])w2v.train_Word2Vec()#Training Against one Datasetw2v=BN_Word2Vec(corpus_file="path to data or txt file")w2v.train_Word2Vec()#Training Against Multiple Dataset'''path->data->1.txt->2.txt->3.txt'''w2v=BN_Word2Vec(corpus_path="path/data")w2v.train_Word2Vec(epochs=25)After training is done the model "w2v_model"  along with it's supportive vector files will be saved to current directory.If you use any pretrained model, specify it while initializing BN_Word2Vec() . Otherwise no model_name is needed.Get Word VectorfrombfeimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_wordVector('আমার')Get SimilarityfrombfeimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_similarity('ঢাকা','রাজধানী')#Output: 67.457879Get n Similar WordsfrombfeimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_n_similarWord(['পদ্মা'],n=10)#Output:'''[('সেতুর', 0.5857524275779724),('মুলফৎগঞ্জ', 0.5773632526397705),('মহানন্দা', 0.5634652376174927),("'পদ্মা", 0.5617109537124634),('গোমতী', 0.5605217218399048),('পদ্মার', 0.5547558069229126),('তুলসীগঙ্গা', 0.5274507999420166),('নদীর', 0.5232067704200745),('সেতু', 0.5225246548652649),('সেতুতে', 0.5192927718162537)]'''Get Middle WordGet the probability distribution of the center word given words list.frombfeimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_outputWord(['ঢাকায়','মৃত্যু'],n=2)#Output:  [("হয়েছে।',", 0.05880642), ('শ্রমিকের', 0.05639163)]Get Odd WordsGet the most unmatched word out from given words listfrombfeimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_oddWords(['চাল','ডাল','চিনি','আকাশ'])#Output: 'আকাশ'Get Similarity PlotCreates a barplot of similar words with their probabilityfrombfeimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_oddWords(['চাল','ডাল','চিনি','আকাশ'])FastTextTrainingGet Word VectorGet SimilarityGet n Similar WordsGet Middle WordGet Odd WordsTrainingfrombfeimportBN_FastText#Training Against Sentencesft=FastText(sentences=[['আমার','প্রিয়','জন্মভূমি'],['বাংলা','আমার','মাতৃভাষা']])ft.train_fasttext()#Training Against one Datasetft=FastText(corpus_file="path to data or txt file")ft.train_fasttext()#Training Against Multiple Dataset'''path->data->1.txt->2.txt->3.txt'''ft=FastText(corpus_path="path/data")ft.train_fasttext(epochs=25)After training is done the model "ft_model"  along with it's supportive vector files will be saved to current directory.If you use any pretrained model, specify it while initializing BN_FastText() . Otherwise no model_name is needed.Get Word VectorfrombfeimportBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_wordVector('আমার')Get SimilarityfrombfeimportBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_similarity('ঢাকা','রাজধানী')#Output: 70.56821120Get n Similar Wordsfrombfe" import BN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_n_similarWord(['পদ্মা'],n=10)#Output:'''[('পদ্মায়', 0.8103810548782349),('পদ্মার', 0.794012725353241),('পদ্মানদীর', 0.7747839689254761),('পদ্মা-মেঘনার', 0.7573559284210205),('পদ্মা.', 0.7470568418502808),('‘পদ্মা', 0.7413997650146484),('পদ্মাসেতুর', 0.716225266456604),('পদ্ম', 0.7154797315597534),('পদ্মহেম', 0.6881639361381531),('পদ্মাবত', 0.6682782173156738)]'''Get Odd WordsGet the most unmatched word out from given words listfrom"package_name"importBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_oddWords(['চাল','ডাল','চিনি','আকাশ'])#Output: 'আকাশ'Get Similarity PlotCreates a barplot of similar words with their probabilityfrombfeimportBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_oddWords(['চাল','ডাল','চিনি','আকাশ'])