Package: ekushey
Ekushey"Ekushey" is the First Structured and Cost-Effective Bangla Natural Language Processing ToolkitCurrent ModulesFeature Extractionfeature_extraction is a Bangla Natural Language Processing based feature extractorFeature ExtractionCountVectorizerHashVectorizerTfIdfWord EmbeddingWord2VecFastTextGloVeInstallationpip install ekusheyExample1. CountVectorizerFit n TransformTransformGet WordsetFit n Transformfromekushey.feature_extractionimportCountVectorizerct=CountVectorizer()X=ct.fit_transform(X)# X is the word featuresOutput:the countVectorized matrix form of given featuresTransformfromekushey.feature_extractionimportCountVectorizerct=CountVectorizer()get_mat=ct.transform("রাহাত")Output:the countVectorized matrix form of given wordGet Wordsetfromekushey.feature_extractionimportCountVectorizerct=CountVectorizer()ct.get_wordSet()Output:get the raw wordset used in training model2. HashVectorizerFit n TransformTransformfromekushey.feature_extractionimportHashVectorizercorpus=['আমাদের দেশ বাংলাদেশ','আমার বাংলা']Vectorizer=HashVectorizer()n_features=8X=Vectorizer.fit_transform(corpus,n_features)corpus_t=["আমাদের দেশ অনেক সুন্দর"]Xf=Vectorizer.transform(corpus_t)print(X.shape,Xf.shape)print("=====================================")print(X)print("=====================================")print(Xf)Output:(2, 8) (1, 8)
=====================================
  (0, 7)	-1.0
  (1, 7)	-1.0
=====================================
  (0, 0)	0.5773502691896258
  (0, 2)	0.5773502691896258
  (0, 7)	-0.5773502691896258Get Wordset3. TfIdfFit n TransformTransformCoefficientsFit n Transformfromekushey.feature_extractionimportTfIdfVectorizerk=TfIdfVectorizer()doc=["কাওছার আহমেদ","শুভ হাইদার"]matrix1=k.fit_transform(doc)print(matrix1)Output:[[0.150515 0.150515 0.       0.      ]
 [0.       0.       0.150515 0.150515]]Transformfromekushey.feature_extractionimportTfIdfVectorizerk=TfIdfVectorizer()doc=["আহমেদ সুমন","কাওছার করিম"]matrix2=k.transform(doc)print(matrix2)Output:[[0.150515 0.       0.       0.      ]
 [0.       0.150515 0.       0.      ]]Coefficientsfromekushey.feature_extractionimportTfIdfVectorizerk=TfIdfVectorizer()doc=["কাওছার আহমেদ","শুভ হাইদার"]k.fit_transform(doc)wordset,idf=k.coefficients()print(wordset)#Output: ['আহমেদ', 'কাওছার', 'হাইদার', 'শুভ']print(idf)'''Output:{'আহমেদ': 0.3010299956639812, 'কাওছার': 0.3010299956639812, 'হাইদার': 0.3010299956639812, 'শুভ': 0.3010299956639812}'''4. Word EmbeddingWord2VecTrainingGet Word VectorGet SimilarityGet n Similar WordsGet Middle WordGet Odd WordsGet Similarity PlotTrainingfromekushey.feature_extractionimportBN_Word2Vec#Training Against Sentencesw2v=BN_Word2Vec(sentences=[['আমার','প্রিয়','জন্মভূমি'],['বাংলা','আমার','মাতৃভাষা'],['আমার','প্রিয়','জন্মভূমি'],['বাংলা','আমার','মাতৃভাষা'],['আমার','প্রিয়','জন্মভূমি'],['বাংলা','আমার','মাতৃভাষা']])w2v.train()#Training Against one Text Corpusw2v=BN_Word2Vec(corpus_file="path_to_corpus.txt")w2v.train()#Training Against Multiple corpuses'''path->corpus->1.txt->2.txt->3.txt'''w2v=BN_Word2Vec(corpus_path="path/corpus")w2v.train(epochs=25)#Training Against a Dataframe Columnw2v=BN_Word2Vec(df=news_data['text_content'])w2v.train(epochs=25)After training is done the model "w2v_model"  along with it's supportive vector files will be saved to current directory.If you use any pretrained model, specify it while initializing BN_Word2Vec() . Otherwise no model_name is needed.Get Word Vectorfromekushey.feature_extractionimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_wordVector('আমার')Get Similarityfromekushey.feature_extractionimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_similarity('ঢাকা','রাজধানী')Output:67.457879Get n Similar Wordsfromekushey.feature_extractionimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_n_similarWord(['পদ্মা'],n=10)Output:[('সেতুর', 0.5857524275779724),
 ('মুলফৎগঞ্জ', 0.5773632526397705),
 ('মহানন্দা', 0.5634652376174927),
 ("'পদ্মা", 0.5617109537124634),
 ('গোমতী', 0.5605217218399048),
 ('পদ্মার', 0.5547558069229126),
 ('তুলসীগঙ্গা', 0.5274507999420166),
 ('নদীর', 0.5232067704200745),
 ('সেতু', 0.5225246548652649),
 ('সেতুতে', 0.5192927718162537)]Get Middle WordGet the probability distribution of the center word given words list.fromekushey.feature_extractionimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_outputWord(['ঢাকায়','মৃত্যু'],n=2)Output:[("হয়েছে।',", 0.05880642), ('শ্রমিকের', 0.05639163)]Get Odd WordsGet the most unmatched word out from given words listfromekushey.feature_extractionimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_oddWords(['চাল','ডাল','চিনি','আকাশ'])Output:'আকাশ'Get Similarity PlotCreates a barplot of similar words with their probabilityfromekushey.feature_extractionimportBN_Word2Vecw2v=BN_Word2Vec(model_name='give the model name here')w2v.get_similarity_plot('চাউল',5)FastTextTrainingGet Word VectorGet SimilarityGet n Similar WordsGet Middle WordGet Odd WordsTrainingfromekushey.feature_extractionimportBN_FastText#Training Against Sentencesft=ft=BN_FastText(sentences=[['আমার','প্রিয়','জন্মভূমি'],['বাংলা','আমার','মাতৃভাষা'],['বাংলা','আমার','মাতৃভাষা'],['বাংলা','আমার','মাতৃভাষা'],['বাংলা','আমার','মাতৃভাষা']])ft.train()#Training Against one Text Corpusft=BN_FastText(corpus_file="path to data or txt file")ft.train()#Training Against Multiple Corpuses'''path->Corpus->1.txt->2.txt->3.txt'''ft=BN_FastText(corpus_path="path/Corpus")ft.train(epochs=25)#Training Against a Dataframe Columnft=BN_FastText(df=news_data['text_content'])ft.train(epochs=25)After training is done the model "ft_model"  along with it's supportive vector files will be saved to current directory.If you don't want to train instead use a pretrained model, specify it while initializing BN_FastText() . Otherwise no model_name is needed.Get Word Vectorfromekushey.feature_extractionimportBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_wordVector('আমার')Get Similarityfromekushey.feature_extractionimportBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_similarity('ঢাকা','রাজধানী')Output:70.56821120Get n Similar Wordsfromekushey.feature_extraction" import BN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_n_similarWord(['পদ্মা'],n=10)Output:[('পদ্মায়', 0.8103810548782349),
 ('পদ্মার', 0.794012725353241),
 ('পদ্মানদীর', 0.7747839689254761),
 ('পদ্মা-মেঘনার', 0.7573559284210205),
 ('পদ্মা.', 0.7470568418502808),
 ('‘পদ্মা', 0.7413997650146484),
 ('পদ্মাসেতুর', 0.716225266456604),
 ('পদ্ম', 0.7154797315597534),
 ('পদ্মহেম', 0.6881639361381531),
 ('পদ্মাবত', 0.6682782173156738)]Get Odd WordsGet the most unmatched word out from given words listfrom"package_name"importBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_oddWords(['চাল','ডাল','চিনি','আকাশ'])Output:'আকাশ'Get Similarity PlotCreates a barplot of similar words with their probabilityfromekushey.feature_extractionimportBN_FastTextft=BN_FastText(model_name='give the model name here')ft.get_similarity_plot('চাউল',5)GloVeTrainingGet n Similar WordsTrainingfromekushey.feature_extractionimportBN_GloVe#Training Against Sentencesglv=BN_GloVe(sentences=[['আমার','প্রিয়','জন্মভূমি'],['বাংলা','আমার','মাতৃভাষা'],['বাংলা','আমার','মাতৃভাষা'],['বাংলা','আমার','মাতৃভাষা'],['বাংলা','আমার','মাতৃভাষা']])glv.train()#Training Against one Text Corpusglv=BN_GloVe(corpus_file="path_to_corpus.txt")glv.train()#Training Against Multiple Corpuses'''path->Corpus->1.txt->2.txt->3.txt'''glv=BN_GloVe(corpus_path="path/corpus")glv.train(epochs=25)#Training Against a Dataframe Columnglv=BN_GloVe(df=news_data['text_content'])glv.train(epochs=25)After training is done the model "glove_model"  along with it's supportive vector files will be saved to current directory.If you don't want to train instead use a pretrained model, specify it while initializing BN_FastText() . Otherwise no model_name is needed.Get n Similar Wordsfromekushey.feature_extraction" import BN_GloVeglv=BN_GloVe(model_name='give the model name here')glv.get_n_similarWord(['পদ্মা'],n=10)Output:[('পদ্মায়', 0.8103810548782349),
 ('পদ্মার', 0.794012725353241),
 ('পদ্মানদীর', 0.7747839689254761),
 ('পদ্মা-মেঘনার', 0.7573559284210205),
 ('পদ্মা.', 0.7470568418502808),
 ('‘পদ্মা', 0.7413997650146484),
 ('পদ্মাসেতুর', 0.716225266456604),
 ('পদ্ম', 0.7154797315597534),
 ('পদ্মহেম', 0.6881639361381531),
 ('পদ্মাবত', 0.6682782173156738)]