Package: bertnlp
BERT NLP toolkitBERT NLP toolkit (https://pypi.org/project/bertnlp) is a Python package that performs various NLP tasks using Bidirectional Encoder Representations from Transformers (BERT) related models.InstallationTo Install this package using pip:pipinstallbertnlp-0.0.x-py3-none-any.whl-fhttps://download.pytorch.org/whl/torch_stable.htmlTo fetch the nightly version that is in-development, visit this project homepage (https://github.com/daishuanglu/bertnlp).Implemented NLP SolutionsBERT tokenizerBERT word embedding and fuzzy matcherBERT sentence embeddingModified BERT sentiment scoreText classifier based on KNN-bert and trainerText classifier based on FastText and trainerMulti-labelled text intent detector based on FastText and trainerUsageTo use this bert package as a SDK,frombertnlp.fuzzy_matcherimportsemanticMatcherfrombertnlp.pipelineimportsentiment,embeddings,tokenizerfrombertnlp.text_classifierimportknnbertasbert_clfcorpus=['The cat sits outside','A man is playing guitar','I love pasta','The new movie is awesome','The cat plays in the garden','A woman watches TV','The new movie is so great','Do you like pizza?','cat','TV']feature_list=['cat','dog','television','guitar','movie','pizza','pasta']matcher=semanticMatcher()sentimentScorer=sentiment(neu_range=0.2)senti_pred=sentimentScorer.score(corpus)forj,sentinenumerate(corpus):features=matcher.match_sent(sent,feature_list,threshold=0.3)feature_mentioned=';'.join(['{:s}, score:{:.4f}'.format(f['label'],f['score'])forfinfeatures])print("[Sentence]{:s}; [Sentiment]{:s}, score:{:.4f}; [Feature Mentioned]{:s}".format(sent,senti_pred[j]['label'],senti_pred[j]['score'],feature_mentioned))print('\n+ Extra pipeline features added 12142020:')senti_pred=sentimentScorer.predict(corpus)senti_score=sentimentScorer.predict_proba(corpus)bert_tok=tokenizer()emb=embeddings()# sentence bert embeddings: Input  - list of sentences, Output - 2D numpy arraysent_emb=emb.sbert_emb(corpus)print('embedding shape:',sent_emb.shape)forj,sentinenumerate(corpus):print("[Sentence]{:s}; [Sentiment proba] :{:s};  [1-5th dimensions of sentence embedding]{:s}".format(sent,str(senti_score[j,:].tolist()),str(sent_emb[j,:5].tolist())))# bert word embeddings: Input  - list of words, Output - 2D numpy arraytokens=bert_tok.token(sent)word_emb=emb.bert_emb(tokens)fori,tokinenumerate(tokens):print("[Token]{}; [1-5th dimension of Word Embeddings]{}".format(tok,word_emb[i,:5]))print('Embedding cosine similarity as text relevancy:')print(emb.cos_sim(sent_emb[:3],sent_emb[-3:]))To train a text classifierfrombertnlp.text_classifierimportknnbert,trainerimportnumpyasnpfrombertnlp.utilsimportget_example_datafrombertnlp.measureimportplotConfMatdefdrop_class(X,y,classname):sel_id=[ifori,yyinenumerate(y)ifyy!=classname]return[X[i]foriinsel_id],[y[i]foriinsel_id]data,senti_cat,subcat,senti_label,featureMent=get_example_data('train','ISO-8859-1')test_data,test_senti_cat,test_subcat,test_senti_label,test_featureMent=get_example_data('test','ISO-8859-1')sbert_model_name='roberta-base-nli-stsb-mean-tokens'cat_data_tr,cat_tr=drop_class(data,senti_cat,"Critique")cat_data_te,cat_te=drop_class(test_data,test_senti_cat,"Critique")cat_model=knnbert(sbert_model_name=sbert_model_name)# using evaluation mode to check training-validation performance statstrainer(cat_data_tr+cat_data_te,cat_tr+cat_te,cat_model,eval_round=10)# enable save_model_path to generate a deployable model on the overall dataset.cat_model=trainer(cat_data_tr+cat_data_te,cat_tr+cat_te,cat_model,save_model_path='./model.pkl')cat_pred=cat_model.predict(cat_data_te)print('senti_cat sbert model test accuracy{}'.format((cat_pred==np.array(cat_te)).mean()))plotConfMat(cat_te,cat_pred,cat_model.classes_,'cat_sbertClf_confmat.png')The bert package support multi-labelled text intent detection, which can adapted to multiple NLP tasks such as (1) intent detection, (2) multi-intent detection, and other text classification tasks.
Different from text classification, the multi-labelled text intent detection can (1) check 'None' class and (2) classify a single text into multiple labels as many as it detects.
To train a multi-labelled text detection model,frombertnlp.text_classifierimportfasttextClfimportnumpyasnpfrombertnlp.utilsimportget_example_datafrombertnlp.measureimportplotPrecisionRecallfrombertnlp.fuzzy_matcherimportfeat_predict_func# load the example data for trainingdata,senti_cat,subcat,senti_label,featureMent=get_example_data('train','ISO-8859-1')test_data,test_senti_cat,test_subcat,test_senti_label,test_featureMent=get_example_data('test','ISO-8859-1')# save all the Mentioned features as a list of feature namesfeaturelist=list(set(sum(test_featureMent,[])+sum(featureMent,[])))X_tr=[data[i]fori,finenumerate(featureMent)if'None'notin' '.join(f)]X_te=test_data# To reduce data imbalance, drop None classes during training.featMent_ftmodel=fasttextClf()tr_featMent=[''.join(f)forfinfeatureMentif'None'notin' '.join(f)]featMent_ftmodel.fit(X_tr,tr_featMent,lr=1.0,epoch=100,wordNgrams=2,loss='ova')featMent_ftmodel.model.save_model('./featMent_ftClf.bin')# mixing a edit-distance based text fuzzy matcher with the multi-intent detector to improve simple cases.combinedModel={'ftmodel':featMent_ftmodel,'featlist':featurelist}defpredict_func(data,model,conf_th):returnfeat_predict_func(data,model,conf_th)[0]feat_pred,_=feat_predict_func(X_te,combinedModel,0.2)print(feat_pred)test_featMent=[''.join(f)forfintest_featureMent]print(test_featureMent)# using test features to check performance stats: Precision-Recall are usedbest_prec_rec,best_thresh=plotPrecisionRecall(X_te,test_featureMent,combinedModel,predict_func,\conf_thresh_range=np.arange(0,1,0.1),fig_path='./aspect_ftClf_prec_rec.png')print('The best Precision- Recall is evaluated at confident threshold{}:'.format(best_thresh))print('Precision: ',best_prec_rec['precision'],'Recall: ',best_prec_rec['recall'])forc,rateinbest_prec_rec['prec_by_class'].items():print('class',c,': ',rate)