Package: kwx
BERT, LDA, and TFIDF based keyword extraction in Pythonkwxis a toolkit for multilingual keyword extraction based on Google'sBERT,Latent Dirichlet AllocationandTerm Frequency Inverse Document Frequency. The package provides a suite of methods to process texts of any language to varying degrees and then extract and analyze keywords from the created corpus (seekwx.languagesfor the various degrees of language support). A unique focus is allowing users to decide which words to not include in outputs, thereby guaranteeing sensible results that are in line with user intuitions.For a thorough overview of the process and techniques see theGoogle slides, and reference thedocumentationfor explanations of the models and visualization methods.ContentsInstallationModelsBERTLDATFIDFWord FrequencyUsageText CleaningKeyword ExtractionVisualsTopic Number Evaluationt-SNEpyLDAvisWord CloudTo-DoInstallation⇧kwx can be downloaded from PyPI via pip or sourced directly from this repository:pipinstallkwxgitclonehttps://github.com/andrewtavis/kwx.gitcdkwx
pythonsetup.pyinstallimportkwxModels⇧Implemented NLP modeling methods withinkwx.modelinclude:• BERT⇧Bidirectional Encoder Representations from Transformersderives representations of words based on nlp models ran over open-source Wikipedia data. These representations are then leveraged to derive corpus topics.kwx usessentence-transformerspretrained models. See their GitHub anddocumentationfor the available models.• LDA⇧Latent Dirichlet Allocationis a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. In the case of kwx, documents or text entries are posited to be a mixture of a given number of topics, and the presence of each word in a text body comes from its relation to these derived topics.Although not as computationally robust as some machine learning models, LDA provides quick results that are suitable for many applications. Specifically for keyword extraction, in most settings the results are similar to those of BERT in a fraction of the time.• TFIDF⇧The user can also computeTerm Frequency Inverse Document Frequencykeywords - those that are unique in a text body in comparison to another that's compared. This is a useful baseline when a user has another text or text body to compare the target corpus against.• Word Frequency⇧Finally a user can simply query the most common words from a text corpus. This method is used in kwx as a baseline to check model efficacy.Usage⇧Keyword extraction can be useful to analyze surveys, tweets and other kinds of social media posts, research papers, and further classes of texts.examples/kw_extractionprovides an example of how to use kwx by deriving keywords from tweets in the KaggleTwitter US Airline Sentimentdataset.The following outlines using kwx to derive keywords from a text corpus withprompt_remove_wordsasTrue(the user will be asked if some of the extracted words need to be replaced):• Text Cleaning⇧fromkwx.utilsimportprepare_datainput_language="english"# see kwx.languages for options# kwx.utils.clean() can be used on a list of liststext_corpus=prepare_data(data="df_or_csv_xlsx_path",target_cols="cols_where_texts_are",input_language=input_language,min_token_freq=0,# for BERTmin_token_len=0,# for BERTremove_stopwords=False,# for BERTverbose=True,)• Keyword Extraction⇧fromkwx.modelimportextract_kwsnum_keywords=15num_topics=10ignore_words=["words","user","knows","they","don't","want"]# Remove n-grams for BERT trainingcorpus_no_ngrams=[" ".join([tfortintext.split(" ")if"_"notint])fortextintext_corpus]# We can pass keywords for sentence_transformers.SentenceTransformer.encode,# gensim.models.ldamulticore.LdaMulticore, or sklearn.feature_extraction.text.TfidfVectorizerbert_kws=extract_kws(method="BERT",# "BERT", "LDA", "TFIDF", "frequency"bert_st_model="xlm-r-bert-base-nli-stsb-mean-tokens",text_corpus=corpus_no_ngrams,# automatically tokenized if using LDAinput_language=input_language,output_language=None,# allows the output to be translatednum_keywords=num_keywords,num_topics=num_topics,corpuses_to_compare=None,# for TFIDFignore_words=ignore_words,prompt_remove_words=True,# check words with usershow_progress_bar=True,batch_size=32,)The BERT keywords are:

['time', 'flight', 'plane', 'southwestair', 'ticket', 'cancel', 'united', 'baggage',
'love', 'virginamerica', 'service', 'customer', 'delay', 'late', 'hour']

Should words be removed [y/n]? y
Type or copy word(s) to be removed: southwestair, united, virginamerica

The new BERT keywords are:

['late', 'baggage', 'service', 'flight', 'time', 'love', 'book', 'customer',
'response', 'hold', 'hour', 'cancel', 'cancelled_flighted', 'delay', 'plane']

Should words be removed [y/n]? nThe model will be rerun until all words known to be unreasonable are removed for a suitable output.kwx.model.gen_filescould also be used as a run-all function that produces a directory with a keyword text file and visuals (for experienced users wanting quick results).Visuals⇧kwx.visualsincludes the following functions for presenting and analyzing the results of keyword extraction:• Topic Number Evaluation⇧A graph of topic coherence and overlap given a variable number of topics to derive keywords from.fromkwx.visualsimportgraph_topic_num_evalsimportmatplotlib.pyplotaspltgraph_topic_num_evals(method=["lda","bert"],text_corpus=text_corpus,num_keywords=num_keywords,topic_nums_to_compare=list(range(5,15)),metrics=True,#  stability and coherence)plt.show()• t-SNE⇧t-SNEallows the user to visualize their topic distribution in both two and three dimensions. Currently available just for LDA, this technique provides another check for model suitability.fromkwx.visualsimportt_sneimportmatplotlib.pyplotaspltt_sne(dimension="both",# 2d and 3d are optionstext_corpus=text_corpus,num_topics=10,remove_3d_outliers=True,)plt.show()• pyLDAvis⇧pyLDAvisis included so that users can inspect LDA extracted topics, and further so that it can easily be generated for output files.fromkwx.visualsimportpyLDAvis_topicspyLDAvis_topics(method="lda",text_corpus=text_corpus,num_topics=10,display_ipython=False,# For Jupyter integration)• Word Cloud⇧Word clouds viawordcloudare included for a basic representation of the text corpus - specifically being a way to convey basic visual information to potential stakeholders. The following figure fromexamples/kw_extractionshows a word cloud generated from tweets of US air carrier passengers:fromkwx.visualsimportgen_word_cloudignore_words=["words","user","knows","they","don't","want"]gen_word_cloud(text_corpus=text_corpus,ignore_words=None,height=500,)To-Do⇧Please see thecontribution guidelinesif you are interested in contributing to this project. Work that is in progress or could be implemented includes:Including more methods to extract keywords(see issue)Adding key phrase extraction as an option forkwx.model.extract_kws(see issues)Adding t-SNE and pyLDAvis style visualizations for BERT models(see issues)Converting the translation feature over to use another translation api rather thanpy-googletrans(see issue)Updates tokwx.languagesas lemmatization and other linguistic package dependencies evolveCreating, improving and sharingexamplesImprovingtestsfor greatercode coverageUpdating and refining thedocumentation