Package: bertsenclu
Bert-SenClu(Bert-)SenCluis a topic modeling technique that leverages sentence transformers to compute topic models. For once, it differs from other topic models by using sentences as unit of analysis, i.e., a sentence is assigned to a topic and not a word (like for LDA, TKM) or an entire document (BertTopic).
Methods that treat documents as a unit can be faster but they only assign the entire document to one topic, which is different from most classical topic models that produce a document-topic distribution, i.e., a document can contain multiple documents. Our topic model also does not do a dimensionality reduction of embeddings. Inference is based on expectation-maximization, e.g., like for TKM (seeTKM PaperandTKM Code).For an in-depth overview of the features ofBert-SenCluyou can check therepositoryor the paperthe paper.InstallationInstallation, with sentence-transformers, can be done usingpypi:pipinstallbertsencluQuick StartWe start by extracting topics from the 20 newsgroups dataset containing English documents:fromsklearn.datasetsimportfetch_20newsgroupsimportnumpyasnpfrombertSenCluimportsenCludocs=fetch_20newsgroups(subset='train',remove=('headers','footers','quotes'))# get raw datafolder="modelOutputs/"topic_model=senClu.SenClu()topics,probs=topic_model.fit_transform(docs,nTopics=20,loadAndStoreInFolder=folder)After generating topics and their probabilities, we can save outputs:>>topic_model.saveOutputs(folder)#Save outputs in folder, i.e. csv-file and visualizationsand look at topics>>forit,tinenumerate(topics):print("Topic",it,t[:10])Topic0[encryption,key,ripem,privacy,rsa,clipper,encrypted,escrow,nsa,secure]Topic1[government,militia,amendment,federal,law,constitution,firearm,regulated,administration,clinton]Topic2[launch,satellite,lunar,space,orbit,mission,spacecraft,larson,probe,shuttle]Topic3[patient,hiv,disease,infection,candida,vitamin,antibiotic,diet,symptom,smokeless]...We can also use an interactive tool for visualization and topic analysis that runs in a browser. It can be called command line with the folder containing topic modeling outputs:You need todownloadthevisual.pyfrom the repo firststreamlit run visual.py -- --folder "modelOutputs/"It can also be called from python:importsubprocessfolder="modelOutputs/"subprocess.run("streamlit run visual.py -- --folder "+folder,shell=True)The interactive visualization looks like this:If you scroll down (or look into the folder where you stored outputs), you see topic relationship information as well, i.e., a TSNE visualization and a hierarchical clustering of topics:We can also access outputs directly by accessing functions from the model>>print("Top 10 words with scores for topic 0",topic_model.getTopicsWitchScores()[0][:10])Top10wordswithscoresfortopic0[('encryption',11.269135),('key',11.173454),('ripem',10.151058),('privacy',10.070835),('rsa',7.3271174),('clipper',6.8211393),('encrypted',6.567956),('escrow',5.993511),('nsa',5.853071),('secure',5.4898496)]>>print("Distribution of topics for document 0",np.round(topic_model.getTopicDocumentDistribution()[0],3))Distributionoftopicsfordocument0[0.0.1....0.0.0.]>>print("Distribution of topics",np.round(topic_model.getTopicDistribution(),3))Distributionoftopics[0.0220.0610.0240.0260.0670.0790.1550.0430.0610.0390.0310.1980.0180.0330.0330.0120.0160.0290.0330.02]>>print("First 4 sentences for top doc for topic 0 with probability and ",topic_model.getTopDocsPerTopic()[0][0][:4])First4sentencesfortopdocfortopic0(['[...]>\n','[...]>\n\n',"If the data isn't there when the warrant comes, you effectively have\n",'secure crypto.  '],1.0,8607)>>print("Top 3 sentences for topic 0 ",topic_model.getTopSentencesPerTopic()[0][:5])Top3sentencesfortopic1[('enforcement.\n\n',0.22597079),('Enforcement.  ',0.22597079),('to the Constitution.\n\n',0.22434217)]#The sentences show that the sentence partitioning algorithm used is not the best... (It splits based on carriage returns. Still topic modeling results are good. It's also easy to use another one, or preprocess the dataHow it worksThe steps for topic modeling withBert-SenCluareSplitting docs into sentencesEmbedding the sentences using pretrained sentence-transformersRunning the topic modelingComputing topic-word distributions based on sentence to topic assignmentsThe outcomes of the first two steps are stored in a user-provided folder if parameter "loadAndStoreInFolder" is set explicitly in "fit_transform". By default this is not the case (i.e., "loadAndStoreInFolder"=None).  **Bert-SenClu** can reuse the stored precomputed sentence partitionen and embeddings, which speeds up re-running the topic modeling, e.g., if you want to change the number of topics. However, if you alter the data, you need to delete the folder, i.e., the files with the precomputed sentence embeddings and partitionings.You can change each algorithm in these steps, especially the algorithm for sentence partitioning as well as the pre-trained sentence embedder. As you saw in the example, the used algorithm for sentence partitioning is not that great for the newsgroup dataset, but the overall result is still good.The (main) function "fit_transform" has a hyperparameter "alpha" (similar to other models like LDA), which guides the algorithm on how many topics a document should contain. Setting it 0, means that a document likely has few topics. Setting it to 1 (or larger) means it is more likely to have many (for longer documents). As default, you can use 0.5/sqrt(nTopics).CitationTo cite theBert-SenClu Paper, please use the following bibtex reference:@article{schneider23,title={Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences},author={Schneider,Johannes},journal={arXiv preprint arXiv:2302.03106},year={2023}}