Package: robics
robicsrobustTopicsis a library targeted atnon-machine learning expertsinterested in building robust
topic models. The main goal is to provide a simple to use framework to check if
a topic model reaches each run the same or at least a similar result.FeaturesSupports sklearn (LatentDirichletAllocation, NMF) and gensim (LdaModel, ldamulticore, nmf) topic modelsCreates samples based on thesobol sequencewhich requires less samples than grid-search and makes sure the whole parameter space is used which is not sure in random-sampling.Simple topic matching between the different re-initializations for each sample using word vector based coherence scores.Ranking of all models based on four metrics:Jaccard distanceof the top n words for each topicSimilarity of topic distributions based on theJensen Shannon DivergenceRanking correlation of the top n words based onKendall's TauWord vector based coherence score (simple version of the TC-W2V)Word based analysis of samples and topic model instances.InstallPython Version:3.5+Package Managers:pippipUsing pip, robics releases are available as source packages and binary wheels:pip install robicsExamplesTest dataset from sklearnfromsklearn.datasetsimportfetch_20newsgroups# PREPROCESSINGdataset=fetch_20newsgroups(shuffle=True,random_state=1,remove=('headers','footers','quotes'))# Only 1000 dokuments for performance reasonsdocuments=dataset.data[:1000]Load word vectors used for coherence computationimportspacynlp=spacy.load("en_core_web_md")Detect robust sklearn topic modelsfromsklearn.feature_extraction.textimportTfidfVectorizer,CountVectorizerfromsklearn.decompositionimportLatentDirichletAllocation,NMFfromrobicsimportRobustTopics# Document vectorization using TFIDFtf_vectorizer=CountVectorizer(max_df=0.95,min_df=2,stop_words='english')tf=tf_vectorizer.fit_transform(documents)tf_feature_names=tf_vectorizer.get_feature_names()# TOPIC MODELLINGrobustTopics=RobustTopics(nlp)# Load  NMF and LDA modelsrobustTopics.load_sklearn_model(LatentDirichletAllocation,tf,tf_vectorizer,dimension_range=[5,50],n_samples=4,n_initializations=3)robustTopics.load_sklearn_model(NMF,tf,tf_vectorizer,dimension_range=[5,50],n_samples=4,n_initializations=3)# Fit the models - Warning, this might take a lot of time based on the number of samples (n_models*n_sample*n_initializations)robustTopics.fit_models()Analysis
We start by looking at the ranking of all modelsrobustTopics.rank_models()# Output:[{'model':sklearn.decomposition._nmf.NMF,'model_id':1,'sample_id':0,'n_topics':27,'params':{'n_components':27},'topic_coherence':0.3184709231908624,'jaccard':0.9976484420928865,'kendalltau':0.9987839560655095,'jensenshannon':0.9999348301145501},{'model':sklearn.decomposition._nmf.NMF,'model_id':1,'sample_id':3,'n_topics':21,'params':{'n_components':21},'topic_coherence':0.31157823571739196,'jaccard':1.0,'kendalltau':1.0,'jensenshannon':0.9999784246484099},...{'model':sklearn.decomposition._lda.LatentDirichletAllocation,'model_id':0,'sample_id':1,'n_topics':38,'params':{'n_components':38},'topic_coherence':0.30808284548733383,'jaccard':0.07074176815158277,'kendalltau':0.1023597955928783,'jensenshannon':0.22596536037699871}]The top model is the NMF model with 27 topics (model_id 1 and sample_id 0). The next step is to look at the model on a word level.robustTopics.display_sample_topic_consistency(model_id=1,sample_id=0)# Output:Wordsin3outof3runs:['edu','graphics','pub','mail','ray','128','send','3d','ftp','com','objects','server','amiga','image','archie','files','file','images','archive','package','section','firearm','weapon','military','license','shall','dangerous','person','division','application','means','device','use','following','issued','state','act',...Wordsin2outof3runs:['know']Wordsin1outof3runs:['goals']Most of the words are in all of the three re-initializations. Only 'know' and 'goals' are inconsistent.In comparison lets look at the model performing worst:robustTopics.display_sample_topic_consistency(model_id=0,sample_id=1)# Output:Wordsin3outof3runs:['know','know','people','use','drive','think','just','like','just','people','just','know','think','good','think','like','people','know','don','think','like','just','don','know','just','don','think','like','windows','like','don']Wordsin2outof3runs:['just','think','people','dc','just','like','want','said','use','said','local','say','god','shall','rights','know','like','window','like','just','time','new','like','just','good','don','used','does','think','like','new','state','like','contact','know','bike','just','like','year','data','use','way','ve','people','don','know','didn','years','little','rocket','like','generation','build','ll','max','ssrt','just','time','using','edu','ftp','file','available','server','10','good','new','know','people','way','know','good','don','just','like','time','like','don','insurance','year','time','car','years','people','say','new','new','need','just','think','like','good','just','don','work','time','government','rights','make','people','edu','com','graphics','time','good','people','need','like','don','know','just','people','server','edu','file','video','support','mit','ftp','linux','time','binaries','information','available','new','greek','10','just','just','does','know','use','need','like','don','use','know','think','like','new','like','just','edu','ve','does','problem','think','just','way','power','wrong','edu','points','point','just','hp','don','good','people','phone','food','just','bit','good','know','just','like','use','people','used','going','people','know','time','say','use','drive','10','like','observations','think','don','want','thing','know','ll','good','like','mm','used','say']Wordsin1outof3runs:['jews','israel','true','state','year',...The worst model has most words in the 1 out of 3 runs section and only filler words are consistent between the runs.
"know" appears five times which means, it is in five different topics in all three runs a top word.Let us look now at the topic in the top and bottom models.# Top modelrobustTopics.display_sample_topics(1,0)# OutputTopic0In3runs:edugraphicspubmailray128send3dftpcomobjectsserveramigaimagearchiefilesfileimagesarchivepackageTopic1In3runs:sectionfirearmweaponmilitarylicenseshalldangerouspersondivisionapplicationmeansdeviceusefollowingissuedstateactdesignedcodeautomaticTopic2In3runs:aidshealthcaresaidchildrenmedicalinfectednewpatientsdisease199310informationresearchaprilnationalstudytrialsservicenumberTopic3In3runs:godgoodpeoplejustsupposebrothersmakesfisherlikedoesjewsjosephdidworshipjudaismrightinsteadjesusendreadTopic4In3runs:serversupportedufile386bsdftpmitbinariesvgainformationsupportedlinuxsvgareadmeavailableosnewfaqfilesvideoTopic5In3runs:educommilnavycsvotemiscvoteshealthcahpnrlgovemailcccreationaujohnthomasukTopic6In3runs:probespacetitanearthorbiterlaunchmissionjupiterorbitatmosphere93saturngravity10surfacesatelliteray12possible97# Bottom modelrobustTopics.display_sample_topics(0,1)# OutputTopic0In3runs:knowTopic1In3runs:knowTopic2Topic3Topic4In3runs:peopleTopic5Topic6In3runs:useTopic7Topic8In3runs:drivethinkjustThe top model produces over all runs consistent topic with words that are connected.
In strong contrast to the bottom model, which has very little consistent words and these are provide little information.We want to take a look at the topics of one of the instances from the top model (model 1, sample 0, instance 0).robustTopics.display_run_topics(1,0,0)# OutputTopic0:edugraphicspubmailray128send3dftpcomobjectsserveramigaimagearchiefilesfileimagesarchivepackageTopic1:sectionfirearmweaponmilitarylicenseshalldangerouspersondivisionapplicationmeansdeviceusefollowingissuedstateactdesignedcodeautomaticTopic2:aidshealthcaresaidchildrenmedicalinfectednewpatientsdisease199310informationresearchaprilnationalstudytrialsservicenumberTopic3:godgoodpeoplejustsupposebrothersmakesfisherlikedoesjewsjosephdidworshipjudaismrightinsteadjesusendreadTopic4:serversupportedufile386bsdftpmitbinariesvgainformationsupportedlinuxsvgareadmeavailableosnewfaqfilesvideoTopic5:educommilnavycsvotemiscvoteshealthcahpnrlgovemailcccreationaujohnthomasukTopic6:probespacetitanearthorbiterlaunchmissionjupiterorbitatmosphere93saturngravity10surfacesatelliteray12possible97If the output is meaningful we can now use this model for further use
or export more indept information for additional analysis.# Use the results in your own workflow# Export specific model based on the ranked models and the analysis (model_id, sample_id, run_id)sklearn_model=robustTopics.export_model(1,0,0)# Look at the full reports inclusing separate values for each initializationrobustTopics.models[1].report_full# Convert the full report to a pandas dataframe for further use or exportimportpandasaspdreport=pd.DataFrame.from_records(robustTopics.models[model_id].report)# or report_fullGensim topic models are also supported. Following is a example how to setup a
simple pipeline. The analysis steps are exactly the same as above.fromgensim.modelsimportLdaModel,nmf,ldamulticorefromgensim.utilsimportsimple_preprocessfromgensimimportcorporafromrobicsimportRobustTopicsdefdocs_to_words(docs):fordocindocs:yield(simple_preprocess(str(doc),deacc=True))tokenized_data=list(docs_to_words(documents))dictionary=corpora.Dictionary(tokenized_data)corpus=[dictionary.doc2bow(text)fortextintokenized_data]# TOPIC MODELLINGrobustTopics=RobustTopics(nlp)# Load 4 different modelsrobustTopics.load_gensim_model(ldamulticore.LdaModel,corpus,dictionary,dimension_range=[5,50],n_samples=4,n_initializations=3)robustTopics.load_gensim_model(nmf.Nmf,corpus,dictionary,dimension_range=[5,50],n_samples=4,n_initializations=3)robustTopics.fit_models()# Same analysis steps as in the sklearn exampleNext StepsVisual interfaceAdding support for more modells if required.Add loggingWriting unit tests.Improving the overall performance.Implementing the Cv coherence measure from thispaperContributionI am happy to receive help in any of the things mentioned above or other interesting feature request.