Package: dtmpy
dtmpyA Python module for doing fast Dynamic Topic Modeling.This module wraps the original C/C++ code by David M. Blei and Sean M. Gerrish.I've refactored the original code to wrap the main function call in a classDTMthat has Python bindings. Other code changes are listed below.UsageBelow is an example of how to use this package. Reference thedtmpy.DTMdocstring for all possible keyword arguments.importdtmpy# Initialize DTM model flagsdtm=dtmpy.DTM(corpus_prefix="model_inputs/file_prefix",output_name="model_outputs",n_topics=10)# Fit the model and write outputs from bag-of-words corpus and timestampsdtm.fit(my_corpus,n_docs_per_timestamp)# The proportion of topic 5 in document 3topic_mixtures=dtm.read_topic_mixtures()print(topic_mixtures[3,5])# The probability of term 32 being in topic 5 at time 7topic5=dtm.read_topic_term_probabilities(5)print(topic5[32,7])There is also a sampleseqandmultfile in theexampledirectory, which can be fit as test case using the below code:importdtmpydtm=dtmpy.DTM(corpus_prefix="example/test",output_name="model_outputs")# Load corpus and time intervals from existing seq and mult filescorpus,time_slices=dtm.read_corpus()# Fit the dynamic topic modeldtm.fit(corpus,time_slices)InstallInstall dependencies:cmakelibgsl-devpybind11-devThese can be installed on Ubuntu with:apt-get install cmake libgsl-dev pybind11-devor on macOS usingHomebrewwith:brew install cmake gsl pybind11After installing the dependencies,dtmpycan be installed with:pip install dtmpyor directly from source:git clone https://github.com/jeffmm/dtmpy
cd dtmpy
pip install .Model inputsFitting a corpusThedtmpy.DTMclass can fit a DTM model to a bag-of-words corpus that is the form of a list of lists (one list for each document), with each document list containing tuples of unique words in the document and their counts. As a simple example:# document 1 is the sentence "foo bar bar baz"# document 2 is the sentence "bar baz bang baz"dictionary={0:"foo",1:"bar",2:"baz",3:"bang"}bow_corpus=[[(0,1),(1,2),(2,1)],[(1,1),(2,2),(3,1)]]Thebow_corpusparameter can be passed to theDTM.fitmethod. The dictionary is only used as a convenience for post processing. The documents in the corpus should be listed in chronological order.DTM.fitalso takes a list of number of documents to include per each time interval (time slice of the corpus) in the DTM model,time_slices. Optionally, you can also pass the number of time slices you want to use,n_time_slices, which will cause the documents to be evenly spread over all time intervals.Mult and seq filesThe original Dynamic Topic Model takes two files as inputs, which are automatically generated from the corpus and time slices when passed to theDTM.fitmethod:foo-mult.dat(themultfile)foo-seq.dat(theseqfile)If the above files are in the directorybarrelative to the working directory, then thecorpus_prefixargument will bebar/foo.If the dataset consists ofNdocs, then themultfile is anN-length file where lineilists the number of unique words in documentiand theindex:countfor each word in the document, like so:unique_word_count_doc_1 index_11:count_11 index_12:count_12 ... index_1n:count_1n
unique_word_count_doc_2 index_21:count_21 index_22:count_22 ...
...
unique_word_count_doc_N index_N1:count_N1 index_N2:count_N2 ...Where each word index corresponds to a unique word in the vocabulary (ie listed in a dictionary somewhere)The order of the docs inmultshould be chronological, in order to correspond to the documents numbered in theseqfile.Theseqfile gives the number of documents that correspond to each time slice in the DTM, and is of the formn_times
number_docs_time_1
number_docs_time_2
...
number_docs_time_n_timesTwo additional files that are useful to have somewhere are:file with all of the words in the vocabulary, arranged in the same order as the word indicesa file with information on each of the documents, arranged in the same order as the docs in themultfile.Loading previous inputsPreviously-generatedmultandseqfiles can be read and used to generate corpus and time slices lists to be used in other model fits. This can be done by using theDTM.read_corpusmethod. Alternatively, theDTM.loadmethod can be used to prepare the class to read previously-generated model outputs.Model outputsReading model outputsThe raw output files (discussed in the next section) can be read intonumpyarrays. TheDTM.read_topic_term_probabilitiesmethod is used to see the estimated probabilities that terms within the corpus appeared in the specified topic at each time interval.DTM.read_topic_mixturesis used to see the estimated proportions of each topic within each document. See the Usage section above for an example.Raw output filesThe model outputs the following files to the directory specified by thecorpus_prefixandoutput_namearguments.topic-xxx-var-e-log-prob.dat: the e-betas (word distributions) for topic xxx for all times. This is in row-major form, e.g. (in R):a=scan("topic-002-var-e-log-prob.dat")b=matrix(a,ncol=10,byrow=TRUE)# The probability of term 100 in topic 2 at time 3:exp(b[100,3])gam.dat: The gammas associated with each document. Divide these by the sum for each document to get expected topic mixtures.a=scan("gam.dat")b=matrix(a,ncol=10,byrow=TRUE)rs=rowSums(b)e.theta=b/rs# Proportion of topic 5 in document 3:e.theta[3,5]If you are running this software in "dim" mode to find document influence, it will also create the following files:influence_time-yyy: the influence of documents at time yyy for each topic, where time is based on in yourseqfile and the document index is given by the ordering of documents in themultfile.a=scan("influence-time-010")b=matrix(a,ncol=10,byrow=TRUE)# The influence of the 2nd document on topic 5:b[2,5]Changes to the original codeRefactored originalmainfunction to befitmethod of classDTMRemoved dependence ofgflagsand converted global flags to use extern variables that are initialized byDTM.Added Python bindings toDTMclass usingpybind11Better error handling to prevent commands likeexitfrom crashing the Python kernelBetter use ofconstto limit number of compiler warningsChanged some useful outputs for monitoring model fitting convergence to redirect to alog.txtfile in the output directory rather thanstderrFixed bug that made LDA initialization finish before converging or reaching its max iterationMore sane defaults:changedcorpus_prefixandoutnameto be required argumentschangedntopicsfrom -1.0 to 10changedalphafrom -10 to 0.01changedinitialize_ldaflag to trueAbout the original Dynamic Topic Model codeDynamic Topic Models and the Document Influence ModelThis implements topics that change over time (Dynamic Topic Models) and a model of how individual documents predict that change.This code is the result of work by David M. Blei and Sean M. Gerrish.(C) Copyright 2006, David M. Blei(C) Copyright 2011, Sean M. GerrishIt includes software corresponding to models described in the
following papers:[1]D. Blei and J. Lafferty. Dynamic topic models. In
Proceedings of the 23rd International Conference on Machine Learning, 2006.[2]S. Gerrish and D. Blei. A Language-based Approach to Measuring
Scholarly Impact. In Proceedings of the 27th International Conference
on Machine Learning, 2010.These files are part of DIM.DIM is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.DIM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA