Package: little-mallet-wrapper
little-mallet-wrapperThis is a little Python wrapper around the topic modeling functions ofMALLET.Currently under construction; please send feedback/requests to Maria Antoniak.Installationpip install little_mallet_wrapperRequirementsPython 3.7MALLETpandasnumpyseaborn(for plotting functions)UsageSee demo.ipynb for a demonstration of how to use the functions in little-mallet-wrapper.Documentationprint_dataset_stats(training_data)Displays basic statistics about the training dataset.NameTypeDescriptiontraining_datalist of stringsDocuments that will be used to train the topic model.process_string(text, lowercase=True, remove_short_words=True, remove_stop_words=True, remove_punctuation=True, numbers='replace', stop_words=STOPS)A simple string processor that prepares raw text for topic modeling. CAUTION: Depending on your data, you might need to write your own processing function. Do not rely on this function for non-English languages; both the stopword list and the punctuation removal assume English as input.NameTypeDescriptiontextstringIndividual document to process.lowercasebooleanWhether or not to lowercase the text.remove_short_wordsbooleanWhether or not to remove words with fewer than 2 characters.remove_stop_wordsbooleanWhether or not to remove stopwords.remove_punctuationbooleanWhether or not to remove punctuation (not A-Za-z0-9)remove_numbersstring'replace' replaces all numbers with the normalized token NUM; 'remove' removes all numbers.stop_wordslist of stringsCustom list of words to remove.RETURNSstringProcessed version of the input text.quick_train_topic_model(path_to_mallet, output_directory_path, num_topics, training_data)Imports training data, trains an LDA topic model using MALLET, and returns the topic keys and document distributions.NameTypeDescriptionpath_to_malletstringPath to your local MALLET installation: .../mallet-2.0.8/bin/malletoutput_directory_pathstringPath to where the output files should be stored.num_topicsintegerThe number of topics to use for training.training_datalist of stringsProcessed documents for training the topic model.RETURNSlist of lists of stringsThe 20 most probable words for each topic.RETURNSlist of lists of integersTopic distribution (list of probabilities) for each document.import_data(path_to_mallet, path_to_training_data, path_to_formatted_training_data, training_data, training_ids=None, use_pipe_from=None)Imports the training data into MALLET formatted data that can be used for training.NameTypeDescriptionpath_to_malletstringPath to your local MALLET installation: .../mallet-2.0.8/bin/malletpath_to_training_datastringPath to where the training data should be stored.path_to_formatted_training_datastringPath to where the MALLET formatted training data should be stored.training_datalist of stringsProcessed documents for training the topic model.training_idslist of stringsUnique identifiers for the training data.use_pipe_fromstringIf you want to import the documents using the same model as a previous set of documents, include the path to the previous MALLET formatted training data.train_topic_model(path_to_mallet, path_to_formatted_training_data, path_to_model, path_to_topic_key, path_to_topic_distributions, path_to_word_weights, num_topics)Trains an LDA topic model using MALLET.NameTypeDescriptionpath_to_malletstringPath to your local MALLET installation: .../mallet-2.0.8/bin/malletpath_to_formatted_training_datastringPath to where the MALLET formatted training data is stored.path_to_modelstringPath to where the model should be stored.path_to_topic_keystringPath to where the topic keys should be stored.path_to_topic_distributionsstringPath to where the topic distributions should be stored.path_to_word_weightsstringPath to where the word weights should be stored.num_topicsintegerThe number of topics to use for training.load_topic_keys(topic_keys_path)Loads the most sets of most probable words for each topic after training a topic model.NameTypeDescriptiontopic_keys_pathstringPath to where the topic keys are stored.RETURNSlist of lists of stringsThe 20 most probable words for each topic.load_topic_distributions(topic_distributions_path)Loads the topic distribution for each document after training a topic model.NameTypeDescriptiontopic_distributions_pathstringPath to where the topic distributions are stored.RETURNSlist of lists of integersTopic distribution (list of probabilities) for each document.load_training_ids(topic_distributions_path)Loads the training IDs. This is either a list of sequential integers or the user-specified training IDs passed toimport_data().NameTypeDescriptiontopic_distributions_pathstringPath to where the topic distributions are stored.RETURNSlist of lists of stringsList of training IDs in the same order as the topic distributions.load_topic_word_distributions(word_weight_path)Loads the topic word distributions. These are the probabilities for each word for each topic.NameTypeDescriptionword_weight_pathstringPath to where the word weights are stored.RETURNSdefaultdict of defaultdict of floatMap of topics to words to probabilities.get_top_docs(training_data, topic_distributions, topic_index, n=5)Gets the documents with the highest probability for the target topic.NameTypeDescriptiontraining_datalist of stringsProcessed documents that was used to train the topic model.topic_distributionslist of lists of integersTopic distribution (list of probabilities) for each document.topic_indexintegerThe index of the target topic.nintegerThe number of documents to return.RETURNSlist of tuples (float, string)The topic probability and document text for the n documents with the highest probability for the target topic.plot_categories_by_topics_heatmap(labels, topic_distributions, topic_keys, output_path=None, target_labels=None, dim=None)If the dataset includes some time of categorical labels, creates a heatmap of the labels x topics.NameTypeDescriptionlabelslist of stringsDocument labels (e.g., authors of the documents, genres of the documents).topic_distributionslist of lists of integersTopic distribution (list of probabilities) for each document.topic_keyslist of lists of stringsThe 20 most probable words for each topic.output_pathstringPath to where the resulting figure should be saved.target_labelslist of stringsA subset oflabelsto use for plotting.dimtuple of integers(x, y) dimensions for the resulting figure.plot_categories_by_topic_boxplots(labels, topic_distributions, topic_keys, output_path=None, target_labels=None, dim=None)If the dataset includes some time of categorical labels, creates a set of boxplots, one plot for each topic.NameTypeDescriptionlabelslist of stringsDocument labels (e.g., authors of the documents, genres of the documents).topic_distributionslist of lists of integersTopic distribution (list of probabilities) for each document.topic_keyslist of lists of stringsThe 20 most probable words for each topic.output_pathstringPath to where the resulting figure should be saved.target_labelslist of stringsA subset oflabelsto use for plotting.dimtuple of integers(x, y) dimensions for the resulting figure.divide_training_data(documents, num_chunks=10)Given a dataset, divides each document into a set of equally sized chunks.NameTypeDescriptiondocumentslist of stringsDocuments to split.num_chunksintegerHow many times to split each document.RETURNStuple (list of strings, list of integers, list of floats)The divided documents, the indices of the input documents, and the positions within the documents (0-1.0).infer_topics(path_to_mallet, path_to_original_model, path_to_new_formatted_training_data, path_to_new_topic_distributions)Get topic distributions for a set of new documents using a model that has been trained on another set of documents.NameTypeDescriptionpath_to_malletstringPath to your local MALLET installation: .../mallet-2.0.8/bin/malletpath_to_original_modelstringPath to where the topic model was stored.path_to_new_formatted_training_datastringPath to where the MALLET formatted training data is stored.path_to_new_topic_distributionsstringPath to where the topic distributions should be stored.plot_topics_over_time(topic_distributions, topic_keys, times, topic_index, output_path=None)Creates lineplots, one for each topic, showing the mean topic probability over document segments.NameTypeDescriptiontopic_distributionslist of lists of integersTopic distribution (list of probabilities) for each document.topic_keyslist of lists of stringsThe 20 most probable words for each topic.timeslist of floatsThe division indices within the document.topic_indexintegerThe index of the target topic.output_pathstringPath to where the resulting figure should be saved.get_js_divergence(topic_index_1, topic_index_2, topic_distributions)Calculates the Jensen-Shannon divergence between the two target topic distributions.NameTypeDescriptiontopic_index_1integerIndex of the first target topic distribution.topic_index_2integerIndex of the second target topic distribution.topic_distributionslist of lists of integersTopic distribution (list of probabilities) for each document.RETURNSfloatJensen-Shannon divergence of the requested topic distributions.