Package: medtop
Documentation is available athttps://amyolex.github.io/medtop/.MedTopExtracting topics from reflective medical writings.RequirementsMedTop is only compatible with 64-bit python. You can check which version of python you're using in your virtual environment with the following code.import platform; platform.architecture()[0];pip install medtopHow to useA template pipeline is provided below using a test dataset. You can read more about the test_data datasethereEach step of the pipeline has configuration options for experimenting with various methods. These are detailed in the documentation for each method. Notably, theimport_docs,get_cluster_topics,visualize_clustering, andevaluatemethods all include the option to save results to a file.Example PipelineImport dataImport and pre-process documents from a text file containing a list of all documents.frommedtop.coreimport*data,doc_df=import_from_files('test_data/corpus_file_list.txt',stop_words_file='stop_words.txt',save_results=False)You can also consolidate your documents into a single, pipe-delimited csv file with the columns "doc_name" and "text".data,doc_df=import_from_csv('test_data/corpus.txt',stop_words_file='stop_words.txt',save_results=False)Transform dataCreate word vectors from the most expressive phrase in each sentence of the imported documents. Seed documents can be passed as a single CSV similar to corpus documents in the import step.NOTE: Ifdoc_dfis NOT passed tocreate_tfidf, you must setinclude_input_in_tfidf=Falseinget_phrases.tfidf,dictionary=create_tfidf(doc_df,path_to_seed_topics_file_list='test_data/seed_topics_file_list.txt')data=get_phrases(data,dictionary.token2id,tfidf,include_input_in_tfidf=True,include_sentiment=True)data=get_vectors("tfidf",data,dictionary=dictionary,tfidf=tfidf)Removed 67 sentences without phrases.Cluster dataCluster the sentences into groups expressing similar ideas or topics. If you aren't sure how many true clusters exist in the data, try runningassign_clusterswith the optional parametershow_chart = Trueto visual cluster quality with varying numbers of clusters. When usingmethod='hac', you can also useshow_dendrogram = Truesee the cluster dendrogram.data=assign_clusters(data,method="hac")cluster_df=get_cluster_topics(data,doc_df,save_results=False)visualize_clustering(data,method="umap",show_chart=False)Evaluate resultsgold_file="test_data/gold.txt"results_df=evaluate(data,gold_file="test_data/gold.txt",save_results=False)Document ClusteringIMPORTANT: This feature is still in alpha, meaning that we have adapted the pipeline to accomodate the clustering of documents, but have made no rigorous efforts the ensure that it works well.To cluster documents, simply import data and create the TF-IDF as above, but extract phrase, create the vectors, and cluster using thedoc_dfdataframe. Passing the parameterwindow_size=-1toget_phrasestells the method to use all tokens instead of selecting a subset of lengthwindow_size.doc_df=get_phrases(doc_df,dictionary.token2id,tfidf,include_input_in_tfidf=True,window_size=-1)doc_df=get_vectors("tfidf",doc_df,dictionary=dictionary,tfidf=tfidf)doc_df=assign_clusters(doc_df,method="kmeans",k=4)cluster_df=get_cluster_topics(data,doc_df,save_results=False)visualize_clustering(data,method="svd",show_chart=False)Removed 0 sentences without phrases.