Package: comid
COMID - Community identification module for Reddit conversationsCOMID is a package of tools for analyzing conversations on RedditWhat does COMID?Collects submissions from a subredditGenerates a corpus based on the O.C. (original content) of the submissions.Assists in the annotation of topics in pre-grouped conversation clusters.Conducts temporal analysis of topics.What does Comid not do?Comid does not conduct topic modeling. For topic modeling, we recommend using the SBMhttps://github.com/martingerlach/hSBM_Topicmodel.Quick StartInstall Comid and download the en_core_web_sm modelpipinstallcomid
python-mspacydownloaden_core_web_smImport RedditScrapper submodule, creates a RedditScraper instance and configure the reddit credentials.If you don’t have the credentials, you can follow the instructions here:How to get client_id and client_secret for Python Reddit API registration ? - GeeksforGeeksfromcomid.scraperimportRedditScraperimportdatetimeasdtrscrap=RedditScraper()#setup reddit credentialsrscrap.config_credentials(client_id="YOUR_CLIENT_ID",client_secret="YOUR_CLIENT_SECRET",password="YOUR_PASSWORD",username="YOUR_USERNAME")Inform the subreddit and date range and download the O.C. ids.subreddit='digitalnomad'# the subredditstart_dt=dt.datetime(2022,1,1)# the initial dateend_dt=dt.datetime(2022,1,2)#the final date# downlaod the O.C ids of given subreddit and date rangerscrap.search_ids_by_datetime(subreddit,start_dt,end_dt)Download all submissions of collected ids, including O.C, comments and repliesrscrap.donwload_by_ids()Import Comid and load the json files of files of Reddit submissions.fromcomidimportComidcm=Comid()# Load all json files as a list. You can use full or relative path for every filefiles=['dataset/submissions.json','dataset/comments.json']cm.load_json_files(files=files)Generate the corpus of submission posts. The corpus includes only the main submission, not will include the comments and replies. This step can expends few minutes if has a large number of posts.cm.generate_corpus()Once generated, The corpus can be accessed in the atributecorpus. Explore the corpus and check the corpus size.print("corpus size: ",len(cm.corpus))For a reasonable runtime topic modelling using SBM, it is recommended that the corpus does not exceed the approximate size of 6000 documents.  If the corpus is too large, perform the next step to shrink the corpus.# reduce the corpus by optimizing the number of tokens and pinning documents that have# at last 10 interactions (comments or replies)cm.reduce_corpus(target_size=6000,optimize_num_interactions=False,min_num_interactions=10)#print the full corpus length vs reduced corpus lengthprint("corpus: ",len(cm.corpus),"corpus_reduced: ",len(cm.corpus_reduced))Save the corpus as a json to used in HSMB# gerating json of reduced corpuscm.save_corpus(reduced=True)Save the current Comid instance in a pickle file to be loaded anytime.cm.save()Use the json file of corpus in a topic modelling. For Comid, it is recommended the use of SBMTopic Model. Here is a guide how to use the corpus in SBMSBM Topic Model. Once the topic modelling is finished, the previous Comid state can be loaded.fromcomidimportComid# replace the comid_file_name with the name of the file to be loadedcomid_file_name="comid_saved_file.pickle"cm=Comid.load(comid_file_name)Load the CSV clusters file generated in topic modelling# replace the cluster_file with the name of the file to be loadedcluster_file='path_to_file/topsbm_level_1_clusters.csv'cm.load_clusters_file(cluster_file)# show clusters dataframecm.df_clusters.head()Print random samples from any Cluster executing the methodprint_cluster_samples# Printing 3 random samples from 'Cluster 1'cm.print_cluster_samples('Cluster 1',3)Also, it is possible to retrieve a conversation flatten text by a document id# replace the doc_id with the id of document to retrievedoc_id='rtsodc'text=c.retrieve_conversation(doc_id)print(text)Save a CSV summary of clusters. The summary file contains four columns: Cluster (cluster id), Num Docs (number of documents in cluster), Percent (percentual of documents in the cluster from all documents) and Topic (To annotate the topic label).cm.save_clusters_summary()Use the saved file in previous step to annotate the topics label in theTopiccolumn.Build the topics data frame from Topics annotated in cluster summary CSV. Clusters not annotated will not be included in the data frame.cm.build_topics("clusters_summary.csv")cm.df_topics.head()Alternatively, it is possible to build the topics data frame without annotate the topic labels. In this case, it this case the method will consider a minimum value for the percentual of documents to include the cluster in the topic data frame.# build the df_topics just for the clusters with a minimal of 7% of documents.# if omitted the parameter min_percent, it will consider 10%cm.build_topics(min_percent=7)Explore the topics grouping them by periods# Group topics by months.# parameter period_type can be 'd' for days, 'w' for weeks, 'm' for months and# 'y' for yearscm.group_by_period(period_type="m")cm.df_periods