Package: communitytopic
Community TopicIntroductionWhat is Community Topic?In this repository we present our novel method called Community Topic for Topic Modelleling as a Pypi library. Our algorithm, Community Topic, is based on mining communities of terms from term-occurrence networks extracted from the documents. In addition to providing interpretable collections of terms as topics, the network representation provides a natural topic structure. The topics form a network, so topic similarity is inferred from the weights of the edges between them. Super-topics can be found by iteratively applying community detection on the topic network, grouping similar topics together. Sub-topics can be found by iteratively applying community detection on a single topic community. This can be done dynamically, with the user or conversation agent moving up and down the topic hierarchy as desired.What problem does it solve? & Who is it for?Unfortunately, the most popular topic models in use today do not provide a suitable topic structure for these purposes and the state-of-the-art models based on neural networks suffer from many of the same drawbacks while requiring specialized hardware and many hours to train. This makes Community Topic an ideal topic modelling algorithm for both applied research and practical applications like conversational agents.Requirement & InstallationSystem requirementPython >= 3.6
commodity hardware
setuptools~=67.6.0
spacy~=3.5.0
numpy~=1.21.5
gensim~=4.2.0
networkx~=2.8.4
igraph~=0.10.4Installation OptionThe easy way to install CommunityTopic is:pip install communitytopicDatasets and Evaluation Metrics UsedWe have used followingdataset for our experiment.Name of the DatasetSourceSource LanguageBBCBBCEnglish20Newsgroups20NewsgroupsEnglishReuters21578Reuters21578EnglishEuroparlEuroparlEnglish, Italian, French, German, SpanishAlso we have used followingmetrics for our Evaluation:1. CoherencesTo compare different topic models, we use two coherence measures: c_v and c_npmi. Both measures have been shown to correlate with human judgements of topic quality with CV having the strongest correlation2. Diversity MeasuresProportion of unique words (PWU): Computes the proportion of unique words for the topicAverage Pairwise Jaccard Diversity (PJD): Coomputes the average pairwise jaccard distance between the topics.Inverted Rank-Biased Overlap (IRBO): Computes score of the rank biased overlap over the topics.3. Hierarchical MeasuresTopic Specialization:  measures the distance of a topicâ€™s probability distribution over terms from thegeneral probability distribution of all terms in the corpus given by their occurrence frequency. We expect topics at higher levels in the hierarchy closer to theroot to be more general and less specialized and topics further down the hierarchy to be more specializedTopic Affinity: measures the similarity between a super-topic and a set of sub-topics. We expect higher affinity between a parent topic and its children and lower affinity between a parent topic and sub-topics which are not its childrenGetting Started (Try it out)This is an example tuotrial which finds topic of BBC dataset using best combination for Pre-Processing and Community Topic Algorithm.Step 1: import necessary class of the libraryfromcommunitytopicimportCommunityTopicfromcommunitytopicimportPreProcessingStep 2: Load raw corpus as the dataset, here we are using BBC dataset.withopen("<Path-To-Dataset>/bbc_train.txt","r",encoding='utf-8')asf:bbc_train=f.read()withopen("<Path-To-Dataset>/bbc_test.txt","r",encoding='utf-8')asf:bbc_test=f.read()Step 3: Performing pre-processing on training and testing corpustokenized_bbc_train_sents,tokenized_bbc_train_docs,tokenized_bbc_test_docs,dictionary=PreProcessing.do_preprocessing(train=bbc_train,test=bbc_test,ner=1,pos_filter=3,phrases="npmi",phrase_threshold=0.35,language="en")Step 4: Applying Community Topic algorithm on pre-processed datacommunity_topic=CommunityTopic(train_corpus=tokenized_bbc_train_sents,dictionary=dictionary)Step 5: Applying Community Topic algorithm on pre-processed datacommunity_topic=CommunityTopic(train_corpus=tokenized_bbc_train_sents,dictionary=dictionary)community_topic.fit()Step 6: Get topic words founded by abovr algorithmtopic_words=community_topic.get_topics_words_topn(10)API UsageFollowing are the API functions that we expose by this library code:MethodCodeFit the flat topic model.fit()Fit the hiearchical topic model.fit_hierarchical()Get flat topic words.get_topics_words()Get topnnflat topic word.get_topics_words_topn(n=10)Get flat topics as dictionary id.get_topics()Get hierarchical topic words.get_topic_words_hierarchical()Get hierarchical topic as dictionary id an ig_graph of that topic.get_topics_hierarchical()Geet firstnlevels in hierarchy.get_n_level_topic_words_hierarchical(n_level=2)Geet hierarchical topic words in a tree-like dictionary format.get_hierarchy_treeConclusion