Package: contextSearching
IntroductionGiven a word token and a corpus where this word appears, this package helps you find and analyze the context in which the word appears. It can be easily leveraged to improve your bag-of-words based analysis.Installationpip install contextSearchingUsageAs an example to illustrate the usage, we choose the term "break" and the Amazon review corpus for Nintendo Switch where people used the term "break".From a simple bag-of-words analysis, we know that whenever people mention "break", the product is likely to receive a low star rating. But we do not know what breaks or any other context around "break."Preparation"""Preparation"""importpandasaspdimportnumpyasnp# read in corpuscorpus=pd.read_csv("data/switch_w_break.csv")# define the target tokentarget="break"corpus.head()starstitlesreviewsdates01.0Already broken parts\nOnly 3 months later and parts are breaking. Th...September 13, 201911.0Dock is broken\nHey. This was supposed to work. Dock is broken...September 11, 201925.0Dependable seller\nArrived on time,  well packed for the trip.  N...August 10, 201931.0Nintendo Does Not Honor Warranty\nMy son used this unit for 7 months.  At which ...August 8, 201944.0Great product, Joycons need work.\nEveryone knows the switch is great. I waited a...August 5, 2019Loading the package and initialize the class"""Loading the package and initialize the class"""fromcontextSearchingimportcontext_searchingcs=context_searching(target_token=target,doc=corpus['reviews'],left_window=5,right_window=5,padding_token="_empty_")In addition to the target token and the corpus, the class requires three more inputs: left/right window and padding token.The algorithm takes in the target token and aggressively collect all the words within the specified window.For example, when left_window is set to 10, it will find the target token within each document of the corpus, then collect all the ten words to the left of the target, recording the relative position. If there are less than 10 words to the left, the algorithm will append the word list with the padding token.Get the Context Probing Matrix"""Get the Context Probing Matrix"""# Get a list of stopwordsfromgensim.parsing.preprocessingimportSTOPWORDSstopwords=list(STOPWORDS)contextPMat=cs.get_context_prob_matrix(stop_words=stopwords,lemmatize=True,stem=False)Assuming we have N documents in the corpus, and left_window and right window are set to 5. The Context Probing Matrix (CPM) is an N by 11 matrix like below:# We can examine the actual CPM like this:cpm_df=pd.DataFrame(np.array(contextPMat.context_prob_matrix))cpm_df.columns=[str(x)forxincontextPMat.position_idx]cpm_df.head()-5-4-3-2-10123450_empty__empty__empty_monthlaterbreakjoybuttonworkreplace_empty_1_empty_heysupposeworkdockbreakreplacedockprettyangry_empty_2_empty_arrivetimepacktripbreakworkgreat_empty__empty__empty_3gadgetyearexpensiverelativefunctionbreakquicklysupportmanufacturerbewarenintendo4issuerealbuttonleavejoyconbreakmonthfortunatelynintendoreplacewarrantyThe column index indicates the relative position. For example, in the first document, the word "button" appears two words to the right of the target term "break".Get the vocabs dictionary"""Get the vocabs dictionary"""contextPMat.vocabs['joycon'][-1, -1, -4, -2, -2, -1, -2, 1]The .vocabs is a dictionary whose keys are unique tokens collected in constructing the CPM, and the values are lists of recorded relative positions to the target token.In the output above, we see the term "joycon" appears 8 times in total within the +- 5 window of the target term. It most often appears on the left side of the target term.Get the statistics table for each term"""Get the statistics table for each term"""cpm_stats=contextPMat.get_cpm_stats_tb()cpm_stats.cpm_stats_tb.head()tokensmeanvarianceabs_meancountmedian0month0.1428573.9795921.857143141.01later0.0000002.0000001.3333333-1.02break0.1864410.6432060.1864411180.03joy-1.4444446.0246912.5555569-2.04button-0.5000002.5833331.5000006-1.0To understand the context, we can look at the statistics of relative positions for each term collected above.For example,When the occurrence of a term is high, we know that it always appears around the target token;When the variance of a term's relative position is low, we know that it always appears at the same relative location;Infer potential N-grams containing the target term"""Infer potential N-grams containing the term"""cpm_stats.guess_ngram(n=5)ngram_candidatestotal_scores0look expensive joycon controller break0.3273521expensive joycon controller break month0.3301292joycon controller break month handle0.3378073controller break month handle inside0.3791444break month handle inside item0.447479based on the statistics table, the algorithm can infer most likely n-grams containing the target termFor example, when we want to infer what is most likely the word appears to the left of "break" (i.e. with relative location = -1), we go through the following stepsstart with a word collected in the CPM constructing process above (e.g. "controller")for the word, take the mean of the observed relative positions, minus the mean by -1 and take the absolute valuefor the word, take the median of the observed relative positions, minus the median by -1 and take the absolute valueCalculate 1/countCalculate the variance of the relative positions of the wordRepeat the above on all the collected words and acquire 4 lists of metrics above (abs median difference, abs mean difference, 1/count, variance)normalize the 4 listsfor each collected word, multiply its 4 metrics with user-defined weights and take the sum to get a final scoreThe best candidate words at location -1 will have the smallest final score.When we want to find the most likely tri-grams, the algorithm considers a trigram with the target token in each possible location. Thus in the example output above, the target term "break" appears as the 5th, 4th, 3rd, 2nd and 1st term on the n-gram respectively.Now we get more context around "break":Expensive Joycon Controller breaks in monthsseem to be the problem.Some NotesCurrently, when the target term appears more than once in a single document, the CPM only takes the first one into consideration. I will try to improve this in the near futureThis method works better when we have more documents while each document is short. It will not work well on, for example, a collection of News articles.