Package: compExtract
IntroductionThis module helps you extract key terms and topics from corpus using a comparative approach.InstallationUsageImport packagesfromcompExtractimportComparativeExtractionLoad sample dataimportpandasaspdimportnumpyasnpPATH="/Users/xiaoma/Desktop/gitrepo/associate-term-search/data/switch_reviews.csv"data=pd.read_csv(PATH)label=[x<=3forxindata['stars']]data<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>starstitlesreviewsdates05.0Worth It\nDefinitely worth the money!\nSeptember 21, 201912.0Nintendo Swich gris joy con\nCon este producto no he sentido mucha satisfac...September 20, 201925.0My kid wont put it down\nCouldnt of been happier, came early.  I was th...September 20, 201933.0Happy\nHappy\nSeptember 20, 201945.0Great\nGreat product\nSeptember 19, 2019...............49951.0One Star\nit is no good, it suck, no work, plz hlp amazon\nDecember 12, 201749965.0A must have gaming system\nThe Nintendo Switch is a versatile hybrid game...December 12, 201749975.0Switch\nThis purchase save me from looking for one.\nDecember 11, 201749985.0Five Stars\nBest babysitter ever!\nDecember 11, 201749995.0Five Stars\nIts a great game console.\nDecember 11, 20175000 rows × 4 columnsdata.columnsIndex(['stars', 'titles', 'reviews', 'dates'], dtype='object')Here we are using online Amazon reviews for Nintendo Switch to illustrate the usages of the module.The module requires a corpus and a set of binary labels as inputs. The labels should be created depending on what type of questions are we trying to answer. The set of labels should be of the same length as the corpus.Here, let's assume that we want to know why people dislike this product and find relevant keywords. To answer this question, we created the label to be a binary variable indicating whether a reviewer gives a 3 star or less.Initialize the module with the review corpus and labelsce=ComparativeExtraction(corpus=data['reviews'],labels=label)Extract the keywordsce.get_distinguish_terms(ngram_range=(1,3),top_n=10)<compExtract.ComparativeExtraction at 0x7ff96f84b588># Get the keywords that are mentioned significantly more in the less than or equal to 3 star reviewsce.increased_terms_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>featurediffpos_proppos_countneg_propneg_count0work0.1949760.2784261910.0834493601switch0.1767640.3513122410.1745487532buy0.1745200.2973762040.1228565303month0.1431290.1588921090.015763684nintendo0.1343160.2900871990.1557726725charge0.1228550.141399970.018544806use0.1184480.2069971420.0885493827new0.1139890.1603501100.0463612008would0.1065400.1647231130.0581832519get0.1040550.2317781590.127724551# Get the keywords that are mentioned significantly less in the less than or equal to 3 star reviewsce.decreased_terms_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>featurediffpos_proppos_countneg_propneg_count0love-0.2169970.080175550.29717212821great-0.1222470.099125680.2213729552fun-0.0481600.046647320.0948084093best-0.0426380.030612210.0732503164amaze-0.0380110.01020470.0482152085awesome-0.0358270.00728950.0431151866son love-0.0355640.00291520.0384791667perfect-0.0325150.00874660.0412611788easy-0.0262820.023324160.0496062149kid love-0.0243700.00437330.028744124If we need more context on a given word, or we need more interpretable topics, we can:Output the reviews that contains the termSwitch the ngram_rangeOutput the reviewsSay we want to know more about the significant term "work", we can directly output all the reviews containing the term.The output class "kw" contains a one-hot encoded document-term-matrix that has all the terms found from the corpus. We can leverage it to find corresponding reviews of each term.# The binary_dtm provides a convenient way to extract reviews with specific termsprint(ce.binary_dtm[['work']])work
0        0
1        0
2        0
3        0
4        0
...    ...
4995     1
4996     0
4997     0
4998     0
4999     0

[5000 rows x 1 columns]reviews_contain_term_work=data['reviews'][[x==1forxince.binary_dtm['work']]]len(reviews_contain_term_work)551forxinpd.Series(reviews_contain_term_work).sample(1):print(x)I bought this as a Christmas present for my son.  After about a month and half of using it.  The switch stopped working.  It wont charge.  The product is an expensive piece of junk.Change the n-gram range to exclude uni-gramsce_ngram=ComparativeExtraction(corpus=data['reviews'],labels=label).get_distinguish_terms(ngram_range=(2,4),top_n=10)/Users/xiaoma/envs/compExtract/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:489: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'
  warnings.warn("The parameter 'token_pattern' will not be used"





<compExtract.ComparativeExtraction at 0x7ff955f23cf8>ce_ngram.increased_terms_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>featurediffpos_proppos_countneg_propneg_count0joy con0.0408570.056851390.015994691brand new0.0205110.027697190.007186312nintendo switch0.0196380.074344510.0547062363buy switch0.0188880.027697190.008809384play game0.0140920.039359270.0252671095game play0.0098120.021866150.012054526year old0.0052430.023324160.018081787christmas gift0.0036820.014577100.010895478battery life0.0018330.024781170.022949999wii u0.0005040.016035110.01553167ce_ngram.decreased_terms_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>featurediffpos_proppos_countneg_propneg_count0son love-0.0355640.00291520.0384791661kid love-0.0243700.00437330.0287441242great game-0.0184420.00728950.0257301113great product-0.0141710.00437330.018544804great console-0.0136410.00583140.019471845best console-0.0136090.00145810.015067656highly recommend-0.0126150.00291520.015531677absolutely love-0.0119870.00145810.013445588game system-0.0117460.021866150.0336111459love switch-0.0114520.01312090.024571106