Package: association-measures
Statistical Association Measures for Python pandasAssociation measures are mathematical formulae that interpret cooccurrence frequency data. For each pair of words extracted from a corpus, they compute an association score, a single real value that indicates the amount of (statistical) association between the two words.http://www.collocations.de/AM/index.htmlInstallationDependenciespandasscipyInstallation using pippython3 -m pip install association-measuresInstallation from source (requires Cython)# Compile Cython code
python3 setup.py build_ext --inplace

# Cython already compiled
python3 setup.py installUsageInputThe module expects a pandas dataframe with reasonably named columns; i.e. the columns must follow one of the following notations:contingency table>>>dfitemO11O12O21O221appreciated11533311766632certain7153271131765513measuring11533371766574particularly215332451766195arrived2153323176661frequency signature (seeEvert 2008: Figure 8)>>>dfitemff1f2N1appreciated11533421919982certain7153341201919983measuring11533481919984particularly215334471919985arrived2153345191998wheref=O11,f1=O11+O12,f2=O11+O21,N=O11+O12+O21+O22.corpus frequencies (“keyword-friendly”)>>>dfitemf1N1f2N21appreciated11533411766642certain7153341131766643measuring11533471766644particularly215334451766645arrived2153343176664wheref1=O11,f2=O21,N1=O11+O12,N2=O21+O22.Observed and Expected FrequenciesGiven a dataframe following one of the notations specified above, you can calculate expected frequencies via>>>importassociation_measures.frequenciesasfq>>>fq.expected_frequencies(df)E11E12E21E2210.15973115333.8402691.840269176662.15973129.58385015324.416150110.416150176553.58385030.63892315333.3610777.361077176656.63892343.75367515330.24632543.246325176620.75367550.39932715333.6006734.600673176659.399327Theobserved_frequencymethod will convert to contingency notation:>>>importassociation_measures.frequenciesasfq>>>fq.observed_frequencies(df)O11O12O21O221115333117666327153271131765513115333717665742153324517661952153323176661Note that all methods return dataframes that are indexed the same way the input dataframe is indexed:>>>dfff1f2Nitemappreciated1153342191998certain715334120191998measuring1153348191998particularly21533447191998arrived2153345191998>>>fq.observed_frequencies(df)O11O12O21O22itemappreciated1153331176663certain715327113176551measuring1153337176657particularly21533245176619arrived2153323176661You can thusjointhe results directly to the input.Association MeasuresThe following association measures are currently implemented (v0.2.2):asymptotic hypothesis tests:z-score(z_score)t-score(t_score)parameter:discDunning's log-likelihood ratio(log_likelihood)parameter:signedsimple-ll(simple_ll)parameter:signedpoint estimates of association strength:Liddell(liddell)minimum sensitivity(min_sensitivity)log-ratio(log_ratio)parameters:disc,discountingDice coefficient(dice)information theory:mutual information(mutual_information)parameter:disclocal mutual information(local_mutual_information)conservative estimatesconservative log-ratio(conservative_log_ratio)parameters:disc,alpha,correct,one_sided,boundary,vocabYou can either calculate specific measures:>>>importassociation_measures.measuresasam>>>am.log_likelihood(df)itemappreciated2.448757certain-0.829802measuring0.191806particularly-1.059386arrived3.879126This assumes thatdfcontains the necessary columns (observed frequencies in contingency notation and expected frequencies).  In most cases, it is most convenient to just usescore():>>>importassociation_measures.measuresasam>>>am.score(df,measures=['log_likelihood'])O11O12O21O22R1R2C1C2NE11E12E21E22log_likelihoodipmipm_referenceipm_expecteditemappreciated11533311766631533417666421919961919980.15973115333.8402691.840269176662.1597312.44875765.2145565.66046310.416775certain715327113176551153341766641201918781919989.58385015324.416150110.416150176553.583850-0.829802456.501891639.632296625.006510measuring11533371766571533417666481919901919980.63892315333.3610777.361077176656.6389230.19180665.21455639.62324041.667101particularly2153324517661915334176664471919511919983.75367515330.24632543.246325176620.753675-1.059386130.429112254.720826244.794217arrived21533231766611533417666451919931919980.39932715333.6006734.600673176659.3993273.879126130.42911216.98138826.041938Note that by default,score()yields observed frequencies in contingency notation (and marginal frequencies) as well as expected frequencies. You can turn off this behaviour settingfreq=False.To calculate all available measures, don't specify any measures:>>>am.score(df,freq=False)z_scoret_scorelog_likelihoodsimple_llmin_sensitivityliddelldicelog_ratiobinomial_likelihoodconservative_log_ratiomutual_informationlocal_mutual_informationitemappreciated2.1024420.8402692.4487571.9879920.0000650.4201390.0001303.5262020.0000000.00.7966110.796611certain-0.834636-0.976603-0.829802-0.7693310.000457-0.0215460.000906-0.4866220.1171170.0-0.136442-0.955094measuring0.4517260.3610770.1918060.1737880.0000650.0451360.0001300.7188470.0000000.00.1945510.194551particularly-0.905150-1.240035-1.059386-0.9889970.000130-0.0373210.000260-0.9656510.2240420.0-0.273427-0.546853arrived2.5330181.1318473.8791263.2431410.0001300.3201430.0002612.9412400.0000000.00.6997011.399402You can also pass constant integer counts as parameters toscore().  This is reasonable for the following notations:frequency signature: integersf1andN(DataFrame contains columnsfandf2)>>>dfff2itemappreciated12certain7120measuring18particularly247arrived25>>>am.score(df,f1=15334,N=191998)corpus frequencies: integersN1andN2(DataFrame contains columnsf1andf2)>>>dff1f2itemappreciated11certain7113measuring17particularly245arrived23>>>am.score(df,N1=15334,N2=176664)Some association measures have parameters (see above). You can pass these parameters as keywords toscore(), e.g.:>>>am.score(df,measures=['log_likelihood'],signed=False,freq=False)log_likelihooditemappreciated2.448757certain0.829802measuring0.191806particularly1.059386arrived3.879126DevelopmentThe package is tested using pylint and pytest.# Installing dev requirementsmakeinstall# Compile Cython codemakecompile# Lintmakelint# Unittestmaketest# Coveragemakecoverage# Performancemakeperformance