Package: asymcat
ASymCatASymCat is a powerful Python library designed to analyze co-occurrence association between categorical
variables using various symmetric and asymmetric measures of association. Whether you are working with
linguistic data, or any other kind of categorical data, ASymCat can help you explore and describe your
data in new and meaningful ways.Using ASymCat, you can input co-occurrence observations such as records, alignments, and matrices of
presence-absence, and generate dictionaries that provide association scores between categories. You can also
focus on the strength or direction of association, or both, depending on your research questions.In addition to the main methods for numeric computation of measures of association, ASymCat includes auxiliary
methods that can help you deal with relational data, n-grams from sequences, alignments, and binary
matrices of presence/absence.ASymCat is a versatile library that can be used by researchers in various fields, and it can help you gain
new insights into your data.BackgroundMeasures of association play a crucial role in statistical analysis as they help to quantify the relationship
between variables. While a variety of measures exist, correlation coefficients are widely used to determine
the strength and direction of the relationship between two variables. Pearson'srho, for example,
is frequently used to assess the strength of the linear relationship between continuous variables, whereas
Spearman'srhois employed to evaluate the strength of the monotonic relationship between ordinal or
ranked variables. The Chi-square measure, on the other hand, is a useful tool for measuring the
association between categorical variables.It is important to note that most measures of association are symmetric, indicating that the association
between anyxandyvalue is equal to that betweenyandx. In general, these measures are
used to investigate either the strength or significance of the association.For researchers in various fields, an understanding of measures of association is essential for
conducting meaningful statistical analyses. By using appropriate measures of association, researchers can
gain valuable insights into the relationships between variables, draw conclusions, and make informed decisions.While symmetric measures are commonly used to measure the relationship between numeric variables, studies
and applications involving categorical variables would typically benefit from the use of asymmetric measures.
Asymmetric measures are useful in determining the fraction of variability inxthat can be explained
by variations iny(Pearson, 2016). This property is easily demonstrated through a modification
of the example provided by Zychlinski (2018), with the introduction of thedythonlibrary:XYObservation 1AcObservation 2AdObservation 3AcObservation 4BgObservation 5BgObservation 6BfIn the given example, it is not possible to determine the categorical value ofywith complete certainty
based onx. However, it is possible to determinexwith certainty based ony. The symmetric version
of Maximum-Likelihood estimation (MLE), which simply divides the number of cases by the total number
of observations (i.e., Cxy/Cx and Cxy/Cy, where C represents the overall count), produces tables that
are transposed versions of each other for XY and YX.X given YABc0.750.75d0.000.00f0.000.00g0.750.75Y given XcdfgA0.750.000.000.75B0.750.000.000.75In contrast to symmetric tables, asymmetric tables can capture the difference in information between variables,
indicating that ifyis known in a dataset,xcan be predicted with certainty. Using the
Maximum-Likelihood estimation scorer, we can see that the asymmetric tables more accurately reflect
this relationship:X given YABc1.000.00d1.000.00f0.001.00g0.001.00Y given XcdfgA0.670.330.000.00B0.000.000.330.67The assessment of categorical association typically involves the use of measures such as Chi-square
and Cramer's V, which are symmetric in nature. However, there are some widely-used asymmetric
measures that include Theil's U and Goodman and Kruskal's tau. The Theil's U measure is
well-suited for fields in the humanities, including linguistic research, as it considers the conditional
entropy betweenxandy, which reflects the number of possible states ofygivenxand
their frequency of occurrence.In the ASymCat library, various scoring methods are available to measure asymmetric associations,
including Maximum-Likelihood Estimation, Pointwise Mutual Information, Normalized Pointwise Mutual
Information, Chi-square (over both 2x2 and 3x2 contingency tables), CramÃ©r's V (over both 2x2
and 3x2 contingency tables), Fisher Exact Odds Ratio (over unconditional MLE), Theil's U
("uncertainty score"), Conditional Entropy, and a new scorertresoldithat combines information
from MLE and PMI, tailored for studies in computational historical linguistics.The most popular methods for measure of categorical association are the
aforementioned Chi-square and Cramer's V, defined as the square root of a
normalized chi-square value. Both are symmetric values. Among the best
known asymmetric measures are Theil's U and Goodman and Kruskal's tau.
The former is particularly useful for domains of the humanities such as
lingustic research, as it is ultimately based on the conditional entropy
betweenxandy, that is, how many possible states ofyare observed
givenxand how often they occur.In addition to the various categorical association scorers, the ASymCat library provides several
useful features. Users can scale the scores with their desired ranges using methods such asminmax,mean, andstdev. The library also includes functions for generating heatmaps of the
scorers. Moreover, it provides a range of smoothing methods for frequency counts, including Laplace
smoothing and more sophisticated approaches like the Certainty Degree scorer, which was specifically
designed for use with this library.Installation and usageThe library can be installed as any standard Python library withpip:pipinstallasymcatDetailed instructions on how to use the library can be found in
theofficial documentation.A show-case example with a subset of themushroomdataset is shown here:importtabulateimportasymcatfromcatcoocc.scorerimportCatScorermushroom_data=asymcat.read_sequences("resources/mushroom-small.tsv")mushroom_cooccs=asymcat.collect_cooccs(mushroom_data)scorer=asymcat.scorer.CatScorer(mushroom_cooccs)mle=scorer.mle()pmi=scorer.pmi()npmi=scorer.pmi(True)chi2=scorer.chi2()chi2_ns=scorer.chi2(False)cramersv=scorer.cramers_v()cramersv_ns=scorer.cramers_v(False)fisher=scorer.fisher()theil_u=scorer.theil_u()cond_entropy=scorer.cond_entropy()tresoldi=scorer.tresoldi()headers=['pair','mle_xy','mle_yx','pmi_xy','pmi_yx','npmi_xy','npmi_yx','chi2_xy','chi2_yx','chi2ns_xy','chi2ns_yx','cremersv_xy','cremersv_yx','cremersvns_xy','cremersvns_yx','fisher_xy','fisher_yx','theilu_xy','theilu_yx','cond_entropy_xy','cond_entropy_yx','tresoldi_xy','tresoldi_yx']table=[]forpairinsorted(scorer.obs):buf=[pair,"%0.4f"%mle[pair][0],"%0.4f"%mle[pair][1],"%0.4f"%pmi[pair][0],"%0.4f"%pmi[pair][1],"%0.4f"%npmi[pair][0],"%0.4f"%npmi[pair][1],"%0.4f"%chi2[pair][0],"%0.4f"%chi2[pair][1],"%0.4f"%chi2_ns[pair][0],"%0.4f"%chi2_ns[pair][1],"%0.4f"%cramersv[pair][0],"%0.4f"%cramersv[pair][1],"%0.4f"%cramersv_ns[pair][0],"%0.4f"%cramersv_ns[pair][1],"%0.4f"%fisher[pair][0],"%0.4f"%fisher[pair][1],"%0.4f"%theil_u[pair][0],"%0.4f"%theil_u[pair][1],"%0.4f"%cond_entropy[pair][0],"%0.4f"%cond_entropy[pair][1],"%0.4f"%tresoldi[pair][0],"%0.4f"%tresoldi[pair][1],]table.append(buf)print(tabulate.tabulate(table,headers=headers,tablefmt='markdown'))Which will output:pairmle_xymle_yxpmi_xypmi_yxnpmi_xynpmi_yxchi2_xychi2_yxchi2ns_xychi2ns_yxcremersv_xycremersv_yxcremersvns_xycremersvns_yxfisher_xyfisher_yxtheilu_xytheilu_yxcond_entropy_xycond_entropy_yxtresoldi_xytresoldi_yx('edible', 'bell')0.384610.43080.43080.31070.31071.83151.83153.58973.58970.20270.20270.19870.1987infinf011.11900.59561('edible', 'convex')0.46150.4615-0.3424-0.3424-0.2844-0.28443.67353.67355.79885.79880.37190.37190.31010.3101000.21470.30710.72730.4486-0.5615-0.5615('edible', 'flat')0.076910.43080.43080.14380.14380.10410.10410.56680.56680000infinf011.11900.45961('edible', 'sunken')0.076910.43080.43080.14380.14380.10410.10410.56680.56680000infinf011.11900.45961('poisonous', 'bell')00-3.5553-3.5553-0.5934-0.59341.83151.83153.58973.58970.20270.20270.19870.1987001100-3.5553-3.5553('poisonous', 'convex')10.53850.43080.43080.41030.41033.67353.67355.79885.79880.37190.37190.31010.3101infinf1000.690210.6779('poisonous', 'flat')00-1.9459-1.9459-0.3248-0.32480.10410.10410.56680.56680000001100-1.9459-1.9459('poisonous', 'sunken')00-1.9459-1.9459-0.3248-0.32480.10410.10410.56680.56680000001100-1.9459-1.9459Similar Projectshttps://github.com/pafoster/pyitlibGriffith, Daniel M.; Veech, Joseph A.; and Marsh, Charles J. (2016)cooccur: Probabilistic Species Co-Occurrence Analysis in R. Journal
of Statistical Software (69). doi: 10.18627/jss.v069.c02https://cran.r-project.org/web/packages/GoodmanKruskal/vignettes/GoodmanKruskal.htmlCommunity guidelinesWhile the author can be contacted directly for support, it is recommended
that third parties use GitHub standard features, such as issues and
pull requests, to contribute, report problems, or seek support.Author and citationThe library is developed by Tiago Tresoldi (tiago.tresoldi@lingfil.uu.se). The library is developed in the context of
theCultural Evolution of Textsproject, with funding from theRiksbankens Jubileumsfond(grant agreement ID:MXM19-1087:1).During the first stages of development, the author received funding from theEuropean Research Council(ERC) under the European Unionâ€™s Horizon 2020
research and innovation programme (grant agreement
No.ERC Grant #715618,Computer-Assisted Language Comparison).If you useasymcat, please cite it as:Tresoldi, Tiago (2023).asymcat, a library for symmetric and asymmetric
analysis of categorical co-occurrences. Version 0.3. Uppsala. Available at:https://github.com/tresoldi/asymcatIn BibTeX:@misc{Tresoldi2023asymcat,author={Tresoldi, Tiago},title={asymcat, a library for symmetric and asymmetric analysis of categorical co-occurrences. Version 0.3.},howpublished={\url{https://github.com/tresoldi/asymcat}},address={Uppsala},year={2023},}