Package: pyitlib
pyitlib is an MIT-licensed library of information-theoretic methods for data analysis and machine learning, implemented in Python and NumPy.API documentation is available online athttps://pafoster.github.io/pyitlib/.pyitlib implements the following 19 measures on discrete random variables:EntropyJoint entropyConditional entropyCross entropyKullback-Leibler divergenceSymmetrised Kullback-Leibler divergenceJensen-Shannon divergenceMutual informationNormalised mutual information (7 variants)Variation of informationLautum informationConditional mutual informationCo-informationInteraction informationMulti-informationBinding informationResidual entropyExogenous local informationEnigmatic informationThe following estimators are available for each of the measures:Maximum likelihoodMaximum a posterioriJames-SteinGood-TuringMissing data are supported, either using placeholder values or NumPy masked arrays.Installation and codebasepyitlib is listed on the Python Package Index athttps://pypi.python.org/pypi/pyitlib/and may be installed usingpipas follows:pipinstallpyitlibThe codebase for pyitlib is available athttps://github.com/pafoster/pyitlib.Notes for getting startedImport the modulediscrete_random_variable, as well as NumPy:importnumpyasnpfrompyitlibimportdiscrete_random_variableasdrvThe respective methods implemented indiscrete_random_variableaccept NumPy arrays as input. Let’s compute the entropy for an array containing discrete random variable realisations, based on maximum likelihood estimation and quantifying entropy in bits:>>>X=np.array((1,2,1,2))>>>drv.entropy(X)array(1.0)NumPy arrays are created automatically for any input which isn’t of the required type, by passing the input to np.array(). Let’s compute entropy, again based on maximum likelihood estimation, but this time using list input and quantifying entropy in nats:>>>drv.entropy(['a','b','a','b'],base=np.exp(1))array(0.6931471805599453)Those methods with the suffix_pmfoperate on arrays specifying probability mass assignments. For example, the analogous method call for computing the entropy of the preceding random variable realisations (with estimated equi-probable outcomes) is:>>>drv.entropy_pmf([0.5,0.5],base=np.exp(1))0.69314718055994529It’s possible to specify missing data using placeholder values (the default placeholder value is-1). Elements equal to the placeholder value are subsequently ignored:>>>drv.entropy([1,2,1,2,-1])array(1.0)In measures expressible in terms of joint entropy (such as conditional entropy, mutual information etc.), equally many realisations of respective random variables are required (with realisations coupled using a common index). Any missing data for random variableXresults in the corresponding realisations for random variableYbeing ignored, and vice versa. Thus, the following method calls yield equivalent results (note use of alternative placeholder valueNone):>>>drv.entropy_conditional([1,2,2,2],[1,1,2,2])array(0.5)>>>drv.entropy_conditional([1,2,2,2,1],[1,1,2,2,None],fill_value=None)array(0.5)It’s alternatively possible to specify missing data using NumPy masked arrays:>>>Z=np.ma.array((1,2,1),mask=(0,0,1))>>>drv.entropy(Z)array(1.0)In combination with any estimator other than maximum likelihood, it may be useful to specify alphabets containing unobserved outcomes. For example, we might seek to estimate the entropy in bits for the sequence of realisations[1,1,1,1]. Using maximum a posteriori estimation combined with the Perks prior (i.e. pseudo-counts of 1/L for each of L possible outcomes) and based on an alphabet specifying L=100 possible outcomes, we may use:>>>drv.entropy([1,1,1,1],estimator='PERKS',Alphabet_X=np.arange(100))array(2.030522626645241)Multi-dimensional array input is supported based on the convention thatleading dimensions index random variables, with the trailing dimension indexing random variable realisations. Thus, the following array specifies realisations for 3 random variables:>>>X=np.array(((1,1,1,1),(1,1,2,2),(1,1,2,2)))>>>X.shape(3,4)When using multi-dimensional arrays, any alphabets must be specified separately for each random variable represented in the multi-dimensional array, using placeholder values (or NumPy masked arrays) to pad out any unequally sized alphabets:>>>drv.entropy(X,estimator='PERKS',Alphabet_X=np.tile(np.arange(100),(3,1)))# 3 alphabets requiredarray([2.03052263,2.81433872,2.81433872])>>>A=np.array(((1,2,-1),(1,2,-1),(1,2,3)))# padding required>>>drv.entropy(X,estimator='PERKS',Alphabet_X=A)array([0.46899559,1.,1.28669267])For ease of use, those methods operating on two random variable array arguments (such asentropy_conditional,information_mutualetc.) may be invoked with a single multi-dimensional array. In this way, we may compute mutual information for all pairs of random variables represented in the array as follows:>>>drv.information_mutual(X)array([[0.,0.,0.],[0.,1.,1.],[0.,1.,1.]])The above is equivalent to setting thecartesian_productparameter toTrueand specifying two random variable array arguments explicitly:>>>drv.information_mutual(X,X,cartesian_product=True)array([[0.,0.,0.],[0.,1.,1.],[0.,1.,1.]])By default, those methods operating on several random variable array arguments don’t determine all combinations of random variables exhaustively. Instead a one-to-one mapping is performed:>>>drv.information_mutual(X,X)# Mutual information between 3 pairs of random variablesarray([0.,1.,1.])>>>drv.entropy(X)# Mutual information equivalent to entropy in above casearray([0.,1.,1.])pyitlib provides basic support for pandas DataFrames/Series. Both these types are converted to NumPy masked arrays internally, while masking those data recorded as missing (based on .isnull()). Note that due to indexing random variable realisations using the trailing dimension of multi-dimensional arrays, we typically need to transpose DataFrames when estimating information-theoretic quantities:>>>importpandas>>>df=pandas.read_csv('https://raw.githubusercontent.com/veekun/pokedex/master/pokedex/data/csv/pokemon.csv')>>>df=df[['height','weight','base_experience']].apply(lambdas:pandas.qcut(s,10,labels=False))# Bin the data>>>drv.information_mutual_normalised(df.T)# Transposition required for comparing columnsarray([[1.,0.32472696,0.17745753],[0.32729034,1.,0.13343504],[0.17848175,0.13315407,1.]])