Package: dit
ditis a Python package for information theory.Tryditlive:IntroductionInformation theory is a powerful extension to probability and statistics, quantifying dependencies
among arbitrary random variables in a way that is consistent and comparable across systems and
scales. Information theory was originally developed to quantify how quickly and reliably information
could be transmitted across an arbitrary channel. The demands of modern, data-driven science have
been coopting and extending these quantities and methods into unknown, multivariate settings where
the interpretation and best practices are not known. For example, there are at least four reasonable
multivariate generalizations of the mutual information, none of which inherit all the
interpretations of the standard bivariate case. Which is best to use is context-dependent.ditimplements a vast range of multivariate information measures in an effort to allow information
practitioners to study how these various measures behave and interact in a variety of contexts. We
hope that having all these measures and techniques implemented in one place will allow the
development of robust techniques for the automated quantification of dependencies within a system
and concrete interpretation of what those dependencies mean.CitingIf you useditin your research, please cite it as:@article{dit,
  Author = {James, R. G. and Ellison, C. J. and Crutchfield, J. P.},
  Title = {{dit}: a {P}ython package for discrete information theory},
  Journal = {The Journal of Open Source Software},
  Volume = {3},
  Number = {25},
  Pages = {738},
  Year = {2018},
  Doi = {https://doi.org/10.21105/joss.00738}
}Basic InformationDocumentationhttp://docs.dit.ioDownloadshttps://pypi.org/project/dit/https://anaconda.org/conda-forge/ditDependenciesPython 3.3+boltonsdebtcollectorlatticesnetworkxnumpyPLTablescipyOptional Dependenciescolorama: colored column heads in PID indicating failure modescython: faster sampling from distributionshypothesis: random sampling of distributionsmatplotlib, python-ternary: plotting of various information-theoretic expansionsnumdifftools: numerical evaluation of gradients and hessians during optimizationpint: add units to informational valuesscikit-learn: faster nearest-neighbor lookups during entropy/mutual information estimation from samplesInstallThe easiest way to install is:pipinstallditIf you want to installditwithin a conda environment, you can simply do:condainstall-cconda-forgeditAlternatively, you can clone this repository, move into the newly createdditdirectory, and then install the package:gitclonehttps://github.com/dit/dit.gitcdditpipinstall.NoteThe cython extensions are currently not supported on windows. Please install
using the--nocythonoption.Testing$gitclonehttps://github.com/dit/dit.git$cddit$pipinstall-rrequirements_testing.txt$py.testCode and bug trackerhttps://github.com/dit/ditLicenseBSD 3-Clause, see LICENSE.txt for details.Implemented Measuresditimplements the following information measures. Most of these are implemented in multivariate & conditional
generality, where such generalizations either exist in the literature or are relatively obvious — for example,
though it is not in the literature, the multivariate conditional exact common information is implemented here.EntropiesShannon EntropyRenyi EntropyTsallis EntropyNecessary Conditional EntropyResidual Entropy /
Independent Information /
Variation of InformationMutual InformationsCo-InformationInteraction InformationTotal Correlation /
Multi-InformationDual Total Correlation /
Binding InformationCAEKL Multivariate Mutual InformationDivergencesVariational DistanceKullback-Leibler Divergence Relative EntropyCross EntropyJensen-Shannon DivergenceEarth Mover’s DistanceOther MeasuresChannel CapacityComplexity ProfileConnected InformationsCopy Mutual InformationCumulative Residual EntropyExtropyHypercontractivity CoefficientInformation BottleneckInformation DiagramsInformation TrimmingLautum InformationLMPR ComplexityMarginal Utility of InformationMaximum CorrelationMaximum Entropy DistributionsPerplexityRate-Distortion TheoryTSE ComplexityCommon InformationsGacs-Korner Common InformationWyner Common InformationExact Common InformationFunctional Common InformationMSS Common InformationPartial Information DecompositionI_{min}I_{wedge}I_{RR}I_{downarrow}I_{proj}I_{BROJA}I_{ccs}I_{pm}I_{dep}I_{RAV}I_{mmi}I_{prec}I_{RA}I_{SKAR}Secret Key Agreement BoundsSecrecy CapacityIntrinsic Mutual InformationReduced Intrinsic Mutual InformationMinimal Intrinsic Mutual InformationNecessary Intrinsic Mutual InformationTwo-Part Intrinsic Mutual InformationQuickstartThe basic usage ofditcorresponds to creating distributions, modifying them
if need be, and then computing properties of those distributions. First, we
import:>>>importditSuppose we have a really thick coin, one so thick that there is a reasonable
chance of it landing on its edge. Here is how we might represent the coin indit.>>>d=dit.Distribution(['H','T','E'],[.4,.4,.2])>>>print(d)Class:DistributionAlphabet:('E','H','T')forallrvsBase:linearOutcomeClass:strOutcomeLength:1RVNames:Nonexp(x)E0.2H0.4T0.4Calculate the probability ofHand also of the combinationH or T.>>>d['H']0.4>>>d.event_probability(['H','T'])0.8Calculate the Shannon entropy and extropy of the joint distribution.>>>dit.shannon.entropy(d)1.5219280948873621>>>dit.other.extropy(d)1.1419011889093373Create a distribution whereZ = xor(X, Y).>>>importdit.example_dists>>>d=dit.example_dists.Xor()>>>d.set_rv_names(['X','Y','Z'])>>>print(d)Class:DistributionAlphabet:('0','1')forallrvsBase:linearOutcomeClass:strOutcomeLength:3RVNames:('X','Y','Z')xp(x)0000.250110.251010.251100.25Calculate the Shannon mutual informationsI[X:Z],I[Y:Z], andI[X,Y:Z].>>>dit.shannon.mutual_information(d,['X'],['Z'])0.0>>>dit.shannon.mutual_information(d,['Y'],['Z'])0.0>>>dit.shannon.mutual_information(d,['X','Y'],['Z'])1.0Calculate the marginal distributionP(X,Z).
Then print its probabilities as fractions, showing the mask.>>>d2=d.marginal(['X','Z'])>>>print(d2.to_string(show_mask=True,exact=True))Class:DistributionAlphabet:('0','1')forallrvsBase:linearOutcomeClass:strOutcomeLength:2(mask:3)RVNames:('X','Z')xp(x)0*01/40*11/41*01/41*11/4Convert the distribution probabilities to log (base 3.5) probabilities, and
access its probability mass function.>>>d2.set_base(3.5)>>>d2.pmfarray([-1.10658951,-1.10658951,-1.10658951,-1.10658951])Draw 5 random samples from this distribution.>>>dit.math.prng.seed(1)>>>d2.rand(5)['01','10','00','01','00']Contributions & HelpIf you’d like to feature added todit, please file an issue. Or, better yet, open a pull request. Ideally, all code should be tested and documented, but please don’t let this be a barrier to contributing. We’ll work with you to ensure that all pull requests are in a mergable state.If you’d like to get in contact about anything, you can reach us through ourslack channel.