Package: embo
EMBO - Empirical BottleneckA Python package for working with the Information Bottleneck[Tishby,
Pereira, Bialek 2001]and the
Deterministic (and Generalized) Information Bottleneck[Strouse and
Schwab 2016]. Embo is especially
geared towards the analysis of concrete, finite-size data sets.Requirementsemborequires Python 3,numpy>=1.7andscipy.InstallationTo install the latest release, run:pipinstallembo(depending on your system, you may need to usepip3instead ofpipin the command above).Testing(requiressetuptools). Ifembois already installed on your
system, look for the copy of thetest_embo.pyscript installed
alongside the rest of theembofiles and execute it. For example:python/usr/lib/python3.X/site-packages/embo/test/test_embo.pyUsageThe Information BottleneckWe refer to[Tishby, Pereira, Bialek
2001]and[Strouse and Schwab
2016]for a general introduction to
the Information Bottleneck. Briefly, if X and Y are two random
variables, we are interested in finding another random variable M
(called the "bottleneck" variable) that solves the following
optimisation problem:min_{p(m|x)} = H(M) - α H(M|X) - β I(M:Y)for any β>0 and 0≤α≤1, and where M is constrained to be independent on
Y conditional on X:p(x,m,y) = p(x)p(m|x)p(y|x)Intuitively, we want to find the stochastic mapping p(M|X) that
extracts from X as much information about Y as possible while
forgetting all irrelevant information. β is a free parameter that sets
the relative importance of forgetting irrelevant information versus
remembering useful information. α determines what notion of
"forgetting" we use: α=1 ("vanilla" bottleneck or IB) implies that we
want to minimise the mutual information I(M:X), α=0 (deterministic
bottleneck or DIB) that we want to make M a goodcompressionof X by
minimising its entropy H(M), and intermediate values interpolate
between these two conditions.Typically, one is interested in the curve described by I(M:Y) as a
function of I(M:X) or H(M) at the solution of the bottleneck problem
for a range of values of β. This curve gives the optimal tradeoff of
compression and prediction, telling us what is the minimum amount of
information one needs to know about X (or minimum amount of entropy
one needs to retain) to be able to predict Y to a certain accuracy, or
vice versa, what is the maximum accuracy one can have in predicting Y
given a certain amount of information about X.UsingemboEmbo can solve the information bottleneck problem for discrete random
variables starting from a set of joint empirical observations. The
main point of entry to the package is theInformationBottleneckclass. In its constructor,InformationBottlenecktakes as arguments an
array of observations for X and an (equally long) array of
observations for Y, together with other optional parameters (see the
docstring for details). In the most basic use case, users can call theget_bottleneckmethod of anInformationBottleneckobject, which will
assume α=1 and return a set of β values and the optimal values of
I(M:X), I(M:Y) and H(M) corresponding to those β. The optimal tradeoff
can then be visualised by plotting I(M:Y) vs I(M:Y).For instance:importnumpyasnpfrommatplotlibimportpyplotaspltfromemboimportInformationBottleneck# data sequencesx=np.array([0,0,0,1,0,1,0,1,0,1])y=np.array([0,1,0,1,0,1,0,1,0,1])# compute the IB bound from the data (vanilla IB; Tishby et al 2001)I_x,I_y,H_m,β=InformationBottleneck(x,y).get_bottleneck()# plot the IB boundplt.plot(I_x,I_y)Embo can also operate starting from a joint (X,Y) probability
distribution, encoded as a 2D array containing the probability
of each combination of states for X and Y.# define joint probability mass function for a 2x2 joint pmfpxy=np.array([[0.1,0.4],[0.35,0.15]]),# compute IBI_x,I_y,H_m,β=InformationBottleneck(pxy=pxy).get_bottleneck()# plot I(M:Y) vs I(M:X)plt.plot(I_x,I_y)The deterministic and generalised bottleneck can be computed by
setting appropriately the parameteralpha:# compute Deterministic Information Bottleneck (Strouse 2016)I_x,I_y,H_m,β=InformationBottleneck(pxy=pxy,alpha=0).get_bottleneck()# plot I(M:Y) vs H(M)plt.plot(H_m,I_y)More examplesTheembo/examplesdirectory contains some Jupyter notebook that
should exemplify most of the package's functionality.Basic-Example.ipynb: basics;
how to compute and plot an IB bound.Markov-Chains.ipynb: using embo
forpast-future bottlenecktype analyses on data from Markov
chains.Deterministic-Bottleneck.ipynb:
Deterministic and Generalized Information Bottleneck. Here we
reproduce a key figure from the Deterministic Bottleneck paper, and
we explore the algorithm's behaviour as α changes from 0 to 1.Compare-embo-dit.ipynb: here
we compare embo withdit[James et
al 2018]. We compare the
solutions found by the two packages on a set of simple IB problems
(including a problem taken from dit's documentation), and we show
that embo is orders of magnitude faster than dit.Further detailsFor more details, please consult the docstrings inInformationBottleneck.ChangelogSee theCHANGELOG.mdfile for a list of changes from
older versions.Authorsembois maintained by Eugenio Piasini, Alexandre Filipowicz and
Jonathan Levine.