Package: bootstrap-stat
bootstrap-statMethods relating to the Bootstrap.Estimates of standard errors, bias, confidence intervals, prediction
errors, and more!Getting StartedBootstrap-Stat is hosted on PyPI. Install as you would any other
library, e.g.:poetry add bootstrap-stat.Documentation is available atConvex Analyticsbut I also recommend reviewing the rest of this README.Brief OverviewQuoting [ET93], "The bootstrap is a data-based simulation method for
statistical inference...The use of the term bootstrap derives from the
phraseto pull oneself up by one's bootstrap..."The bootstrap is a collection of methodologies for estimating errors
and uncertainties associated with estimates and predictions. For
example, it can be used to compute the bias and standard error of a
particular estimator. It can be used to estimate the prediction error
of a particular ML model (as a competitor to Cross Validation). And it
can be used to compute confidence intervals. Most of the techniques
described in [ET93] have now been implemented. (Notable exceptions
include importance resampling, likelihood-based methods, and better
support for the parametric bootstrap.)Basic TerminologyTo apply bootstrap techniques, it is important to understand the
following terminology.Distribution: an entity assigning probabilities to observations or
sets of observations.Population: the complete set of data, typically only partially observed.Sample: the observed data, assumed to be drawn with replacement from
the population. In real life, samples are usually drawnwithoutreplacement from the population, but provided the sample is a small
fraction of the population, this is a negligible concern.Empirical Distribution: the distribution assigning probability 1/n
to each observation in a sample of size n.Parameter: some function of a distribution. Seestatistic.Statistic: some function of a collection of observations. We will
assume all statistics are real-valued scalars, such as mean, median,
or variance. Parameters and statistics are similar: it makes sense
to talk about the mean of a distribution, which is a parameter of
that distribution, and it also makes sense to talk about the mean of
a collection of numbers, which is a statistic. For this reason, it
is important to keep it straight whether we are talking about a
parameter or a statistic! For example, it makes sense to talk about
the bias of a statistic, but it does not make sense to talk about
the bias of a parameter.Plug-in estimate of a parameter: an estimate of a parameter
calculated by "plugging-in" the Empirical Distribution. For example,
to estimate the mean of an unobserved distribution, simply calculate
the mean of the (observed) Empirical Distribution. The plug-in
estimate is astatistic.Bootstrap sample: a sample drawn with replacement from the Empirical
Distribution, having size equal to the size of the original dataset.Standard Error: the square root of the variance of a statistic,
typically used to quantify accuracy.Bias: the difference between the expected value of a statistic and
the parameter it purports to estimate.Confidence Interval: a range of plausible values of a parameter
consistent with the data.ExamplesThis library includes some datasets that can be used for trying out
methods. The test cases themselves (intests/) contain many
practical examples.>>> import numpy as np
>>> from bootstrap_stat import bootstrap_stat as bp
>>> from bootstrap_stat import datasets as d
>>>
>>> df = d.law_data()
>>> print(df)
    LSAT   GPA
0    576  3.39
1    635  3.30
2    558  2.81
3    578  3.03
4    666  3.44
5    580  3.07
6    555  3.00
7    661  3.43
8    651  3.36
9    605  3.13
10   653  3.12
11   575  2.74
12   545  2.76
13   572  2.88
14   594  2.96The law data are a collection of N = 82 American law schools
participating in a large study of admissions practices. Two
measurements were made on the entering classes of each school in 1973:
LSAT, the average score for the class on a national law test, and GPA,
the average undergraduate grade-point average for the class. Both the
full data set, and a sample are available. The above is a sample of 15
schools. The law data are taken from [EF93].Suppose we are interested in the correlation between LSAT and
GPA. Numpy can be used to compute the observed correlation for the
sample (astatistic), but we hope to draw inferences about the
population (all 82 schools) correlation coefficient (aparameter)
based just on the sample. In this case, the entire population is
available, and we could just compute the parameter directly. In most
cases, the entire population is not available.To use the bootrap method, we need to specify the statistic as well as
the dataset. Specifically, we need to be able to sample with
replacement from the Empirical Distribution.bootstrap_stathas a class
facilitating just that.>>> dist = bp.EmpiricalDistribution(df)
>>> dist.sample(reset_index=False)
    LSAT   GPA
14   594  2.96
3    578  3.03
0    576  3.39
6    555  3.00
10   653  3.12
12   545  2.76
6    555  3.00
0    576  3.39
10   653  3.12
8    651  3.36
3    578  3.03
4    666  3.44
13   572  2.88
5    580  3.07
8    651  3.36Generating the Empirical Distribution is as simple as feeding either
an array, pandas Series, or pandas DataFrame into the
constructor. Under the hood, the bootstrap methods make frequent use
of thesamplemethod, which samples with replacement from the
original dataset. Such samples are calledbootstrap samples. Notice
in the example above how school 0 makes multiple appearances in the
bootstrap sample. Since the sampling is random, if you run the above
you will likely get different results than in this example. (In some
of the more exotic use cases, we need to reset the index for technical
reasons relating to pandas indexing, so the default behavior is to
reset, hence thereset_index=Falsein this example.)Next we need to implement the statistic, which will be applied to
bootstrap samples.>>> def statistic(df):
...     return np.corrcoef(df["LSAT"], df["GPA"])[0, 1]
...
>>> obs_correlation = statistic(df)  # Observed correlation coefficient
>>> print(obs_correlation)
0.776374491289407Notice how we can apply the statistic to the original dataset to
calculate the observed value. The statistic should take as input
either an array or a pandas Series or DataFrame, whatever was used to
generate the Empirical Distribution. It should output a single
number. Other than that, it can be anything: a simple calculation like
a mean, a parameter from a linear regression model, or even a
prediction from a neural network.Now we can compute the standard error, which is a way of quantifying
the variability of a statistic:>>> se = bp.standard_error(dist, statistic)
>>> print(se)
0.13826565276176475Since the bootstrap involves random sampling, you will likely get a
slightly different answer than above, but it should be within 1% or
so.Or we can compute a confidence interval, a range of plausible values
for the parameter consistent with the data.>>> ci_low, ci_high = bp.bcanon_interval(dist, statistic, df)
>>> print(ci_low, ci_high)
0.44968698948896413 0.9230026418265834These represent lower and upper bounds on a 90% confidence interval,
the default behavior ofbcanon_interval. We can do a 95% confidence
interval by specifyingalpha:>>> ci_low, ci_high = bp.bcanon_interval(dist, statistic, df, alpha=0.025)
>>> print(ci_low, ci_high)
0.3120414479586675 0.9425059323691073In general,bcanon_intervalreturns a 100(1-2alpha)% confidence
interval. Thebcanonterminology is a nod to the S implementation
discussed in [ETF93]. (BCa is an algorithm for Bias-Corrected and
Accelerated confidence intervals, and the function is NONparametric.)Basic multicore functionality is implemented, allowing parallel
calculation of bootstrap samples. Simply specify thenum_threadsargument in applicable functions. See the function documentation for
details.Running the test cases$ poetry shell
$ python -m pytestDocumentationDocumentation is built using Sphinx and is hosted atConvex Analytics.To update the docs (e.g. after updating the code), just change
directory todocsand typemake html. You'll need to be in a
poetry shell.ArchitectureWe use Poetry to manage dependencies, pytest as our test runner, black
for code formatting, and sphinx for generating documentation.Basic multicore functionality is implemented, using thepathosversion of
multiprocessing. We chose this version over the official python
multiprocessing library since pathos usesdillinstead ofpickleto manage shared memory, andpicklecannot be used with locally
defined functions. For users of this library, hopefully that
implementation detail is irrelevant.LicensingBootstrap-Stat is licensed under the Apache License, Version 2.0. SeeLICENSE.txtfor the full license text.References[ET93] Bradley Efron and Robert J. Tibshirani, "An Introduction to the
Bootstrap". Chapman & Hall, 1993.