Package: bayeshist
Bayesian histogramsBayesian histogramsare a nifty tool for data mining if:you want to know how theevent rate(probability) of a binaryrare eventdepends on a parameter;you have millions or evenbillions of data points, but few positive samples;you suspect the event rate dependshighly non-linearlyon the parameter;you don't know whether you haveenough data, so you needuncertainty information.Thanks to an adaptive bin pruning algorithm, you don't even have to choose the number of bins, and you should get good results out of the box.This is how they look in practice (see full example below):Installation$pipinstallbayeshistUsage exampleAssume you have binary samples of a rare event like this:Compute and plot a Bayesian histogram:>>>frombayeshistimportbayesian_histogram,plot_bayesian_histogram# compute Bayesian histogram from samples>>>bin_edges,beta_dist=bayesian_histogram(X,y,bins=100,pruning_method="bayes")# beta_dist is a `scipy.stats.Beta` object, so we can get the# predicted mean event rate for each histogram bin like this:>>>bin_mean_pred=best_dist.mean()# plot it up>>>plot_bayesian_histogram(bin_edges,beta_dist)The result is something like this:See alsodemo.ipynbfor a full walkthrough of this example.But how do they work?Here's the blog post.API referencebayesian_histogramdefbayesian_histogram(x:np.ndarray,y:np.ndarray,bins:Union[int,Iterable]=100,x_range:Optional[Tuple[float,float]]=None,prior_params:Optional[Tuple[float,float]]=None,pruning_method:Optional[Literal["bayes","fisher"]]="bayes",pruning_threshold:Optional[float]=None,max_bin_size:Optional[float]=None,)->Tuple[np.ndarray,FrozenDistType]:"""Compute Bayesian histogram for data x, binary target y.The output is a Beta distribution over the event rate for each bin.Parameters:x:1-dim array of data.y:1-dim array of binary labels (0 or 1).bins:int giving the number of equally spaced intial bins,or array giving initial bin edges. (default: 100)x_range:Range spanned by binning. Not used if `bins` is an array.(default: [min(x), max(x)])prior_params:Parameters to use in Beta prior. First value relates to positive,second value to negative samples. [0.5, 0.5] represents Jeffrey's prior, [1, 1] a flatprior. The default is a weakly informative prior based on the global event rate.(default: `[1, num_neg / num_pos]`)pruning_method:Method to use to decide whether neighboring bins should be merged or not.Valid values are "bayes" (Bayes factor), "fisher" (exact Fisher test), or None(no pruning). (default: "bayes")pruning_threshold:Threshold to use in significance test specified by `pruning_method`.(default: 2 for "bayes", 0.2 for "fisher")max_bin_size:Maximum size (in units of x) above which bins will not be merged(except empty bins). (default: unlimited size)Returns:bin_edges: Coordinates of bin edgesbeta_dist: n-dimensional Beta distribution (n = number of bins)Example:>>> x = np.random.randn(1000)>>> p = 10 ** (-2 + x)>>> y = np.random.rand() < p>>> bins, beta_dist = bayesian_histogram(x, y)>>> plt.plot(0.5 * (bins[1:] + bins[:-1]), beta_dist.mean())"""plot_bayesian_histogramdefplot_bayesian_histogram(bin_edges:np.ndarray,data_dist:FrozenDistType,color:Union[str,Iterable[float],None]=None,label:Optional[str]=None,ax:Any=None,ci:Optional[Tuple[float,float]]=(0.01,0.99))->None:"""Plot a Bayesian histogram as horizontal lines with credible intervals.Parameters:bin_edges:Coordinates of bin edgesdata_dist:n-dimensional Beta distribution (n = number of bins)color:Color to use (default: use next in current color cycle)label:Legend label (default: no label)ax:Matplotlib axis to use (default: current axis)ci:Credible interval used for shading, use `None` to disable shading.Example:>>> x = np.random.randn(1000)>>> p = 10 ** (-2 + x)>>> y = np.random.rand() < p>>> bins, beta_dist = bayesian_histogram(x, y)>>> plot_bayesian_histogram(bins, beta_dist)"""Questions?Feel free to open an issue.