Package: orderstats
OrderStatisticsA repository focused on outlier detection and analysis of heavy-tailed distributions using order statistics. Here you can find:Simualations of order statistics for outlier detection and the statistic provided in our paper titledOn a Notion of Outliers Based on Ratios of Order Statistics.Single and double bootstrap methods (& more) for tail index estimation of heavy-tailed data.Special kernel density estimation methods for heavy-tailed data.Also:Useful plotting functions for outlier detection.Easy reporting tools for binary classification such asget_classification_report(), which reports all classification metrics from confusion matrix to AUC and can plot ROC curve.Installation:You can install the repository with:pip install orderstatsAlternatively, you can download a copy of the repository from this page.
After downloading, you can dopip install -r requirements.txtto install the requirements.General explanation about Random Variables:We usescipy.statspackage for generating random variables.
A random variable instance can be created just by giving the appropriate parameters.
For example for X ~ N(0, 1), we can do:>>> X = stats.norm(0, 1)Once an instance is created, we can calculate pdf, or cdf using.pdf(),.cdf()methods, or,
we can take a sample using.rvs():>>> X.pdf(1.96)for pdf of X ~ N(0, 1) at x = 1.96;>>> X.cdf(1.96)for cdf of X ~ N(0, 1) at x = 1.96;>>> X.rvs(1000)for an i.i.d sample of 1000 from X ~ N(0, 1).Some problems:Not every random variable function inscipy.statsis intuitive.For instanceexponfunction creates an instance of an exponential distribution.
However, if we wish to get an instance of exponential distribution with lambda = 2
(i.e. with pdf f(x) = 2e^(-2x)), then we would need to useexpon(0, 1/2).
In distributions.py, there are examples given for some popular distributions to clarify any ambiguities.ExamplesOutlier Detection:Here is an example use of our method for a 1D datasetX:fromorderstatsimportscoringfromorderstats.distributionsimportmoving_average_unscaled_kappafromorderstats.plot_utilsimportplot_anomaliesscores,scores_sorted=scoring.get_anomaly_scores(X,scoring_func=moving_average_unscaled_kappa)threshold=scoring.get_kappa_threshold(scores_sorted)predictions=scores>thresholdplot_anomalies(X,predictions=predictions)Order Statistics Simulation:In general, we will useOrderSimulationclass in distributions for simulations. Any random variable from thescipy.statspackage can be given to this class as an argument.
For example, if you wish to simulate the sums of firstmorder statistics from a sample of exponential distribution of sizen:fromorderstatsimportOrderSimulationsimulate_normal_dist=OrderSimulation(stats.expon(0,1),calculate_S_m)simulation=simulate_normal_dist(10000,n,m)For studying change point detection, we provide theMixSimulationclass. For getting a mixed sample with corresponding ids:fromorderstatsimportMixSimulationsimulate_mixture=MixSimulation(dist1=expon(0,1),dist2=stats.pareto(2.))mixed_array,idx=simulate_mixture(n1,n2)Tail Index Estimation:Most of the methods for tail index estimation mentioned in [1] is implemented intail_estimation.
As an example for the double bootstrap method:importnumpyasnpN=1000pareto_sample=np.random.pareto(2,1000)sample_to_estimate_index=np.sort(X,)[::-1]# Sort decreasingdouble_bootsrap=DoubleBootstrap()tail_index=double_bootstrap(N,sample_to_estimate_index)References:[1] Markovich, N. (2008). Nonparametric analysis of univariate heavy-tailed data: research and practice (Vol. 753). John Wiley & Sons.