Package: runstats
RunStatsis an Apache2 licensed Python module for online statistics and
online regression. Statistics and regression summaries are computed in a single
pass. Previous values are not recorded in summaries.Long running systems often generate numbers summarizing performance. It could
be the latency of a response or the time between requests. It’s often useful to
use these numbers in summary statistics like the arithmetic mean, minimum,
standard deviation, etc. When many values are generated, computing these
summaries can be computationally intensive. It may even be infeasible to keep
every recorded value. In such cases computing online statistics and online
regression is necessary.In other cases, you may only have one opportunity to observe all the recorded
values. Python’s generators work exactly this way. Traditional methods for
calculating the variance and other higher moments requires multiple passes over
the data. With generators, this is not possible and so computing statistics in
a single pass is necessary.There are also scenarios where a user is not interested in a complete summary
of the entire stream of data but rather wants to observe the current state of
the system based on the recent past. In these cases exponential statistics are
used. Instead of weighting all values uniformly in the statistics computation,
an exponential decay weight is applied to older values. The decay rate is
configurable and provides a mechanism for balancing recent values with past
values.The PythonRunStatsmodule was designed for these cases by providing classes
for computing online summary statistics and online linear regression in a
single pass. Summary objects work on sequences which may be larger than memory
or disk space permit. They may also be efficiently combined together to create
aggregate summaries.FeaturesPure-PythonFully Documented100% Test CoverageNumerically StableOptional Cython-optimized Extension (5-100 times faster)Statistics summary computes mean, variance, standard deviation, skewness,
kurtosis, minimum and maximum.Regression summary computes slope, intercept and correlation.Developed on Python 3.9Tested on CPython 3.6, 3.7, 3.8, 3.9Tested on Linux, Mac OS X, and WindowsTested using GitHub ActionsQuickstartInstallingRunStatsis simple withpip:$ pip install runstatsYou can access documentation in the interpreter with Python’s built-in help
function:>>>importrunstats>>>help(runstats)# doctest: +SKIP>>>help(runstats.Statistics)# doctest: +SKIP>>>help(runstats.Regression)# doctest: +SKIP>>>help(runstats.ExponentialStatistics)# doctest: +SKIPTutorialThe PythonRunStatsmodule provides three types for computing running
statistics: Statistics, ExponentialStatistics and Regression.The Regression
object leverages Statistics internally for its calculations. Each can be
initialized without arguments:>>>fromrunstatsimportStatistics,Regression,ExponentialStatistics>>>stats=Statistics()>>>regr=Regression()>>>exp_stats=ExponentialStatistics()Statistics objects support four methods for modification. Usepushto add
values to the summary,clearto reset the summary, sum to combine Statistics
summaries and multiply to weight summary Statistics by a scalar.>>>fornuminrange(10):...stats.push(float(num))>>>stats.mean()4.5>>>stats.maximum()9.0>>>stats+=stats>>>stats.mean()4.5>>>stats.variance()8.68421052631579>>>len(stats)20>>>stats*=2>>>len(stats)40>>>stats.clear()>>>len(stats)0>>>stats.minimum()nanUse the Python built-inlenfor the number of pushed values. Unfortunately
the Pythonminandmaxbuilt-ins may not be used for the minimum and
maximum as sequences are expected instead. Therefore, there areminimumandmaximummethods provided for that purpose:>>>importrandom>>>random.seed(0)>>>for__inrange(1000):...stats.push(random.random())>>>len(stats)1000>>>min(stats)Traceback(mostrecentcalllast):...TypeError:...>>>stats.minimum()0.00024069652516689466>>>stats.maximum()0.9996851255769114Statistics summaries provide five measures of a series: mean, variance,
standard deviation, skewness and kurtosis:>>>stats=Statistics([1,2,5,12,5,2,1])>>>stats.mean()4.0>>>stats.variance()15.33333333333333>>>stats.stddev()3.915780041490243>>>stats.skewness()1.33122127314735>>>stats.kurtosis()0.5496219281663506All internal calculations use Python’sfloattype.Like Statistics, the Regression type supports some methods for modification:push,clearand sum:>>>regr.clear()>>>len(regr)0>>>fornuminrange(10):...regr.push(num,num+5)>>>len(regr)10>>>regr.slope()1.0>>>more=Regression((num,num+5)fornuminrange(10,20))>>>total=regr+more>>>len(total)20>>>total.slope()1.0>>>total.intercept()5.0>>>total.correlation()1.0Regression summaries provide three measures of a series of pairs: slope,
intercept and correlation. Note that, as a regression, the points need not
exactly lie on a line:>>>regr=Regression([(1.2,1.9),(3,5.1),(4.9,8.1),(7,11)])>>>regr.slope()1.5668320150154176>>>regr.intercept()0.21850113956294415>>>regr.correlation()0.9983810791694997Both constructors accept an optional iterable that is consumed and pushed into
the summary. Note that you may pass a generator as an iterable and the
generator will be entirely consumed.The ExponentialStatistics are constructed by providing a decay rate, initial
mean, and initial variance. The decay rate has default 0.9 and must be between
0 and 1. The initial mean and variance default to zero.>>>exp_stats=ExponentialStatistics()>>>exp_stats.decay0.9>>>exp_stats.mean()0.0>>>exp_stats.variance()0.0The decay rate is the weight by which the current statistics are discounted
by. Consequently, (1 - decay) is the weight of the new value. Like theStatisticsclass,
there are four methods for modification:push,clear, sum and
multiply.>>>fornuminrange(10):...exp_stats.push(num)>>>exp_stats.mean()3.486784400999999>>>exp_stats.variance()11.593430921943071>>>exp_stats.stddev()3.4049127627507683The decay of the exponential statistics can also be changed. The value must be
between 0 and 1.>>>exp_stats.decay0.9>>>exp_stats.decay=0.5>>>exp_stats.decay0.5>>>exp_stats.decay=10Traceback(mostrecentcalllast):...ValueError:decaymustbebetween0and1The clear method allows to optionally set a new mean, new variance and new
decay. If none are provided mean and variance reset to zero, while the decay is
not changed.>>>exp_stats.clear()>>>exp_stats.decay0.5>>>exp_stats.mean()0.0>>>exp_stats.variance()0.0CombiningExponentialStatisticsis done by adding them together. The mean and
variance are simply added to create a new object. To weight eachExponentialStatistics, multiply them by a constant factor. If twoExponentialStatisticsare added then the leftmost decay is used for the new
object. Thelenmethod is not supported.>>>alpha_stats=ExponentialStatistics(iterable=range(10))>>>beta_stats=ExponentialStatistics(decay=0.1)>>>fornuminrange(10):...beta_stats.push(num)>>>exp_stats=beta_stats*0.5+alpha_stats*0.5>>>exp_stats.decay0.1>>>exp_stats.mean()6.187836645All internal calculations of the Statistics and Regression classes are based
entirely on the C++ code by John Cook as posted in a couple of articles:Computing Skewness and Kurtosis in One PassComputing Linear Regression in One PassThe ExponentialStatistics implementation is based on:Finch, 2009, Incremental Calculation of Weighted Mean and VarianceThe pure-Python version ofRunStatsis directly available if preferred.>>>importrunstats.core# Pure-Python>>>runstats.core.Statistics<class'runstats.core.Statistics'>When importing fromrunstatsthe Cython-optimized version_coreis
preferred and thecoreversion is used as fallback. Micro-benchmarking
Statistics and Regression by callingpushrepeatedly shows the
Cython-optimized extension as 20-40 times faster than the pure-Python
extension.Reference and IndicesRunStats DocumentationRunStats API ReferenceRunStats at PyPIRunStats at GitHubRunStats Issue TrackerLicenseCopyright 2013-2021 Grant JenksLicensed under the Apache License, Version 2.0 (the “License”); you may not use
this file except in compliance with the License.  You may obtain a copy of the
License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied.  See the License for the
specific language governing permissions and limitations under the License.