Package: batchedmoments
pyBatchedMomentspyBatchedMomentsis a Python library for computing (batch-wise) sample statistics,
such as mean, variance, standard deviation, skewness and kurtosis.In certain applications it is needed to compute simple statistics of a population, but withtextbookformulae
the calculation can suffer from loss of precision and can be numerically unstable.
Additionally, for large populations only a single pass over the values is feasible, therefore,
an incremental (batch-wise) approach is needed.InstallationTo install the current release, runpipinstallbatchedmomentsFrom SourceTo install the latest development version (e.g. ineditable mode), rungitclonehttps://github.com/sbrodehl/pyBatchedMoments.git
pipinstall-epyBatchedMomentsExamplesWe start with the simple use case of sample statistics of some (random) numbers.frombatchedmomentsimportBatchedMomentsdata=[2,8,0,4,1,9,9,0]bm=BatchedMoments()bm(data)# use computed values# bm.mean, bm.std, ...The result is equivalent tonumpy(mean,stdandvar)
andscipy.stats(skewandkurtosis).Batched ComputationWherepyBatchedMomentsreally shines is when the data is not available at once.
In this case, the data can be batched (split inusableparts), and the statistics can be computed batch-wise.frombatchedmomentsimportBatchedMoments# a generator function which returns batches of datadata_iter=iter(list(range(n,n+10))forninrange(0,1000,10))bm=BatchedMoments()forbatchindata_iter:bm(batch)# use computed values# bm.mean, bm.std, ...Distributed / Parallel ComputationThe sample statistics of single batches can be computed independently and later be combined with theaddoperator.
The following example shows a multiprocessing use case, but the batches can be computed distributed among different
computers (nodes) as well.importmultiprocessingfrommultiprocessingimportPoolfrombatchedmomentsimportBatchedMoments# a generator function which returns batches of datadata=iter(list(range(n,n+10))forninrange(0,1000,10))# create object and initialize with first batch of databm=BatchedMoments()(next(data))withPool(processes=multiprocessing.cpu_count())aspool:fordbminpool.imap_unordered(BatchedMoments(),data):bm+=dbm# use computed values# bm.mean, bm.std, ...Reduction of AxesTheaxis=...keyword allows specifying axis or axes along which the sample statistics are computed.
The default (None) is to compute the sample statistics of the flattened array.Working with data of shape(1000, 3, 28, 28)and specifyingaxis=0the computed statistics will have shape(3, 28, 28).
Ifaxis=(0, 2, 3)the computed statistics will have shape(3,).Using thereducemethod the shape of the computed statistics can be further reduced at a later stage.
E.g. with data of shape(1000, 3, 28, 28)andaxis=(2, 3)the computed statistics will have shape(1000, 3).
By usingreduce(0)the computed statistics will be reduced to shape(3,).Machine Learning Use CaseA prime example, wherepyBatchedMomentscan be used, is to compute sample statistics of machine learning data sets.
Here we usetorchvision.datasetsto compute sample mean and sample standard deviation needed for normalization of the data set.fromtorch.utils.dataimportDataLoaderfromtorchvisionimporttransforms,datasetsfrombatchedmomentsimportBatchedMomentsimage_data=datasets.FashionMNIST("/tmp/FashionMNIST",download=True,train=True,transform=transforms.Compose([transforms.ToTensor()]))data_loader=DataLoader(image_data,batch_size=1024,)bm=BatchedMoments(axis=(0,2,3))forimgs,_indata_loader:bm(imgs.numpy())# use computed values# bm.mean, bm.std, ...# mean=0.28604060219395394 std=0.35302424954262396LicensepyBatchedMoments uses a MIT-style license, as found inLICENSEfile.