Package: npstreams
npstreamsnpstreams is an open-source Python package for streaming NumPy array
operations. The goal is to provide tested routines that operate on
streams (or generators) of arrays instead of dense arrays.Streaming reduction operations (sums, averages, etc.) can be implemented
in constant memory, which in turns allows for easy parallelization.This approach has been a huge boon when working with lots of images; the
images are read one-by-one from disk and combined/processed in a
streaming fashion.This package is developed in conjunction with other software projects in
theSiwick research group.Motivating ExampleConsider the following snippet to combine 50 images from an iterablesource:importnumpyasnpimages=np.empty(shape=(2048,2048,50))forindex,iminenumerate(source):images[:,:,index]=imavg=np.average(images,axis=2)If thesourceiterable provided 1000 images, the above routine would
not work on most machines. Moreover, what if we want to transform the
images one by one before averaging them? What about looking at the
average while it is being computed? Let's look at an example:importnumpyasnpfromnpstreamsimportiaveragefromscipy.miscimportimreadstream=map(imread,list_of_filenames)averaged=iaverage(stream)At this point, the generatorsmapandiaverageare 'wired' but
will not compute anything until it is requested. We can look at the
average evolve:importmatplotlib.pyplotaspltforavginaverage:plt.imshow(avg);plt.show()We can also uselastto get at the final average:fromnpstreamsimportlasttotal=last(averaged)# average of the entire streamStreaming Functionsnpstreams comes with some streaming functions built-in. Some examples:Numerics :isum,iprod,isub, etc.Statistics :iaverage(weighted mean),ivar(single-pass
variance), etc.More importantly, npstreams gives you all the tools required to build
your own streaming function. All routines are documented in theAPI
Reference on readthedocs.io.Benchmarkingnpstreams provides a function for benchmarking common use cases.To run the benchmark with default parameters, from the interpreter:fromnpstreamsimportbenchmarkbenchmark()From a command-line terminal:python-c'import npstreams; npstreams.benchmark()'The results will be printed to the screen.Future WorkSome of the features I want to implement in this package in the near
future:Optimize the CUDA-enabled routinesMore functions : more streaming functions borrowed from NumPy and
SciPy.API ReferenceTheAPI Reference on readthedocs.ioprovides API-level documentation, as well as tutorials.InstallationThe only requirement is NumPy. To have access to CUDA-enabled routines,
PyCUDA must also be installed. npstreams is available on PyPI; it can be
installed withpip.:python-mpipinstallnpstreamsnpstreams can also be installed with the conda package manager, from the
conda-forge channel:condaconfig--addchannelsconda-forge
condainstallnpstreamsTo install the latest development version fromGithub:python-mpipinstallgit+git://github.com/LaurentRDC/npstreams.gitTests can be run using thepytestpackage.CitationsIf you find this software useful, please consider citing the following
publication:L. P. René de Cotret, M. R. Otto, M. J. Stern. and B. J. Siwick,An open-source software ecosystem for the interactive exploration of ultrafast electron scattering data, Advanced Structural and Chemical Imaging 4:11 (2018)DOI: 10.1186/s40679-018-0060-y.Support / Report IssuesAll support requests and issue reports should befiled on Github as an
issue.Licensenpstreams is made available under the BSD License, same as NumPy. For
more details, seeLICENSE.txt.