Package: numpy-groupies
numpy-groupiesThis package consists of a small library of optimised tools for doing things that can roughly
be considered "group-indexing operations". The most prominent tool isaggregate, which is
described in detail further down the page.InstallationIf you havepip, then simply:pip install numpy_groupiesNote thatnumpy_groupiesdoesn't have any compulsory dependencies (evennumpyis optional)
so you should be able to install it fairly easily even without a package manager.  If you just
want one particular implementation ofaggregate(e.g.aggregate_numpy.py), you can download
that one file, and copy-paste the contents ofutils.pyinto the top of that file (replacing
thefrom .utils import (...)line).aggregateimportnumpyasnpimportnumpy_groupiesasnpggroup_idx=np.array([3,0,0,1,0,3,5,5,0,4])a=np.array([13.2,3.5,3.5,-8.2,3.0,13.4,99.2,-7.1,0.0,53.7])npg.aggregate(group_idx,a,func='sum',fill_value=0)# >>>          array([10.0, -8.2, 0.0, 26.6, 53.7, 92.1])aggregatetakes an array of values, and an array giving the group number for each of those values.
It then returns the sum (or mean, or std, or any, ...etc.) of the values in each group. You have
probably come across this idea before - seeMatlab'saccumarrayfunction, orpandasgroupby concept, orMapReduce paradigm, or simply thebasic histogram.A couple of implemented functions do not reduce the data, instead it calculates values cumulatively
while iterating over the data or permutates them. The output size matches the input size.group_idx=np.array([4,3,3,4,4,1,1,1,7,8,7,4,3,3,1,1])a=np.array([3,4,1,3,9,9,6,7,7,0,8,2,1,8,9,8])npg.aggregate(group_idx,a,func='cumsum')# >>>          array([3, 4, 5, 6,15, 9,15,22, 7, 0,15,17, 6,14,31,39])InputsThe function accepts various different combinations of inputs, producing various different shapes of output.
We give a brief description of the general meaning of the inputs and then go over the different combinations
in more detail:group_idx- array of non-negative integers to be used as the "labels" with which to group the values ina.a- array of values to be aggregated.func='sum'- the function to use for aggregation. See the section below for more details.size=None- the shape of the output array. IfNone, the maximum value ingroup_idxwill set the size of the output.fill_value=0- value to use for output groups that do not appear anywhere in thegroup_idxinput array.order='C'- for multidimensional output, this controls the layout in memory, can be'F'for fortran-style.dtype=None- thedtypeof the output.Nonemeans choose a sensible type for the givena,func, andfill_value.axis=None- explained below.ddof=0- passed through into calculations of variance and standard deviation (see section on functions).Form 1 is the simplest, takinggroup_idxandaof matching 1D lengths, and producing a 1D output.Form 2 is similar to Form 1, but takes a scalara, which is broadcast out to the length ofgroup_idx. Note that this is generally not that useful.Form 3 is more complicated.group_idxis the same length as thea.shape[axis]. The groups are broadcast out along the other axis/axes ofa, thus the output is of shapen_groups x a.shape[0] x ... x a.shape[axis-1] x a.shape[axis+1] x ... a.shape[-1], i.e. the output has two or more dimensions.Form 4 also produces output with two or more dimensions, but for very different reasons to Form 3.  Hereais 1D andgroup_idxis exactly2D, whereas in Form 3aisND,group_idxis1D, and we provide a value foraxis.  The length ofamust matchgroup_idx.shape[1], the value ofgroup_idx.shape[0]determines the number of dimensions in the output, i.e.group_idx[:,99]gives the(x,y,z)group indices for thea[99].Form 5 is the same as Form 4 but with scalara. As with Form 2, this is rarely that helpful.Note on performance.Theorderof the output is unlikely to affect performance ofaggregate(although it may affect your downstream usage of that output), however the order of multidimensionalaorgroup_idxcan affect performance:  in Form 4 it is best if columns are contiguous in memory withingroup_idx, i.e.group_idx[:, 99]corresponds to a contiguous chunk of memory; in Form 3 it's best if all the data inaforgroup_idx[i]is contiguous, e.g. ifaxis=1then we wanta[:, 55]to be contiguous.Available functionsBy default,aggregateassumes you want to sum the values within each group, however you can specify another
function using thefunckwarg.  Thisfunccan be any custom callable, however you will likely want one of
the following optimized functions. Note that not all functions might be provided by all implementations.'sum'- sum of items within each group (see example above).'prod'- product of items within each group'mean'- mean of items within each group'var'- variance of items within each group. Useddofkwarg for degrees of freedom. The divisor used in calculations isN - ddof, whereNrepresents the number of elements. By defaultddofis zero.'std'- standard deviation of items within each group. Useddofkwarg for degrees of freedom (seevarabove).'min'- minimum value of items within each group.'max'- maximum value of items within each group.'first'- first item inafrom each group.'last'- last item inafrom each group.'argmax'- the index inaof the maximum value in each group.'argmin'- the index inaof the minimum value in each group.The above functions also have anan-form, which skip thenanvalues instead of propagating them to the result of the calculation:'nansum','nanprod','nanmean','nanvar','nanstd','nanmin','nanmax','nanfirst','nanlast','nanargmax','nanargmin'The following functions are slightly different in that they always return boolean values. Their treatment of nans is also different from above:'all'-Trueif all items within a group are truethy. Note thatnp.all(nan)isTrue, i.e.nanis actually truethy.'any'-Trueif any items within a group are truethy.'allnan'-Trueif all items within a group arenan.'anynan'-Trueif any items within a group arenan.The following functions don't reduce the data, but instead produce an output matching the size of the input:'cumsum'- cumulative sum of items within each group.'cumprod'- cumulative product of items within each group. (numba only)'cummin'- cumulative minimum of items within each group. (numba only)'cummax'- cumulative maximum of items within each group. (numba only)'sort'- sort the items within each group in ascending order, use reverse=True to invert the order.Finally, there are three functions which don't reduce each group to a single value, instead they return the full
set of items within the group:'array'- simply returns the grouped items, using the same order as appeared ina. (numpy only)ExamplesCompute sums of consecutive integers, and then compute products of those consecutive integers.group_idx=np.arange(5).repeat(3)# group_idx: array([0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4])a=np.arange(group_idx.size)# a: array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14])x=npg.aggregate(group_idx,a)# sum is default# x: array([ 3, 12, 21, 30, 39])x=npg.aggregate(group_idx,a,'prod')# x: array([ 0, 60, 336, 990, 2184])Get variance ignoring nans, setting all-nan groups tonan.x=npg.aggregate(group_idx,a,func='nanvar',fill_value=nan)Count the number of elements in each group. Note that this is equivalent to doingnp.bincount(group_idx),
indeed that is how the numpy implementation does it.x=npg.aggregate(group_idx,1)Sum 1000 values into a three-dimensional cube of size 15x15x15. Note that in this example all three dimensions
have the same size, but that doesn't have to be the case.group_idx=np.random.randint(0,15,size=(3,1000))a=np.random.random(group_idx.shape[1])x=npg.aggregate(group_idx,a,func="sum",size=(15,15,15),order="F")# x.shape: (15, 15, 15)# np.isfortran(x): TrueUse a custom function to generate some strings.group_idx=np.array([1,0,1,4,1])a=np.array([12.0,3.2,-15,88,12.9])x=npg.aggregate(group_idx,a,func=lambdag:' or maybe '.join(str(gg)forgging),fill_value='')# x: ['3.2', '12.0 or maybe -15.0 or maybe 12.9', '', '', '88.0']Use theaxisarg in order to do a sum-aggregation on three rows simultaneously.a=np.array([[99,2,11,14,20],[33,76,12,100,71],[67,10,-8,1,9]])group_idx=np.array([[3,3,7,0,0]])x=npg.aggregate(group_idx,a,axis=1)# x : [[ 34, 0, 0, 101, 0, 0, 0, 11],#      [171, 0, 0, 109, 0, 0, 0, 12],#      [ 10, 0, 0,  77, 0, 0, 0, -8]]Multiple implementationsThere are multiple implementations ofaggregateprovided. If you usefrom numpy_groupies import aggregate,
the best available implementation will automatically be selected. Otherwise you can pick a specific version directly
likefrom numpy_groupies import aggregate_nb as aggregateor by importing aggregate from the implementing modulefrom numpy_groupies.aggregate_weave import aggregate.Currently the following implementations exist:numpy- This is the default implementation. It uses plainnumpy, mainly relying onnp.bincountand basic indexing magic. It comes without other dependencies exceptnumpyand shows reasonable performance for the occasional usage.numba- This is the most performant implementation, based on jit compilation provided by numba and LLVM.pure python- This implementation has no dependencies and uses only the standard library. It's horribly slow and should only be used, if there is no numpy available.numpy ufunc-Only for benchmarking.This implementation uses the.atmethod of numpy'sufuncs (e.g.add.at), which would appear to be designed for performing exactly the same calculation thataggregateexecutes, however the numpy implementation is rather incomplete.pandas-Only for reference.The pandas'groupbyconcept is the same as the task performed byaggregate. However,pandasis not actually faster than the defaultnumpyimplementation. Also, note that there may be room for improvement in the way thatpandasis utilized here. Most notably, when computing multiple aggregations of the same data (e.g.'min'and'max') pandas could potentially be used more efficiently.All implementations have the same calling syntax and produce the same outputs, to within some floating-point error.
However some implementations only support a subset of the valid inputs and will sometimes throwNotImplementedError.BenchmarksScripts for testing and benchmarking are included in this repository. For benchmarking, runpython -m numpy_groupies.benchmarks.genericfrom the root of this repository.Below we are using500,000indices uniformly picked from[0, 1000). The values ofaare uniformly picked from
the interval[0,1), with anything less than0.2then set to 0 (in order to serve as falsy values in boolean operations).
Fornan-operations another 20% of the values are set to nan, leaving the remainder on the interval[0.2,0.8).The benchmarking results are given in ms for an i7-7560U running at 2.40GHz:functionufuncnumpynumbapandassum1.9501.7280.70811.832prod2.2792.3490.70911.649min2.4722.4890.71611.686max2.4572.4800.74511.598len1.4811.2700.63510.932all37.1863.0540.89212.587any35.2785.1570.89012.845anynan5.7832.1260.762144.740allnan7.9714.3670.774144.507mean----2.5000.82513.284std----4.5280.96512.193var----4.2690.96912.657first----1.8470.81111.584last----1.3090.58111.842argmax----3.5041.411293.640argmin----6.9961.347290.977nansum----5.3881.56915.239nanprod----5.7071.54615.004nanmin----5.8311.70014.292nanmax----5.8471.73114.927nanlen----3.1701.52914.529nanall----6.4991.64015.931nanany----8.0411.65615.839nanmean----5.6361.58315.185nanvar----7.5141.68215.643nanstd----7.2921.66615.104nanfirst----5.3182.09614.432nanlast----4.9431.47314.637nanargmin----7.9771.779298.911nanargmax----5.8691.802301.022cumsum----71.7131.1198.864cumprod--------1.12312.100cummax--------1.06212.133cummin--------0.97311.908arbitrary----147.85346.690129.779sort----167.699--------Linux(x86_64), Python 3.10.12, Numpy 1.25.2, Numba 0.58.0, Pandas 2.0.2DevelopmentThis project was started by @ml31415 and thenumbaandweaveimplementations are by him. The pure
python andnumpyimplementations were written by @d1manson.The authors hope thatnumpy'sufunc.atmethods or some other implementation ofaggregatewithinnumpyorscipywill eventually be fast enough, to make this package redundant. Numpy 1.25 actually
contained majorimprovements on ufunc speed,
which reduced the speed gap between numpy and the numba implementation a lot.