JuliaStats/DataArrays.jl — pull request #101 by simonster (2014-07-02 21:12:54+00:00)
Title: Implement reductions with optional skipna argument

This implements reductions with an optional `skipna` argument, addressing #3 and mostly superseding #32 (except for `skewness` and `kurtosis`). The following reductions are implemented, all in terms of `mapreduce`:

- sum
- prod
- maximum
- minimum
- Base.sumabs
- Base.sumabs2
- var
- varm
- std
- stdm

With `skipna=false`, for some reductions that are guaranteed to return NA when any input is NA, we first check for NAs and then call `_mapreduce` from Base on the `data` Array if there are none. This has basically no overhead. For other reductions, we simply call the implementations in Base on the DataArray, which is slow due to type instability in indexing, but is the most obvious way to guarantee correctness.

With `skipna=true`, we first check if there are NA values. If not, we call `_mapreduce` from Base on the `data` Array. If there are, we use either an algorithm that branches on NA or an algorithm that does not branch on NA depending on the types and functors involved. For summation, we use a pairwise algorithm that divides blocks along BitArray chunk boundaries based on the number of non-NA elements. For blocks that contain no NAs, we call the implementation in Base. This has little overhead when there are few NA elements.

The tests currently fail on Travis because I based the implementation of `var` off of JuliaLang/julia#7502 and so it has slightly different semantics than the current Julia master, but the tests here are pretty comprehensive, and as soon as that is merged this should be good to go.