square/crossfilter — pull request #92 by esjewett (2013-09-12 20:02:54+00:00)
Title: When initially building groups, reduce add all values to allow determining unfiltered group state

We have a situation where dimensions and groups are dynamically added during an application lifecycle (on a single page application). This means that new dimensions and groups are often created when filters have already been applied to other dimensions.

When new groups are created we need to be able to determine what the group values would be without any filters in place. We use this information to display the unfiltered count for these groups (though we use custom map-based reducers so as not to double-count values, so it's a bit more complicated than the default count). There doesn't appear to be a way to do this without keeping a handle to the full data set around and then recreating the grouping logic or creating a new non-filtered Crossfilter, since Crossfilter doesn't allow access to the unfiltered data and currently the reduceAdd function is only called for selected records when a new group is created.

This patch changes the functionality of resetMany and resetOne so that upon group creation or other reset reduceAdd is called for all records and then reduceRemove is called for filtered records. This lets us use custom reducers to track the maximum value a group reaches, and thereby record the unfiltered value. (This only works, of course, for counting or for summing on groups with non-negative values.)

Performance impact is minimal for our use-case and is not measurable using 'make benchmark'. Theoretically, when building groups on unfiltered Crossfilters, performance should be identical. For building groups on filtered Crossfilters, performance should be no worse than 2x as bad, assuming reduceAdd and reduceRemove have identical complexity.

A test is included.