urbanairship/datacube — pull request #36 by dgoldin (2012-08-08 14:18:15+00:00)
Title: Slice functionality

This pull request was made to discuss an approach to slicing.
**This is work-in-progress. It's unfinished and exemplifies one approach.**

We want to add [slicing](http://en.wikipedia.org/wiki/OLAP_cube#Operations) to datacube in order to retrieve the top datapoints for a slice.

Concrete example: Having the three dimensions time, zipcode and gender, we want to retrieve the top 10 zipcodes for a fixed time and gender. Currently, this would involve querying for all zipcodes (or another dimension for which not all values are known - due to the way addresses/keys are constructed atm). Not only for HBaseDbHarness, this would be a serious performance issue. Therefore, we want to extend datacube.

In our approach, a DataCube takes a set of slices (in addition to dimensions and rollups). Slices are used in a similar way to Rollups. Slices are defined as one (in our case) or multiple _variable_ (called wildcard for now - zipcode in the above example) and multiple _fixed_ dimensions. Maps are saved as values of slices, either serialized (MapDbHarness) or in HBase columns. When writing, the DbHarness has to check if a given Op is of type ColumnOp (which wraps a regular Op and a "key") and then increment the counter in the map. DataCubeIo has a getSlice method that returns a map. Sorting and obtaining the top entries has to be done outside of datacube.

So far we only implemented this functionality on MapDbHarness (OVERWRITE and READ_COMBINE_CAS) to test our approach. An Hbase implementation will be done soon.

Known issues:
- Column values do not support substitution through IdService (read and written through serialization to byte[])
- ColumnOp feels a bit clumsy - maybe every Op should just have an optional key/column property (so far it uses an empty column in hbase)
- Modified interfaces (serialization) require modifications of already implemented Ops, CSerializables and Bucketers
- WriteBuilder and ReadBuilder protect from directly accessing the slice records, but using Address its possible.

Comments and suggestions are very much appreciated, so we can get the implementation more in line with the envisioned design.