cfpb/qu — pull request #95 by cndreisbach (2013-09-02 17:45:49+00:00)
Title: All aggregations now use map-reduce

This requires us to compute aggregations behind the scenes and cache
their results. When computed, we give the result like normal. When not
yet computed, we give the user an indication that it is currently
processing.

From an architectual angle, this changes the application significantly.
We now have another process running watching for new map-reduce jobs
that need to be done. These jobs are stored in Mongo, and atomically
claimed, so any app server can claim it. (`lein run` starts a worker to
claim and work these jobs.) The aggregation framework in Mongo is not
used at all, which gives us more flexibility.

Is this perfect? No way. Two queries with the exact same information
might have different cache keys because they are addressed differently
(`GROUP BY state, county` as opposed to `GROUP BY county, state` for
example.) We also do not give an indication of how backed up jobs are,
and we don't have a way to expire caches that are not being used. Those
can all come later, though: for now, this makes aggregations work.