kairosdb/kairosdb — pull request #94 by davidvgalbraith (2014-09-10 07:36:20+00:00)
Title: Drop points on the floor beyond 100k PROD-2653

My changes are in MetricsResource.java. When it queries the Cassandra, it gets the result back in a List of DataPointGroup objects, which implement Iterator, one DataPointGroup for each combination of tags. My change just burns through these things until there are <= 100000 points left. I think it'll place the desired cap on points, although some knowledgeable party will have to educate me in the ways of connecting this hacked branch to our data engine for testing.

I don't think it would be totally inconceivable to put together an ElasticSearch-like streaming thing that gives you all the points 10k at a time or whatever, but it would be pretty ugly and slow, basically jumbling all the streams together into a huge list (affixing tag values to points), sorting them by time, and sending chunks of that until it's gone. That'd be a big task.

@demmer @andrew-swan