clarencen/lshift — Issue #62: Segment scans
Author: clarencen
Created: 2011-11-14 22:41:33+00:00
State: open

To avoid the cost of a running a complete scan, it should be possible to segment a scan for configured bounds into smaller sub-scans. It should then be possible to initiate a scan that will only request participant queries which lie within the confines of the constrained view. This means that the configuration for a pair can have very wide bounds (or even be boundless in some dimensions). Pairings could have a number of views configured into them which can be triggered separately and either manually, as a scheduled job or both.

This means that real time events can be fed into into a pair with no regard of views, the partitioning consideration would be the incoming event fits into the configured bounds of the pair, which now could be quite wide or even unbounded.

It should also be possible to declare the pair to be unscannable - this would provide protection against somebody accidentally triggering the widest possible scan.

Potential extensions to this work could include:

* Auto-segmenting the scan of a wide partition into n sequential subscans, configured by some kind of portioning methodology (e.g. scan a pair in monthly/weekly/daily segments)
* Specifying views on the fly - these need not necessarily be persistent, they could just be one-shot views;
* Specifying a view date range as a period offset to the current system time, e.g. a view for the last 24 hours, for the last week, etc
* Being able to trigger a view in the UI;
* Triggering a one-shot view from the UI;