clarencen/lshift — Issue #304: Generalize entity partition criteria and granularity function
Author: clarencen
Created: 2011-11-14 22:42:27+00:00
State: open

## Motivation

ATM the business date is hard baked into the protocol as a partitioning scheme. Whilst this works for a number of applications (e.g. trading), it is not general enough for applications that don't really have this notion (e.g. consider synchronizing ref data, which realistically doesn't have a business date). To do this, we are going to have to change the protocol in these ways:

* Replace the business date attribute with a generalized partition attribute
* Generalize the Range Granularity concept to be able to communicate any aggregation function on the network - these function names can be categorized into:
* Well known function names (e.g. DAILY, MONTHLY, YEARLY for timestamp based partitions, nth character of a GUID based partitions )
* User defined function names that are configured into the agent for a particular participant

Given the fact the detection date is going to get persisted (see later comments on ticket #1), the Hibernate-mapped Correlation entity will get a last update timestamp, and as part of this issue, the date field of the Correlation entity should be migrated to be a category string.

## Use Cases

* Specifying which attribute of an entity to use as a partition criterion should be configurable
* Support range queries on:
* Dates
* Sequence numbers
* Support list queries of a specified set of attributes
* Allow multiple constraints to be queried simultaneously

## Proposed Design

The solution is composed of of the following concepts:

* Category - this is name of the attribute an entity that is configured as a partitioning variable
* Constraint - a query for digests contains 0 or more constraints. A constraint has the following structure:
* Category Name
* A well known function name to apply in the participant
* EITHER a list of values OR a range of values TO apply
* Digest - this is similar to the current structure, but is now changed to carry categories. A digest has the following structure:
* Id (as before)
* Version (as before)
* [{k,v}] - a list of key-value pairs that represent category-name-value pairs

The intended approach is to maintain a Lucene based index so that categories can be added and indexed dynamically without having to manipulate the DDL.