WebFilings/furious — pull request #17 by johnlockwood-wf (2013-01-13 06:10:35+00:00)
Title: Ndb persist

A Context will split it's Asyncs up into a tree with each branch having no more then ten leaf nodes. It does a persistance of the tree that writes out each node then It persists the whole structure as json in an ndb model MarkerTree(de-coupling from ndb needed, but it's only used for visualization for now, but could be used for error recovery).

Each Async gets a _persistence_id during the creation of the tree and the context gets the _persistence_id of the root of the tree.

When each Async is processed, if it has a _persistence_id, a MarkerPersist(de-coupling from ndb needed) is retrieved by a matching id and if it exists, it's marked as done and then an update_done process happens where it bubbles up the a 'done' checking process.

The 'done' checking process works like we white-boarded, where a grouped node get's it's child markers and if they are all done, then it marks itself as done and then bubble's up the process.

The aggregation of results is hard-coded to just extend a results list since there isn't yet a mechanism to specify the context level callback/aggregation method.

When the root of the tree is done, the result is put in a Result entity with an id the same as the context's _persistence_id and then the persisted nodes of the tree are deleted, leaving a Result and a MarkerTree.

Each grouping node of the tree is given a key of a different uuid. The leaf keys are the group id with their index as a suffix(well, comma joined). The reason the whole she-bang doesn't use the same uuid with incrementing suffixes is because http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/. It shards the data into chunks of ten or less, so ten keys at most could be written to the same datastore tablet and should make for some fast work.

Known issues:
Coupled too tightly with ndb in places.
Hard-coded result aggregation
Happy path
If you add more than 100 Asyncs to a Context, it fails because more than 100 tasks cannot be inserted at a time, also the speed of persisting the tree could be an issue.
Bubble up process happens in the stack and not with a task
Size of tree could exceed 1MB, preventing it's whole persistence in MarkerTree
I realize the update done process should probably be initiated in _process_results instead of _execute_callback
Robert talked about having sub contexts be connected to the grouping nodes of the tree below the root, but I thought I'd leave that for Robert and discussion.