fluo-io/fluo — pull request #317 by keith-turner (2014-11-07 03:13:28+00:00)
Title: fixes #19 #282  Major improvement in how notifications to process are fo...

...und

This commit does a few things.

* Reorganizes code to separate finding work and executing work
* Now uses a single thread per worker to find work (could easily run multiple finder threads)
* Introduces new method of partitioning work by hash+mod of notifications

Using hashes should completely avoid the case of multiple workers processing
the same notifcation, in the case of a stable set of workers. As worker
processes come and go, there is the possibility of small amount of duplicate
processing, however this should be for a short time period. Each worker reacts
aggressively to the total number of workers changing. Using hash mod also
evenly distributes work.

A possible draw back of the hash technique is that each worker process has to
scan the entire table locality group to find all of its work. This may not
scale well for really large clusters, however I think this new approach will
scale much much better than the code it replaced. I feel like is a really good
and simple solution for up to medium sized clusters. Uncertain about large
clusters. This patch does use an Accumulo iterator to the the actual hash mod,
so that avoids all workers bringing all notifications to the client side. I
suspect that if Accumulo compacted locality groups separately, that the delete
markers in the notification localitly groupd could be cleaned up making
hash+mod scale well.

Now that its possible to implement multiple work finders. Could implement the
method mentioned in the percolator paper in the future. The papers mentions
chosing a random table, random row in the tablet, and then using a light weight
lck service to avoid collisions. The hash+mod approach allowed me to avoid
collisions w/o implementing a LWLS (light weight lock service) :) I do not
think ZK would be well suited for this LWLS (there is no need to persisted and
would need to support many more locks per second than I think ZK could offer)