rackerlabs/otter — pull request #366 by manishtomar (2013-09-10 23:46:42+00:00)
Title: scheduler schema change - AUTO-600 (in progress)

This code is not tested at all. Its for review:

Events are stored based on minute they will execute. So, each row is keyed based on trigger and contains list of policy ids. On every interval (say 10 or 20 seconds), scheduler gets the batch (say 100) of events that need to execute and deletes them. Instead of locking the whole table, it locks the row. This way if processing of events of a particular row is taking time (> minute), another node will start processing events of another minute. Also, instead of locking the whole processing of events, the lock is only acquired while fetching and deleting. The grace time for compaction can also be increased (if needed) after this change since deleted rows will not be queried.

**Another idea:**
Events are stored based on minute like earlier. Make scheduler to strictly wake on beginning of each minute (i.e say 10:21:00, 10:22:00) and process all the events within that minute in one go. The lock is acquired on the row like above. Here, the deletion of events is one simple lightweight query to delete the whole row after fetching. The consistency level of read and delete can be ONE that increases availability since no other node will be working on that row once a node starts processing it. There is chance that the event was added few seconds ago which may not be available right away due to consistency. To handle this, scheduler caches the events processed in a set and fetches the events again after 20-30 seconds and checks the returned events against the cache. If any of the events are not in cache, it means it was added newly and processes them. The big disadvantage here is that if there are many events on a particular minute (+10,000) then one particular node is completely loaded which does not exist in previous case.