pulp/pulp — pull request #963 by bmbouter (2014-05-14 19:31:31+00:00)
Title: Bmbouter 1096311

This is a large commit. Too large, but things kept getting added to it as I discovered issues along the way.

Currently, the tests do not pass. I will finish them tonight, and ensure all of them pass before I merge. This PR is created without all tests passing so that a review can begin. Please look only at the code, and not at the tests.

Note: this does alter the schema of available_queues collection, but it does not include a migration of any kind. This is because a new install or or upgrade of 2.4 won't have that collection, so we can still adjust the schema without any issue. The removal and addition of those records should automatically fix the schema as workers leave and return in a dev/QE environment. If there are any issues, stopping all services, and dropping all documents from pulp_database.available_queues collection will fix any schema related issues, and the new code will pick up correctly from there.

Things this code does:
* class separation of the FailureWatcher functionality and the event handler functionality through the introduction of a new class called EventMonitor
* adds some new event handlers for worker-heartbeat and worker-shutdown
* Changes the behavior of the event listener to ask workers to send a gratuitous heartbeat when it starts up. This will handle "existing" workers correctly.
* heartbeats update to the database when they come in
* worker shutdown events now gracefully remove workers
* complete removal of babysit dispatch, task, etc. Workers also do not run babysit on startup.
* Change the layout and object model of the available_queues collection, and its associated AvailableQueues model. missing_since has been renamed to last_heartbeat, which represents the last time a worker was seen to heartbeat.
* A duplicate threading bug is fixed whereby Scheduler.__init__ was being called twice.
* a new WorkerTimeoutMonitor thread is introduced that wakes up periodically and looks for missing workers. This is thread based, not task based.
* Additional logging both in new and existing code.
* removed redundant setting of the celery app in the Scheduler __init__
* removal of the manager 'get_or_create_available_queue' it is no longer used or needed.
* PEP8 fixes and copyright removal along the way