rackerlabs/qonos — pull request #318 by broble (2014-07-11 15:10:45+00:00)
Title: Qonos job max retried

Currently, when a job's retry_count reaches/exceeds max_retry count, we do not update the job status to denote that the job has failed. As a result there is no way to send job.failed notification.

This patch addresses the issue by introducing a new job status, 'MAX_RETRIED' which will get updated whenever a job's retry_count reaches/exceeds max_retry.

Some of the key changes done as part of the patch are as follows:

moved the config param 'max_retry' from qonos-api.conf to qonos-worker.conf.
made the job processor on qonos-worker responsible for validating the max_retry count and based on that update the job status
made changes to the '_job_get_next_by_action' query to ignore jobs whose status are set to 'MAX_RETRIED' and also removed the criteria that checks if the job's retry_count is lesser than max_retry
removed the job.failed notification logic from qonos.api.jobs.
made the job.failed notification for 'MAX_RETRIED' to be sent by qonos-worker.

This patch is dependent on the following PR's:
#311 (introduce new job state, HARD_TIMED_OUT)
#305 (implements qonos.job.failed notification)