rackerlabs/qonos — pull request #323 by broble (2014-07-22 18:20:12+00:00)
Title: introduce new job states, HARD_TIMED_OUT and MAX_RETRIED.

HARD_TIMED_OUT changes:

Currently, when a job hard times out, we do not update the status to
denote that the job has failed and hence there is no way to send a
job.failed notification for hard timed out jobs.
With this patch we introduce a new job state (HARD_TIMED_OUT) and based
on this new job state a job.failed notification is sent for the hard
timed out jobs.

Implementation Details:
- introduce a new job state - HARD_TIMED_OUT
- remove the filter for ignoring the hard_timed_out jobs from db
call, _job_get_next_by_action
- instead update query criteria to ignore jobs whose status is already
set to 'HARD_TIMED_OUT'
- implement calls for updating and sending job.failed notification in
JobProcessor
- include tests for the changes done.

MAX_RETRIED changes:

Currently, when a job's retry_count reaches/exceeds max_retry count, we
do not update the job status to denote that the job has failed. As a
result there is no way to send job.failed notification.

This patch addresses the issue by introducing a new job status,
'MAX_RETRIED' which will get updated whenever a job's retry_count
reaches/exceeds max_retry.
Some of the key changes done as part of the patch are as follows:
- moved the config param 'max_retry' from qonos-api.conf to
qonos-worker.conf.
- made the job processor on qonos-worker responsible for validating the
max_retry count and based on that update the job status
- made changes to the '_job_get_next_by_action' query to ignore jobs
whose status are set to 'MAX_RETRIED' and also removed the criteria
that checks if the job's retry_count is lesser than max_retry
- removed the job.failed notification logic from qonos.api.jobs.
- made the job.failed notification for 'MAX_RETRIED' to be sent by
qonos-worker.