rackerlabs/qonos — pull request #316 by CrashenX (2014-07-01 19:39:57+00:00)
Title: implement sliding next_timeout

Problem: While the worker is processing a job, it polls for image_status
and as part of it, it updates the job.status and job.next_timeout based
on the following params: [job_timeout_update_increment_min,
job_timeout_max_updates]. The timeout will be increased by
job_timeout_update_increment_min up to job_timeout_max_updates number of
times. This can push the timeout out quite a bit (i.e. typically 9
hours) since the job_timeout_update_increment_min is commonly set to 3
hours. Additionally, the timeout isn't udpated until it is expired. This
seems like a risk for allowing another worker to pick up a job that is
still being processed.

It is also worth noting that the timeout can be extended by quite a bit
by the timeout backoff logic that executes in certain error conditions.
This patch does not deal with the backoff logic. It does, however, fix a
broken window and DRY violation nearby.

Fix: This change introduces a new option job_timeout_window_increment
that is used to increase the timeout when the timeout is near
expiration. The worker can increase the timeout up to
job_timeout_max_updates number of times.

Purpose of Sliding Window: An initial timeout is set so that the job can
be picked up by another worker if it is not completed within the
timeout. The sliding window allows the worker to continue working and
keep the job from being picked up by another worker. The reason a single
static timeout is not used is so that if the worker fails in some way
the job will get back into the queue for processing in a more expedient
manner.

RM4877