alphagov/email-alert-service — pull request #32 by rboulton (2014-11-25 12:51:38+00:00)
Title: Rework the locking so we don't drop messages

With the previous algorithm, only one attempt would have been made to
send an email. If this failed (eg, because email-alert-api was broken
temporarily), the email would never be sent.

Instead, we do a two-phase algorithm. First, we use a redis lock to
ensure that only one worker tries to process a given message at a time.
This lock is removed when the worker has finished its work, and is also
given a short timeout (2 minutes) so that if the worker crashes without
the chance to clean up its lock will be removed automatically. An
attempt to acquire the lock when it's already held results in an
exception, causing the message to be nackked - so it should get retried
again automatically soon afterwards.

Once the lock is held, we check for a marker which indicates if the
message has already been processed successfully. If this is set, or if
the message is more than 3 months old, we don't do anything further.
Otherwise, we process the message, send it to email-alert-api, and only
after that has returned a successful result we add a 'done marker' to
redis. As in the previous algorithm, we set a 3 month timeout on the
done marker to ensure that the memory used by redis doesn't grow
unboundedly over time.

Note: we always set a 3 month timeout on the done marker, even if the
message is nearly 3 months old, because this keeps the code and tests
simpler. It'll still be cleaned up eventually, and normally messages
will be processed only when they're new.

Further note: in the tests, I've changed various helpers to return times
as Time objects, rather than converting them to strings or seconds.
This makes performing calculations on them, and passing them around,
simpler.

Pivotal story: https://www.pivotaltracker.com/story/show/83204578