Package: actionqueues
Action Queuesactionqueuesis a lightweight way to queue up commands for execution and
rollback on failures.The idea is that it provides a framework for safely
executing sequences of action with side-effects, like database writes, that
might need later rolling back if later actions fail. In addition, it provides
a standardised way for actions to be retried.For example, a user sign up process may write to several different systems.
If one system is down, then the other systems modified so far need cleaning
up before the failure is propagated back to the user. Usingactionqueueswith an action for each external system to be modified enables this pattern,
along with simple retry semantics for likely-transient failures such as network
blips.Installingpip install actionqueuesUsing Action QueuesIt's barebones, the main point is to provide a framework to work within for
actions that have side effects. It's basically the Command pattern, with a
tiny execution framework.AnActionis the lowest-level building block. It's any object withexecuteandrollbackmethods. TheActionis what handles executing each step of the
overall workflow, and rolling back any changes made to external state. It's
a single object so it can save state for rollback -- for example, primary
keys for any created database rows so they can be deleted during rollback.The main task of a user ofactionqueuesis to create theActionclasses
which implement the tasks their workflows require.OnceActionclasses are written, they can be executed. AnActionQueueholdsActionsfor execution and rollback. AddActionobjects to anActionQueuefor execution. CallActionQueue#executeto start running each action'sexecutemethod in the order theActionobjects were added to theActionQueue. Behaviour after this point is controlled by theexecuteandrollbackmethods on theActionobjects being executed by the queue.Normal operationThe default case is that no exception is raised during anexecuteand the
next action in the queue is hasexecutecalled. This is shown below for a
sequence ofActionobjects within anActionQueue.Exceptions duringexecutecause rollbackIf anAction#executeraises an exception, the ActionQueue notes where it's
up to in the Actions queued up and then propagates the exception
back up to the caller.It is then the caller's responsibility to catch the exception and then to callActionQueue#rollback. This is so the caller can know that the queue of
actions failed and is able to log the exception before callingrollback.CallingActionQueue#rollbackwill execute therollbackmethod on all
actions where theexecutemethod was called, including the one raising the
exception, in the reverse order to that which theexecutemethod was called.Rollback will not be called on actions whereexecutehas not been called.Again, the default case at this point is thatrollbackmethods succeed and
don't throw exceptions, leading to each being executed in succession.Exceptions duringrollbackIn contrast to a raised exception fromexecute, if an exception is raised
during therollbackmethod, theActionQueuewill
silently swallow the exception and continue executing therollbackmethods
of earlierActionobjects in the queue.This is because, in the rollback scenario, it's most likely that all rollback
actions should happen so the library assumes this. Thereforerollbackmethods
should do their own logging of exceptions before re-raising them.Retrying failed operationsThere is an exception to the above rules. If theexecuteorrollbackmethod
raises aactionqueue.ActionRetryExceptionthen theexecuteorrollbackmethod will be called again. TheActionRetryExceptioninit method takes an
optionalms_backoffargument to specify a time to sleep before trying the
method again, in milliseconds.TheActionQueuewill retry as long as the action keeps raisingActionRetryException, so the action must maintain a retry count
to avoid endless retries. Seebelowfor some
helper classes which cover common cases.Exampleimportrandomfromactionqueuesimportactionqueue,actionSUCCEED=0RETRY=1FAIL=2classMyAction(action.Action):def__init__(self,id):self._id=idself._value=0defexecute(self):"""Called if actions before it in the queue complete successfully.Raise any exception to indicate failure."""action=random.choice([SUCCEED,RETRY,FAIL])ifaction==RETRY:printself._id,"Throwing retry exception"raiseactionqueue.ActionRetryException(ms_backoff=0)elifaction==FAIL:printself._id,"Throwing failure exception"raiseException()else:printself._id,"Executing success action"self._value=1defrollback(self):"""Called in reverse order for all actions queued whose executemethod was called when the ActionQueue's rollback method is called."""printself._id,"Rolling back action"ifself._value==1:self._value=0q=actionqueue.ActionQueue()q.add(MyAction("a"))q.add(MyAction("b"))try:q.execute()except:q.rollback()Retry exception helpersIt can be tedious to keep track of the backoff and retry count for an action.
Thereforeactionqueuesprovides helpers for this called exception factories.
These are created when theActionis initialised, and when anexecutemethod hits a retriable exception, it calls the factory'sraise_exception()method. In general, this will throwActionRetryExceptionexceptions for a
given number of retries, then throw a generic exception, or one provided by
theActionobject.Using separate ExceptionFactory objects forexecuteandrollbackis usually
required.The available exception factories are:DoublingBackoffExceptionFactorywhich will throw a configurable numberActionRetryExceptionexceptions, each doubling its backoff time.In this example, theZeroDivisionErrorwill cause 5 retries, at 100, 200,
400, 800 and 1600ms delays, by using aDoublingBackoffExceptionFactory:fromactionqueuesimportactionqueue,actionfromactionqueues.exceptionfactoryimportDoublingBackoffExceptionFactoryclassMyFailingAction(action.Action):def__init__(self):self._run=1self._execute_ex_factory=DoublingBackoffExceptionFactory(retries=5,ms_backoff_initial=100)self._rollback_ex_factory=DoublingBackoffExceptionFactory(retries=10,ms_backoff_initial=100)defexecute(self):"""Execute an always failing action, but have it retried 5 times."""print"Executing action",self._runself._run+=1try:1/0exceptZeroDivisionError,ex:self._execute_ex_factory.raise_exception(original_exception=ex)defrollback(self):print"Rollback action",self._runself._run+=1try:1/0exceptZeroDivisionError,ex:self._rollback_ex_factory.raise_exception(original_exception=ex)q=actionqueue.ActionQueue()q.add(MyFailingAction())try:q.execute()except:print"boom"