Package: cs.taskqueue
A general purpose Task and TaskQueue for running tasks with
dependencies and failure/retry, potentially in parallel.Latest release 20230401:
Add missing requirement to DISTINFO.ClassBaseTask(cs.fsm.FSM, cs.gvutils.DOTNodeMixin, cs.resources.RunStateMixin)A base class subclassingcs.fsm.FSMwith aRunStateMixin.Note that this class and theFSMbase class does not provide
aFSM_DEFAULT_STATEattribute; a defaultstatevalue ofNonewill leave.fsm_stateunset.This behaviour is is chosen mostly to support subclasses
with unusual behaviour, particularly Django'sModelclass
whoserefresh_from_dbmethod seems to not refresh fields
which already exist, and setting.fsm_statefrom aFSM_DEFAULT_STATEclass attribute thus breaks this method.
Subclasses of this class andModelshouldnotprovide aFSM_DEFAULT_STATEattribute, instead relying on the field
definition to provide this default in the usual way.ClassBlockedError(TaskError, cs.fsm.FSMError, builtins.Exception, builtins.BaseException)Raised by a blockedTaskif attempted.Functionmain(argv)Dummy main programme to exercise something.Functionmake(*tasks, fail_fast=False, queue=None)Generator which completes all the suppliedtasksby dispatching them
once they are no longer blocked.
Yield each task fromtasksas it completes (or becomes cancelled).Parameters:tasks:Tasks as positional parametersfail_fast: defaultFalse; if true, cease evaluation as soon as a
task completes in a state with is notDONEqueue: optional callable to submit a task for execution later
via some queue such asLateror celeryThe following rules are applied by this function:if a task is being prepared, raise anFSMErrorif a task is already running or queued, wait for its completionif a task is pending:if any prerequisite has failed, fail this taskif any prerequisite is cancelled, cancel this taskif any prerequisite is pending, make it firstif any prerequisite is not done, fail this taskotherwise dispatch this task and then yield itiffail_fastand the task is not done, returnExamples:>>> t1 = Task('t1', lambda: print('doing t1'), track=True)
>>> t2 = t1.then('t2', lambda: print('doing t2'), track=True)
>>> list(make(t2))    # doctest: +ELLIPSIS
t1 PENDING->dispatch->RUNNING
doing t1
t1 RUNNING->done->DONE
t2 PENDING->dispatch->RUNNING
doing t2
t2 RUNNING->done->DONE
[Task('t2',<function <lambda> at ...>,state='DONE')]Functionmake_later(L, *tasks, fail_fast=False)Dispatch thetasksviaL:Laterfor asynchronous execution
if it is not already completed.
The caller can wait ont.resultfor completion.This callsmake_now()in a thread and usesL.deferto
queue the task and its prerequisites for execution.Functionmake_now(*tasks, fail_fast=False, queue=None)Run the generatormake(*tasks)to completion and return the
list of completed tasks.ClassTask(BaseTask, cs.fsm.FSM, cs.gvutils.DOTNodeMixin, cs.resources.RunStateMixin, cs.threads.HasThreadState, cs.context.ContextManagerMixin)A task which may require the completion of other tasks.The model here may not be quite as expected; it is aimed at
tasks which can be repaired and rerun.
As such, ifself.run(func,...)raises an exception fromfuncthen thisTaskwill still block dependentTasks.
Dually, aTaskwhich completes without an exception is
considered complete and does not block dependentTasks.Keyword parameters:cancel_on_exception: if true, cancel thisTaskif.runraises an exception; the default isFalse, allowing repair
and retrycancel_on_result: optional callable to test theTask.resultafter.run; if the callable returnsTruetheTaskis marked
as cancelled, allowing repair and retryfunc: the function to call to complete theTask;
it will be called asfunc(*func_args,**func_kwargs)func_args: optional positional arguments, default()func_kwargs: optional keyword arguments, default{}lock: optional lock, default anRLockstate: initial state, default fromself._state.initial_state,
which is initally 'PENDING'track: defaultFalse;
ifTruethen apply a callback for all states to print task transitions;
otherwise it should be a callback function suitable forFSM.fsm_callbackOther arguments are passed to theResultinitialiser.Example:t1 = Task(name="task1")
t1.bg(time.sleep, 10)
t2 = Task(name="task2")
# prevent t2 from running until t1 completes
t2.require(t1)
# try to run sleep(5) for t2 immediately after t1 completes
t1.notify(t2.call, sleep, 5)Users wanting more immediate semantics can supplycancel_on_exceptionand/orcancel_on_resultto control
these behaviours.Example:t1 = Task(name="task1")
t1.bg(time.sleep, 2)
t2 = Task(name="task2")
# prevent t2 from running until t1 completes
t2.require(t1)
# try to run sleep(5) for t2 immediately after t1 completes
t1.notify(t2.call, sleep, 5)ClassTaskError(cs.fsm.FSMError, builtins.Exception, builtins.BaseException)Raised byTaskrelated errors.ClassTaskQueueA task queue for managing and running a set of related tasks.UnlikemakeandTask.make, this is aimed at a "dispatch" worker
which dispatches individual tasks as required.Example 1, put 2 dependent tasks in a queue and run:>>> t1 = Task("t1", lambda: print("t1"))
 >>> t2 = t1.then("t2", lambda: print("t2"))
 >>> q = TaskQueue(t1, t2)
 >>> for _ in q.run(): pass
 ...
 t1
 t2Example 2, put 1 task in a queue and run.
The queue only runs the specified tasks:>>> t1 = Task("t1", lambda: print("t1"))
 >>> t2 = t1.then("t2", lambda: print("t2"))
 >>> q = TaskQueue(t1)
 >>> for _ in q.run(): pass
 ...
 t1Example 2, put 1 task in a queue withrun_dependent_tasks=Trueand run.
The queue pulls in the dependencies of completed tasks and also runs those:>>> t1 = Task("t1", lambda: print("t1"))
 >>> t2 = t1.then("t2", lambda: print("t2"))
 >>> q = TaskQueue(t1, run_dependent_tasks=True)
 >>> for _ in q.run(): pass
 ...
 t1
 t2MethodTaskQueue.__init__(self, *tasks, run_dependent_tasks=False):
Initialise the queue with the suppliedtasks.Release LogRelease 20230401:
Add missing requirement to DISTINFO.Release 20230331:Task: subclass BaseTask instead of (FSM, RunStateMixin).BaseTask.init: use @uses_runstate to ensure we've got a RunState.Release 20230217:
Task: subclass HasThreadState, drop .current_task() class method.Release 20221207:Pull out core stuff from Task into BaseTask, aids subclassing.BaseTask: explainatory docustring about unusual FSM_DEFAULT_STATE design choice.BaseTask.tasks_as_dot: express the edges using the node ids instead of their labels.BaseTask: new tasks_as_svg() method like tasks_as_dot() but returning SVG.Release 20220805:
Initial PyPI release.