Package: rework
REWORKreworkis a distributed execution system for the execution of tasks
that can belong to independant python environments and code bases,
even hosted on different computers.The only constraint is that postgres must be accessible from all nodes
of a givenreworkinstallation.PrinciplesRework might interest people who:want Postgres (andonly Postgres) as a Task Queue Manager,
Input/Output store and task log storehave Python long-running tasks to run, with the ability to
preemptively kill taskswant a tiny, self-contained tool with great functional test
abilities.The most common python library for such things is Celery. If you don't
mind depending on RabbitMQ and the Celery API and feature set, Celery
is for you. It is mature, probably well-tested, and largely used.Basic usageSetting up a databaseYou need a postgresql database. Rework will install its tables into
its ownreworknamespace schema, so you can use either a dedicated
database or an exising one, with little risk of conflict.If you don't already have a database, create a fresh one with:createdbjobstoreTo install rework inside:reworkinit-dbpostgres://babar:password@localhost:5432/jobstoreDeclaring and scheduling a taskAll the features are covered in thetest suite, which can
henceforth be regarded as a reasonnable source of
documentation. However here's a simple example:fromtimeimportsleepfromreworkimportapifromsqlalchemyimportcreate_engine@api.taskdefmy_first_task(task):withtask.capturelogs(std=True):print('I am running')somevalue=task.input*2task.save_output(somevalue)print('I am done')defmain(dburi):engine=create_engine(dburi)# record the decorated tasksapi.freeze_operations(engine)# now, schedule taskst1=api.schedule(engine,'my_first_task','hello')t2=api.schedule(engine,'my_first_task',100)# wait til they are completedt1.join()t2.join()assertt1.output=='hellohello'assertt2.output==200if__name__=='__main__':main('postgres://babar:password@localhost:5432/jobstore')If you put this into atest_rework.pymodule and typepython test_rework.pyit shouldhangforever. Hold on, what's missing ?On another terminal, one needs tostartthe workers that will
execute the tasks. Do as follows:reworkmonitorpostgres://babar:password@localhost:5432/jobstoreThen, the script will quickly terminate, as both tasks have been
executed.Specifying inputsHaving a formal declaration of the task input can help validate them
and also, inrework_uiit will provide an interactive web
form allowing subsequent launch of the task.fromreworkimportapi,io@api.task(inputs=(io.file('myfile.txt',required=True),io.string('name',required=True),io.string('option',choices=('foo','bar')),io.number('weight'),io.datetime('birthdate'),io.moment('horizon')))defcompute_things(task):inp=task.inputassert'name'ininp...... and then, later:task=api.schedule(engine,'compute_things',{'myfile.txt':b'file contents','birthdate':datetime(1973,5,20,9),'name':'Babar','weight':65,'horizon':'(shifted (today) #:days 7)'})asserttask.input=={'myfile.txt':b'file contents','birthdate':datetime(1973,5,20,9),'name':'Babar','weight':65,'horizon':datetime(2021,1,7)}Specifying outputsAs for the inputs, and for the same reasons, we can provide a spec for
the outputs.fromreworkimportapi,io@api.task(outputs=(io.string('name'),io.datetime('birthdate')))defcompute_things(task):...task.save_output({'name':'Babar','birthdate':datetime(1931,1,1)})And this will of course be fetched from the other side:t=api.schedule(engine,'compute_things')assertt.output=={'name':'Babar','birthdate':datetime(1931,1,1)}SchedulingWhile the base api provides aschedulecall that schedules a task
for immediate execution, there is also apreparecall that allows to
define the exact moment the task ought to be executed, using acrontablike notation.Example:api.prepare(engine,'compute_things',{'myfile.txt':b'file contents','birthdate':datetime(1973,5,20,9),'name':'Babar','weight':65},rule='0 0 8,12 * * *')This would schedule the task every day at 8:00 and 12:00. The extended
crontab notation also features a field for seconds (in first
position).DebuggingIf you need to debug some task, the standard advice is:write your task content in plain functions and have them unit-tested
with e.g.pytest@api.taskdefmy_fancy_task(task):the_body_of_my_fancy_task(task.input)you can also you use print-based logging as shown there:@api.taskdefmy_fancy_task(task):withtask.capturelogs(std=True):print('starting')# do stuffprint('done',result)finally, it may happen that a task is "stuck" because of a deadlock,
and in this case, starting the monitor with--debug-portwill help:$pipinstallpystuck
$reworkmonitorpostgres://babar:password@localhost:5432/jobstore--debug-port=666Then launchingpystuck(possibly from another machine) is done as such:$pystuck-h<host>-p666Organize tasks in codeA common pattern is to have aproject/tasks.pymodule.One can manage the tasks using theregister-operationsandunregister-operationcommands.$reworkregister-operations<dburi>/path/to/project/tasks.pyand also```reworkunregister-operation<dburi><opname>
delete<opname><domain>/path/to/project/tasks.py<hostid>
reallyremovethose[y/n]?[y/N]:yThis pair of operations can be used also whenever a task input or
output specifications have changed.APITheapimodule exposes most if what is needed. Thetaskmodule
and task objects provide the rest.apimoduleThree functions are provided: thetaskdecorator, thefreeze_operations,scheduleandpreparefunctions.Defining tasks is done using thetaskdecorator:fromrework.apiimporttask@taskdefmy_task(task):passIt is also possible to specify a non-defaultdomain:@task(domain='scrapers')defmy_scraper(task):passAtimeoutparameter is also available:fromdatetimeimporttimedelta@task(timeout=timedelta(seconds=30)defmy_time_limited_task(task):passTo make the tasks available for use, they must be recorded within
the database referential. We usefreeze_operationsfor this:fromsqlalchemyimportcreate_enginefromrework.apiimportfreeze_operationsengine=create_engine('postgres://babar:password@localhost:5432/jobstore')api.freeze_operations(engine)Finally, one can schedule tasks as such:fromsqlalchemyimportcreate_enginefromrework.apiimportscheduleengine=create_engine('postgres://babar:password@localhost:5432/jobstore')# immediate executionntask=api.schedule(engine,'my_task',42)# execution every five minutesapi.prepare(engine,'my_task',42,rule='0 */5 * * * *')Theschedulefunction wants these mandatory parameters:engine: sqlalchemy engineoperation: stringinputdata: any python picklable object (if no input specification
is provided, else the input formalism provides ways for numbers,
strings, dates and files)It also accepts two more options:domain: a domain identifier (for cases when the same service is
available under several domains and you want to force one)hostid: an host identifier (e.g. '192.168.1.1')metadata: a json-serializable dictionary (e.g. {'user': 'Babar'})Thepreparefunction takes the same parameters asscheduleplus aruleoption usingcrontabnotation with seconds in first position.Task objectsTask objects can be obtained from thescheduleapi call (as seen in the
previous example) or through thetaskmodule.fromtaskimportTasktask=task.byid(engine,42)The task object provides:.stateattribute to describe the task state (amongst:queued,running,aborting,aborted,failed,done).join()method to wait synchronously for the task completion.capturelogs(sync=True, level=logging.NOTSET, std=False)method to
record matching logs into the db (synccontrols whether the logs
are written synchronously,levelspecifies the capture level,stdpermits to also record prints as logs).inputattribute to get the task input (yields any object).save_output(<obj>)method to store any object.abort()method to preemptively stop the task.log(fromid=None)method to retrieve the task logs (all or from a
given log id)Command lineOperationsIf you read the previous chapter, you already know theinit-dbandmonitorcommands.Thereworkcommand, if typed without subcommand, shows its usage:$rework
Usage:rework[OPTIONS]COMMAND[ARGS]...

Options:--helpShowthismessageandexit.

Commands:abort-taskimmediatelyabortthegiventaskThiswillbedone...export-scheduledimport-scheduledinit-dbinitializethedatabaseschemaforreworkinitsown...kill-workerasktopreemptivelykillagivenworkertoits...list-monitorslist-operationslist-scheduledlist-taskslist-workerslog-taskmonitorstartamonitorcontrollingmin/maxworkersnew-workerspawnanewworker--thisisapurely*internal*...register-operationsregisteroperationsfromapythonmodulecontaining...shutdown-workeraskaworkertoshutdownassoonasitbecomesidle...unregister-operationunregisteranoperation(orseveral)usingitsname...vacuumdeletenon-runingworkersorfinishedtasksOf those commands,new-workeris for purely internal purposes, and
unless you know what you're doing, you should never use it.One can list the tasks:reworklist-taskspostgres://babar:password@localhost:5432/jobstore1my_first_taskdone[2017-09-1317:08:48.306970+02]2my_first_taskdone[2017-09-1317:08:48.416770+02]It is possible to monitor the output of a given task:$reworklog-taskpostgres://babar:password@localhost:5432/jobstore1stdout:INFO:2017-09-1317:08:49:Iamrunning
stdout:INFO:2017-09-1317:08:49:IamdoneThe last argument1is the task identifier as was shown by thelist-taskscommand.Notice how we capture the standard output (print calls) using thetask.capturelogscontext manager. This is completely optional of
course but quite handy. The line shown above actually capturestandard output,standard errorandall logs. It accepts alevelparameter, like e.g.capturelogs(level=logging.INFO).Lastly,list-workerswill show the currently running workers:$reworklist-workerspostgres://babar:password@localhost:5432/jobstore14889896@192.168.1.230Mb[running]24889748@192.168.1.230Mb[running]ExtensionsIt is possible to augment thereworkcommand with new subcommands (or
augment, modify existing commands).Any program doing so must define a new command and declare a setup
tools entry point namedrework:subcommandas in e.g.:entry_points={'rework.subcommands':['view=rework_ui.cli:view']}For instance, therework_uipython package provides such aviewsubcommand to launch a monitoring webapp for a given rework job
store.