Package: woflo
wofloOverviewwoflois a Python local-first no-bloat extensible task orchestration framework.Okay, that's a lot of buzz. So what is actually the point?Main goal is to abstract a lot of functionality related to Task orchestration and execution away while keeping the "API" clear and dead-simple.Currently this includes:retriesretry timeoutparallelismloggingInstallationTo download from PyPI use:pipinstallwofloTo install from source:gitclonehttps://github.com/petereon/woflo.gitcdwoflo
poetrybuildcddist
pipinstall./woflo-<version>-py3-none-any.whlExamplesIntended usage is by utilizing a decorator@task, consider a very simple example which would run 10 sleepy workers in parallel without blocking the main thread:importtimefromwofloimporttask@taskdefsleepy_worker():time.sleep(5)print('I am done')for_inrange(10):sleepy_task_run=sleepy_worker()You can also include retries for tasks that might fail at times. Following would attempt to run the decorated function for 3 times in total with 5 second delay between attempts.fromwofloimporttask@task(retries=2,retry_sleep_time=5)deffetch_data_from_unstable_api():...Furthermore, you can also provide a runner within a@taskdecorator. For example theSequentialTaskRunif prefer your tasks to run sequentially and like to wait around a computer a lot. For example:fromwofloimporttaskfromwoflo.runnersimportSequentialTaskRun@task(runner=SequentialTaskRun)defsequential_sleepy_worker():time.sleep(5)print('I am done')for_inrange(10):sleepy_task_run=sequential_sleepy_worker()EachTaskRunshould also expose a few methods that enable you to handle it:.get_result()to fetch the return value of the finished task.wait()to block main thread till the task finishes (irrelevant forSequentialTaskRunwhich will block until it finishes anyway).stop()to stop the task while its running (irrelevant forSequentialTaskRunwhich will block until it finishes anyway).is_running()to check if the task is still running (irrelevant forSequentialTaskRunwhich will block until it finishes anyway)Let us define an example task:importtimefromwofloimporttask@taskdefquick_nap(duration):time.sleep(duration)ifduration<10raiseException("Ouch oof")else:return'Well rested'After you run it,napping=quick_nap(10)you can check on it to monitor it's state and receive results,assertnapping.is_running()napping.wait()assertnapping.get_result()=="Well rested"Task RunnersIt is designed to be easily extended by developing a customTaskrunners. Library itself currently exposes two such runners,MultiprocessTaskRunandSequentialTaskRun.Additionallywoflomakes available aBaseTaskRun, an interface against which custom runners can be developed.The defualt task runner isMultiprocessTaskRun, which can run multiple tasks, or even multiple instances of the same task at the same time in parallel in separate Python process.MultiprocessTaskRunThe defualt task runner, which can run multiple tasks, or even multiple instances of the same task at the same time in parallel in separate Python process.It offers two modes of operation:ForkProcess, which forks a main process and inherits all of its state. Forking is default on Darwin and Linux (it is not available on Windows)SpawnProcess, which spawns a new process with same global stateThis behavior can be configured by setting theprocess_type:fromwoflo.runners.multiprocessimportSpawnProcess,MultiprocessTaskRunMultiprocessTaskRun.process_type=SpawnProcess@task(runner=MultiprocessTaskRun)defsleepy_worker():time.sleep(5)print('I am done')RoadmapSetup GitHub Actions, SonarCloud monitoring and CodecovMake a PyPI PackageDecide on final API and create a version 1.x.xImplement a Dask runnerImplement a Thread runnerImplement an Async runnerKnown issuesProcesses potentially inherint a large in-memory state in MultiprocessTaskRunSpawnProcessrunning intoOSError: [Errno 9] Bad file descriptoronmacOS 12.6when usingmultiprocess.sharedctypes.Valueas reflected inthis issueImports need some refactoring