Package: dagio
dagio: Asynchronous I/O - with DAGs!dagiois an embarassingly simple Python package for running directed acyclic
graphs of asynchronous I/O operations.  It is built using and to be used with
Python's built-inasynciomodule, and provides aveeeerythin layer of functionality on top of it.
:sweat_smile:Git repository:https://github.com/brendanhasz/dagioBug reports:https://github.com/brendanhasz/dagio/issuesGetting StartedSuppose you have a set of potentially long-running I/O tasks (e.g. hit a web
service, query a database, read a large file from disk, etc), where some of the
tasks depend on other tasks having finished.  That is, you've got a directed
acyclic graph (DAG) of tasks, where non-interdependent tasks can be run
asynchronously.For example, if you've got a taskGwhich depends onEandF, butEdepends onD, andFdepends on bothCandD, etc:A
|
B   C
 \ /|
  D |
 / \|
E   F
 \ /
  GCoding that up using rawasynciomight look something like this:importasyncioclassMyDag:asyncdeftask_a(self):# does task a stuff...asyncdeftask_b(self):# does task b stuff...asyncdeftask_c(self):# does task c stuff...asyncdeftask_d(self):# does task d stuff...asyncdeftask_e(self):# does task e stuff...asyncdeftask_f(self):# does task f stuff...asyncdeftask_g(self):# does task g stuff...asyncdefrun():obj=MyDag()task_a=asyncio.create_task(obj.task_a())task_c=asyncio.create_task(obj.task_c())awaittask_aawaitobj.task_b()awaittask_cawaitobj.task_d()task_e=asyncio.create_task(obj.task_e())task_f=asyncio.create_task(obj.task_f())awaittask_eawaittask_fawaitobj.task_g()asyncio.run(run())Which is...fine, I guess :roll_eyes:  But, you have to be careful about what
task you start before what other task, and which tasks can safely be run
asynchronously vs those which can't. And then you have to type out all that
logic and ordering manually!  With the confusingasyncioAPI!  So: a lot of
thought has to go into it, especially for complex DAGs.And thinking is hard!  Less thinking! :fist:Withdagio, you just use thedependsdecorator to specify what methods any
other given method depends on, and it'll figure everything out for you, and run
them in the correct order, asynchronously where possible:importasynciofromdagioimportdependsclassMyDag:asyncdeftask_a(self):# does task a stuff...@depends("task_a")asyncdeftask_b(self):# does task b stuff...asyncdeftask_c(self):# does task c stuff...@depends("task_b","task_c")asyncdeftask_d(self):# does task d stuff...@depends("task_d")asyncdeftask_e(self):# does task e stuff...@depends("task_c","task_d")asyncdeftask_f(self):# does task f stuff...@depends("task_e","task_f")asyncdeftask_g(self):# does task g stuff...asyncdefrun():obj=MyDag()awaitobj.task_g()asyncio.run(run())Note that:Each task in your DAG has to be a method of the same classTask methods must beasyncmethodsCalling a task method decorated withdependsruns that taskand all its dependenciesTask methods should not take arguments nor return values.  You can handle
inter-task communication using object attributes (e.g.self._task_a_output = ...).
If you need a lock, you can set up anasyncio.Lockin your class's__init__.You can also run a non-async method asynchronously in a thread pool using therun_asyncdecorator:importasynciofromdagioimportdepends,run_asyncclassMyDag:@run_asyncdeftask_a(self):# a sync method which does task a stuff...@run_asyncdeftask_b(self):# a sync method which does task b stuff...@depends("task_a","task_b")asyncdeftask_c(self):# does task c stuff...asyncdefrun():obj=MyDag()awaitobj.task_c()#runs a and b concurrently, then cThat's it.  That's all this package does.Installationpip install dagioSupportPost bug reports, feature requests, and tutorial requests inGitHub
issues.ContributingPull requestsare totally
welcome! Any contribution would be appreciated, from things as minor as fixing
typos to things as major as adding new functionality. :smile:Why the name, dagio?It's for making DAGs of IO operations. DAG IO. Technically it'sasynchronousDAG-based I/O, and the nameadagiowould have been siiiick, but it wasalready taken! :sob: