Package: luigi-tools
Luigi-toolsThis package extends and adds new features to theluigi package.
Here are a few examples of these features:add a newBoolParameterthat automatically switch to explicit parsing when the default value isTrue(otherwise it is not possible to set it toFalseusing the CLI).add several types of optional parameters.add aOutputLocalTargetclass to help building an output tree.add a mixin that adds a--rerunparameter that forces a given task to run again even if its targets exist, and also rerun all the tasks that depend on this one.add a mixin to remove the output of failed tasks which is likely to be corrupted or incomplete.
This feature applies the default behaviour of asnakemakerule (Task).add a new@copy_paramsmechanism to copy the parameters from a task to another (the@inheritsgives the same object to all the inheriting tasks while@copy_paramsonly copies the definition of the parameter so each inheriting task can be given a different value).add functions to get and display the dependency graph of a given task.add a mechanism to setup templates for theluigi.cfgfiles, so the user just has to update specific values instead of copying the entireluigi.cfg.Please read thecomplete API documentationfor more details.InstallationThis package should be installed using pip:pipinstallluigi-toolsUsageTheLuigipackage describes itself as follow:Luigi is a Python package that helps you build complex pipelines of batch
jobs. It handles dependency resolution, workflow management, visualization, handling failures,
command line integration, and much more.Theluigi-toolspackage is supposed to make luigi easier for developers.
The following presents a few examples of the main features of the package.Boolean parameterTheluigi.BoolParametercan be parsed in two ways: implicit or explicit. The explicit way requires the user to enter a
value:TrueofFalse. On the contrary, the implicit way requires no value and will just set
the value toTrueif the parameter is given. This is not compatible with a default value set toTrue, as it is not possible to set the value toFalseusing the CLI in this case.If you want to automatically set the parsing to explicit when the default value isTrue:fromluigi.taskimportTaskfromluigi_toolsimportBoolParameterclassMyTask(Task):a_boolean_parameter=BoolParameter(default=True)defrun(self):passTarget with prefixTheLuigiworkflows are based onTargetobject that represents the state of a step
of the workflow. These targets can be anything but are often files in a result directory tree. In
order to not having to specify the result directory to each target, one can use theOutputLocalTargetclass and give it aprefix. So all targets based on this class will be
located in the same directory.fromluigi.taskimportTaskfromluigi_tools.targetimportOutputLocalTargetclassMyTask(Task):defrun(self):passdefoutput(self):# The target will point to the file result_directory/filename.extreturnOutputLocalTarget("filename.ext")# Set the default prefix (it could also be called inside another Task)OutputLocalTarget.set_default_prefix("result_directory")# Run the task (the task can also be called with the CLI as usual)luigi.build([MyTask()],local_scheduler=True)Rerunable taskInLuigi, the states of the tasks are deducted from their targets. If the targets exist, the task
is assumed to have already been completed and is thus skipped if the workflow is run again. This
behavior is usually good to avoid performing computations that are already completed. Nevertheless,
sometimes it is desirable to overwrite a former result, especially during the development process.
For this reason, a mixin that adds a--rerunparameter to a task is introduced. When this
parameter is set toTrue, all the targets of this task are deleted as well as the targets of the
tasks that depend on this one. So when all the tasks that are related to this task will run again.
As for any mixin, it must be go on the left of theTaskclass in the inheritance list.fromluigi.taskimportTaskfromluigi_tools.taskimportRerunMixinclassMyTask(RerunMixin,Task):defrun(self):passNow the taskMyTaskhas a boolean parameter--rerunwhich can be called in the CLI:luigi-mmy_modulemytask--rerun
luigi-mmy_moduleanother_task_that_depends_on_mytask--MyTask-rerunClear the output of failed tasksWhen a task fails unexpectedly, it may leave an incomplete or corrupted output
that leads to wrong results in the downstream. With the RemoveCorruptedOutputMixin,
Luigi automatically removes the output targets of the tasks that failed. This is the default behaviour
in other workflow management systems such asSnakemake.fromluigi_tools.taskimportRemoveCorruptedOutputMixinclassTaskA(RemoveCorruptedOutputMixin,luigi.Task):"""TaskA can remove its output upon failure."""passTheclean_failedisfalseby default and it must explicitly be set totrue.
This allows users to set it to false to debug the output without changing the code.luigi-mmy_moduleTaskA--clean_failedtrueCopy parametersIn some situations, several tasks have a few parameters in common. This can lead to painful
situations, and luigi provides some dedicated tools to deal with this,as described here.
Nevertheless, the tools provided byLuigihave a major drawback: all the tasks with
the inherited parameter will have the same value for this parameter. In some situations, one want
to be able to give different values to a task with an inherited parameter, especially during the
development process. This is possible with the@copy_paramsdecorator:fromluigi.taskimportTaskfromluigi_tools.taskimportcopy_paramsclassTaskA(Task):a=luigi.Parameter(default="default_value_a")@luigi_tools.task.copy_params(a=luigi_tools.task.ParamRef(TaskA))classTaskB(Task):b=luigi.Parameter(default="b")Here the classTaskBhas two parameters:awithdefault_value_aas default value.bwithbas default value.It also possible to change the name of the parameter or to change the default value:fromluigi.taskimportTaskfromluigi_tools.taskimportcopy_paramsclassTaskA(Task):a=luigi.Parameter(default="default_value_a")@luigi_tools.task.copy_params(a=luigi_tools.task.ParamRef(TaskA),aa=luigi_tools.task.ParamRef(TaskA,"a"),a_default=luigi_tools.task.ParamRef(TaskA,"a","given_default_value"),a_none=luigi_tools.task.ParamRef(TaskA,"a",None),)classTaskB(Task):b=luigi.Parameter(default="b")In this case the classTaskBhas 5 parameters:awithdefault_value_aas default value.aawithaas default value.a_defaultwithgiven_default_valueas default value.a_nonewithNoneas default value.bwithbas default value.Note that the second parameter ofParamRefis the name of the inherited parameter in the parent
class. If it is not given, it is supposed that the parameter has the same name in both the
inheriting and the parent classes.Global parametersIn addition to the@copy_paramsdecorator, it is possible to use theGlobalParamMixinmixin.
A task with this mixin has a new feature for the parameters inherited using@copy_params: if the
default value is not changed inParamRefand if no specific value is given for the task, then the
task would take the same value as one of the inherited parameter. This combination of the@copy_paramsdecorator andGlobalParamMixinallows many ways of dealing with the parameters.fromluigi.taskimportTaskfromluigi_tools.taskimportcopy_paramsfromluigi_tools.taskimportGlobalParamMixinclassTaskA(Task):a=luigi.Parameter(default="default_value_a")@luigi_tools.task.copy_params(a=luigi_tools.task.ParamRef(TaskA))classTaskB(GlobalParamMixin,Task):b=luigi.Parameter(default="b")Now ifTaskBis called with the following configuration:[TaskA]a = "value for a"[TaskB]b = "value for b"then the parameteraofTaskBhas the valuevalue for a.
IfTaskBdid not inherit fromGlobalParamMixin, then it would have the valuedefault_value_a.Dependency graphTheluigi-toolspackage provides several functions to get the dependency graph of a task and to
render it using GraphViz. This can be very useful to show how the tasks of a workflow are
organized.Funding & AcknowledgmentThe development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.For license and authors, seeLICENSE.txtandAUTHORS.mdrespectively.Copyright © 2021-2022 Blue Brain Project/EPFL