A Tutorial on Luigi, the Spotify’s Pipeline | Towards Data Science

Luigi is a Python (2.7, 3.6, 3.7 tested) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
Intro
Pros and Cons
Pipeline Structure:. Tasks. Targets. Parameters
Building Blocks
Execute it
My experience with Luigi started a few years ago, when the sole person in charge of the company pipeline left, and I got “gifted” with a massive legacy codebase.
At the time, I had no knowledge about pipelines in general, or Luigi in particular, so I started looking for tutorials and reading the official documentation in full.
I didn’t find it particularly useful; it is, of course, factually correct, but it is not a tutorial. Nor, at the time, were many courses or blogs about it.
This article, therefore, is my attempt to fill this gap. I’ve collected what learnt, and it will hopefully help you during your first steps with the Luigi package
Before we dive into the code, let’s see the strengths and weaknesses of this package.
The Luigi pipeline library has been designed and still used by Spotify. Over the years it has been adopted by other companies, notably like Deliveroo. But, exactly, because it was tailor-made for Spotify, it might not be suited to solve your need and I advise you to consider carefully if this system can work for you.
It is Python! Thus, first, you already know how to code in it, plus you can blend the process that you want to automatize (your original code) with the pipeline infrastructure (thus, Luigi)
Its “backward” structure allows it to recover from failed tasks without re-running the whole pipeline.
An AWS batch wrapper to work directly with Amazon ECS.
A GUI showing status of the tasks and graphical representation of your pipeline.
It is hard to test.
It has a central scheduler that makes it hard to parallelize tasks.
It needs to be kept simple, or better linear (try to keep it <1k tasks). Too many branches and forks can slow the run time incredibly.
There is no trigger. Thus, the pipeline won’t start when the needed files are in place, but you need to set up a cronjob to check them and make the pipeline start
If you have decided to start using Luigi, run
pip install luigi
on your terminal, and let’s start to see how it works
The structure of a pipeline in Luigi, like one of many pipeline systems, resembles that of a graph. It contains nodes, where information is processed and edges connecting the nodes, transferring the information to the next node.
This process would, normally, follow a linear flow, in which the first node is considered the start node, this is the first to be executed, followed by a second, then a third until the end-node completes the workflow. Something resembling this:
Start -> Node A -> Node B -> Node C -> End
Instead, in Luigi, this process is reversed!
The program starts from the last task, it checks if it can be executed or if it requires anything.
If that is the case, it moves up to the pipeline until it finds a node that has all requirements satisfied, and only then, it starts running the nodes already visited. Like in the following:
Start -> Node C -> Node B -> Node A -> Node B -> Node C -> End
This methodology could seem counterintuitive at first, but it actually helps to avoid rerunning nodes already executed after a failure or error appears.
Let’s image to run our pipeline a first time, and in this first run Node B fails:
Start -> Node C-> Node B -> Node A (Done) -> Node B (Error) -X-> Node C -> End
After we have manually fixed the error in Node B the pipeline can run again but, in this occasion, it will not consider Node A
Start -> Node C-> Node B -> Node C-> Node.
Avoiding running it entirely.
As mentioned, the Luigi package works like a graph. The nodes in Luigi are called Tasks and the edges are called Targets
Tasks are the basic building blocks of Luigi and where the real work actually happens.
To create a task you have to create a class and inherit the class luigi.Task.
Within the new class, there are contained at least one or all, of the following methods:
requires()
run()
output()
targets
The method requires() is the first method that is executed, if present.
It contains all the instances that have been executed before the current task. Most commonly this method makes a call to another Luigi Task above the workflow, allowing the code to move backward to the beginning of the pipeline.
If we saw earlier that Targets are what connect a Task to the next requires() is what connects the Task to the previous one.
Targets move the code to the end, required to the beginning.
Only when a method requires is satisfied that Task can execute the second method run:
In this method is contained the action that the task has to execute. It can be anything, a call to another method, the running of a script, etc.
An example of a Task in Luigi with the two previous model would be:
class MakeTaskA(luigi.Task):    def requires(self):        return MakeTaskB()    def run(self):        return luigi.LocalTarget(DoSomething())
The method output() as the name suggests return one or more Target objects. Although, it is recommended that any Task only return one single Target in output.
Example
A good example is present in the Luigi documentation:
In this example,
The first Task is CountLetters(), it requires the class GenerateWords() so it moves there
GenerateWords() has no requires so it calls it’s run() method and it creates a file calls “words.txt”
It’s the turn of output(). This method finds the file word.txt and it returns it in the form of a LocalTarget class object, back to the class CountLetters
CountLetters() can execute run(), now. It takes what needed from self.input()
Afterword, it execute output()
Task.input() is a wrapper around Task.requires() that returns the corresponding Target objects instead of Task objects.
Target is the edge connecting a Tasks to the next.
These are usually represented as a file. Indeed, we have already seen one in the previous example. The file ‘hello.txt’ was a Target.
A Task can be considered complete if and only if each of its output Targets exist. They can be as simple as empty files, an S3 or whatever.
The last important element that we saw in the example, but we did not consider yet, are the Parameters.
Parameters are in Luigi, the equivalent of creating a constructor for each Task. Those are instance variables of a class. Only one instance is created for each set of parameters.
Personally, it reminds me of when in Java or C++, you have to declare variables, I know they are not the same thing, but it can help to understand the use of it.
You can declare a parameter in the following way:
class MakeSomething(luigi.Task):    string = luigi.Parameter(default='hello world!')
Now, we have a class with a parameter attribute ‘hello world!’ string.
You can pass parameters back and forth between tasks.
Besides the one seen above, there are different subclasses of Parameters:
There are a few of them: DateParameter, DateIntervalParameter, IntParameter, FloatParameter, etc.
These provide methods for serialization and reading, you can make your own ones. However, you are not obligated to specify the type of parameter.
You can simply use the base class Parameter. The reason you would use a subclass like DateParameter is that Luigi needs to perform a conversion for the command line interaction.
That is how it knows how to convert a string provided on the command line to the corresponding type (i.e. datetime.date instead of a string).
To execute the code, type the following from your command line:
python -m luigi — module file_name TaskName — local-schedule
I hope that this article helps you to move your first steps with Luigi