The Flawless Pipes of Python/ Pandas | by Dr. Gregor Scheithauer | Towards Data Science

By trade I am an R person. Especially the Tidyverse is such a powerful, clean, easy-to-understand and well documented data science platform. I highly recommend to every beginner the free online book R for Data Science.
However, my team’s programming language of choice is Python/ Pandas — which is also a wonderful data science platform. One of the major differences (to me, at least) is how we write Python code, which is very different to R code — that has nothing to do with the syntax in itself.
One of R’s elegances is using the pipe functionality programming metaphor. This allows you write more efficient and readable code. One good example is the post by Soner Yıldırım The Flawless Pipes of Tidyverse.
I would like to recreate his post and to show you how to use pipes (or method chaining) for Pandas.
According to the r magrittr package [1] pipes allow you to write code with the following benefits:
structuring sequences of data operations left-to-right (as opposed to from the inside and out),
avoiding nested function calls,
minimizing the need for local variables and function definitions, and
making it easy to add steps anywhere in the sequence of operations.
R example without using pipes (cf. [2])
What you see in the code below is a typical example. We might save a result of a function call in a variable, e.g. foo_foo_1 with the only reason to pass it into the very next function call, e.g., scoop().
This results in a number of variables with maybe not so meaningful names and as a consequence might increase the complexity of your code.
foo_foo_1 <- hop(foo_foo, through = forest)foo_foo_2 <- scoop(foo_foo_1, up = field_mice)foo_foo_3 <- bop(foo_foo_2, on = head)
R example with pipes (cf. [2])
Pipes in R are used with the syntax %>%. It allows you to chain your functions. In the example below try to read it in the following manner:
I want to evaluate/ manipulate variable foo_fooI want foo_foo to hop trough the forest, and then,I want foo_foo to scoop up a field mice, and finally,I want foo_foo to bop on its head
I want to evaluate/ manipulate variable foo_foo
I want foo_foo to hop trough the forest, and then,
I want foo_foo to scoop up a field mice, and finally,
I want foo_foo to bop on its head
foo_foo %>%                    hop(through = forest) %>%  scoop(up = field_mice) %>%  bop(on = head)
Please note, how the data set is a noun and the functions are verbs. You might get the idea how it increases the code’s readability.
For a better understanding how the piped code works the interpreted version is shown below:
bop(  scoop(    hop(foo_foo, through = forest),    up = field_mice  ),   on = head)
Since there is no magrittr package in Python it has to work in a different way. We will make use of the fact that in Pandas most data frame functions return the data set itself. This is called method chaining. Let’s stick with the foo_foo example.
A possible Python code might look like this:
foo_foo_1 = hop(foo_foo, through = forest)foo_foo_2 = scoop(foo_foo_1, up = field_mice)foo_foo_3 = bop(foo_foo_2, on = head)
Chaining it would look like this:
foo_foo.hop(through = forest).scoop(up = field_mice).bop(on = head)
This is close, but not exactly. Add one little adjustment, i.e., () brackets around the code to make it the way we want it. Please see the code below that shows how this looks like.
(    foo_foo    .hop(through = forest)    .scoop(up = field_mice)    .bop(on = head))
I love using this way of coding. Here are some of the benefits I see in using this programming metaphor:
Your code is much more readable for other data scientists in your team (and your future self)You avoid the need for more or less meaningless local variablesYou are able to add or remove functions in your data evaluation quicklyYour code follows you line of thoughts (noun-verb-paradigma) during data evaluation and analysis
Your code is much more readable for other data scientists in your team (and your future self)
You avoid the need for more or less meaningless local variables
You are able to add or remove functions in your data evaluation quickly
Your code follows you line of thoughts (noun-verb-paradigma) during data evaluation and analysis
[1] magrittr package[2] Pipes chapter in R for Data Science book
I will follow the article of Soner Yıldırım in order for you to compare how to use pipes/ method chaining in R as well as in Python. Soner uses the Kaggle data set on Melbourne housing. You may download it and follow along with me.
# import libsimport pandas as pd# read datamelb = pd.read_csv("../01-data/melb_data.csv")# Have a quick look at the data(    melb    .head())
The next example filters houses by Distances below 2, groups them by Type, and calculates the averages Prices for each Type. Then do some formatting.
(    melb    .query("Distance < 2") # query equals filter in Pandas    .filter(["Type", "Price"]) # select the columns Type and Price    .groupby("Type")    .agg("mean")    .reset_index()     .set_axis(["Type", "averagePrice"], axis = 1, inplace = False))
In the next example we will filter with more than one criteria and calculate additional features. Please note, it is possible to use the function agg (for aggregation). Personally, I mostly use assign in combination with lambda. Please see it here in action.
(    melb    .query("Distance < 2 & Rooms > 2")     .filter(["Type", "Price"])     .groupby("Type")    .agg(["mean", "count"])    .reset_index()     .set_axis(["Type", "averagePrice", "numberOfHouses"], axis = 1, inplace = False)    .assign(averagePriceRounded = lambda x: x["averagePrice"].round(1)))
The next example shows how to sort the houses data frame by average distance by region (only regions beginning with the string South).
(    melb    .query('Regionname.str.startswith("South")', engine = 'python')     .filter(["Type", "Regionname", "Distance"])    .groupby(["Regionname", "Type"])    .agg(["mean"])    .reset_index()    .set_axis(["Regionname", "Type", "averageDistance"], axis = 1, inplace = False)    .sort_values(by = ['averageDistance'], ascending = False))
The beauty is, this concept does not stop with evaluating or manipulating the data. You may also use it with plotting. Personally, I highly recommend the plotting library plotnine — a great implementation of the grammar of graphics in python that is more a less a clone of the wonderful ggplot2 r package. Please find an extended article about plotnine here on Medium.
But for a first look at your data Pandas plot function is very worthwhile.
(    melb    #.query('Regionname.str.startswith("South")', engine = 'python')     .filter(["Regionname", "Distance"])    .groupby(["Regionname"])    .agg(["mean"])    .reset_index()    .set_axis(["Regionname", "averageDistance"], axis = 1, inplace = False)    .set_index("Regionname")    .sort_values(by = ['averageDistance'], ascending = False)    .plot(kind = "bar"))
(    melb    .Price # getting one specific variable     .hist())
In this article I motivated the usage of the pipes in R and method chaining in your Python code to improve readability and efficiency. I highlighted some benefits and together we applied the concept to a housing data frame. In particular I showed how to read data frames, filter and group data, calculating new variables, and how to plot. Again please make sure to check out the plotnine package for much nicer visualization results.
Please feel free to reach out to me if you have ideas to improve this solution! Thank you!
Find more articles from me here:
Learn how I plan my articles for MediumLearn how to write clean code in Python using chaining (or pipes)Learn how to analyze your LinkedIn data using RLearn how to create charts in a descriptive way in Python using grammar of graphicsLearn how to set up logging in your python data science code in under 2 minutes
Learn how I plan my articles for Medium
Learn how to write clean code in Python using chaining (or pipes)
Learn how to analyze your LinkedIn data using R
Learn how to create charts in a descriptive way in Python using grammar of graphics
Learn how to set up logging in your python data science code in under 2 minutes
Gregor Scheithauer is a consultant, data scientist, and researcher. He is specialized in the topics of Process Mining, Business Process Management, and Analytics. You can connect with him on LinkedIn, Twitter, or here on Medium. Thank you!
Melbourne Housing Snapshot | Kaggle
Tidyverse
The Flawless Pipes of Tidyverse. Exploratory data analysis made easy | by Soner Yıldırım | Mar, 2021 | Towards Data Science
Welcome | R for Data Science (had.co.nz)
18 Pipes | R for Data Science (had.co.nz)
Data visualization in Python like in R’s ggplot2 | by Dr. Gregor Scheithauer | Medium