Python Pandas vs. R Dplyr. The Full Cheatsheet | by Martin Šiklar | Towards Data Science

Pandas for Python and Dplyr for R are the two most popular libraries for working with tabular/structured data for many Data Scientists. There is always this big and partly heated discussion on which framework is better. Honestly, does it really matter? In the end, it’s about getting the job done and both pandas and dplyr offer great tools for data wrangling. No worries, this article is not yet another comparison that tries to prove a point for either library! The purpose of this article therefore is:
To help others with the transition from one language/framework to the other
To explore new tools that you can add to your repertoire as a Data Scientist
To create a reference cheat sheet in case you need to look up the most frequently used data wrangling functions in either language
In this tutorial we will be working with the iris dataset which is part of both Pythons sklearn and base R. After some homogenisation our data in R / Python looks like this:
Sepal_length  Sepal_width  Petal_length     Petal_width  Species          5.1         3.5           1.4             0.2   setosa          4.9         3.0           1.4             0.2   setosa          4.7         3.2           1.3             0.2   setosa          4.6         3.1           1.5             0.2   setosa          5.0         3.6           1.4             0.2   setosa
Disclaimer: long read
And here it already gets confusing. First of all, there are multiple ways on how to select columns from a dataframe in each framework. In Pandas you can either simply pass a list with the column names or use the filter() method. This is confusing because the filter() function in dplyr is used to subset rows based on conditions and not columns! In dplyr we use the select() function instead:
Pandas
#Pass columns as listdataframe[[“Sepal_width”, “Petal_width”]]#Use Filter Functiondataframe.filter(items=['Sepal_width', 'Petal_width'])
Dplyr
dataframe %>% select(Sepal_width, Petal_width)
Again, there are multiple ways on how to filter records in a dataframe based on conditions across one or multiple columns.
Pandas
In Pandas you can either use the indexing approach or try out the handy query API, which I personally prefer.
#indexingdataframe[(dataframe["Sepal_width"] > 3.5) & (dataframe["Petal_width"] < 0.3)]#query APIdataframe.query("Sepal_width > 3.5 & Petal_width < 0.3")
Dplyr
The standard way of filtering records in dplyr is via the filter function().
dataframe %>% filter(Sepal_width > 3.5 & Petal_width < 0.3)
Renaming sounds like an easy task, but be cautious and note the subtle difference here. If we want to rename our column from Species to Class in Pandas we supply a dictionary that says {‘Species’: ‘Class’} and in Dplyr it is the exact opposite way Class=Species:
Pandas
dataframe.rename(columns = {'Species': 'Class'}, inplace = True)
Dplyr
dataframe <- dataframe %>% rename(Class=Species)
Let us say we want to rename multiple columns at once based on a condition. For example, convert all our feature columns (Sepal_length, Sepal_width, Petal_length, Petal_width) to upper case. In Python that's actually quite tricky and you need to first import another library and iterate manually over each column. In Dplyr there is a much cleaner interface if you want to access/change multiple columns based on conditions.
Pandas
import re#prepare pattern that columns have to match to be converted to upper casepattern = re.compile(r".*(length|width)")#iterate over columns and covert to upper case if pattern matches.for col in dataframe.columns:  if bool((pattern.match(col))):    dataframe.rename(columns = {col: col.upper()}, inplace = True)
Dplyr
dataframe <- dataframe %>% rename_with(toupper, matches("length|width"))
Result
Note the upper case feature column names:
SEPAL_LENGTH  SEPAL_WIDTH  PETAL_LENGTH     PETAL_WIDTH  Species          5.1         3.5           1.4             0.2   setosa          4.9         3.0           1.4             0.2   setosa
Let us say we want to recode/alter cell values based on conditions: In our example, we will try to recode the Species strings “setosa, versicolor and virginica” to integers from 0 to 2:
Pandas
dataframe.loc[dataframe['Species'] == 'setosa', "Species"] = 0dataframe.loc[dataframe['Species'] == 'versicolor', "Species"] = 1dataframe.loc[dataframe['Species'] == 'virginica', "Species"] = 2
Dplyr
dataframe <- dataframe %>%  mutate(Species = case_when(Species == 'setosa' ~ 0,                             Species == 'versicolor' ~ 1,                             Species == 'virginica' ~ 2))
Sometimes we want to see which values distinct/unique values we have in a column. Note how different the function call is in both frameworks: Pandas uses the unique() method and dplyr() uses the distinct() function to get to the same result:
Pandas
dataframe.Species.unique()#array(['setosa', 'versicolor', 'virginica'], dtype=object)
Dplyr
dataframe %>% select(Species) %>% distinct()# Species   # setosa    # versicolor# virginica
If you want to count how many entries the dataframe has in total or get a count for a certain group, you can do the following:
Pandas
# Total number of records in dataframelen(dataframe)#150# Number of records per Groupdataframe.value_counts('Species')#Species#virginica     50#versicolor    50#setosa        50# Note that you can also use the .groupby() method followed by size()dataframe.groupby(['Species']).size()
Dplyr
# Total number of records in dataframedataframe %>% nrow()#[1] 150# Number of records per Group (count and tally are interchangeable)dataframe %>% group_by(Species) %>% count()dataframe %>% group_by(Species) %>% tally()#  Species        n#  <fct>      <int>#1 setosa        50#2 versicolor    50#3 virginica     50
If you want to create descriptive statistics for one or multiple columns in your data frame, you can do the following:
Pandas
#get mean and min for each columndataframe.agg(['mean', 'min'])#      Sepal_length  Sepal_width  Petal_length  Petal_width Species#mean      5.843333     3.057333         3.758     1.199333     NaN#min       4.300000     2.000000         1.000     0.100000  setosa
Dplyr
Unfortunately, I didn't find a way on how to use multiple aggregation functions over multiple columns at once. That is why you need to call the summarise function multiple times to achieve the same result:
#first aggregation over all columns using meandataframe %>% summarise(across(everything(), mean))#  Sepal_length Sepal_width Petal_length Petal_width Species#         5.84        3.06         3.76        1.20      NA#second aggregation over all columns using mindataframe %>% summarise(across(everything(), min))#Sepal_length Sepal_width Petal_length Petal_width Species#        5.84        3.06         3.76        1.20      NA
If you want to have aggregate statistics for by group in your dataset, you have to use the groupby() method in Pandas and the group_by() function in Dplyr. You can either do this for all columns or for a specific column:
Pandas
Note how Pandas uses multilevel indexing for a clean display of the results:
# aggregation by group for all columnsdataframe.groupby(['Species']).agg(['mean', 'min'])#                 Sepal_length      Sepal_width       ...#                   mean  min        mean min         ...#Species                                                         #setosa            5.01  4.3       3.43             ...#versicolor        5.94  4.9       2.77             ...#virginica         6.59  4.9       2.97             ...# aggregation by group for a specific columndataframe.groupby(['Species']).agg({'Sepal_length':['mean']})#   Sepal_length#                   mean#Species                #setosa            5.01#versicolor        5.94#virginica         6.59
Dplyr
Since Dplyr doesn’t support multilevel indexing, the output of the first call looks a little bit messy compared to Pandas. In this output, the statistics for the first function are displayed (mean-fn1) followed by the statistics of the second function (min-fn2).
# aggregation by group for all columnsdataframe %>% group_by(Species) %>% summarise_all(list(mean,min))Species    Sepal_length_fn1  Sepal_width_fn1         ...setosa                 5.01            3.43          ...versicolor             5.94            2.77          ...         virginica              6.59            2.97          ...# aggregation by group for a specific columndataframe %>% group_by(Species) %>% summarise(mean=mean(Sepal_length))#Species     mean# setosa      5.01# versicolor  5.94# virginica   6.59
Sometimes you want to create a new column and combine the values of two or more existing columns with some mathematical operation. Here is how to do it in both Pandas and Dplyr:
Pandas
dataframe["New_feature"] = dataframe["Petal_width"]* dataframe["Petal_length"] / 2
Dplyr
dataframe <- dataframe %>% mutate(New_feature= Petal_width*Petal_length/2)
To clean up a dataframe, deleting columns can sometimes be quite handy:
Pandas
In Pandas you can delete a column with drop(). You can also use inplace=True to overwrite the current dataframe.
dataframe.drop("New_feature", axis=1, inplace=True)
Dplyr
In Dplyr you specify the column name you want to remove inside the select() function with a leading minus.
dataframe <- dataframe %>% select(-New_feature)
To sort values you can use sort_values() in Pandas and arrange() in Dplyr. The default sorting for both is ascending. Note the difference in each function call on how to sort descending:
Pandas
dataframe.sort_values('Petal_width', ascending=0)
Dplyr
dataframe %>% arrange(desc(Petal_width))
Renaming sounds like an easy task, but be cautious and note the subtle difference here. If we want to rename our column from Species to Class in Pandas we supply a dictionary that says {‘Species’: ‘Class’} and in Dplyr it is the exact opposite way Class=Species:
Pandas
dataframe.rename(columns = {'Species': 'Class'}, inplace = True)
Dplyr
dataframe %>% relocate(Species)dataframe %>% relocate(Species, .before=Sepal_width)
I don’t use this functionality often, but sometimes it comes in handy if I want to create a table for a presentation and the ordering of the columns doesn’t make logical sense. Here is how to move around columns:
Pandas
In Python Pandas you need to reindex your columns by making use of a list. Let’s say we want to move the column Species to the front.
#change order of columnsdataframe.reindex(['Species','Petal_length','Sepal_length','Sepal_width','Petal_Width'], axis=1)
Dplyr
In Dplyr you can use the handy relocate() function. Again let’s say we want to move the column Species to the front.
dataframe %>% relocate(Species)#Note that you can use .before or .after to place a columne before or after another specified column - very handy!dataframe %>% relocate(Species, .before=SEPAL_WIDTH)
Slicing is a whole topic on its own and there is a lot of ways on how to do it. Let us step through the most frequently used slicing operation below:
Sometimes you know the exact row number you want to extract. Although the procedure in Dplyr and Pandas is quite similar please note that Indexing in Python starts at 0 and in R at 1.
Pandas
dataframe.iloc[[49,50]]#  Sepal_length  Sepal_width  Petal_length  Petal_width    Species#          5.0          3.3           1.4          0.2      setosa#          7.0          3.2           4.7          1.4  versicolor
Dplyr
dataframe %>% slice(50,51)#  Sepal_length Sepal_width Petal_length Petal_width Species     #1            5         3.3          1.4         0.2 setosa    #2            7         3.2          4.7         1.4 versicolor
Sometimes we want to see either the first or last records in a dataframe. This can be either done by providing a fixed number n or a proportion prop of values.
Pandas
In Pandas you can use the head() or tail() method to get a fixed amount of records. If you want to extraction a proportion, you have to do some math on your own:
#returns the first 5 recordsdataframe.head(n=5)#returns the last 10% of total recordsdataframe.tail(n=len(dataframe)*0.1)
Dplyr
In Dplyr there are two designated functions for this use case: slice_head() and slice_tail(). Please note how you can either specify a fixed number or a proportion:
#returns the first 5 recordsdataframe %>% slice_head(n=5)#returns the last 10% of total recordsdataframe %>% slice_tail(prop=0.1)
Sometimes it is useful to select records with the highest or lowest values per column. Again this can be done by providing a fixed number or a proportion.
Pandas
In Pandas this is a little more tricky than in Dplyr. Imagine for example you want 20 records with the longest “Petal_length” or 10% of the total records with the shortest “Petal_length”. To do the second operation in Python, we have to do some math and first sort our values:
#returns 20 records with the longest Petal_length (for returning the shortest you can use the function nsmallest)dataframe.nlargest(20, 'Petal_length')#returns 10% of total records with the shortest Petal_lengthprop = 0.1 dataframe.sort_values('Petal_length', ascending=1).head(int(len(dataframe)*prop))
Dplyr
In Dplyr this is much simpler since there are designated functions for this use case:
#returns 20 records with the longest Petal_lengthdataframe %>% slice_max(Petal_length, n = 20)#returns 10% of total records with the shortest Petal_lengthdataframe %>% slice_min(Petal_length, prop = 0.1)
Sometimes it is useful to select records with the highest or lowest values per column but separated by group. Again this can be done by providing a fixed number or a proportion. Imagine for example we want 3 records with the shortest Petal_length per Species.
Pandas
In Pandas, this is again a little bit more tricky than in Dplyr. We first group our dataframe by Species and then apply a lambda function that makes use of the above-described nsmallest() or nlargest() function:
#returns 3 records with the shortest Petal_length per Species(dataframe.groupby('Species',group_keys=False)        .apply(lambda x: x.nsmallest(3, 'Petal_length')))#Sepal_length  Sepal_width  Petal_length  Petal_width     Species#        4.6          3.6           1.0          0.2      setosa#        4.3          3.0           1.1          0.1      setosa#        5.8          4.0           1.2          0.2      setosa#        5.1          2.5           3.0          1.1  versicolor#        4.9          2.4           3.3          1.0  versicolor#        5.0          2.3           3.3          1.0  versicolor#        4.9          2.5           4.5          1.7   virginica#        6.2          2.8           4.8          1.8   virginica#        6.0          3.0           4.8          1.8   virginica#returns 5% of total records with the longest Petal_length per Speciesprop = 0.05(dataframe.groupby('Species',group_keys=False)        .apply(lambda x: x.nlargest(int(len(x) * prop), 'Petal_length')))#Sepal_length  Sepal_width  Petal_length  Petal_width     Species#         4.8          3.4           1.9          0.2      setosa#         5.1          3.8           1.9          0.4      setosa#         6.0          2.7           5.1          1.6  versicolor#         6.7          3.0           5.0          1.7  versicolor#         7.7          2.6           6.9          2.3   virginica#         7.7          3.8           6.7          2.2   virginica
Dplyr
In Dplyr this is much simpler since there are designated functions for this use case. Note how with_ties=FALSE can be provided to specify if ties (records with equal values) should not be returned.
#returns 3 records with the shortest Petal_length per Speciesdataframe %>% group_by(Species) %>% slice_min(Petal_length, n = 3, with_ties = FALSE)#returns 5% of total records with the longest Petal_length per Speciesdataframe %>% group_by(Species) %>% slice_max(Petal_length, prop = 0.05, with_ties=FALSE)
Slicing random records can also be called sampling. Again this can be done by proving a fixed number or a proportion. Furthermore, this can be done on the entire dataset or also equally distributed based on a group. Since this is quite a frequent use case, there are functions for this in both frameworks:
Pandas
In Pandas you can use the sample() function and either specify n for a fixed amount of records or frac for a proportion of records. Furthermore, you can specify replace to allow or disallow sampling of the same row more than once.
#returns 20 random samplesdataframe.sample(n=20)#return 20% of total recordsdataframe.sample(frac=0.2, replace=True)#returns 10% of total records split by groupdataframe.groupby('Species').sample(frac=0.1)
Dplyr
The interface in Dplyr is very similar. You can use the slice_sample() function and either specify n for a fixed amount of records or prop for a proportion of records. Furthermore, you can specify replace to allow or disallow sampling of the same row more than once.
#returns 20 random samplesdataframe %>% slice_sample(n=20)#return 20% of total recordsdataframe %>% slice_sample(prop=0.2, replace=True)#returns 10% of total records split by groupdataframe %>% group_by(Species) %>% slice_sample(prop=0.1)
Joining dataframes is also a frequent use case. (There is a wide range of join operation but I am not going to get into details here)
Subsequently, however, you will learn how to perform a full (outer) join in both Pandas and Dplyr.
Image you have two dataframes that share a common variable “key”:
#Python PandasA = dataframe[[“Species”, “Sepal_width”]]B = dataframe[[“Species”, “Sepal_length”]]#R Dplyr:A <- dataframe %>% select(Species, Sepal_width)B <- dataframe %>% select(Species, Sepal_length)
Pandas
For all join operations you can use the “merge” function in Pandas and specify, what you want to join, how (outer, inner, left, right,..) you want to join and on which key:
#Join dataframe A and B (WHAT), with a full join (HOW) by making use of the key "Species" (ON) pd.merge(A, B, how="outer", on="Species")
Dplyr
In Dplyr the syntax is very similar, however, you have separate functions for each join type. In this example, we will again perform a full join with the full_join() function:
#Join dataframe A and B (WHAT), with a full join (HOW) by making use of the key "Species" (ON)A %>% full_join(B, by="Species")
Sometimes we don’t want to join our dataframe but just append two existing dataframes either by rows or columns. Both Pandas and Dplyr have a nice interface for achieving this:
Pandas
In Pandas you can concatenate two dataframes with the concat() method. The default value concatenates dataframes by rows. By specifying the axis (e.g. axis = 1) you can concatenate two dataframes by columns.
Note that if a value doesn’t appear in one of the dataframes it's automatically filled with NA.
#Concatenate by rowspd.concat([A,B])#       Species  Sepal_width  Sepal_length#0       setosa          3.5           NaN#1       setosa          3.0           NaN#2       setosa          3.2           NaN#3       setosa          3.1           NaN# ...#Concatenate by columns pd.concat([A,B], axis=1)#       Species  Sepal_width    Species  Sepal_length#0       setosa          3.5     setosa           5.1#1       setosa          3.0     setosa           4.9#2       setosa          3.2     setosa           4.7#3       setosa          3.1     setosa           4.6# ...
Dplyr
In Dplyr there are two separate functions for binding dataframe: bind_rows() and bind_columns().
Note that if a value doesn’t appear in one of the dataframes it's automatically filled with NA if you apply bind_rows(). Also, note how R automatically changes the column names (to avoid duplicates). This behaviour can be changed with the .name_repair argument.
#Bind by rowsA %>% bind_rows(B)#   Species Sepal_width Sepal_length# 1 setosa          3.5           NA# 2 setosa          3             NA# 3 setosa          3.2           NA# 4 setosa          3.1           NA# ...#Bind by columnsA %>% bind_cols(B)#  Species...1 Sepal_width Species...3 Sepal_length# 1 setosa              3.5 setosa               5.1# 2 setosa              3   setosa               4.9# 3 setosa              3.2 setosa               4.7
Pfew! Congratulations! You might be the first person ever to reach the end of this article/cheat sheet. You can claim your reward by sending me a clap or leaving me a comment below :)