Pandas (Python) vs Data.table (R) | by Soner Yıldırım | Towards Data Science

Data science ecosystem is full of highly effective and practical tools and frameworks. New ones are introduced and the existing ones are improved continuously.
Having a variety of selections is a good thing and likely to increase the efficiency in most cases. However, it might also make it hard to decide which one to pick.
In this article, we will compare popular data manipulation libraries in arguably the two most commonly used programming languages in data science domain.
We will see how basic operations are done in Pandas (Python) and Data.table (R). The goal is not to determine if one is superior to or better than the other. I just want to make you familiar with the syntax and show similar or different approaches.
There are many options to use these packages. I’m using R-studio IDE for R and VSCode for Python.
We start with importing the libraries and creating the basic data structures by reading data from a csv file.
In Pandas, we use the read_csv function to create a dataframe from a csv file.
import numpy as npimport pandas as pd #importing pandasinsurance = pd.read_csv("/home/soner/Downloads/datasets/insurance.csv")
In Data.table, the equivalent of the read_csv is the fread function.
> library(data.table) #importing data.table > insurance = fread("/home/soner/Downloads/datasets/insurance.csv")
The head function displays the top n number of rows. The name of the function is same for both packages but the syntax is slightly different.
The default n value is 5 for both so they will display the first 5 rows by default.
We usually check the size of data in terms of the number of rows and columns. Both libraries have intuitive ways of obtaining the size related information.
#Pandasinsurance.shape(1338, 7)#Data.tablenrow(insurance)1338ncol(insurance)7
In some cases, we just need a certain set of columns from a table. Both Pandas and Data.table provides convenient ways for doing this operation.
sub = insurance[['age','sex','charges']]sub.head()
The same operation is done with Data.table as follows:
> sub = insurance[, .(age, sex, charges)]> head(sub)
The rename and setname functions can be used to rename columns in Pandas and Data.table, respectively.
Pandas:
sub.rename(columns={'sex':'gender'}, inplace=True)sub.columnsIndex(['age', 'gender', 'charges'], dtype='object')
Data.table:
> setnames(sub, c("sex"), c("gender"))> names(sub)[1] "age"     "gender"  "charges"
As we see from the examples, the columns method on dataframe returns a list of columns. Same operation is done with the names function in Data.table library.
We can filter a dataframe or table based on row values. In the following examples, we create a subset that contains customer who are older than 40.
Pandas:
sub_over40 = sub[sub.age > 40]
Data.table:
> sub_over40 <- sub[age > 40] #can also use "<-" instead of "="
One of the most common operations in data analysis is to comparing numerical variables based on different values in a categorical variable.
In Pandas, this kind of transformations and calculations are done with the group by function. For instance, we can calculate the average charge of data points in each category based on the gender and smoker columns.
insurance[['gender','smoker','charges']]\.groupby(['gender','smoker']).mean().round(2)
It is possible to apply multiple aggregate functions. For instance, we may want to see both the number of observations in each group in addition to the average charge value.
insurance[['gender','smoker','charges']]\.groupby(['gender','smoker']).agg(['mean','count'])
In Data.table, the same calculation is done as follows:
> insurance[, .(mean(charges)), by = .(gender, smoker)]
Just like with Pandas, Data.table provides a straightforward way to apply multiple aggregations. Both the number of observations and average charge value are calculated as follows:
> insurance[, .(mean(charges), .N), by = .(gender, smoker)]
We have covered some simple tasks in Pandas and Data.table libraries. Due to the complexity level of operations, the syntax seems quite similar for both libraries. However, as the complexity increases and we do more advance data analysis and manipulation operations, the difference between them become more noticeable.
The best way to learn how to use these libraries is through practice. As you adapt them in your data analysis tasks, you will have some preference for using a particular one in certain tasks.
In general, I think both are more than enough to perform typical data analysis tasks.
Thank you for reading. Please let me know if you have any feedback.