Top 3 Alternative Python Packages for Pandas | by Cornellius Yudha Wijaya | Towards Data Science

If you enjoy my content and want to get more in-depth knowledge regarding data or just daily life as a Data Scientist, please consider subscribing to my newsletter here.
For many modern data scientists, Python is the programming language that was used for their everyday work — as a consequence, the data analysis would be done using one of the most data packages, which are Pandas. Many online courses and lectures would introduce Pandas as the basis for every data analysis with Python.
In my opinion, Pandas is still the most useful and viable package to do your data analysis in Python. However, for comparison purposes, I want to introduce you to several Pandas package alternatives. I don’t intend to convince people to switch from Pandas to another package, but I only want people to know that there are alternatives for Pandas package.
So, what are these Pandas alternative packages? Let’s get into it!
Polars is a DataFrame library designed to processing data with a fast lighting time by implementing Rust Programming language and using Arrow as the foundation. Polars premise is to give the users a swifter experience in comparison to Pandas package. The ideal situation to use the Polars package is when you have data that were too big for Pandas but too small for using Spark.
For you who familiar with the Pandas workflow, Polars would not be that different — there is some extra functionality, but overall they are pretty similar. Let’s try to experiment with the Polars package. First, we need to install Polars in your Python environment.
pip install polars
After that, I would create a DataFrame sample using the Polars package.
import polars as pldf= pl.DataFrame({'Patient':['Anna','Be','Charlie','Duke','Earth','Faux','Goal','Him'],                  'Weight':[41,56,78,55,80,84,36,91],                  'Segment':[1,2,1,1,3,2,1,1] })df
Similar to Pandas DataFrame, the interface and the data analysis workflow is feels familiar. A little bit different is how the data type is on the top of the data, and no index was showing. Let’s try using some of the functions that available in Polars.
#Print DataFrame datatypeprint(df.dtypes)#Print DataFrame columnsprint(df.columns)#Print DataFrame top 5print(df.head())#Conditional Selectionprint(df[df['Weight'] < 70])
There are many available functions in Polars that have the same function as Pandas functions, and there is a possibility of conditional selection. I didn’t say that all the function that available in Pandas is available on Polars. For example, when I want to do index selection using iloc.
df.iloc[1]
The error is raised when I want to make index selection. This is because the Pandas Index concept did not apply in Polars.
One special concept that applicable in Polars is Expression. In Polars, you could produce a consecutively executed function (which is called Expression) and piped together. Let’s use an example to understand the concept.
#Parallel Expression df[[         #Expression for executing the stats (sum) and create the column names "sum"        pl.sum("Weight").alias("sum"),        pl.min("Weight").alias("min"),        pl.max("Weight").alias("max"),        pl.std("Weight").alias("std dev"),        pl.var("Weight").alias("variance"),]]
If we borrow the Pandas concept, often what we did above is something similar to the Pandas conditional selection. However, in our case with Polars, an expression execution resulted in another dataset when we were passing the expression.
The main selling point for using Polars is still their faster execution time. So, how fast the Polars execution time compared to Pandas? In this case, let me borrow the image example from the Polars documentation to give you the comparison.
The image above was the time comparison when Inner joined using three different methods: Polars Inner Join, Pandas Inner Join, and Polars Inner Join Parallel. As you can see, Pandas execution time is slower when the data size becomes bigger when you compared it to Polars. It might not have that much difference in execution time in a smaller data size, but it becomes clearer when the data size is bigger.
Dask is a Python package for parallel computing in Python. There are two main parts in Dask, there are:
Task Scheduling. Similar to Airflow, it is used to optimized the computation process by automatically executing tasks.Big Data Collection. Parallel data frame like Numpy arrays or Pandas data frame object — specific for parallel processing.
Task Scheduling. Similar to Airflow, it is used to optimized the computation process by automatically executing tasks.
Big Data Collection. Parallel data frame like Numpy arrays or Pandas data frame object — specific for parallel processing.
In simpler terms, Dask offers a data frame or array object like you could find in Pandas, but it is processed in parallel for faster execution time, and it offers a task scheduler.
As this article only cover the alternative for Pandas, we would not test the Task Scheduling functionality. Instead, we would focus on the Dask data frame object. Let’s try a few simple Dask functions example — but first, we need to install the Dask package (Anaconda has it by default).
#If you want to install dask completelypython -m pip install "dask[complete]"#If you want to install dask core onlypython -m pip install dask
After you have the dask installed, let’s try Dask functions. Initially, we could initiate a dashboard to monitor the data processing.
import daskfrom dask.distributed import Client, progressclient = Client(processes=False, threads_per_worker=4,                n_workers=1, memory_limit='2GB')client
Dask provides you a client dashboard in a different port that you could access after you initiate it. When you click on the Dashboard link, it will look like the following image.
I would say that this dashboard is easy to understand, and it includes all your current activities with dask. You could try to check out all the nice tabs that the dask dashboard has, but let’s continue for now. Let’s try to create a dask array for now. (something like what Numpy have)
import dask.array as dax = da.random.random((10000, 10000), chunks=(1000, 1000))x
When the dask array was created, you would receive the matrix information just like above. This array would become your base for the dask data frame. Let’s try to create a dataset from dask to have a more detailed example.
df = dask.datasets.timeseries()df
Dask data frame has a similar structure with Pandas data frame object, but because the Dask data frame is lazy, it would not print the data on your Jupyter Notebook. However, you could execute many functions that existed in Pandas on Dask. Let’s take a few examples for that.
df.head()
If you are using the .head function, you could print the top five data. If you want to make the conditional selection, it is still similar to Pandas. Although, it needs a special function to execute it. Let me show you an example code.
df[df['x'] >0.9]
If you do it normally, you will acquire the dask data frame structure. In case you want to obtain the Pandas data frame, you need to use the .compute function.
df[df['x'] >0.9].compute()
This is the actual data you get when you compute the dask data frame conditional selection. Everything on dask is about parallel and fast computation, so if you decide to use dask, it is better to have familiarity with Pandas. The ideal situation for dask is when you have a high computational dataset, but if your data is comfortably processed with your computer RAM, then stick with Pandas. There are still so many things you could do with dask; I suggest you visit their website to explore more.
Vaex is a Python package used for processing and exploring big tabular datasets with interfaces similar to Pandas. Vaex documentation shows that it can calculate statistics such as mean, sum, count, standard deviation, etc., on an N-dimensional grid up to a billion (109) objects/rows per second. It means Vaex is Pandas alternative that is also used to improve the execution time.
The Vaex workflow is similar to Pandas API, which means if you already familiar with Pandas, then it would not be hard for you to using Vaex. Let’s start the example by installing Vaex.
pip install vaex
When you have finished installing the Vaex package, we could try to use the dataset example to looks at the Vaex data frame.
import vaexdf = vaex.example()df
Similar to Pandas application, you could do the conditional selection.
df[df['x'] > 88]
However, just the Vaex documentation stated, “A core feature of Vaex is the extremely efficient calculation of statistics on N-dimensional grids. This rather useful for making visualisations of large datasets.” Let’s try to use simple statistical methods.
df.count(), df.max('x'), df.mean('L')
The result is presented as an array object with the detail of the value data type. Another function that Vaex selling is the plotting function, where you could apply a statistical function to the plot function. For example, instead of using the usual count statistic, I use the mean of the ‘E’ column with a limit of 99.7% data only that was shown.
df.plot1d(df['x'], what='mean(E)', limits='99.7%')
For more advanced plotting functions, you could visit the Vaex documentation. What is important to know is that Vaex selling point is their fast statistical computational and big data visualization exploration functionality. However, if you have a smaller dataset — you could stick with Pandas.
As a Data Scientist who uses Python as their analysis tool, you would certainly use the Pandas package to help your data exploration. However, there are a few Pandas alternative you could use — especially when you are dealing with big data, they are:
PolarsDaskVeux
Polars
Dask
Veux
I hope it helps!
Visit me on my LinkedIn or Twitter
If you are not subscribed as a Medium Member, please consider subscribing through my referral.