Is something better than pandas when the dataset fits the memory? | by Vaclav Dekanovsky | Medium | Towards Data Science

The tabular format is still the most typical way how to store data and there’s arguably no better tool how to explore data-tables than python’s pandas. Despite its broad range of capabilities, pandas has limits. In those cases, you have to opt for an alternative. Though, does it make sense to abandon pandas and use another tool, if the dataset fits comfortably into your memory?
Pandas is a convenient tabular data processor offering a variety of methods for loading, processing, and exporting datasets to many output formats. Pandas can handle a sizeable amount of data, but it’s limited by the memory of your PC. There was a golden rule of data science. If the data fits into the memory, use pandas. Is this rule still valid?
Use pandas when data fits your PC’s memory
Pandas alternatives were only recommended in these cases:
processing in pandas is slow
data doesn’t fit available memory
Let’s explore a few of these alternatives on a medium-size dataset to see if we can get any benefit or to confirm that you simply use pandas and sleep without doubts.
You can review complete code on GitHub
pandas_alternatives_POC.ipynb — exploring dask, spark, vaex and modin
julia_POC.ipynb — exploring julia and julia performance test
Performance_test.py — run python perf. tests in console
Results_and_Charts.ipynb — processing performance tests logs and creating the charts
Let’s first explore the arguments against the pandas alternatives.
they are not as universal as pandasdocumentation, tutorial, and community support is smaller
they are not as universal as pandas
documentation, tutorial, and community support is smaller
We will review several of the alternatives one by one and compare their syntax, computational approach, and performance. We will look at Dask, Vaex, PySpark, Modin (all in python) and Julia. These tools can be split into three categories:
Parallel/Cloud computing — Dask, PySpark, and Modin
Memory efficient — Vaex
Different programming language — Julia
For each of the tools, we will compare the speed of the basic operations using Kaggle fraud detection dataset. It contains two files train_transaction.csv (~700MB) and train_identity.csv (~30MB) which we will load, merge, aggregate, and sort to see how fast is the performance. I’ll do these operations on my 4 core notebook with 16MB RAM.
We will load, merge, sort and aggregate the data
Dask’s primary purpose is to parallelize any type of python computations — data processing, parallel message handling, or machine learning. The way how to spread up the calculations is to use the power of a cluster of computers. Even on a single PC, the computation can run faster, leveraging multiple processing cores.
Dask’s module way to work with the dataframes is prosaically called DataFrame. Its power originates in parallelism, which comes at a cost you must be aware:
Dask API is not so rich as pandas’sThe results must be materialized
Dask API is not so rich as pandas’s
The results must be materialized
Dask’s syntax is very similar to pandas.
As you can see, many methods are exactly the same in both libraries. You notice that dask basically lacks options for sorting. That’s because parallel sorting is special. Dask offers just one method, and that’s set_index. Index is sorted by definition.
The idea is to use the Dask to do the heavy lifting and then move the reduced and thus smaller dataset to pandas for final polishing. That brings us to the second caveat. Dask results must be materialized with .compute() command.
As you will see in the PySpark introduction, dask doesn’t calculate anything until it’s prompted to do so. All the steps are prepared and waiting for your command to start the work which comes with .compute() call.
You may wonder why don’t we get the results immediately, like when you run a pandas operation? The reason is simple. Dask is mainly used in case the data doesn't fit your memory and for that reason, results of the initial operations, like a load of the huge data frame cannot be materialized, because you don’t have enough memory to do so.
That’s why you prepare the computational steps, then let the cluster calculate and bring back a much smaller set containing only the results.
# the dask code goes for example like this:df = dd.read_csv(path)d2 = dd.read_csv(path2)re = df.merge(d2, on="col")re = re.groupby(cols).agg(params).compute()
It’s not straightforward how to compare the speed of two platforms meant for different purposes. The results might be skewed by the data as well. One tool can merge string columns very fast, while the other can excel in integer merging.
To show how fast these libraries can be, I’ve picked 5 operations, and I compared their speed. I repeated the performance test 7 times and I’ve measured that my cpu and memory usage never exceeded 50% of the PC (i7–5600 @ 2.60Ghz, 16GB Ram, SSD disk). No other process was running, besides the operational system and the performance test.
load_transactions — reading ~700MB CSV file
load_identity — reading ~30MB CSV file
merge — joining these two dataset on a string column
aggregation — grouping over 6 columns and calculating sum and mean
sorting — sort the merge dataset 3 times (if the library allows)
It looks that Dask loads the CSV files very fast, but the reason is the lazy operation mode of Dask. The loads are postponed until I materialize the results during the aggregation. It means that Dask only prepares for loading and merging, but performs it together with the aggregation.
Dask has very little support for sorting. Even the official guidelines say to run parallel computations and then pass the calculated (and much smaller results) to pandas.
Even if I try to compute the read_csv results, Dask was about 30% slower on my test datasets. That only confirms the original assumption, that Dask is beneficial mainly when your dataset doesn’t fit into the memory.
It’s python API for Spark, an analytic big data engine. Spark has evolved on the top of the Hadoop platform and might be the most popular cloud computing tool. It’s written in Scala, but many methods in the pySpark API let you compute without losing speed for python interpretation.
Similarly like Dask, you first define all the operations, and then you run .collect() command to materialize the results. There are more options than collect and you can read about them in the spark documentation — Actions.
Spark is using resilient distributed datasets (RDD) to perform the calculation and the syntax to operate them is quite similar to pandas. Often there are alternatives that yield the same or similar results like sort or orderBy methods.
In the beginning, you must initialize the Spark session. Then you prepare the steps using python API or you can benefit from Spark SQL and write the code in SQL.
You don’t have to install spark if you want to try it, because PySpark package comes with a spark instance. You must install java on your PC nevertheless.
The same performance tests were run using pySpark that I’ve described in the Dask section and the results were similar.
The difference is that spark reads part of the csv in order to infer the schema of the dataframe. It takes more time than to load the whole dataset to pandas in this case.
Spark is a great platform for massive computations leveraging the power of big clusters to get insights from a huge dataset. To employ spark on relatively small data won’t produce desired speed increase.
So far we have seen the platforms which split the work between more computer cores and often many computers in the cluster. They usually can not beat pandas for the dataset which fit the memory. Vaex aims to do that though.
The library was created to make the basic analysis of a dataset blazingly fast. Vaex doesn’t support the full range of pandas functionalities, but it can calculate basic statistics and create some chart types very quickly.
There are not many differences between pandas and vaex syntax.
Unlike the previous two tools, Vaex speed is similar to pandas on the example dataset, and in some areas, it’s even faster.
Often you will do well with pandas, but there can be cases when you struggle and that’s the time to try vaex. In this article, Jonathan Alexander did a bigger performance test to find out that Vaex can beat other approaches. It mainly masters join.
Julia is somewhat popular in the data science community. It was predicted to have a grand future, though the break-through did not happen yet. There are however many people who fell in love with the way Julia deals with things. I’m one of them.
Contrary to python, Julia is a compiled language. That often results in better performance. Both languages run in Jupyter notebook and that’s why Julia is popular for data science proof of concepts.
Julia was developed especially for mathematicians and data scientists. You can read what benefits of many existing programming languages Julia combines in the Why we created Julia article.
Despite Julia being a different language it does many things the pythonic way. On top of that, it uses its own tricks when suitable. I love the bang ! sign determining that it’s a mutable operation. It’s elegant and compact (compared to pandas’s inplace=True).
On the other hand, in python, many things have developed a preferred way of conding overtime. Julia often offers several ways how to do even the basic things like reading the csv. The syntax is quite stable since 1.0 release, but it can still happen that your favorite approach will be deprecated in the future.
Let’s compare how data loading, merging, aggregating and sorting look in pandas and julia.
Let’s see if Julia’s DataFrames can beat pandas.
Considering the speed of Julia isn’t that straightforward. When you first run any Julia code, the Just-in-Time compiler needs to translate it to a computer language, and that takes some time. That’s why the first run of any code takes longer than the follow-up runs.
On the chart below you can see that the first run is significantly longer than the average of the remaining six measurements. I’ve also tried to run Julia on a single core (julia) and on 4 processor cores (julia-4).
You can run julia with more cores by setting environmental variable JULIA_NUM_THREADS to the number of cores you want to use. Since 1.5 you can start julia by julia -t n or julia --threads n where n is the desired number of cores.
Processing using more cores is usually faster and julia has very good support for out-of-the-box parallelization. You might be concerned by the compiling speed, but you don’t have to be. The code is compiled once and changing the parameters won’t force the recompilation. After you compile CSV.read(joinpath(folder,file), DataFrame) the following calls are processed without compilation even in case you change the path to the source file. That is why you don’t see any delay in load_identity step because CSV reading was already compiled before.
Before we close our discussion about pandas alternatives, I must mention the Modin library. Its authors claim modin leverage parallelism to speed up 80% of pandas functions. I have unfortunately struggled to see these benefits. Sometimes it breaks during the initial import command. On other occasions modin said: “not supported, defaulting to pandas”, though the operation eventually crashed leaving 4 python processes each consuming a significant amount of memory. It took some time to kill these processes afterward.
I must say I like the idea behind modin and I hope one day these gaps will be covered which would elevate modin to a position of an alternative worth considering.
We have explored several popular pandas alternatives to find out whether it makes sense to use something else if the dataset is small enough to fit into the memory.
None of the parallel computing platforms was able to beat pandas in speed. Considering their more complex syntax, additional installation requirements and lack of some data processing capabilities eliminated these tools as an ideal alternative to pandas.
Vaex shows the potential to speed up some tasks during data exploration. The benefit can become more significant with the larger datasets.
Julia was developed with data scientists’ needs in mind. It might not be as popular as pandas and probably doesn't offer all the tricks pandas does. For some operations, it can provide a performance boost and I must say that some code is more elegant in julia. Even though Julia doesn’t get into the top 20 most popular programming languages, I guess it has a future and you won’t make a mistake if you keep an eye on its development.
Did you like this article, check other tutorials I wrote:
Alternative ways how to persist pandas dataframe
Demonstrative way how to use line charts
Turn a list of addresses into a map