Accelerating Pandas concatenation | by Philippe Cotte | Towards Data Science

I was recently faced with the problem of concatenating a fair amount of MultiIndexed Pandas Series (stacked DataFrames) into one single DataFrame. This can take a fair amount of time if you have many and/or large Series, and because of the MultiIndex, Dask cannot be used. I first present a sample of code using Dask’s logic to concatenate the Series pairwise in parallel jobs. I then show the acceleration performance for different parallel processing methods, for several number of CPUs ranging from 8 to 80 and several number of Series ranging from 10 to 90.
An optimal computation time of 37 % compared to the benchmark is found with only 8 CPUs. A higher number of CPUs does not reduce the relative computation time significantly.
In order to give some context to this study, let us specify how the data look like: we need to concatenate 90 time-indexed DataFrames of roughly 2000 lines and 500 columns, by stacking them into Series of 2000 x 500 lines and then concatenating those Series together with each Series being a column of the new DataFrame (see Figure 1). This can take, on my local machine (Intel® CoreTM i9–9900K CPU @ 3.60 GHz × 16, 32 GB memory), up to 10 minutes. It is worth mentioning that after stacking the initial DataFrames into Series, not a single index was found that was present in every Series, so the concatenation process will have a lot of re-indexing to do.
Usually, for this kind of acceleration problem, one would use Dask. Here it is not possible, for Dask does not support MultiIndex. But we can still use its basic logic, which is the following (see Figure 2): instead of concatenating N DataFrames at once, we split the N DataFrames into N/2 pairs and parallelize the concatenation of the 2 DataFrames in each pairs. Then we end up with N/2 DataFrames, which we split into N/4 pairs, and so on.
If N is not even, we just do the previous steps with the first N-1 DataFrames, and concatenate the remaining DataFrame with the resulting DataFrame.
Here is the sample of code doing that :
You can notice that the step of loading the files was included in the concat method, in order to make it parallel too.
The algorithm presented above was executed on a GCP machine of type n2-highcpu-80 (80 vCPUs, 80 GB memory). The benchmark used is the standard pandas concatenation, where the process of loading the files is parallelized:
with ProcessPoolExecutor(max_workers=max_workers) as e:    features = pd.concat(e.map(get_feature, features), axis=1)features = features.loc[:, sorted(features.columns)]
Figure 3 shows the absolute (left axis, plain lines) and relative (right axis, dashed lines) computation time against the number of CPUs used, for 90 DataFrames and various parallelization methods. The error bars are obtained by running each point 5 times and taking the standard deviation of the 5 resulting times.
Several things can be said about that graph:
There is a fast decrease in computation time for all methods between 8 and 16 CPUs. This can be attributed to the process of opening files being efficiently parallelized, for the benchmark line follows this trend too but only implements parallel jobs for opening the files.
The computation time is back to its initial value at 64 CPUs for all the methods, most likely because of the overhead of the multithreading (this has also been observed at the end of this study by one of my coworkers)
towardsdatascience.com
The most effective parallel computation method seems to be ThreadPool
There is a clear speed up of the concatenation process between ThreadPool and the benchmark, with a gain of 63 % +/- 1.5 %.
This gain is achieved with 8 CPUs, and is not significantly better with more CPUs
Figure 4 is similar to the previous one but when concatenating 30 DataFrames instead of 90.
The curves behave similarly compared to the previous graph. The absolute computation times are smaller as expected. So is the relative gain, and that too was to be expected. However the best method changed: ProcessPoolExecutor achieves the best gain, and not ThreadPool anymore.
Figure 5 shows the same absolute and relative time, with 8 CPUs, against the number of DataFrames to concatenate.
This graph tells us two things:
Even with as few as 10 DataFrames, the parallelization gives significant decrease in computation time.
ThreadPool is the best method only above 70 DataFrames. Below that, ProcessPoolExecutor shows better results.
My knowledge of the gears and tricks involved in parallel computation is not enough to allow me to propose any explanation on that last point, but if you have an idea about it do not hesitate to leave a comment! It would seem that ThreadPool, even though being a bit slower with fewer DataFrames, handles better a load ramping up than ProcessPoolExecutor.
I proposed here a method to accelerate the process of concatenating many Pandas Series into one DataFrame by using a Dask-like approach, when Dask itself can not be used because of the MultiIndex. This approach allowed for a gain of 63 % in computation time with only 8 CPUs, so it is worth implementing on desktop machines.
The results provided here have only been tested with a fixed size for input Pandas Series (2000x500 rows). It remains to be studied how well this method would fare with bigger or smaller data sets. Furthermore, after stacking the initial DataFrames into Series, not a single index was found that was present in every Series, so the concatenation process had a lot of re-indexing to do, which would have benefited a lot from the parallelization. For “cleaner” data sets, sharing a common index, the gain in computational time might be smaller.
Advestis is a European Contract Research Organization (CRO) with a deep understanding and practice of statistics, and interpretable machine learning techniques. The expertise of Advestis covers the modeling of complex systems and predictive analysis for temporal phenomena.