Pandas DataFrame vs. Spark DataFrame: When Parallel Computing Matters | by Kevin C Lee | Towards Data Science

Note: The code for this post can be found here
Python is famous for its vast selection of libraries and resources from the open-source community. As a Data Analyst/Engineer/Scientist, one might be familiar with popular packages such as Numpy, Pandas, Scikit-learn, Keras, and TensorFlow. Together these modules help us extract value out of data and propels the field of analytics. As data continue to become larger and more complex, one other element to consider is a framework dedicated to processing Big Data, such as Apache Spark. In this article, I will demonstrate the capabilities of distributed/cluster computing and present a comparison between the Pandas DataFrame and Spark DataFrame. My hope is to provide more conviction on choosing the right implementation.
Pandas has become very popular for its ease of use. It utilizes DataFrames to present data in tabular format like a spreadsheet with rows and columns. Importantly, it has very intuitive methods to perform common analytical tasks and a relatively flat learning curve. It loads all of the data into memory on a single machine (one node) for rapid execution. While the Pandas DataFrame has proven to be tremendously powerful in manipulating data, it does have its limits. With data growing at an exponentially rate, complex data processing becomes expensive to handle and causes performance degradation. These operations require parallelization and distributed computing, which the Pandas DataFrame does not support.
Apache Spark is an open-source cluster computing framework. With cluster computing, data processing is distributed and performed in parallel by multiple nodes. This is recognized as the MapReduce framework because the division of labor can usually be characterized by sets of the map, shuffle, and reduce operations found in functional programming. Spark’s implementation of cluster computing is unique because processes 1) are executed in-memory and 2) build up a query plan which does not execute until necessary (known as lazy execution). Although Spark’s cluster computing framework has a broad range of utility, we only look at the Spark DataFrame for the purpose of this article. Similar to those found in Pandas, the Spark DataFrame has intuitive APIs, making it easy to implement.
When comparing computation speed between the Pandas DataFrame and the Spark DataFrame, it’s evident that the Pandas DataFrame performs marginally better for relatively small data. With size as the major factor in performance in mind, I conducted a comparison test between the two (script in GitHub). I found that as the size of the data increased, notably beyond 1 millions rows and 1000 columns, the Spark DataFrame can outperform the Pandas DataFrame. Below is an Animated 3D Wireframe Plot to illustrate the comparison for univariate (mean) and bivariate (correlation) calculations. Note that this doesn’t include the high overhead associated with setting up the Spark DataFrame. Additionally, the comparison is conducted on relatively simple operations. In reality, more complex operations are used, which are easier to perform with Pandas DataFrames than with Spark DataFrames.
The result is unsurprising given the single node nature of Pandas DataFrames vs. the distributed nature of Spark DataFrames. That is, since execution is done on a single server for the Pandas DataFrame, the in-memory computing speed and capability take a hit for very large data sets. Of course this is contingent upon the capabilities of the hardware.
Considering modern hardware specifications and how Pandas optimizes calculations, it is unlikely that a typical exploratory project warrants the implementation of Spark. However, there are cases where Spark makes obvious sense.
Spark is useful for applications that require a highly distributed, persistent, and pipelined processing. It might make sense to begin a project using Pandas with a limited sample to explore and migrate to Spark when it matures. This is used today in the development of market trend prediction, personalized customer experience, and weather forecast engines, to name a few.Spark is useful for Natural Language Processing and Computer Vision applications, which typically entail a great deal of calculations with data that is both wide and long.Spark has a Machine Learning library, MLlib, which aims to provide high-level APIs to create Machine Learning Pipelines. Machine Learning Pipelines are ideally run on Spark’s framework due to the iterative nature of model tuning.
Spark is useful for applications that require a highly distributed, persistent, and pipelined processing. It might make sense to begin a project using Pandas with a limited sample to explore and migrate to Spark when it matures. This is used today in the development of market trend prediction, personalized customer experience, and weather forecast engines, to name a few.
Spark is useful for Natural Language Processing and Computer Vision applications, which typically entail a great deal of calculations with data that is both wide and long.
Spark has a Machine Learning library, MLlib, which aims to provide high-level APIs to create Machine Learning Pipelines. Machine Learning Pipelines are ideally run on Spark’s framework due to the iterative nature of model tuning.
As demonstrated in the comparison analysis, the popular Pandas DataFrame is very capable of performing rapid calculation. Although the implementation of Spark DataFrame has become easier since the syntax is intended to match its Pandas counterpart, it’s not always desirable for ad-hoc analysis. It does become very powerful with data processes that are well established, particular when the data is expected to be large.
Here’s is the walk-through of how I created the animated 3D Wireframe Plot. At a high level, it can be broken down into the following steps:
Capture some data — I used the limited data points to infer additional simulations in steps 2–4Fit Regression Model to dataCreate a grid of the desired X, Y planeUse Regression Model to calculate Z coordinates for the gridGenerate 3D Wireframe PlotRepeat step 5 by rotating the axis of the X, Y plane and store each plot as individual PNG filesUse sequential PNG files to generate GIF
Capture some data — I used the limited data points to infer additional simulations in steps 2–4
Fit Regression Model to data
Create a grid of the desired X, Y plane
Use Regression Model to calculate Z coordinates for the grid
Generate 3D Wireframe Plot
Repeat step 5 by rotating the axis of the X, Y plane and store each plot as individual PNG files
Use sequential PNG files to generate GIF
I simulated and collect 26 trials at varying data dimensions shown below (only 5 shown).
The above gist is used to generate the additional points for the Wireframe Plot since the Scatter Plot doesn’t do a complete job for the visualization I’m looking for. I used a for-loop to iterate through different rotations of the x, y axis to generate different frames of the GIF.
import matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3Dax=plt.axes(projection='3d')X, Y, pandas_Z, spark_Z, _=get_plane(df, 2, 20)ax.plot_wireframe(X, Y, pandas_Z)ax.plot_wireframe(X, Y, spark_Z)# rotate x, y plane with the azimuth parameter and save figure at every 10 degrees# intentionally generate more frames between 180 and 270 degrees to create a slow down effect in GIFfor ii in range(0,360,1):    if ii>=180 and ii<=270 and ii%3==0:         ax.view_init(elev=10., azim=ii)        plt.savefig("movie/movie%d.png" % ii)    elif (ii<=180 or ii>=270) and ii%10==0:         ax.view_init(elev=10., azim=ii)        plt.savefig("movie/movie%d.png" % ii)