Speed Testing Pandas vs. Numpy
By: Eunjoo Byeon
Tags: Programming, Numpy, Python, Pandas, Data Science

Speed Testing Pandas vs. Numpy

Is Numpy Always Faster?

Photo by Tim Gouw on Unsplash

For Data Scientists, Pandas and Numpy are both essential tools in Python. We know Numpy runs vector and matrix operations very efficiently, while Pandas provides the R-like data frames allowing intuitive tabular data analysis. A consensus is that Numpy is more optimized for arithmetic computations. Is this always the case? I decided to put them to the test.

Method

First, I tested the vector array in Numpy with Pandas Series along with the Python list object. Then I also tested the matrix in Numpy with Pandas Dataframe and the nested list object. The list object’s performance was generally not comparable, so I will not discuss it. Each of these objects contained the same random numbers between 0 and 1 and were tested at two different scales: 1000 x 10 and 500000 x 1000. For the vector comparisons, only row counts were used (1000 and 500000).

# 1D

np_arr = np.random.random(n)

pd_ser = pd.Series(np_arr)

list_ = list(np_arr) # 2D

np_arr_2d = np.random.random(size = (n, m))

pd_db = pd.DataFrame(np_arr_2d)

list_2d = list(np_arr_2d)

I used the built-in IPython magic function %timeit to find the average time each function took. The syntax is as below.

np_result = %timeit -o np.mean(np_arr, axis = 1)

This returns a TimeitResult object that contains information on the best performance, average performance, standard deviations, number of runs, and number of loops it tested.

I tested the time to calculate mean, sum, and median for both row-wise and column-wise, as well as the time to query values using row numbers and column numbers.

Results

Querying with Index

Indexing for a small dataset

Indexing for a large dataset

When querying the dataset using the row numbers and column numbers, regardless of the size of a dataset, Numpy performed much better. But of course, Pandas allow indexing using column or row labels, which is a very useful feature regardless of the speed.

Vector Operation

Vector Operation on a small array (n = 1000)

When calculating straightforward arithmetic, such as summation and averaging, on vectors, Numpy again performed much better. But when it comes to computing the median, Numpy was shown to be quite slow. On the other hand, Pandas performed very well when computing a median.

Let’s see how this scales as the number of observations grow.

On a larger scale, the performance on summation was pretty similar for both Numpy and Pandas. But the median calculation started to get significantly worse for Pandas after about 60k rows.

Matrix Operation

Then I compared the column-wise and row-wise operations on a matrix in Numpy and Pandas data frame. Please note that the column size was consistent throughout, while the number of rows increased.

Row-wise and column-wise sum

The results on the summation were pretty comparable between the two (not too surprisingly, as Pandas uses Numpy on its backend). For both, there was no advantage in computing row-wise vs. column-wise, even though the columns were not increasing.

Row-wise and column-wise median

Pandas started to suffer greatly at computing medians as the number of observations increased. If we look closer into the smaller datasets, we can see that the Pandas performance on row-wise computation starts to decrease pretty early on. But time on column-wise calculation grows at a slower rate. Considering how the number of columns is the same, Pandas seem to be more affected by indexing median in each iteration than Numpy.