Different ways to iterate over rows in a Pandas Dataframe — performance comparison | by A Hung | Towards Data Science

Why I am interested in the performance of iteration
If you are a python user constantly dealing with structured datasets like myself, you probably use pandas a lot for many good reasons. This package includes plenty of useful functions that are already optimised for its usages and purposes (e.g. descriptive statistics, group-by aggregation). Most of the time, these functions suffice what you need to achieve. When it comes to time series data though, I often need to iterate through the data frame and perform ad-hoc sliding window calculations in my python code.
That gets me thinking — what would be the most time-efficient way to iterate through a pandas data frame?
Let’s compare performance of various iteration methods using this simple problem:
N = 1000repeats = 100a = np.repeat(1000, N)pd_dataset = pd.DataFrame({'a': a})# Goal is compute the column b where b(i) = a(i) + 1b = a + 1
This can actually be solved very quickly by applying a operator on the entire column to generate the new column as shown above. The operation is not important here, the idea is to compare the alternative ways for getting the same results if column operation is not feasible. In this exercise, I have implemented 6 alternative methods and compared them.
I have borrowed the timeit method from this blog post to measure the execution time of each method by annotating the method with @ timeit . This is my modified version of the timeit method — that appends the execution time to a list passed in as an input variable named log_time:
# Credits to Fahim Sakridef timeit(method):    def timed(*args, **kw):        ts = time.time()        result = method(*args, **kw)        te = time.time()        kw['log_time'].append(int((te - ts) * 1000))        return result    return timed
Let’s encapsulate the a + 1 operation into its own method, so it can be timed. Using the column operation as an example, here is the inital code for obtaining the average execution time using a given method:
def my_compute(x):    return x + 1@timeitdef use_column(dataset, **kwargs):    dataset['b'] = my_compute(dataset.a)def time_this(func, method_name):    """ Execute the given function 100 times and measure the execution time for each run.        Returns a dictionary containing the statistics based on the execution times    """    N = 1000    repeats = 100    a = np.repeat(1000, N)    pd_dataset = pd.DataFrame({'a': a})    timing = []    for i in range(repeats):        func(pd_dataset.copy(), log_time=timing)    return {'method': method_name, 'average': np.average(timing), 'min': np.min(timing), 'max': np.max(timing)}results = pd.DataFrame()results = results.append([time_this(use_for_loop_loc,'use_for_loop_loc')])
Ways to iterate over rows
In total, I compared 8 methods to generate a new column of values based on an existing column (requires a single iteration on the entire column/array of values). These were implemented in a single python file.
The first two are ways to apply column-wise functions on a dataframe column:
use_column: use pandas column operationuse_panda_apply: use pandas apply function
use_column: use pandas column operation
use_panda_apply: use pandas apply function
Next are the three different approaches for accessing the variable by using pandas indexing methods inside a for-loop:
3. use_for_loop_loc: uses the pandas loc function
4. use_for_loop_at: use the pandas at function(a function for accessing a single value)
5. use_for_loop_iat: use the pandas iat function(a function for accessing a single value)
There are other approaches without using pandas indexing:
6. use_numpy_for_loop: get the underlying numpy array from column, iterate , compute and assign the values as a new column to the dataframe
7. use_iterrows: use pandas iterrows function to get the iterables to iterate
8. use_zip: use python built-in zip function to iterate, store results in a numpy array then assign the values as a new column to the dataframe upon completion
Here are the average execution duration in seconds for each method, the test is repeated using different dataset sizes (N=1000,10000,10000):
N = 1000               method  average  min  max          use_column     0.00    0    0     use_panda_apply     1.95    1    9    use_for_loop_loc   409.69  398  474     use_for_loop_at    13.01   12   14    use_for_loop_iat    13.39   13   15  use_numpy_for_loop     9.59    9   11  use_panda_iterrows    58.31   57   60             use_zip     0.02    0    1N = 10000              method  average   min   max          use_column     0.06     0     1     use_panda_apply     1.01     1     2    use_for_loop_loc  4220.30  4182  4418     use_for_loop_at   127.79   126   130    use_for_loop_iat   134.03   132   138  use_numpy_for_loop    94.64    93    96  use_panda_iterrows   574.54   564   579             use_zip     3.00     3     3N = 100000              method   average    min    max          use_column      0.43      0      1     use_panda_apply      6.01      4     10    use_for_loop_loc  54340.47  53415  55124     use_for_loop_at   1263.17   1238   1431    use_for_loop_iat   1326.79   1313   1365  use_numpy_for_loop    939.02    926    956  use_panda_iterrows   5676.55   5609   6184             use_zip     28.00     27     29
What I expected and turned out to be correct:
Column operation and apply are both relatively fast
Select using at() and iat() is faster than loc()
Location-based indexing on numpy array is faster than locating-based indexing on a pandas dataframe
What are surprising:
zip() is relatively fast for small dataset - even faster than apply() for N < 1000
iat() and at() indexing can be 30 times faster than loc()
loc() is slower than expected even when access using the index
What I will be doing differently from now on based on these results:
If I cannot achieve what I want using column operation or apply(), I will use zip() instead (not iterrows()!)
I will avoid using loc() for updating or access single value, use iat() and at() instead
Consider extracting the underlying values as a numpy array then perform the processing/analysing