Appending One Row at a Time in Empty Pandas DataFrame | Towards Data Science

A very common question in the context of pandas is whether you can actually create an empty DataFrame and then iteratively fill it in by appending -say- one row at a time. However, this approach tends to be quite inefficient and should be avoided at all costs.
In today’s article we are going to discuss an alternative approach that will give you the same result but much more efficiently than creating an empty DataFrame and then using loops to append rows in it.
Of course, it is actually possible to create an empty pandas DataFrame and then append rows in an iteratively fashion. This approach in particular would look like below:
import numpy as npimport pandas as pdfrom numpy.random import randint# Make sure results are reproduciblenp.random.seed(10)# Instantiate an empty pandas DFdf = pd.DataFrame(columns=['colA', 'colB', 'colC'])# Fill in the dataframe using random integersfor i in range(7):    df.loc[i] = [i] + list(randint(100, size=2))print(df)  colA colB colC0    0    9   151    1   64   282    2   89   933    3   29    84    4   73    05    5   40   366    6   16   11
Even though the approach above will do the trick, it must be avoided since it is quite inefficient and there are certainly way more efficient approaches rather than creating an empty DataFrame and then build it up using iterative loops.
An even worse approach, is the use of append() or concat() methods inside loops.
It is worth noting that concat() (and therefore append()) makes a full copy of the data, and that constantly reusing this function can create a significant performance hit. If you need to use the operation over several datasets, use a list comprehension.
— pandas docs
Instead of appending rows in an iterative fashion using loc[] property or append/concat methods, you can actually append the data into a list and finally instantiate a new pandas DataFrame directly from the pre-created list. This is even mentioned in the official pandas documentation.
Iteratively appending rows to a DataFrame can be more computationally intensive than a single concatenate. A better solution is to append those rows to a list and then concatenate the list with the original DataFrame all at once.
— pandas docs
import numpy as npimport pandas as pdfrom numpy.random import randint# Make sure results are reproduciblenp.random.seed(10)data = []for i in range(7):    data.append([i] + list(randint(100, size=2))df = pd.DataFrame(data, columns=['colA', 'colB', 'colC'])print(df)   colA  colB  colC0     0     9    151     1    64    282     2    89    933     3    29     84     4    73     05     5    40    366     6    16    11
Working with lists (either appending or removing elements) is much more efficient and you must always prefer this approach when it comes to iteratively appending rows to pandas DataFrames.
In today’s article, we discussed why it is important to avoid creating empty pandas DataFrames and iteratively filling them up as this is going to significantly affect the performance.
Instead, we explored how to iteratively build such constructs using lists and finally create new pandas DataFrames out of the created lists.
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read. You’ll also get full access to every story on Medium.
You may also like