How to load huge CSV datasets in Python Pandas | by Angelica Lo Duca | Towards Data Science

It may happen that you have a huge CSV dataset which occupies 4 or 5 GBytes (or even more) in your hard disk and you want to process it with Python pandas. Maybe you don't need all the data contained in the dataset, but only some records satisfying some criteria. In this short tutorial I show you how to deal with huge datasets in Python Pandas.
We can apply four strategies:
vertical filter
horizontal filter
bursts
memory.
You can download the full jupyter notebook from my Github repository.
In this case we load only some columns of the whole dataset. We can use the parameter usecols of the read_csv() function to select only some columns.
import pandas as pddf = pd.read_csv('hepatitis.csv', usecols=['age','sex'])
In this case we load only some rows of the dataset. We can choose the starting row and how many rows we must load. Keep in mind that if we skip the first row, we must pass the names of the columns as further parameter.
srow = 5nrows = 50columns = ['age','sex','steroid','antivirals','fatigue','malaise','anorexia','liver_big','liver_firm','spleen_palpable','spiders','ascites','varices','bilirubin','alk_phosphate','sgot','albumin','protime','histology','class']df = pd.read_csv('hepatitis.csv', skiprows=srow, nrows=nrows, names=columns)df.head()
It may happen that we need to load the full dataset, but we don’t have enough memory to load it. Thus we can load it in bursts, then we apply some filters and store the results into another (smaller) dataset. Filters may include dropping operations and conversion from bigger to smaller types.
Firstly, we define a function called read_burst() which reads the i-th burst, performs some filters and then stores the results into another output csv file. In our case we can drop rows where malaise = False. We pass the starting row srow, the burst size burst the names of columns columns as parameters of the read_burst() function. When the starting row is equal to ,1 we must also write the header to the output csv file. Thus we define a variable called header, which is set to True if srow = 1.
def read_burst(srow,burst,columns):    header = False    if srow == 1:        header = True    df = pd.read_csv('hepatitis.csv', skiprows=srow, nrows=burst, names=columns)    df = df.drop(df[df['malaise'] == False].index)    df.to_csv('hepatitis_small.csv', mode='a',header=header)
Now we can loop throughout the dataset. Note that we must know in advance the number of lines of the csv file. This can be done by using a command line tick. We can use the wc unix shell command. In order to run this command within the jupyther notebook, we must use the ! operator.
! wc -l hepatitis.csv
which gives the following output:
156 hepatitis.csv
Our file contains 156 rows, thus we can set the maximum number of lines to be read to 156, since the first line corresponds to the header. We set burst = 10. Thus we read 100 at times. At the end, we run again the read_burst() function to load the last remaining rows.
burst =10srow = 1nrows = 156columns = ['age','sex','steroid','antivirals','fatigue','malaise','anorexia','liver_big','liver_firm','spleen_palpable','spiders','ascites','varices','bilirubin','alk_phosphate','sgot','albumin','protime','histology','class']while srow < nrows:    print('srow: ' + str(srow))    read_burst(srow,burst,columns)    srow = srow + burstread_burst(srow,nrows,columns)
Now we can load the smaller dataset.
df = pd.read_csv('hepatitis_small.csv')df.head()
We can drop the Unnamed: 0 column.
df.drop(columns=['Unnamed: 0'],axis=1,inplace=True)
Another option while reading huge datasets in Python pandas could be increasing the memory associated to the reading operation. This can be done through the low_memory parameter.
df = pd.read_csv('hepatitis.csv', low_memory=False)
In this short tutorial I have illustrated how to deal with huge datasets in Python pandas. We have learnt that four strategies could be applied:
vertical filter
horizontal filter
bursts
memory configuration.
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and and Github.