Best (fastest) ways to import CSV files in python for production environments (pandas, csv, dask)
By: Dipansh Girdhar
Tags: Data Analysis, Machine Learning, Pandas, Data Science, Csv

Best (fastest) ways to import CSV files in python for production environments (pandas, csv, dask) Dipansh Girdhar Follow Jun 21 · 6 min read

In the life of an Engineer working around data, working on csv files is an everyday task. While dealing with analytics tasks, we sometimes choose to ignore the time consumed in importing the data, but it becomes rather important in production environments. And it becomes somewhat problematic while dealing with large files. It turns out, as it will become clear further down in this post that if you face the issue of larger input files (like several several hundred MBs or more) it is probably best to use some sort of partitioning or parallel processing for the import.

This post will not focus on super accurate benchmarking, but rather on how to perform each of these CSV import options and how to run some simple benchmarking ourselves. We can draw some conclusions afterwards on the basis of this casual analysis. This post focuses only on CSV files, as these are quite often the format of choice for tabular data.

The options that I will cover here are: csv.DictReader() , pandas.read_csv(), dask.dataframe.read_csv(). This is by no means an exhaustive list of all methods for CSV import. However, it is probably a somewhat exhaustive list of the most common methods to do that.

How I measured the time efficiency

Here I am interested in how much time it takes to import a CSV file using various methods. This is pretty straightforward. Get the time before and after a certain action and then get the difference in seconds.

The results given by this method can be very volatile. Every split second different processes run on the computer with different intensity and therefore they interfere with the results of the measurement. For the purposes of this post we can ignore this interference, since we are more interested in time efficiency in relative rather than absolute terms.

Let us move on to the main main comparison.

csv.DictReader()

The DictReader is a Python class which maps the data read as a dictionary, whose keys, unless specified are the first row of the CSV. All values in the subsequent rows will be dictionary values and can be accessed with the respective dictionary key. However, these values will be imported as strings. I personally do not use this method a lot for data analysis tasks, but it is valuable in certain other cases (working with the JSON format or dealing with csvs through the terminal). You would go about it in the following way:

The snippet above will create the CSV object (data).

pandas.read_csv()

pandas is a popular library that has data structures suited for tabular data with heterogeneously-typed columns (integers, floats, strings like the random data we already created), as well as time series.Pandas are my preferred data structure for data analysis tasks as pandas are very easy to manipulate and transform. In a sense they are very similar to data.frame and data.table in R.

According to many opinions I have come across, the pandas package is very well optimized and thus very efficient in many tasks including basic ones as data import.

An importnat point here is that pandas.read_csv() can be run with the chunksize option. This will break the input file into chunks instead of loading the whole file into memory. This will reduce the pressure on memory for large input files and given an optimal chunksize found through trial and error, there can be significant increase in efficiency. The code below will split the input file into chunks of 100 000 lines.

Another important functionality present in pandas is the usecols parameter. This allows you to read only the selected columns to be read and skip the non-relevant columns in the data. This can save us time that was earlier used to read all the unnecessary columns.

I found this post by Goutham Balaraman very useful, where it is also explained how separating the input file in chunks can help speed up further processing as well. Later he edited his post saying that he finds the dask.dataframe superior to this method, which brings us to the next method.

dask.dataframe

The dask.dataframe is a collection of smaller pandas data frames split by the index (the row labels used for identification of data), which can be processed in parallel on a single machine or on multiple machines on a cluster. I learnt about dask from this useful post by Eric Brown.

Apparently, unlike pandas with dask the data is not fully loaded into memory, but is ready to be processed. Also certain opperations can be performed again without loading the whole dataset into memory. Another advantage is that the most functions used with pandas can be also use with dask. The differences arise from the parallel nature of dask.

Benchmarking

I perform the comparison for the Python methods with the following code all of which was already covered above:

The csv file used for the above code has around 500,000 rows and has a size of 420 MB.

The results I get are the following:

csv.DictReader took 0.000013709068298339844 seconds

pd.read_csv took 11.0141019821167 seconds

pd.read_csv with chunksize took 11.9807119369507 seconds

dask.dataframe took 0.079848103523254395 seconds

csv.DictReader is by far the fastest, but as I pointed out earlier it imports everything as strings, while the other methods try to guess the data types of each column separately and possibly do multiple other validations upon import. This means that if you use csv.DictReader for data analysis tasks you will possibly have to do a number of transformations on it before you can analyze the contents. But in cases when you know that the columns present in the data are all in string format already, csv.DictReader is the method to go for.

Ignoring csv.DictReader, dask.dataframe is by far the quickest method. This is understandable since it does not load the whole data set to memory, which the pandas methods do.

Summary

Pandas is very popular for data science work and has integration with a lot of other libraries. I would recommend looking into the chunksize option, if you are dealing with a large dataset and you are low on RAM. Dask is an option which gives tremendous improvements in import speed. However, I have not used it very much on data analysis tasks, but from what I understand it should be quite the same as pandas, which is promising.

Csv.DictReader remains the quickest option, but as I mentioned earlier I would not recommend it for data analysis tasks. Possibly working with the JSON format is one such area where this method will be quite useful. In production environments, when you have already analyzed the data, say using pandas and just need to use a final preprocessed data once in a while to either retrain the model or to upload the updated data to a search index like elasticsearch index, using csv.DictReader to import the data makes more sense as it is way faster than the other options.

This was a short overview of what I observed while working recently. Please reach out to me if you have any suggestions.

Cheers!