Quickest ways to load CSV file data in R
By: Rafael Belokurows
Tags: Data Analysis, R, Data Import And Export, Data Science, R Programming Language

Dataset 1: Starting Small

Let’s have a look at the first dataset. It’s got 8K rows and 23 categorical columns with few levels each, thus only using 1.44 Mbs of memory.

For more information and to download the dataset, click here.

Memory usage of the small dataset

It should probably be loaded in less than a second into a data frame regardless of the function used.

Time to load the small dataset

We see that function fread was the fastest in loading the first dataset, with little variation, taking 9 to 11 milliseconds each time. The base read.csv function came second, followed by readr’s function. Although we almost wouldn´t notice a difference in real time between 10 and 100 milliseconds, the fread function showed a lot of potential, importing the dataset 10 times faster than the others.

🏆Winner: data.table::fread

Dataset 2: Still going easy on them

The second dataset contains 19 attributes for 170K songs and it was extracted from Spotify.

For more information and to download the dataset, click here.

As we can see in the following plot, the numerical columns take up way less memory than the categorical columns with lots of levels such as the song ids, names of the songs, and artists.

Memory usage of the medium dataset

And when we run the benchmark on this dataset, we get:

Time to load the medium dataset

Although our unit is still milliseconds, we can still see quite a good difference when using each of the methods. Only this time, readr::read_csv was way faster than the R’s base function. The win goes easily for data.table::fread again, taking an average of 300 ms to load the dataset, 3 times faster than readr and 15 times faster than base R.

🏆Winner: data.table::fread

Dataset 3: Now we’re talking

The third dataset was also obtained from Kaggle and it’s a dataset of 240K rows and 29 columns of information on gun violence that happened in the US between 2013 and 2018.

For more information and to download the dataset, click here.

Although the number of rows is not much larger than the previous dataset, it takes 5 times more memory and although metrics may vary on the definition, I consider this a large dataset — or at least a larger one.

Memory usage of the large dataset

Our unit now is seconds as this is a much larger dataset compared to the previous ones. The order pretty much remained the same when it comes to loading times: 1st: fread, 2nd: readr, then 3rd: base R.

Time to load the large dataset

There’s just one outlier here as we can see pretty easily on the plot of the loading times. One of the executions of the fread function took a whopping 13 seconds to run, much more than any of the times from the other functions. Still, the median was pretty consistent with what we have seen before, as it took in average (median) 1,45 seconds vs. 3 and 6,5 of the other functions. Even with this clear outlier, fread is the winner again.

🏆Winner: data.table::fread

Dataset 4: Getting our hands dirty

The big guy out of the 4 datasets is a 1,3Gb CSV file that contains 3.5 Million rows and 49 columns with a variety of data types ranging from date to categorical to numerical.

For more information and to download the dataset, click here.

Memory usage of the huge dataset

It should present a good challenge to all our functions so let’s see how they did.

Time to load the huge dataset

Last but not least, we run the benchmarks when loading the larger dataset of the bunch, which I call “huge” for sake of convention. And for the surprise of exactly zero people, fread wins the crown yet again, with a median time of 12 seconds vs. 40 secs and 80 secs from the other contestants.

🏆Winner: data.table::fread

TLDR and Final Thoughts

This comes as a foregone conclusion, but out of these 3 packages/functions, one was obviously superior to the others:

data.table::fread is the winner! 🏆🏆🏆🏆

It loaded the datasets 2 to 10 times faster than the other functions that we tested, regardless of the size of the data.

In future work, I’d like to compare the loading times on a faster and more powerful machine as soon as I get my hands on one (should I start a crowdfunding campaign? 😂).

Although the results are very clear in my case, feel free to comment, to let me know any other packages you would rather use to import data into R and whether you were able to reproduce the same type of results in your environment or not.