What is .npy files and why you should use them... | by Peter Nistrup | Towards Data Science

First of all thanks a lot to my followers for sticking with me these last few months, I’ve been terribly busy and haven’t had a lot of time to pump out articles. I’ve decided that a partial remedy for this is to make some shorter and easier to digest articles which will be easier to produce! Therefore this is my first attempt at making a short-and-to-the-point article. I hope you find it useful!
Make sure to follow my profile if you enjoy this article and want to see more!
Reading a 10 million data-point file from storage:
This is by far the fastest method of loading in data.
Pandas proved that .csv files aren’t useless, but it’s still lacking in speed.
This is just so slow compared to the others that it’s painful.
If you’ve ever done any kind of data processing in Python you’ve undoubtedly come across Numpy and Pandas. These are the giants of Data Science in Python and stand as the foundation for a lot of other packages, namely Numpy provides the fundamental objects used by the likes of Scikit-Learn and Tensorflow!
So why am I talking about these packages and why Numpy in particular? Well as you might know, the “industry standard” with regard to data-files is .csv files. Now while convenient, these files are highly un-optimized when compared to the alternatives, like the .npy files provides as courtesy of Numpy.
“Who cares, let’s see the code and evidence!”
Right, on with the show!
Let’s start off by simply creating 10 million random points of data and save it as Comma Separated Values:
Now let’s load this by traditional means and do a simple reshaping of the data:
This is the output I get:
Now, you might think that the reshaping is preventing a faster load but even if we don’t do any reshaping we get a similar time!
np.save('data.npy', data_array)
Right, that was pretty easy right? So now that we have our array in .npy format let’s see how fast we can read it in:
Which gives me the following output:
A LOT faster, also notice that we didn’t need to reshape the data since that information was contained in the .npy file.
Another “minor” feature of using .npy files is the reduced storage the file occupies. In this case it’s more than a 50% reduction in size. This can wary a lot though but in general the .npy files are more storage friendly.
“What about Pandas and their .csv handling?”
First let’s create a proper .csv file for Pandas to read, this would be the most likely real-life scenario.
data = pd.DataFrame(data_array)data.to_csv('data.csv', index = None)
This simply saves the ‘data_array’ we created before as a standard .csv file without index.
Now let’s load it and see what kind of time we get:
Which gives me the following output:
Now you might think this is cheating because we’re also loading into a Pandas DataFrame, but it turns out that the time-loss for that is negligible, if we read in like this:
data_array = np.load('data.npy')data = pd.DataFrame(data_array)
And time it we get the following:
You’re probably used to loading and saving data as .csv but the next time you do a data-science project try getting into the habit of loading and saving to .npy files instead! It’ll save you a lot of downtime and annoyance when you’re waiting for the kernel to load your file!
I hope this very short piece was helpful to you!
If you want to see and learn more, be sure to follow me on Medium🔍 and Twitter 🐦