17 Strategies for Dealing with Data, Big Data, and Even Bigger Data
By: Jeff Hale
Tags: Machine Learning, Python, Artificial Intelligence, Technology, Data Science

Python is the most popular language for scientific and numerical computing. Pandas is the most popular for cleaning code and exploratory data analysis.

Using pandas with Python allows you to handle much more data than you could with Microsoft Excel or Google Sheets.

SQL databases are very popular for storing data, but the Python ecosystem has many advantages over SQL when it comes to expressiveness, testing, reproducibility, and the ability to quickly perform data analysis, statistics, and machine learning.

Unfortunately, if you are working locally, the amount of data that pandas can handle is limited by the amount of memory on your machine. And if you’re working in the cloud, more memory costs more money. 💵

Regardless of where you code is running you want operations to happen quickly so you can GSD (Get Stuff Done)! 😀

Things to always do

If you’ve ever heard or seen advice on speeding up code you’ve seen the warning. ⚠️ Don’t prematurely optimize! ⚠️

This is good advice. But it’s also smart to know techniques so you can write clean fast code the first time. 🚀

Getting after it! Source: pixabay.com

The following are three good coding practices for any size dataset.

Avoid nested loops whenever possible. Here’s a brief primer on Big-O notation and algorithm analysis. One for loop nested inside another for loop generally leads to polynomial time calculations. If you have more than a few items to search through, you’ll be waiting for a while. See a nice chart and explanation here. Use list comprehensions (and dict comprehensions) whenever possible in Python. Creating a list on demand is faster than repeatedly loading and appending attributes to a list — hat tip to the Stack Overflow answer here. However, in general, don’t sacrifice clarity for speed, so be careful with nesting list comprehensions. ⚠️ In pandas, use built-in vectorized functions. The principle is same as the one behind list and dict comprehensions. Applying a function to a whole data structure at once is much faster than repeatedly calling a function.

If you find yourself reaching for apply , think about whether you really need to. apply is looping over rows or columns. Vectorized methods are usually faster and less code, so they are a win on multiple fronts. 🚀

Likewise, avoid other pandas Series and DataFrame methods that loop over your data, such as applymap , itterrows , and ittertuples . If it makes sense, use the map or replace methods on a DataFrame instead of any of those other options to save lots of time.

Notice that these suggestions might not hold for very small amounts of data, but in that case, the stakes are low, so who cares. 😉

This brings us to our most important rule

If you can stay in pandas, stay in pandas. 🐼

It’s a happy place. 😀

Don’t worry about these speed and memory issues if you aren’t having problems and you don’t expect your data or memory footprint to balloon.

But at some point storm clouds will gather. ⛈ You’ll encounter a big dataset and then you’ll want to know what to do. Let’s see some tips.

Things to do with pretty big data (roughly millions of rows)

Like millions of grains of sand. Source: pixabay.com

Use a subset of your data to explore, clean, and make a baseline model if you’re doing machine learning. Solve 90% of your problems fast and save time and resources. This technique can help you get a good model so much faster! ⏳ Load only the columns that you need with the usecols argument when reading in your DataFrame. Less data in = win! Use dtypes efficiently. Downcast numeric columns to the smallest dtypes that makes sense with pd.to_numeric() . Convert columns with low cardinality (just a few values) to a categorical dtype. Here’s a pandas guide on efficient dtypes. Parallelize model training in scikit-learn to use more processing cores whenever possible. By default, scikit-learn uses just one of your machine’s cores. Many computers have 4 or more cores. You can use them all for parallelizable tasks by passing the keyword argument n_jobs=-1 . Save pandas DataFrames in feather or pickle formats for faster reading and writing. Hat tip to Martin Skarzynski, who links to evidence and code here. Use pd.eval to speed up pandas operations. Pass the function your usual code in a string. It does the operation much faster. Here’s a chart from tests with a 100 column DataFrame.

Image from this good article on the topic by Tirthajyoti Sarkar

df.query is basically same as pd.eval , but as a DataFrame method instead of a top-level pandas function.

See the docs because there are some gotchas. ⚠️

Pandas is using numexpr under the hood. Numexpr also works with NumPy. Hat tip to Chris Conlan in his book Fast Python for pointing me to @Numexpr. Chris’s book is an excellent read for learning how to speed up your Python code. 👍

Things do with really big data (roughly tens of millions of rows and up)

Even more data! Source: pixabay.com

Use numba. Numba gives you a big speed boost if you’re doing mathematical calcs. Install numba and import it. Then use the @numba.jit decorator function when you need to loop over NumPy arrays and can't use vectorized methods. It only works withNumPy arrays. Use .to_numpy() on a pandas DataFrame to convert it to a NumPy array. Use SciPy sparse matrices when it makes sense. Scikit-learn outputs sparse arrays automatically with some transformers, such as CountVectorizer. When your data is mostly 0s or missing values, you can convert columns to sparse dtypes in pandas. Read more here. Use Dask to parallelize the reading of datasets into pandas in chunks. Dask can also parallelize data operations across multiple machines. It mimics a subset of the pandas and NumPy APIs. Dask-ML is a sister package to parallelize machine learning algorithms across multiple machines. It mimics the scikit-learn API. Dask plays nicely with other popular machine learning libraries such as XGBoost, LightGBM, PyTorch, and TensorFlow. Use PyTorch with or without a GPU. You can get really big speedups by using PyTorch on a GPU, as I found in this article on sorting.

Things to keep an eye on/experiment with for dealing with big data in the future

Keep an eye on them! Source: pixabay.com

The following three packages are bleeding edge as of mid-2020. Expect configuration issues and API changes. If you are working locally on a CPU, these packages are unlikely to fit your needs. But they all look very promising and are worth keeping an eye on. 👀

Do you have access to lots of cpu cores? Does your data have more than 32 columns (necessary as of mid-2020)? Then consider Modin. It mimics a subset of the pandas library to speed up operations on large datasets. It uses Apache Arrow (via Ray) or Dask under the hood. The Dask backend is experimental. Some things weren’t fast in my tests — for example reading in data from NumPy arrays was slow and memory management was an issue. You can use jax in place of NumPy. Jax is an open source google product that’s bleeding edge. It speeds up operations by using five things under the hood: autograd, XLA, JIT, vectorizer, and parallelizer. Jax works on a CPU, GPU, or TPU and might be simpler than using PyTorch or TensorFlow to get speed boosts. Jax is good for deep learning, too. It works withNumPy version but not pandas as of mid-2020. However, you could convert a DataFrame to a TensorFlow or NumPy object and then use jax. Read more here. Rapids cuDF uses Apache Arrow on GPUs with a pandas-like API. It’s an open source Python package from NVIDIA. Rapids plays nicely with Dask, so you could get multiple GPUs processing data in parallel. For the biggest workloads, it should provide a nice boost.

Other stuff to know about code speed and big data

Timing operations

If you want to time an operation in a Jupyter notebook, you can use

%time or %%timeit magic commands. They both work on a single line or an entire code cell.

%time runs once and %%timeit runs the code multiple times (the default is seven). Do check out the docs to see some subtleties.

If you are working in a Python script or notebook you can import the time module, check the time before and after running code, and find the difference.

When testing for time, note that different machines and software versions can cause variation. Also, caching will sometimes mislead if you are doing repeated tests. As with all experimentation, hold everything you can constant. 👍

Storing big data

GitHub’s maximum file size is 100MB. You can use Git Large File Storage extension if you want to version large files with GitHub.

Also, make sure you aren’t auto-uploading files to Dropbox, iCloud, or some other auto-backup service, unless you want to be.

Want to learn more?

The pandas docs have sections on enhancing performance and scaling to large datasets. Some of my ideas are adapted from those sections.

Have other tips? I’d love to hear them over on Twitter. 🎉

Wrap

You’ve seen how to write faster code. You’ve also seen how to deal with big data and really big data. Finally, you saw some new libraries that will likely continue to become more popular for processing big data.

I hope you’ve found this guide to be helpful. If you did, please share it on your favorite social media so other folks can find it, too. 😀

I write about Python, SQL, Docker, and other tech topics. If any of that’s of interest to you, sign up for my mailing list of awesome data science resources and read more to help you grow your skills here. 👍