How to process a DataFrame with billions of rows in seconds
By: Roman Orac
Tags: Pandas, Analytics, Data Science, Big Data, Python

How to process a DataFrame with billions of rows in seconds

Yet another Python library for Data Analysis that You Should Know About — and no, I am not talking about Spark or Dask.

Photo by Christian Englmeier on Unsplash

Big Data Analysis in Python is having its renaissance. It all started with NumPy, which is also one of the building blocks behind the tool I am presenting in this article. In 2006, Big Data was a topic that was slowly gaining traction, especially with the release of Hadoop. Pandas followed soon after with its DataFrames. 2014 was the year when Big Data became mainstream, also Apache Spark was released that year. In 2018 came Dask and other libraries for data analytics in Python.

Each month I find a new tool, which I am eager to learn. It is a worthy investment of spending an hour or two on tutorials as it can save you a lot of time in the long run. It's also important to keep in touch with the latest tech. While you might expect that this article will be about Dask you are wrong. I found another Python library for data analysis that you should know about.

Here are a few links that might interest you:

Some of the links above are affiliate links and if you go through them to make a purchase I’ll earn a commission. Keep in mind that I link courses because of their quality and not because of the commission I receive from your purchases.

In case you’ve missed my article about Dask:

Big Data Analysis in Python is having its renaissance

Meet Vaex

Vaex is a high-performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. It can calculate basic statistics for more than a billion rows per second. It supports multiple visualizations allowing interactive exploration of big data.

Vaex vs Dask

Vaex vs Dask logos

Vaex is not similar to Dask but is similar to Dask DataFrames, which are built on top pandas DataFrames. This means that Dask inherits pandas issues, like high memory usage. This is not the case Vaex.

Vaex doesn’t make DataFrame copies so it can process bigger DataFrame on machines with less main memory.

Both Vaex and Dask use lazy processing. The only difference is that Vaex calculates the field when needed, wherewith Dask we need to explicitly use the compute function.

Data needs to be in HDF5 or Apache Arrow format to take full advantage of Vaex.

Install Vaex

To install Vaex is as simple as installing any other Python package:

pip install vaex

Let's take it to a test drive

Photo by Eugene Chystiakov on Unsplash

Let’s create a pandas DataFrame with 1 billion rows and 1000 columns to create a big data file.

import vaex

import pandas as pd

import numpy as np n_rows = 1000000

n_cols = 1000

df = pd.DataFrame(np.random.randint(0, 100, size=(n_rows, n_cols)), columns=['col%d' % i for i in range(n_cols)]) df.head()

How much main memory does this DataFrame uses?

df.info(memory_usage='deep')

Let’s save it to disk so that we can read it later with Vaex.

file_path = 'big_file.csv'

df.to_csv(file_path, index=False)

By reading the whole CSV directly with Vaex we wouldn’t gain much as the speed would be similar to pandas. Both need approximately 85 seconds on my laptop.

We need to convert the CSV to HDF5 (the Hierarchical Data Format version 5) to see the benefit with Vaex. Vaex has a function for conversion, which even supports files bigger than the main memory by converting smaller chunks.

If we cannot open a bigger file with pandas, because of memory constraints, we can covert it to HDF5 and process it with Vaex.

dv = vaex.from_csv(file_path, convert=True, chunk_size=5_000_000)

This function automatically created an HDF5 file and persist it to disk.

Let’s check the dv type.

type(dv) # output

vaex.hdf5.dataset.Hdf5MemoryMapped

Now, let’s read the 7.5 GB dataset with Vaex — We wouldn’t need to read it as we already have it in dv variable above. This is just to test the speed.

dv = vaex.open('big_file.csv.hdf5')

Vaex needed less than 1 second to execute the command above. But Vaex didn’t actually read the file, because of lazy loading, right?

Let’s force to read it by calculating a sum of col1.

suma = dv.col1.sum()

suma # array(49486599)

I was really surprised by this one. Vaex needed less than 1 seconds to calculate the sum. How is that possible? By using memory mapping.

Plotting

Vaex is also fast when plotting data. It has special plotting functions plot1d, plot2d and plot2d_contour.

dv.plot1d(dv.col2, figsize=(14, 7))

Virtual columns

Vaex creates a virtual column when adding a new column, — a column that doesn’t take the main memory as it is computed on the fly.

dv['col1_plus_col2'] = dv.col1 + dv.col2

dv['col1_plus_col2']

Efficient filtering

Vaex won’t create DataFrame copies when filtering data, which is more memory efficient.

dvv = dv[dv.col1 > 90]

Aggregations

Aggregation works slightly differently than in pandas, but more importantly, they are blazingly fast.

Let’s calculate a binary column where col1 ≥ 50.

dv['col1_50'] = dv.col1 >= 50

Vaex combines group by and aggregation in a single command. The command below groups data by the “col1_50” column and calculates the sum of the col3 column.

dv_group = dv.groupby(dv['col1_50'], agg=vaex.agg.sum(dv['col3']))

dv_group

Joins

Vaex joins data without making memory copies, which can save the main memory. Pandas users will be familiar with the join function:

dv_join = dv.join(dv_group, on=’col1_50')

Before you go

Follow me on Twitter, where I regularly tweet about Data Science and Machine Learning.