Process Dataset with 200 Million Rows using Vaex | by Satyam Kumar | Towards Data Science

Pandas is one of the most popular libraries used for data science case studies. It is one of the best tools for exploratory data analysis and data wrangling. Pandas works efficiently well with small or medium-size datasets which fit best into the memory. For out of core dataset or large dataset pandas is inefficient to perform operations. One needs to spend a lot of time performing exploratory data analysis for a large size dataset using a pandas data frame.
Here Vaex comes into rescue that has a similar API that of pandas and is very efficient to perform out of memory datasets.
Vaex is a high-performance Python library in replacement to pandas that uses expression system and memory mapping to allow developers to perform operations on out of memory or large-sized datasets on standard machines. It covers some of the pandas' API’s and is focused more on data exploration and visualization.
Vaex is not fully compatible with Pandas API, but most of the data wrangling and exploration functionality is also available with Vaex. Visualization of dataset features is done using histograms, density plots, and 3d volume rendering.
Vaex is very efficient compared to Pandas when compared to perform operations on a large-sized dataset. In the further article, we’ll see benchmark numbers for experiments for the wrangling of large-sized data with Vaex. Some of the key efficiency of Vaex are:
Capable to read about 1.2TB of data instantly on your machine.
Can easily handle and perform operations on over 1Billion rows on your laptop
Capable of speedup string processing 10–1000x compared to pandas.
Vaex can load a very large size dataset (almost 1.2TB) and has the capability to perform exploration and visualization on your machine. Instead of loading the entire data into memory, Vaex just memory maps the data and creates an expression system.
From Vaex Documentation:
Vaex uses memory mapping, zero memory copy policy, and lazy computations for best performance hence no memory is wasted. It actually refers to the same data every time you make any changes to the data frame, it just adds a new state to the expression.
I have generated an artificial dataset having 200 million rows and 4 columns (‘id’, ‘name’, ‘x’, ‘y’) with ‘timestamp’ as the index. The total size of the dataset is 12GB.
timestamp: index, 1s time frequency from [2014-09-01, 2020-12-31]id: intname: stringx: floaty: float
Some popular data explorations experiments are performed for 200 million rows dataset on a windows OS with 8GB of RAM:
Read Data
Data Shape
Data describe
Value Counts
Group by column and aggregation
10th percentile computation
Visualizing a column
Apply function
Adding a new column
Filter data frame
The experiment was designed in a way that follows best practices for each tool — this is using binary format HDF5 for Vaex. Need to convert the CSV file to HDF5 format so that Vaex can perform its best. Vaex required 33minutes to convert 2313 partitions of the CSV file to its HDF5 format.
Now to read the HDF5 data from disk:
df = vaex.open("200M_data_hdf5/analysis_*.hdf5")
Vaex requires 6 minutes to read the entire dataset.
To compute the number of rows in the dataset using Vaex requires no time (0 ns). The entire data has around 200 million rows.
To generate descriptive statistics including the central tendency, dispersion, and shape of a dataset’s distribution, excluding NaN values using .describe() function.
Vaex took around 15 minutes to compute the descriptive statistics of each column.
To compute the frequency distribution of the categorical column ‘name’ using the function .value_counts() in Vaex data frame.
Vaex took around 3.5 minutes to return with the frequency distribution of the ‘name’ column.
Similar to pandas API, Vaex also comes up with a function to compute grouping and aggregation. The below command groups the ‘name’ column and aggregates the mean of column ‘x’.
Vaex took around 2.5 minutes to compute the above grouping and aggregation command.
To compute grouping for the ‘id’ column and mean aggregation for two columns ‘x’ and ‘y’.
Vaex took around 11.5 minutes to compute the above grouping and aggregation command.
Vaex has percentile_approx function to compute an approximation of a given percentile.
Vaex took 46.8 secs to compute the 10th percentile of the ‘id’ column.
To plot a histogram of large-sized data is problematic as traditional tools for data analysis are not optimized to handle them.
Using plot1d function in Vaex to plot a histogram of numerical vector, it took 3.5mins to return with the plot.
Similar to Pandas API, Vaex has apply function to apply a function along an axis of the DataFrame. Function to return list vowels in name column:
Vaex took almost no time (132 ms), to process 200 million records of name column.
Vaex actually takes no time to add a column to the dataset, as it does not add the new column instantly, rather uses an expression system to generate just the expression of the new column.
To add a new column Vaex took nearly 251ms.
Similar to pandas API, Vaex has a similar concept of selection, to filter the data based on any given condition. Vaex does not instantly filter the data frame, instead generates an expression.
Vaex takes almost no time (273 ms), to apply the filter, and as observed from the above image, the shape of the data frame reduced from 200 million to 98 million.
In this article, we have generated 200 million records of time-series artificial data having 4 columns of the size of nearly 12GB. Using Pandas library it’s impossible to read the dataset and perform exploration and visualization on it.
Vaex data frame can easily read the data and perform the required exploration and visualization. The only requirement on the Vaex data frame, it works well with HDF5 data. So the CSV files need to convert to the required HDF5 format.
Also, most of the popular Pandas API is available in the Vaex library, hence it makes it the most useful library to work with a large-sized dataset.
[1] Vaex Documentation: https://vaex.readthedocs.io/en/latest/
Thank You for Reading