Data Processing In Rust With DataFusion (Arrow)
By: Chengzhi Zhao
Tags: Programming, Data, Rust, Data Science, Data Engineering

Data Processing In Rust With DataFusion (Photo By Author)

Rust is the most beloved language, according to StackOverflow, it is on the top of the list for four years! Data processing is getting simpler and faster with a framework like Apache Spark. However, the field of data processing is competitive. DataFusion (part of Arrow now) is one of the initial attempts of bringing data processing to the Rust. If you are interested in learning some aspects of data processing in Rust with DataFusion, I will show some code examples in Rust with DataFusion, as well as compare the query performance between DataFusion and Pandas.

Update: I initially wrote this article in DataFusion version 0.15.0. With release 1.0.0 of DataFusion and Arrow, I have added both the code and benchmark so we can see the improvement.

DataFusion

Andy Grove created DataFusion, and he had some great articles about building modern distributed computing, for example, How To Build A Modern Distributed Compute Platform. The DataFusion project is not for the production environment yet, as Andy mentioned,

“This project is a great way to learn about building a query engine, but this is quite early and not usable for any real-world work just yet.”

The project was donated to the Apache Arrow project in February 2019, and more people start to contribute to the Arrow version of DataFusion.

DataFusion is an in-memory query engine that uses Apache Arrow as the memory model. It supports executing SQL queries against CSV and Parquet files as well as querying directly against in-memory data.

The project description may not deliver too much excitement here, but since the entire project is done in Rust, it provides you ideas about writing your analytics SQL in Rust. Additionally, you can bring DataFusion as a library to your Cargo file for your Rust project easily.

Initial Setup

To test run some code with DataFusion, first, we need to create a new Rust package

cargo new datafusion_test --bin

Then bring DataFusion as a dependency in Cargo.toml file

[dependencies]

arrow = "1.0.0"

datafusion = "1.0.0"

Test Dataset

There are many datasets available online, Kaggle is one of the places that I usually go to and explore the new dataset. We are going to use The Movies Dataset, and the complete version of this dataset is about 676.68 MB. The movie dataset has the following schema

userId: int movieId: int rating: double timestamp: long

To work with CSV file format, DataFusion used to requires us to provide the schema here, as version 1.0.0 introduced schema infer, this is not needed any more.

let schema = Arc::new(Schema::new(vec![ Field::new(“userId”, DataType::UInt32, false), Field::new(“movieId”, DataType::UInt32, false), Field::new(“rating”, DataType::Float64, false), Field::new(“timestamp”, DataType::Int16, false) ]));

Initial Select (version 1.0.0)

As more features introduced in version 1.0.0, DataFusion API brings lots exciting enhancement. You’d see improvements like schema infer, easier to print result and more. The Code is much concise and easy to read.

Code Example for DataFusion 1.0.0

Performance Comparison Between DataFusion vs. Pandas

It won’t be a fair comparison since DataFusion is entirely new and lack lots of optimization. But it’s still interesting to see the current state of DataFusion and compare it with mature data processing package like pandas.

Disclaimer: I am running it on my personal Mac 13 (2 GHz Quad-Core Intel Core i5) to perform the benchmark, the result could be biased. Since the initial benchmark was published in debug mode, noticed the performance is significant different in release mode.

Query 1: SELECT Top 10 Rows

Query: “SELECT userId,movieId,rating FROM ratings LIMIT 10” DataFusion: 0.7s Pandas: 6.15s

DataFusion is running very fast on random access ten rows here. On the other hand, pandas is about 6s slower.

Query 2: Get Each User’s Average Rating

Query: “SELECT userId, AVG(rating) FROM ratings GROUP BY userId” DataFusion: 18.57s Pandas: 6.24s

As Pandas uses NumPy under the hood, it is not surprising to see good performance on the Pandas side. On DataFusion side, though it is slower than Pandas, the performance is also reasonable to perform those types of aggregations.

Query 3: What is the Highest Rating Score

Query: “SELECT MAX(rating) FROM ratings” DataFusion: 15.28s Pandas: 5.97s

As the previous analytics query seems is slow, this query would also be slow on DataFusion side.

Final Thoughts

As we discussed first, DataFusion is an exciting attempt in Rust to get into the competitive data compute market. As the DataFusion project is still an early stage and requires more contributions, it is not surprising to see some slow performance on certain types of queries. Also, as described in the project README, some key features are missing, so you have to be careful about the SQL command you write and double-check to see if it is currently supported or not.

Overall, DataFusion is an attractive beginning with Rust for the data world, and especially it is part of Apache Arrow now, DataFusion can leverage features easily from the Arrow ecosystem. I’d expect to see considerable performance improvement and more supported SQL features in the future version of DataFusion.