Oxidizing your Python with some Rust
By: Bradley Bonitatibus
Tags: Extension, Rust, Python, Benchmark

Python and its ecosystem are filled with a plethora of libraries and packages that make up some of Stack Overflow’s most popular tools & technologies [1]. The ecosystem is thoroughly embedded and the ease of use and learning curve is beginner-friendly. This is Python’s main strength: someone who is curious about the language can pick it up with little to no headache, especially with the vast number of resources to help kick-start their learning. However, with most things in life, the ease of use has its drawbacks.

Figure 1: Energy, Time, and Memory Consumption across programming languages

Making Sense of Energy, Time, and Memory Performance

The figure above displays the energy, time, and memory performance across several programming languages. Python lags in energy, and time performance, but ranks slightly higher in the memory performance category[2]. Although Python is easy to use, its performance is not as strong as many other languages.

These runtime attributes are critical to modern engineering and are almost positively associated with improved software. When software can run faster, with less memory, and use less power (good for the environment when thinking of computing at-scale), it’s almost always better.

On the opposite end of the spectrum, Rust ranks second in energy, and time performance — only trailing behind C — and reasonably well in memory consumption [2]. Rust’s values, from the rust-lang website, give us a sense of what their design goals are: [3]

Performance

Reliability

Productivity

Rust competes with C and C++ with respect to performance. Rust has some very key guarantees that make it reliable. An excerpt from the Rust website states, [3]

“Rust’s rich type system and ownership model guarantee memory-safety and thread-safety — enabling you to eliminate many classes of bugs at compile-time.”

An analysis of chromium , the core of Google Chrome & Chrome OS, found that 70% of their bugs are caused by memory safety issues [4]. Rust eliminates these classes of bugs at compile-time with the help of the Borrow Checker removing this class of bugs at compile time [5]. You can have confidence that a successful compilation will remain reliable and perform well.

Figure 2: Chromium High Severity Bug Classification

Getting Started — Building Extensions for Python

The pyo3 Rust crate (package) lets us build Python extensions that can allow us to build highly efficient, highly performant, and safe code in Rust. At the same time, Python users can utilize these tools without learning a whole new language.

In this example, I will go over a simple example of building a Python extension in Rust. We will be using the maturin build system by pyo3. This build system works seamlessly for building python packages from Rust. To start this tutorial I will be creating the package oxidizer that contains a function that calculates the sum of primes to a given value n . Wherever you see oxidizer , please note you should replace this with your own project / package / module name.

Please install Rust, Python, and maturin before continuing.

New Package

First, we need to create a new Rust library using

cargo new oxidizer --lib

This will generate our Rust library with a ./src directory and some Cargo files. Next, let’s add the pyo3 dependency to our ./Cargo.toml file and specify the crate-type to be a cdylib .

Next, create a pyproject.toml file to specify the build system for our python project.

Next, we’ll need to specify our setup.py file with the following:

Implementation

Here is a trivial implementation of calculating the sum of primes from 0 to n. The key parts are the #[pyfunction] and #[pymodule] macros. These macros are what will be publicly visible to the python package. I will make use of using the primal crate to calculate the primality of a number. Both Rust and Python will be making use of the Miller-Rabin primality test [6].

Now, all we need to do is create the remaining folders and files for maturin . I’ll create the ./oxidizer folder and create the __init__.py file inside with the following content:

from .oxidizer import sum_primes

Now we build our Python package with maturin

maturin build --release

This builds the python wheels into ./target/wheels/ . We can now install this package with:

cd target/wheels && pip install oxidizer-{...}

Alternatively, maturin can install locally for you with the develop command

maturin develop

I highly recommend going through the maturin documentation for an in depth explanation.

Benchmarks

Below I’ve implemented a pytest micro-benchmark that compares the two implementations.

Here are the results.