Modin — Speed up your Pandas Notebooks, Scripts, and Libraries | by Satyam Kumar | Towards Data Science

Pandas is an open-source library providing high performance, easy to use data structures and data analytics tools for the Python language. Pandas is a fundamentally high-level building block for doing real-world data analysis in Python. It has the broader goal of becoming the most powerful and flexible open-source data analysis and manipulation tool.
Pandas library has 40+ API’s, some of the most used are pandas DataFrame, pandas Series, pandas plotting, pandas Input/Output. Important highlights of pandas libraries are:
A fast and efficient DataFrame object for data manipulation with integrated indexing
Tools for reading and writing data
Intelligent data alignment and integrated handling of missing data
Intelligent label-based slicing, fancy indexing, and subsetting of large datasets
Aggregating or transforming data with a powerful group by engine.
High-performance merging and joining of data sets
Time series-functionality
Pandas library is highly used in academics and commercial domains including Finance, Neuroscience, Economics, Statistics, Advertising, Web Analytics, and more.
Pandas being such a great library that enables data scientists/data analysts to interact with the data. But it takes time to interact with large size data. Pandas mainly use 1 core of CPU to process instructions and return the result. Here Modin comes in the picture, that uses distributed computing to accelerate Pandas workflow.
Modin is an early access multi-processing data frame library with identical API to pandas, that bridges the gap between data analytics and large size data. It provides an effortless way to speed up pandas notebooks, scripts, and libraries and manages data partitioning and shuffling so that data scientists/data analysts can interact with the data efficiently.
Modin speed-up to 4x on a laptop with 4 cores.
Modin accelerates Pandas queries, only requiring users to change a single line of code in their notebooks. Pandas use just 1 core of CPU for processing instructions and the rest of the cores are not utilized. Whereas, Modin utilizes all the cores of available in the system, to speed up the Pandas workflow.
As discussed above, Modin accelerates Pandas queries by distributed computing. It utilizes all the cores of the system. For a 4 core system, the below image compares the utilization of Pandas and Modin libraries.
On a large server with comparatively, more number of cores compared to a standard Laptop, the performance of Modin exponentially increases. From (Image 1), it is observed it takes few seconds to load 18gb of CSV file.
Modin is open-source and you can find the GitHub repository here. Install Modin from PyPI:
pip install modin
If you don’t have Ray or Dask installed, you will need to install Modin with one of the targets:
# Install Modin dependencies and Ray to run on Raypip install modin[ray]# Install Modin dependencies and Dask to run on Daskpip install modin[dask]# Install all of the abovepip install modin[all]
Modin will automatically detect which engine you have installed and use for scheduling computation.
Use Modin to accelerate your Pandas queries with just one line of code, that is only by importing the Modin library.
import modin.pandas as pd
Pandas is a massive library with a large number of APIs, some of the famous ones being DataFrame, Series, etc. Pandas DataFrame and Series have 280+ functions each. Modin does not support all the Pandas functions, but it supports the most famous ones.
The developers of Modin figured out the most popular Pandas methods used in Kaggle competitions. A list of the top 20 is mentioned in (Image 4).
They started implementing distributed computing Modin functions in order of their popularity.
The functions that are not implemented in Modin, is defaulted to pandas. In case you try to use a function not distributed in Modin, the library uses the default implementation of that function in Pandas, by converting the Modin data frame to Pandas data frame, performing the operations, then reconverting to the Modin data frame.
You can use Modin in replacement of Pandas, as it parallelizes the implementation and reduces the computation time, even if some pandas API is not parallelized, it is still defaulting to pandas.
Modin uses Ray or Dask to parallelize as much of the pandas API as is possible. It is intended to be used as a drop-in replacement for pandas. In the long-term, Modin is planned to become a DataFrame library that supports the popular APIs (SQL, pandas, etc.) and runs on a variety of compute engines and backends.
[1] Pandas Documentation, (08 Sep 2020): https://pandas.pydata.org/
[2] Modin Documentation: https://modin.readthedocs.io/en/latest/
[3] PyData YouTube video “Scaling Interactive Pandas Workflows with Modin — Devin Petersohn”, (01 Feb 2019)
Thank You for Reading