How to Speedup Pandas with with One-Line change using Modin ? - GeeksforGeeks

15 Mar, 2021
In this article, we are going to see how to increase the speed of computation of the pandas using modin library. Modin is a python library very similar to pandas (almost identical in terms of syntax) capable of handling a huge dataset that cannot fit into RAM in one go. Pandas are good enough in terms of speed of execution for datasets in size of in MB’s and few GB’s but when we are dealing with really large datasets speed to process the data becomes the bottleneck.
Pandas library was designed to work on single-core and therefore with modern age compute power every personal laptop comes with now at least 2 cores and Modin just exploits this opportunity of executing the operations on all available cores thus speeding up the whole process.
To install Modin and all it’s dependencies use any of the below pip commands.
pip install modin[ray] 
Or,
pip install modin[dask] 
Or,
pip install modin[all] 
To limit the number of CPUs to use we can add the below 2 lines of code in your script
import os

# this specifies the number of
# CPUs to use. 
os.environ["MODIN_CPUS"] = "2"
Example 1: Dataframe Append Operation:
Append() operations are very common in pandas and in the code below here we have demonstrated this by running it 10 times using both pandas and Modin and timed it against each other to see the speedup difference. Clearly, Modin beats pandas as it uses all the cores available on my system. Also using the time module to measure the operations speed to compare with each other, and it turns out that Modin is 25x Times faster than pandas in this case.
Code:
Python3
import pandas as pdimport modin.pandas as mpdimport time  start = time.time()  # Creating a Custom Dataframedata = {'Name': ['Tom', 'nick', 'krish', 'jack',                 'ash', 'singh', 'shilpa', 'nav'],                  'Age': [20, 21, 19, 18, 6, 12, 18, 20]}  df = pd.DataFrame(data)  # Appending the dataframe to itself 10 times.for _ in range(10):    df = df.append(df)      end = time.time()print(f"Pandas Appending Time :{end-start}")  start = time.time()modin_df = mpd.DataFrame(data)  # Appending the dataframe to itself 10 times.for _ in range(10):    modin_df = modin_df.append(modin_df)      end = time.time()print(f"Modin Appending Time :{end-start}")
Output:
Pandas Appending Time :0.682852745056152
Modin Appending Time :0.027661800384521484
Example 2: Modin is 4.4x Times faster than pandas.
Here we are using a CSV file of size 602 MB which can be downloaded from this link. Also renamed the file as demo.csv to keep it short. In the code below here we used fillna() method which goes through the entire DataFrame and fills all NaN values with the desired value in my example it’s 0.
Code:
Python3
import pandas as pdimport modin.pandas as mpd  # Reading demo.csv file into pandas dfdf = pd.read_csv("demo.csv")  s = time.time()df = df.fillna(value=0)  e = time.time()print(f"Pandas fillna Time: {e-s})  # Reading demo.csv file into modin dfmodin_df = mpd.read_csv("demo.csv")s = time.time()        modin_df = modin_df.fillna(value=0)e = time.time()print(f"Modin fillna Time: {e - s})
Output:
Pandas fillna Time: 1.2 sec
Modin fillna Time: 0.27 sec
python-modules
Python-pandas
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
How to Install PIP on Windows ?
How to drop one or multiple columns in Pandas Dataframe
How To Convert Python Dictionary To JSON?
Check if element exists in list in Python
Python | Pandas dataframe.groupby()
Defaultdict in Python
Python | Get unique values from a list
Python Classes and Objects
Python | os.path.join() method
Create a directory in Python