Data Science and Parallel Computing With Dask | Towards Data Science

In this post we discuss the basics of leveraging Dask in python, use it to execute some simple tasks that are trivial to parallelize (Embarrassingly Parallel), understand some of the most common possible use cases (data munging, data exploration, machine learning) and then touch upon some of the more complex workflows that can be built by combining different ML libraries with Dask. In this post, we’ll focus on running an embarrassingly parallel task with Dask.
What’s parallel computing?
You might have heard the phrase, but what it actually implies at its core other than all the actual complex algorithmic stuff is this: You execute the work you want to do such that multiple steps happen at the same time instead of always waiting for the previous task to finish. The level of difficulty rises with the nature of what you actually want to accomplish.
Operations/tasks which are independent of each other i.e. except for the start and end for each task, there isn’t any data dependency — are usually easiest to parallelize and are termed as embarrassingly parallel, while tasks involving a lot of data transfer/communication between start and end — are difficult to parallelize.
As a data scientist/analyst you don’t always have either the luxury or the skillset required for thinking about how to parallelize your task and implement a solution in some other language from the ground up — e.g. spending weeks, may be months to find out the optimal way to do a group by isn’t what’s expected from you!
What’s Dask?
Dask : A framework for parallelizing your computational tasks in Python
Dask is a library designed to help facilitate :
Data munging with very large datasets, and
Distribution of computation across lots of cores ( local cluster with multiple workers )or physical computers ( cluster with multiple nodes )
It is similar to Apache Spark in the functionality it provides, but it is tightly integrated into numpy and pandas, making it much easier to learn and utilize than spark for users of those libraries.
Parallelizing a task, ANY task is hard if you are the one building the framework from scratch for such a task. There are python modules (e.g. multiprocessing )which help you skip this part and execute and manage ( to some extent ) different threads. Though, even this takes a lot of effort and specific skillset from the user’s perspective. Monitoring, logging etc also usually take a backseat in this scenario.
Dask takes abstracts away all the complex ( though important ) internals and gives you a clean api to parallelize your existing code within python. depending on how you have written your code — you may not even need to modify your existing functions for them to work with Dask!
Let’s get started on some basics of using Dask below and over the next few posts.
Getting Dask :
I would recommend 1 of the following choices :
Install Anaconda for your python environment ( My preferred choice, Dask comes installed with the base setup. You can also create a new environment to keep your Dask environment separate while you are learning, scheduling your jobs etc )
conda create --name dask python=3.7 anaconda dask
Create a new python virtual environment and install Dask within it
Install Dask within your current environment
# Anaconda environmentconda install dask# pip based installationpip install "dask[complete]"
Create your Dask client :
import daskfrom dask.distributed import Client, progresstry:    client.close()    client = Client(threads_per_worker=1,                     n_workers=5,                     memory_limit='2GB')except:    client = Client(threads_per_worker=1,                     n_workers=5,                     memory_limit='2GB')client
Here, we have created a Local cluster, defined how many local workers to run and the resources to allocate for each worker. This client can at most runs 5 tasks at a time (n_workers) and Dask handles all the details of defining individual threads, coordinating between different threads as one task finishes, the next one begins, data transfer from our environment to each worker and back etc
Task: Calculate Mean and Standard Deviation for each category in a given dataset
Step 01: Create a sample simulationIn our case, I have simulated a dataset of k operations and g categories. The output of this simulation is mean and standard deviation for each from the simulation.
def simulate_dataset(i):        g = 10    k = 100000        categories = np.random.choice(a = np.arange(g), size= k)    values = [(j+1)*np.random.random(k//g) for j in range(g) ]    values = np.concatenate(values)        data     = pd.DataFrame({'category':categories,                             'values':values})    data_out = data.groupby('category').apply(lambda df: [                                         df.loc[:,'values'].mean(),                                         df.loc[:,'values'].std()                                         ])return(data_out)
THIS function could be anything, any complex analysis/simulation task that you want — as long as it is self-contained. Each iteration can be run independently and doesn’t require any communication with other threads and only inputs at the beginning of your simulation.
Step 02: Run simulation in a for loop n times
%%timeitresults = {}for i in range(n):        results[i] = simulate_dataset(i)
Step 03: Run simulation with Dask n times
%%timeitresults_interim = {}for i in range(n):    results_interim[i] = delayed(simulate_dataset)(i)results_scheduled = delayed(list)(results_interim)results = results_scheduled.compute()
In this stand alone example, we see a speedup of ~3 times. The speedup is much more apparent when you make the computation/simulation heavier (increase either unique categories or total sample size in our case)
The increase isn’t linear because of the overheads involved in launching and managing different workers. The heavier computation with each worker is, the less impact this overhead has.
Visualizations for progress on Dask dashboard: