Dask: Parallelize Everything
By: Haoran Chen
Tags: Machine Learning, Data Science, Dask, Image Processing, Blog Post

Dask is a one-stop-shop for general big data processing. Whether you are a Python developer looking to speed up existing codebases, a data scientist aiming to extract insight from complex data, or a microbiologist who wants to analyze terabytes of images, Dask got you covered.

Built from the ground up in Python, Dask is truly the only one of its kind. Since it is co-developed with Pandas, scikit-learn and Jupyter teams, it offers many things its competitor PySpark does not have. With Dask, Python developers no longer need to read complicated Java error messages, constantly switch between different syntax, or rewrite the entire codebase to benefit from distributed computing.

Dask also simplifies the big data workflow. Its excellent single-machine performance speeds up the prototyping stage, and leads to faster model deployment. For anyone with experience in Pandas, NumPy or SciPy, parallelizing existing workflow using Dask is painless and only requires small changes. Dask provides the easiest way to deploy not only statistic data analysis, but also machine learning and imaging processing pipeline on clusters.

Dask: Parallelized Python Programming

What is Dask?

A flexible library for parallel computing in Python.

Dask is a Python library leveraging task scheduling for computational problems. Dask provides the most widely-used data structures inherited from Pandas and Numpy, as well as basic parallel computing interfaces based on its self-developed task scheduling system, in order to make large-scale data computing happen.

Why choose Dask?

Code in Python, compute in parallel.

As a data scientist or machine learning engineer, you might face several challenges during your project:

The dataset is extremely large, causing your computer out of memory. You are expecting to switch between a cluster and your home workstation. Multi-processing or multi-thread calculation is what you always dream of. There are other frameworks available, but their APIs are different from what you regularly use. Understanding the task order of computation is necessary. You want Python for everything in your project.

Dask solves ALL OF THESE for you!

When using Dask, you can use all your familiar Python libraries and toolkits, and make the computation running in parallel. Dask customizes some of your most-used data structures to fit large datasets requirements, and supports computing and processing everything in a scheduled order. Whether you are using a laptop, or you own a cluster of 1,000 CPUs, Dask works with the exact same strategy, offering high accuracy and robustness.

Lazy evaluation, a method for minimizing the work done for computing, is used to schedule and optimize all the tasks in the program before getting the final result.

The entire task graph can be easily visualized with one line of code, and it helps you to figure out where to optimize your process from the start to the end.

How to use Dask?

Do whatever you do with Python.

Dask can be installed with either conda or pip.

# Install with conda

conda install -c conda-forge dask # Install with pip

pip install dask

The two most important functions in Dask

To understand and run Dask code, the first two functions you need to know are .visualize() and .compute() .

.visualize() provides the visualization of the task graph, a graph of Python functions and the relationships between each other. Based on these dependencies, the task scheduler in Dask determines how to run functions in parallel. Parameter rankdir="LR" is helpful if the graph is expected to be viewed from left to right.

Dask uses lazy evaluation strategy, so the program only computes the results after .compute() is called. To avoid calling .compute() multiple times to get results in collections, there is also a .compute() function taking multiple collections and returning multiple results.

y = (x + 1).sum()

z = (x + 1).mean()

# compute results of y and z at once

da.compute(y, z)

High-Level Collections

Array

dask.array splits a large array into small blocks of ndarray .

A quick example of visualizing this is to create a 2D array of 100,000 * 100,000 numbers with 10,000 chunks of size 1000 * 1000.

import dask.array as da x = da.random.random((100000, 100000), chunks=(1000, 1000))

The huge array is split into small chunks to minimize RAM usage

The original array is about 80GB! It could be extremely difficult for nparray to handle this on most personal computers. But as we see, it is only 8MB for each chunk, which is much smaller and easier to process.

Most functions you would like to call on nparray are supported here, for example:

y = x + x.T

z = y[::2, 5000:].mean(axis=1)

2. Dataframe

dask.dataframe is implemented based on pandas.dataframe , combining a number of Pandas dataframes by index into a huge dataframe.

A Dask Dataframe containing several Pandas Dataframes ordered by date

The functions of dask.dataframe is a subset copied from pandas :

import dask.dataframe as dd df = dask.datasets.timeseries()

df2 = df[df.y > 0]

df3 = df2.groupby('name').x.std()

df3.head(20)

3. Bag

dask.bag is implemented from python.list , which is designed for simple parallel computing for unstructured or semi-structured datasets, like text files and JSON objects.

You can make function calls like what you do with pyspark.rdd or pytoolz :

import dask.bag as db b = db.read_text('data/*.json').map(json.loads) b.map(lambda record: record['occupation']).take(2)

b.filter(lambda record: record['age'] > 30).take(2)

b.count().compute()

Low-Level Interface

Sometimes you may want to parallelize your algorithm on some small tasks, but Array , DataFrame or Bag are not sufficient to use, or you are aiming to construct some functions by yourself. Then it is the time to use dask.delayed() . It is much simpler to use .delayed() for parallel programming, which is only calling dask.delayed(func)(parameters) .

dask.delayed() works pretty well with loops, for example:

def inc(x):

return x + 1 def mul(x, y):

return x * y def add(x, y):

return x + y results = []

for x in [1, 2, 3, 4]:

a = dask.delayed(inc, pure = True)(1)

b = dask.delayed(mul, pure = True)(2, x)

c = dask.delayed(add, pure = True)(a, b)

results.append(c)



total = dask.delayed(sum, pure = True)(results) total.visualize(rankdir="LR")

Task Graph of a scheduled loop