Set up a Dask Cluster for Distributed Machine Learning | by Aadarsh Vadakattu | Towards Data Science

Collaboratively written by Pablo Salvador Lopez and Aadarsh Vadakattu
If you know what Dask is capable of and how it can distribute your machine learning processes, you are in the right place! This article will explain how to create a simple SSH cluster, which you can use as a foundation to implement a Dask environment that can distribute your machine learning workflow.
A Dask cluster can be spun up in multiple ways:
Using SSH connectionsOn a Hadoop cluster, essentially running with the power of HDFS and YARNWith the help of Kubernetes and a bunch of Docker containersOn a supercomputerIn a cloud environment like AWS or Azure
Using SSH connections
On a Hadoop cluster, essentially running with the power of HDFS and YARN
With the help of Kubernetes and a bunch of Docker containers
On a supercomputer
In a cloud environment like AWS or Azure
In this article, we will show how we spun up a working Dask test environment by using SSH connections.
Setting up a Dask cluster using SSH connections might seem like the easiest of the bunch, but it is also the most unstable, insecure and tedious to build. Use this approach for testing purposes only and not for somewhere in production. Based on your production environment, you can either try implementing Dask with Hadoop or with Kubernetes or with cloud solutions as they are much more organized and stable.
Dask runs on a Scheduler-Worker network where the scheduler assigns the tasks and the nodes communicate with each other to finish the assigned task. So, every machine in the network must be able to connect and contact each other. Dask sometimes also tries to connect from a source node to the same source node, so we should make sure every connection gateway on all machines, including a connection to itself available.
After you find a bunch of machines that you can use to implement a distributed cluster,
Firewall rules must be set up, to make connections possible
Password-less login must be set up using SSH
A unique mounted directory on all machines must be set up, which will contain our training/testing data to be fed to Dask
Python must be installed on all machines with identical versions of libraries to be installed.
The following test setup is implemented on a bunch of machines running on Red Hat Enterprise Linux 7 (RHEL 7). You can develop or deploy this architecture anywhere by starting a bunch of Virtual Machines either locally or in the cloud or on real machines that are accessible through a network.
Make sure you either turn off the firewall or allow access to all other machines through the firewall.
If you would like to make your test setup secure, allow access from each machine to every other machine, including itself. Assuming you are giving access to the IP 10.10.10.10 from any of your other machines, execute a command like the one below. Here, 24 determines the general TCP port, and 8786 is the port Dask usually needs for its communication. You can open any additional port as needed by using the same commands and changing the port number.
firewall-cmd --permanent --zone=public --add-source=10.10.10.10/24firewall-cmd --permanent --zone=public --add-source=10.10.10.10/8787
You can confirm that your connection is not obstructed by using the ping command. If you do not receive the transmitted packets by ping, there is an issue with your network.
If security is not significant for your test setup, you can disable the firewall to reduce some hassle. To do this, run this command on all machines. This will shut down the firewall on the machine you run.
systemctl stop firewalld
You can check the firewall status by running the following command:
systemctl status firewalld
After all your machines are made discoverable from each other machine, you need to let them connect using SSH so that they can communicate and transfer data.
Usually, SSHing to a machine would need an IP to the system and a password for authentication. This type of login can be done manually, but Dask will not know how and where to enter a password if the destination machine requests it. To overcome this, we can make all the other possible destination machines in the planned cluster as an SSH known host, making the login secure and password-less.
If you have not created an SSH key before, run the ssh-keygen command as shown. You can either put in a passphrase if needed or you can leave it blank.
Now run the following command to copy the SSH ID of your destination machine. Assuming it is 10.1.1.93,
ssh-copy-id 10.1.1.93
Running the above command may show you an output similar to this:
You can also confirm if all possible destinations in your cluster are made as hosts by checking the ~/.ssh/known_hosts file.
Make sure you run this step for each machine you are planning to include in the cluster(Including the IP of the current machine you might be working on. As mentioned, Dask sometimes tries to connect to itself and it fails because there is no SSH authorization to do it.)
After you run these for all machines, you should be able to log in from one machine to the other seamlessly without the use of a password. The following example shows two machines where 10.1.1.107 has no SSH ID copied, while 10.1.1.93 has it’s ID copied on the current working machine.
Now you can set up a place where Dask can access your data. You can do this with the help of an application called SSHFS, which creates mount points on the directory you specify on any machine. When you specify a mount source on a destination machine, this transfers data from source to destination and from destination to the source to keep all files and folders in that specific directory in sync across machines. The good thing with this approach is — this copies and syncs the data from both locations and not only where the mount point source is created.
Dask sometimes gets confused if you place your data to be processed in different directory paths on the machines you have. Thus, make sure you choose a specific directory that is common across machines. A simple example can be /tmp/daskdata/ .
To designate this directory as a mount point,
Create the common directory on all machines
Install SSHFS on all machines using these following commands(If you are on RHEL)
wget https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-12.noarch.rpmyum install fuse-sshfs
Login to all machines and create a mount point at this directory, pointing it to the source machine’s directory. Imagining your source machine to be 10.10.10.10, type in a command like this:
sshfs 10.10.10.10:/tmp/daskdata /tmp/daskdata
The above command assumes you already logged in to a machine and you are pointing your /tmp/daskdata/ on this machine to /tmp/daskdata/ on 10.10.10.10 which can be the source.
After you implement the mount points correctly, make sure it is running by creating a file in your mounted directory in any machine. You can easily create a test file with touch dummy which creates an empty file with the name “dummy”. Creating this file at any one location must copy the file to all locations, which means it should be found in /tmp/daskdata/ directories in all machines.
Finally, you have to make sure all Python libraries that you need are running with the same versions across all clusters. To do this,
Try to run and debug a simple Dask distributed program locally, by using the scheduler machine. If this works, you can identify the specific set of versions of the libraries that Dask needs to run. Often, this step fails as Dask needs other versions of libraries to be installed. Grab the necessary libraries based on the errors this program throws, and try to install them.
from dask.distributed import Clientimport dask# Import a sample datasetdf = dask.datasets.timeseries()# Connect to distributed cluster. To run locally, pass no arguments # in the Client function.client = Client()# This now runs on the distributed systemdf.groupby('name').x.mean().compute()
Here is a link to the dependencies that Dask might require, with the minimum supported versions. Check this and make sure you have satisfied all requirements.
After you have the versions of the libraries, you can gather all installed versions using the pip freeze command and install all these requirements as shown below:
pip freeze > requirements.txt #on a machine where all libraries are #installedpip install -r requirements.txt #on another machine where all #libraries should be installed
Also to reduce hassle, you can always create either a pip or conda environment and export the libraries to other machines by using venv-pack or conda-pack. This duplicates your created environment in the destination machine. There are very good video explanations on how to do this here:
https://conda.github.io/conda-pack/
https://jcrist.github.io/venv-pack/
You are now ready to put data in your mounted folder and write a Dask application!
Pros:
This is a cluster completely built from scratch and can run without installing HDFS, which can be tedious. If you have a small cluster with a bunch of machines, Dask can be quickly tested using this way.
Cons:
The mount point needs to be exactly at the same location on all machines. HDFS gets a big win here, as Dask only needs to know where the file exists in HDFS if it is implemented that way.
Massive files can break disks on each machine and can break SSHFS connections over the network.
SSHFS mounts can be hard to track, and the SSHFS application does not do a good job keeping track of running mount points.
Stay tuned for other ways to run Dask effectively, including Setting up via HDFS and YARN, Setting up using Kubernetes and Setting up in Azure Cloud.
Thanks a lot for reading! Now that you have a cluster set up, you can create and run a Dask program and see all the run statistics in the Dask dashboard!