Scalable Machine Learning with Dask on Google Cloud | by Jiahao Weng | Towards Data Science

Updated on 13 Nov 2021 to reflect latest steps to setup Dask cluster on Google Cloud
Dask has been reviewed by many and compared to various other tools, including Spark, Ray and Vaex. Developed in coordination with other community projects like Numpy, Pandas, and Scikit-Learn, it is definitely a great tool for scaling machine learning.
Hence, the purpose of this article is not to compare the pros and cons of Dask (for that, you can refer to the reference links at the end of this article), but rather to add to existing documentation on the deployment of Dask on cloud and specifically Google Cloud. It definitely also helps that Google Cloud has a free trial for new signups, so you can experiment at no cost.
We list down first the general steps to take before detailing each of the steps with screenshots (feel free to click on each step to navigate). Having a Google Cloud account is the only prerequisite for following this article.
Creating a Kubernetes clusterSetting up HelmDeploying Dask processes and JupyterConnecting to Dask and JupyterConfiguring environmentRemoving your cluster
Creating a Kubernetes cluster
Setting up Helm
Deploying Dask processes and Jupyter
Connecting to Dask and Jupyter
Configuring environment
Removing your cluster
Our first step is to set up a Kubernetes Cluster through Google Kubernetes Engine (GKE).
a) Enable the Kubernetes Engine API after logging in to your Google Cloud console
b) Start Google Cloud Shell
You should see a button similar to the one in red box below in the top right corner of your console page. Click on it and a terminal will pop out. The virtual machine behind this terminal has various tools preinstalled, most importantly kubectl, which is a tool for controlling Kubernetes clusters.
c) Create a managed Kubernetes cluster
Key in the following into Google Cloud Shell to create a managed Kubernetes cluster, replacing <CLUSTERNAME> with a name that can be referred to later.
gcloud container clusters create \  --machine-type n1-standard-4 \  --num-nodes 2 \  --zone us-central1-a \  --cluster-version latest \  <CLUSTERNAME>
A brief description of the parameters in the code above:
machine-type specifies the amount of CPU and RAM for each node. You can choose other types from this list.
num-nodes determines the number of nodes to spin up.
zone refers to the data center zone that your cluster resides in. You can choose somewhere that is not too far away from your users.
While your cluster is initializing, you can also see it spinning up on the Kubernetes Clusters page:
Key in kubernetes clusters in the search bar at the top of your console page.
Select Kubernetes Clusters from the drop down list.
Your cluster with the <CLUSTERNAME> specified can be seen spinning up. Wait till a green tick appears and your cluster is ready.
Alternatively, you can also verify if your cluster is initialized by running:
kubectl get node
When your cluster is deployed, you should see the status as Ready.
d) Provide account permissions to cluster
kubectl create clusterrolebinding cluster-admin-binding \  --clusterrole=cluster-admin \  --user=<GOOGLE-EMAIL-ACCOUNT>
Replace <GOOGLE-EMAIL-ACCOUNT> with the email of the Google account you used to login to Google Cloud.
We will use Helm for installing, upgrading and managing applications on a Kubernetes cluster.
a) Install Helm by running installer script in Google Cloud Shell
curl https://raw.githubusercontent.com/helm/helm/HEAD/scripts/get-helm-3 | bash
b) Verify that Helm is installed properly
helm version
Make sure the version is at least 3.5.
We are almost there... Just a couple more steps before we can start running our machine learning code.
a) Add and update packages information with Dask’s Helm chart repository
helm repo add dask https://helm.dask.org/helm repo update
b) Launch Dask on Kubernetes cluster
helm install my-dask dask/dask --set scheduler.serviceType=LoadBalancer --set jupyter.serviceType=LoadBalancer
This deploys a dask-scheduler, three dask-workers, and also a Jupyter server by default.
Depending on your use case, you may amend the options in the code above:
my-dask is used to reference your Dask setup and you can change to other names.
— set will set the parameters scheduler.serviceType and jupyter.serviceType to the value LoadBalancer. This is necessary to have external IP addresses that we can use to access the Dask dashboard and Jupyter server. Without this option, only cluster IP will be set up by default as mentioned in this Stack Overflow post.
In the previous step, we launched Dask on the cluster. However, it may take a minute to deploy and you can check the status with kubectl after a while:
kubectl get services
Once ready,the external IPs will show up for your Jupyter server (my-dask-jupyter) and Dask scheduler (my-dask-scheduler). If you see <pending> under EXTERNAL-IP, just wait a while more before running the above code again.
Entering the external IP addresses for my-dask-jupyter and my-dask-scheduler in your web browser will allow you to access your Jupyter server and Dask dashboard respectively.
For the Jupyter server, you can log in with default password dask. To change this password, please see the next section.
Congratulations! You can now start running your Dask code :)
Note: If you face 404 error when accessing Jupyter, just click on the Jupyter logo at the top to be directed to the login page.
You may be able to perform some basic Dask code after step 4 but what if you would like to run dask-ml? That is not installed by default. And what if you would like to launch more than the default three workers? How about changing your Jupyter server password?
Hence, we need a way to customize our environment and we can configure it by creating a yaml file. The values in this yaml file will then overwrite the default values of the corresponding parameters in the standard configuration file.
For our illustration, we shall be using the values.yaml below. In general, the configurations are separated into three main sections; one each for the Scheduler, Worker and Jupyter.
To update the configurations, simply perform the following:
In your Google Cloud Shell, run nano values.yaml to create the file values.yaml.
Copy paste the template above (feel free to amend accordingly) and save.
Update your deployment to use this configuration file:
helm upgrade my-dask dask/dask -f values.yaml
Note that you may need to wait a while for the updates to be ready.
Overview of configurations
We also provide below a general description of the commonly used configurations in our template.
a) Install libraries
Under Worker and Jupyter, you can find the sub-section on env. Notice that installation can be via conda or pip and packages are separated by space.
env:  # Environment variables.  - name: EXTRA_CONDA_PACKAGES    value: dask-ml shap -c conda-forge  - name: EXTRA_PIP_PACKAGES    value: dask-lightgbm --upgrade
b) Number of workers
Number of workers can be specified through replicas parameter. In our case, we requested 4 workers.
worker:  replicas: 4  # Number of workers.
c) Resource allocated
Depending on your needs, you can increase the amount of memory or CPUs allocated to your scheduler, workers and/or Jupyter through the resources sub-section.
resources:  limits:    cpu: 1    memory: 4G  requests:    cpu: 1    memory: 4G
c) Jupyter password
The Jupyter password is a hashed value under password parameter. You can change your password by replacing this field.
jupyter:  password: 'sha1:aae8550c0a44:9507d45e087d5ee481a5ce9f4f16f37a0867318c'
To generate the hashed value of your new password,
Launch a terminal in your Jupyter Launcher first.
Run jupyter notebook password in the command-line and key in your new password. The hashed password will be written to a file named jupyter_notebook_config.json.
View and copy the hashed password.
Replace the password field in values.yaml.
To remove your Helm deployment, execute in Google Cloud Shell:
helm del --purge my-dask
Note that this does not destroy the Kubernetes cluster. To do so, you can delete your cluster from the Kubernetes Cluster page.
Through the guide above, we hope that you are now able to deploy Dask on Google Cloud.
Thanks for reading and I hope the article was useful :) Please also feel free to comment with any questions or suggestions that you may have.