How to Connect a Local or Remote Machine to a Databricks Cluster | Towards Data Science

When you start working with Databricks, you will reach the point that you decide to code outside of the Databricks and remotely connect to its computation power, a.k.a. Databricks Cluster. Why? Mostly because one of the main features of Databricks is its Spark job management that can make your life easy. Using this service, you can submit a series of Spark jobs to a large-scale dataset and get back your results in a matter of seconds; thanks to its Spark engines. In this article, I want to describe how you can configure your local or remote machine to connect to a Databricks Cluster as the first step in this process.
Databricks is an abstract layer sitting on cold cloud infrastructures like AWS and Azure that let you easily manage computation power, data storage, job scheduling, and model management. It provides you a development environment to obtain preliminary results of your data science or AI project. Databricks is powered by Spark, an open-source data-processing engine that is specially designed for large-scale data.
The simplicity that Databricks provides, comes with some limitations. For example, Databricks enables you as a data science newbie to generate results fast since you don’t need to deal with cloud configuration or visualization tools extensively. However, if you want to have control over every aspect of your development, Databricks certainly restricts you.
Each Databricks Cluster must be run with a specialized operating system called Databricks Runtime. You will find many versions of the Databricks Runtime when you configure a cluster. If you want to remotely connect to a Databricks cluster, you must cautiously select the Databricks Runtime version. Why? Because a limited number of Databricks Runtime versions are supported by the databricks-connect client, the Spark client library enabling this remote connection.
Databricks Runtime version must be compatible with the Databricks Connect library.
In other words, you can not remotely connect to clusters whose Databricks Runtime are not supported by the databricks-connect. You can find the updated list of Databricks Runtime versions in the original documentation of Databricks Connect.
docs.databricks.com
After selecting a Databricks Runtime version that is compatible with databricks-connect, you must ensure using the compatible Python version on your machine. As instructed in the original documentation: “The minor version of your client Python installation must be the same as the minor Python version of your Databricks Cluster.”
The Python version of development environment must be compatible with Databricks Runtime version working on the Databricks Cluster.
Let’s say you selected 7.3 LTS for the Databricks Runtime version. In that case, you must install Python 3.7 on your local or remote machine to be compatible with it. You must also install Java Runtime Environment (JRE) 8 on your machine. These are instructed by the Databricks website.
www.oracle.com
As recommended for Python development, it is better to create an isolated virtual environment to ensure having no conflict. You can easily build a virtual environment with a library of your choice such as venv, pipenvor conda. I selected the latest one here.
If you don’t want to create a new virtual environment, make sure to not install (or, uninstall)pyspark on the existing environment. You should uninstall the existing version of pyspark since it conflicts with databricks-connectclient. The databricks-connect has its own methods equivalent to pyspark that makes it run standalone. By the following code, you create a virtual environment with Python 3.7 and a version of databricks-connect.
conda create --name ENVNAME python=3.7conda activate ENVNAMEpip3 uninstall pysparkpip3 install -U databricks-connect==7.3.*
You may not need pip3 uninstall pyspark in the above since the virtual environment is clean. I put it there just for sake of completeness.
If you want to build a docker image with Python 3.7 and Java 8, and a version of databricks-connectyou can use the following Dockerfile.
FROM ubuntu:20.04RUN apt-get update && apt-get -y install sudoRUN sudo apt-get -y install software-properties-common### INSTALL PYTHONRUN sudo apt-get -y install libssl-dev opensslRUN sudo apt-get -y install libreadline-gplv2-dev libffi-devRUN sudo apt-get -y install libncursesw5-dev libsqlite3-devRUN sudo apt-get -y install tk-dev libgdbm-dev libc6-dev libbz2-dev RUN sudo apt-get -y install  wgetRUN apt-get update && apt-get install makeRUN sudo apt-get -y install zlib1g-devRUN apt-get -y install gcc mono-mcs && \    rm -rf /var/lib/apt/lists/*RUN wget https://www.python.org/ftp/python/3.7.10/Python-3.7.10.tgzRUN tar xzvf Python-3.7.10.tgzRUN ./Python-3.7.10/configureRUN sudo make installRUN alias python=python3### INSTALL JAVARUN sudo add-apt-repository ppa:openjdk-r/ppaRUN sudo apt-get install -y openjdk-8-jre### INSTALL DATABRICKS-CONNECTRUN pip3 install --upgrade pipRUN pip3 uninstall pysparkRUN pip3 install -U databricks-connect==7.3.*
Using databricks-connect configure , it is easy to configure the databricks-connect library to connect to a Databricks Cluster. After running this command, it interactively asks you questions about the Host, Token, Org Id, Port, and Cluster ID. For more information, you can check the official documentation below.
docs.databricks.com
However, if you want to automatedly configure the connection properties in the Docker image, you can add the below code to the end of the above Dockerfile.
RUN export DATABRICKS_HOST=XXXXX && \    export DATABRICKS_API_TOKEN=XXXXX && \    export DATABRICKS_ORG_ID=XXXXX && \    export DATABRICKS_PORT=XXXXX && \    export DATABRICKS_CLUSTER_ID=XXXXX && \    echo "{\"host\": \"${DATABRICKS_HOST}\",\"token\": \"${DATABRICKS_API_TOKEN}\",\"cluster_id\":\"${DATABRICKS_CLUSTER_ID}\",\"org_id\": \"${DATABRICKS_ORG_ID}\", \"port\": \"${DATABRICKS_PORT}\" }" >> /root/.databricks-connectENV SPARK_HOME=/usr/local/lib/python3.7/site-packages/pyspark
The above code creates the configuration file artificially and saves it in the proper address. Moreover, it sets the SPARK_HOME address to its correct value. When you run databricks-connect configure , these steps are being executed without your involvement. When you want to automatedly configure the connection properties, you should do it in this way.
Last, but not least, you can check the health of the connection by running databricks-connect test .
If you like this post and want to support me...
Follow me on Medium!
Check out my books on Amazon!
Become a member on Medium!
Connect on Linkedin!
Follow me on Twitter!