Getting Started with Databricks — Analyzing COVID-19
By: Renan Ferreira
Tags: Spark, Machine Learning, Databricks, Data Science, Big Data

A typical Data Science workflow is composed of the following steps:

Identify Business Needs -> Data Ingestion -> Data preparation -> Data analysis -> Sharing data insights

Each of these steps requires a set of expertise that can be divided in:

Data engineers: develop, construct, test, and maintain data pipelines

Data scientists: use a variety of methods to extract insights (machine learning models)

Data analysts: take data prepared by data engineers to extract insights from it in the form of graphs, charts, and dashboards (Business Intelligence)

Platform administrators: responsible for managing and supporting data infrastructure (DevOps)

Databricks is a unified platform that provides the tools necessary for each of these jobs. In this article, we will analyze the COVID-19 data of Brazil by creating a data pipeline and indicating the responsibilities of each team member.

Preparation

To complete the next steps, you will need to have access to a Databricks account. The simplest way of starting one is by creating a Databricks Community account at https://community.cloud.databricks.com

Cluster creation (Platform Administrator)

The first step is to configure a cluster. Databricks is a Spark-based platform, which is one of the most popular Big Data analytics framework. Spark is, by nature, a distributed system, and that’s why the cluster creation process involves the selection of virtual machines for the driver and the worker nodes. Basically, the driver is the orchestrator of the cluster, and the worker nodes are responsible for the heavy work.

The platform administrator is responsible for selecting the appropriate family of virtual machines based on the use case, the spark version, number of worker nodes, and autoscaling configurations. ETL processes may need memory-optimized devices, and heavy Machine Learning training processes may run on GPUs, for example.

Cluster creation process

Go to the Clusters page and create a new cluster using the 6.6 ML runtime. If you are using Azure Databricks or AWS, you will need to select the VM family of the driver and the worker nodes. For this tutorial, you can choose the cheapest ones.

Data Ingestion (Data Engineer)

Data ingestion can be a challenging area. Usually, companies have data stored in multiple databases, and nowadays is really common the use of streams of data. Fortunately, Databricks, in conjunction to Spark and Delta Lake, can help us with a simple interface for batch or streaming ETL (extract, transform and load).

In this tutorial, we will start with the most straightforward type of ETL, loading data from a CSV file.

First, we need to download our Dataset. Visit the following URL:

https://github.com/relferreira/databricks-tutorial/tree/master/covid

And download the file caso.csv.gz

Decompress the file, access the Data menu, and click on the Add Data button. Next, upload the CSV file downloaded previously.

After the upload is completed, we will create a new table with the information presented in the dataset. Click the Create Table UI , rename the table to covid , set the first row as the header, and finally, click on the create button.

Table creation process

Data analytics (Data Analyst)

With the table created, we can start to analyze our dataset. First, we need to create a new python Notebook. Go to the

Worspace > Users > YOUR EMAIL

Click on the arrow and create a new Notebook

Notebook creation process

Although been a python notebook, Databricks supports multiple languages inside your notebook. In our case, we will mostly use SparkSQL. If you are familiar with SQL, SparkSQL will feel like home to you.

Let’s start by viewing our new table:

%sql

SELECT * FROM covid

You should see a table like this: