Running an Apache Beam Data Pipeline on Databricks | Towards Data Science

A brief walk-through on how to execute your Apache Beam Pipeline on Databricks
When we think about data-parallel pipelines, Apache Spark immediately comes to mind, but there are also promising and fresher models able to achieve the same results and performances.
This is the case of Apache Beam, an open source, unified model for defining both batch and streaming data-parallel processing pipelines. It gives the possibility to define data pipelines in a handy way, using as runtime one of its distributed processing back-ends (Apache Apex, Apache Flink, Apache Spark, Google Cloud Dataflow and many others).
Apache Beam’s great capabilities consist in an higher level of abstraction, which can prevent programmers from learning multiple frameworks. Currently, the usage of Apache Beam is mainly restricted to Google Cloud Platform and, in particular, to Google Cloud Dataflow.
However, when it comes to moving to other platforms, it can be tricky to find some useful references and examples that could help us running our Apache Beam pipeline.
That’s why I would like to tell you my experience on how to run an Apache Beam Pipeline on Databricks.
Note: the code of this walk-through is available at this Github repository.
I decided to start off from official Apache Beam’s Wordcount example and change few details in order to execute our pipeline on Databricks.
The official code simply reads a public text file from Google Cloud Storage, performs a word count on the input text and writes the output to a given path. In order to simplify this process, we will replace these operations by simply reading the input text from an in-code mocked string, finally printing the word count results to the standard output.
The input string will be defined as a List:
We then create a simple Beam custom DoFn Transform to print our results:
Our final pipeline will look like this:
We now have a working Beam Pipeline that can be executed in local mode. If you try to run it, you should see this printed to your standard output:
20/06/01 13:14:13 INFO transforms.PrintFN: against: 120/06/01 13:14:13 INFO transforms.PrintFN: and: 120/06/01 13:14:13 INFO transforms.PrintFN: of: 220/06/01 13:14:13 INFO transforms.PrintFN: troubles: 120/06/01 13:14:13 INFO transforms.PrintFN: nobler: 120/06/01 13:14:13 INFO transforms.PrintFN: arrows: 120/06/01 13:14:13 INFO transforms.PrintFN: suffer: 120/06/01 13:14:13 INFO transforms.PrintFN: sea: 120/06/01 13:14:13 INFO transforms.PrintFN: The: 120/06/01 13:14:13 INFO transforms.PrintFN: Or: 120/06/01 13:14:13 INFO transforms.PrintFN: not: 120/06/01 13:14:13 INFO transforms.PrintFN: slings: 120/06/01 13:14:13 INFO transforms.PrintFN: that: 120/06/01 13:14:13 INFO transforms.PrintFN: is: 120/06/01 13:14:13 INFO transforms.PrintFN: arms: 120/06/01 13:14:13 INFO transforms.PrintFN: Whether: 120/06/01 13:14:13 INFO transforms.PrintFN: a: 120/06/01 13:14:13 INFO transforms.PrintFN: fortune: 120/06/01 13:14:13 INFO transforms.PrintFN: take: 120/06/01 13:14:13 INFO transforms.PrintFN: question: 120/06/01 13:14:13 INFO transforms.PrintFN: To: 120/06/01 13:14:13 INFO transforms.PrintFN: mind: 120/06/01 13:14:13 INFO transforms.PrintFN: to: 320/06/01 13:14:13 INFO transforms.PrintFN: outrageous: 120/06/01 13:14:13 INFO transforms.PrintFN: or: 120/06/01 13:14:13 INFO transforms.PrintFN: tis: 120/06/01 13:14:13 INFO transforms.PrintFN: in: 120/06/01 13:14:13 INFO transforms.PrintFN: the: 220/06/01 13:14:13 INFO transforms.PrintFN: be: 2
Now we would like to execute our pipeline on our Databricks instance. To achieve this, we need to modify few more things in our code. First of all, we modify our WordCountOptions, which has to extend the SparkContextOptions class. These Beam options are necessary in order to manipulate Beam’s SparkContext. A Databricks cluster has its own SparkContext which is crucial to retrieve, in order to scale out the application. Once we retrieve the SparkContext we can directly inject it into Beam’s SparkContextOptions as shown below:
With this final version of our Beam code, we are now ready to launch our Databricks workspace in Azure and to proceed by creating a new Job. We package our project into a fat jar (in this example, I will be using the standard maven life cycle to package my application) and we upload it to our Job by clicking on “Upload Jar”.
Notice that if you have any Spark dependencies in your pom.xml file, remember to mark them as “provided”, since our Databricks cluster will provide them to our application through the execution context.
After specifying the main class, it’s essential to use these parameters that will be parsed by SparkContextOptions:
--runner=SparkRunner --usesProvidedSparkContext
Finally, we can setup the cluster that will be associated to our Job by clicking on edit:
In this way, we define a “New Automated Cluster” with 2 workers on Databricks runtime version 6.4. If you prefer, you can also create an “Interactive Cluster”, which gives you more control over cluster’s execution.
Now, we’re good to go! If your job looks similar to the one below, just click on “run now” and wait for its termination.
Please note that if you have written your Beam pipeline in python the procedure to make it work on Databricks should look more or less the same: just remember to inject Databricks’ SparkContext into Beam and execute your Pipeline with the right set of parameters.
I hope you liked my walk-through on how to run an Apache Beam pipeline on Azure Databricks and feel free to contact me if you find out more useful insights on this topic!
Some rights reserved