Building a Dynamic data pipeline with Databricks and Azure Data Factory | by Paul Simpson | Towards Data Science

TL;DR A few simple useful techniques that can be applied in Data Factory and Databricks to make your data pipelines a bit more dynamic for reusability. Passing parameters, embedding notebooks, running notebooks on a single job cluster.
-Simple skeletal data pipeline
-Passing pipeline parameters on execution
-Embedding Notebooks
-Passing Data Factory parameters to Databricks notebooks
-Running multiple ephemeral jobs on one job cluster
This section will break down at a high level of basic pipeline
The main idea is to build out a shell pipeline in which we can make any instances of variables parametric. In this instance we look at using a get metadata to return a list of folders, then a foreach to loop over the folders and check for any csv files (*.csv) and then setting a variable to True. Then *if* the condition is true inside the true activities having a Databricks component to execute notebooks.
This goes without saying, completing a pipeline to make sure as many values are parametric as possible. This is so values can be passed to the pipeline at run time or when triggered. Reducing as many hard coded values will cut the amount of changes needed when utilizing the shell pipeline for related other work.
For maintainability reasons keeping re-usable functions in a separate notebook and running them embedded where required. A quick example of this; having a function to trim all columns of any additional white space.
Child Notebook (Contains Functions)
def trim_all_columns(x):    """    Trim white space from ends of each value across all series in   dataframe    """    trim_strings = lambda x: x.strip() if type(x) is str else x    return x.applymap(trim_strings)
Parent Notebook (Calls functions)
%run /Users/..Username../ParentFunctionsdf = trim_all_columns(data)
There is the choice of high concurrency cluster in Databricks or for ephemeral jobs just using job cluster allocation. After creating the connection next step is the component in the workflow. Below we look at utilizing a high-concurrency cluster.
Adjusting base parameter settings here as in fig1 will allow for the Databricks notebook to be able to retrieve these values. This is achieved by using the getArgument(“BlobStore”) function. These parameters can be passed from the parent pipeline. This makes it particularly useful because they can be scheduled to be passed using a trigger.
A crucial part is to creating this connection to the Blob store is the azure-storage library. This will allow us to create a connection to blob, so this library has to be added to the cluster.
The next part will assume that you have created a secret scope for your blob store in databricks CLI. Here you can store SAS URIs for blob store. This may be particularly useful if you are required to have data segregation, and fencing off access to individual containers in an account.
from azure.storage.blob import (BlockBlobService,ContainerPermissions)import pandas as pdfrom importlib import reloadimport sys, re, iotrigger = getArgument("DepartmentName")scope = "" # Insert Secret scope name heredbutils.secrets.list(scope=scope)keyC = trigger+"-home"Secrets = dbutils.secrets.get(scope = scope ,key = keyC)storage_account_name = getArgument("BlobStore")container_name = trigger+"-home-exp"blobService = BlockBlobService(account_name=storage_account_name, account_key=None, sas_token=Secrets[1:])generator = blobService.list_blobs(container_name)for blob in generator:  print(blob.name)blobs = [b for b in generator]
Above is one example of connecting to blob store using a Databricks notebook. Take it with a grain of salt, there are other documented ways of connecting with Scala or pyspark and loading the data into a Spark dataframe rather than a pandas dataframe.
After creating the code block for connection and loading the data into a dataframe. You can now carry out any data manipulation or cleaning before outputting the data into a container.
Last step of this is sanitizing the active processing container and shipping the new file into a blob container of its own or with other collated data.
For efficiency when dealing with jobs smaller in terms of processing work (Not quite big data tasks), dynamically running notebooks on a single job cluster. This option is used if for any particular reason that you would choose not to use a job pool or a high concurrency cluster.
The idea here is you can pass a variable or pipeline parameter to these values. For the simplicity in demonstrating this example I have them hard coded. Where the name dataStructure_*n* defining the name of 4 different notebooks in Databricks.
The code below from the Databricks Notebook will run Notebooks from a list nbl if it finds an argument passed from Data Factory called exists. A use case for this may be that you have 4 different data transformations to apply to different datasets and prefer to keep them fenced.
Embedded Notebooks
nbl = ['dataStructure_1', 'dataStructure_2',        'dataStructure_3', 'dataStructure_4']arg1 = "exists"exitParam =[]def argCheck(list: nbl):    return [getArgument(i) for i in nbl]var_s = argCheck(nbl)def notebookExe(list: nbl):  for x,y in zip(nbl,var_s):    try:      if y == arg1:        output = dbutils.notebook.run("/..NotebookLocation/JobA/"+ x, 60,                             {"key": x})        print(x,y)        print(output)        exitParam.append(output)      else:        pass    except:      passnotebookExe(nbl)dbutils.notebook.exit(exitParam)
Hopefully you may pickup something useful from this, or maybe have some tips for me. Please feel free to reach out.