How to add metadata to your Azure Data Lake | by René Bremer | Towards Data Science

A lot of companies consider setting up an Enterprise Data Lake. The idea is to store data in a centralized repository. In this way, it becomes easier for teams to create business value with data. To prevent that a Data Lake becomes a Data Swamp with untrusted data, metadata is key. In this, the following types of metadata are distinguished:
Business metadata: Data owner, data source, privacy level
Technical metadata: Schema name, table name, field name/type
Operational metadata: Timestamp, size of data, lineage
In the remainder of this blog, it is discussed how an Azure Data Lake can be set up and how metadata is added. For more details how to secure data orchestration in your Azure Data Lake, see my follow-up blog here. For a solution how to prevent data loss in your Data lake using snapshots and incremental backups, see this blog.
In the architecture of this project, the functionality of popular Azure services is combined to create a data lake, where data/metadata is copied from SQLDB to Azure Data Lake Store. The setup of the project can be found below.
The Azure services and its usage in this project are described as follows:
Metadata store is used to store the business metadata. In this project, a blob storage account is used in which the data owner, privacy level of data is stored in a json file.
SQLDB is used as source system that contains the table data that will be copied. It also contains the technical metadata that will be used to describe the data.
Azure Data Factory v2 (ADFv2) is used as orchestrator that copies the data from source to destination. It also gathers the business metadata and technical metadata and produces the operational metadata.
Azure Function Python is used by Azure Data Factory to create a single metadata file adhering to the Common Data Model (CDM) format. In this, it uses the business/technical/operational metadata as input and creates a model.json using the jsonschema of CDM.
Azure Data Lake Store gen2 (ADLS gen2) is used to store the data from 10 SQLDB tables. A File System is created and each table is a root folder in the File System. Next to the data itself, the metadata is stored using the model.json in CDM format created by the Azure Function Python.
In the next three chapters, this architecture is realized. In chapter 3, the prerequisites to create the Data Lake are deployed. Then in chapter 4, a pipeline is deployed and run. Finally, a conclusion is drawn in chapter 5. For more details how to secure an ADFv2 pipeline, see my other blog here.
In this chapter, an Azure Data Lake with metadata is realized. The following steps are executed:
3a. Create SQLDB
3b. Create storage account with metadata
3c. Create ADLS gen2 account
3d. Create Azure Function in python
3e. Create an Azure Data Factory instance
3f. Grant access rights to ADLS gen2 using Managed Identities
Follow the steps in the quickstart how to create is SQLDB database. In this make sure that you select the basic database, see below.
Also, make sure that you install the AdventureWorksLT sample database, see also below.
In this step, a regular Storage account is created. Follow the steps in this quickstart. Make sure that the storage account is stored in the same resource group in which SQLDB is installed. Also, create a container in the storage account and name the container processmetadata. Then download the following business metadata file that contains data owner, privacy level of data using the following link:
https://raw.githubusercontent.com/rebremer/adfv2_cdm_metadata/master/procesdataexample/MetadataProcess.json
Upload the MetadataProcess.json to your storage container, see also below.
In this step, a ADLS gen2 storage account is created. Follow the steps in this quickstart. Make sure that hierarchical namespace is yes, this will create a Hierarchical File System (HFS) in which folders can be created (instead of an object based file system in which all information is stored in the object itself). See also below.
Subsequently, create a File System in ADLS gen 2, see below.
Make sure that the File System is named as sqldbdata.
In this step, an Azure Function in Python is created. Follow the steps in this quickstart that creates a Function in Azure Python. After you published your initial Azure Function Python using an HTTP trigger, update the __init__.py and requirement.txt with the following code and add cdmschema.py to your project next to __init__.py. For links, see below.
https://raw.githubusercontent.com/rebremer/adfv2_cdm_metadata/master/modeljson/modeljsonwrite/__init__.pyhttps://raw.githubusercontent.com/rebremer/adfv2_cdm_metadata/master/modeljson/modeljsonwrite/cdmschema.pyhttps://raw.githubusercontent.com/rebremer/adfv2_cdm_metadata/master/modeljson/requirements.txt
The __init__.py receives metadata input as parameters, creates a model.json object adhering to the CDM jsonschema and then returns the JSON object to ADFv2. ADFv2 will then write the data to the ADLS gen2 File System. Requirement.txt contains the Pypi libraries that the Azure Function in Python needs to functions. Subsequently, publish the code by using the following command.
func azure functionapp publish <APP_NAME> --build remote
When your Azure Function is deployed successfully, you can see the code in your function, see below.
3e. Create an Azure Data Factory instance
In this step, an Azure Function in Python is created. Follow the steps in this quickstart that creates an Azure Data Factory. After the Data Factory is created, find your ADFv2 resource and click on author & monitor. Then select to set up a code repository and import the following GitHub repository rebremer and project adfv2_cdm_metadata, see below.
In the discussed Architecure, ADFv2 is used to copy data from SQLDB to ADLS gen2. Furthermore, business metadata is read from a blob storage and written to ADLS gen 2 using an Azure Python Function. For that purpose, access need to be granted to ADLS gen2, blob storage and SQLDB. In this tutorial, it is explained how RBAC rules can be assigned to storage. Assign the following RBAC rules as follows:
Grant your ADFv2 instance the role “Storage Blob Data Contributor” to you storage account with business metadata (created in 3b)
Grant your ADFv2 instance the role “Storage Blob Data Contributor” to you ADLS gen account (created in 3c)
It can be verified in Access control tab if RBAC are successfully added, see below for ADFv2 instance (3c) and Azure Function (3d).
For simplicity, Managed Identies are not used in the following two steps:
ADFv2 uses SQL authentication to authenticate to SQLDB, however, this could also have been done with MI, refer to this link.
ADFv2 uses a key to authenticate to the Azure Function. Refer to this tutorial how Managed Identity could have been used for this as well
All prerequisites are now deployed. In the next chapter, the pipeline is built an run in ADFv2.
In this chapter, a build and run pipeline in ADFv2 is realized. The following steps are executed:
4a. Set parameters and linked services
4b. Configure applications insights in Azure Function
4c. Run pipeline
In the first step, the pipeline needs to be customized to your storage accounts and SQLDB. Make sure that ADFv2 is running in Data Factory mode and fill in the parameters of the pipeline using your parameters, see below
Subsequently, refer in the linked services to your own storage account for logging, ADLS gen 2, SQLDB and Azure Function. See linked services that need to be changed below:
It is likely that the pipeline does not correctly run the first time you run it. Error messages can be seen in ADFv2, however, ADFv2 also refers to an external Azure Function. To see the logging of the Azure Function, Applications insights need to be enabled. Go the monitor tab and follow the wizard to set up Applications Insights, see below
When an Azure Function is run, all logging.info and raised exceptions are logged, see example below.
In the final step, the pipeline can be run in ADFv2. Go to your pipeline and click debug. When everything goes well, all green checks will appear in the output, see below.
Go to your ADLS gen storage account and verify that the data and metadata is copied. See the example below for table dboversio
When you open the model.json, the metadata can be found using the CDM format, see snippet below.
{  "name": "BlogMetaData",  "description": "Example model.json using CDM json schema",  "version": "1.0",  "annotation": {    "retentionPeriod": 62,    "sourceSystem": "bremerov",    "IngestionDate": "2019-09-09 16:56:00Z",    "privacyLevel": "10"  },  "entities": [    {      "$type": "LocalEntity",      "name": "SalesLTProductModel",      "description": "",       "attributes": [        {          "name": "ProductModelID",          "dataType": "int64"        },...
A lot of companies consider setting up an Enterprise Data Lake. To prevent that a Data Lake becomes a Data Swamp, metadata is key. In this, the following types of metadata are distinguished:
Business metadata: Data owner, data source, privacy level
Technical metadata: Schema name, table name, fields, field type
Operational metadata: Timestamp that ingestion starts/ends
In this blog, it was discussed how to add metadata to your datalake using the architecture below.