How to secure your Azure Data Factory pipeline | by René Bremer | Towards Data Science

Azure Data Factory (ADFv2) is a popular tool to orchestrate data ingestion from on-premises to cloud. In every ADFv2 pipeline, security is an important topic. Common security aspects are the following:
Azure Active Directory (AAD) access control to data and endpoints
Managed Identity (MI) to prevent key management processes
Virtual Network (VNET) isolation of data and endpoints
In the remainder of this blog, it is discussed how an ADFv2 pipeline can be secured using AAD, MI, VNETs and firewall rules. For more details on security of Azure Functions, see my other blog. For a solution how to prevent data loss in your Data lake using snapshots and incremental backups, see this blog.
Update 2020–07–26: It is now possible to run ADFv2 Azure hosted Integration Runtime in a Managed VNET . This is in public preview and this blog has not yet been updated yet with this feature.
In this ADFv2 pipeline, data is read from SQLDB, transformed using an Azure Function and written to an ADLS gen2 account. The architecture of the project can be found below.
The Azure services and its usage in this project are described as follows:
SQLDB is used as source system that contains the table data that will be copied.
Azure Data Factory v2 (ADFv2) is used as orchestrator to copy data from source to destination. ADFv2 uses a Self-Hosted Integration Runtime (SHIR) as compute which runs on VMs in a VNET
Azure Function in Python is used to parse data. Function has no access to other Azure resources
Azure Data Lake Store gen2 (ADLS gen2) is used to store the data from 10 SQLDB tables and the metadata file created by the Azure Function
Azure Active Directory (AAD) is Microsoft’s identity and access management service to authenticate and authorize to Azure resources
The following security aspects are part of this project.
AAD access control: SQLDB, ADLS gen 2 and Azure Function only allow the MI of ADFv2 to access the data. This means that no keys need to be stored in ADFv2 or key vaults.
Firewall rules: SQLDB, ADLS gen 2 and Azure Function all have firewall rules in which only the VNET of the SHIR is allowed as inbound network
In the next chapter, the ADFv2 pipeline architecture is realized.
In this chapter, the following steps are executed to create and deploy the ADFv2 pipeline.
3a. Install preliminaries and resources
3b. Create Self-hosted integration runtime in ADFv2
3c. Secure ADLS gen2 account
3d. Secure SQLDB database
3e. Deploy and secure Azure Function
3f. Configure and run ADFv2 pipeline
In this tutorial, deployment of resources will be done using code as much as possible. The following resources need to be installed:
Install Azure CLI
Install Azure PowerShell
Install Visual Studio Code
Install Azure Function Python libraries in Visual Studio Code
After the preliminaries are installed, the basic resources can be installed. Open your Visual Studio Code, create a new terminal session and execute commands below.
# Login with your AzureAD credentialsaz login# set parameters$rg = "<<Resource group>>"$loc = "<<Location, e.g. westeurope>>"$adfv2 = "<<Adfv2 name>>"$sqlserv = "<<Logical sql server>>"$sqldb = "<<SQLDB name>>"$sqluser = "<<SQLDB username>>"$pass = "<<SQLDB password, use https://passwordsgenerator.net/>>"$adls = "<<adls gen 2 account, only alphanumeric>>"$funname = "<<Azure Function name>>"$funstor = "<<Azure Function storage>>"$funplan = "<<Azure Function plan>>"$vnet = "<<Azure VNET name>>"# create resource groupaz group create -n $rg -l $loc# create Azure Data Factory instanceaz resource create --resource-group $rg --name $adfv2 --resource-type "Microsoft.DataFactory/factories" -p {}# create logical SQL server and SQLDBaz sql server create -l $loc -g $rg -n $sqlserv -u sqluser -p $passaz sql db create -g $rg -s $sqlserver -n $sqldb --service-objective Basic --sample-name AdventureWorksLT# create ADLS gen 2 account and containeraz storage account create -n $adls -g $rg -l $loc --sku Standard_LRS --kind StorageV2 --hierarchical-namespace trueaz storage container create --account-name $adls -n "sqldbdata"# create Azure Functionaz storage account create -n $funstor -g $rg --sku Standard_LRSaz appservice plan create -n $funplan -g $rg --sku B1 --is-linuxaz functionapp create -g $rg --os-type Linux --plan $funplan --runtime python --name $funname --storage-account $funstor# create VNETaz network vnet create -g $rg -n $vnet -l $loc --address-prefix 10.100.0.0/16 --subnet-name shir --subnet-prefix 10.100.0.0/24
In this part, the self-hosted integration runtime in ADFv2 is configured. This will be done using VMs running in the VNET that was created earlier. Execute the Azure CLI script below.
# use the same parameters in step 3a, plus additional params below:$shir_rg="<<rg name>>"$shir_name="<<shir name>>"$shir_vm="<<name of VMs on which SHIR runs>>"$shir_admin="<<name of VMs on which SHIR runs>>"$shir_pass="<<VM password, use https://passwordsgenerator.net/>>"az group deployment create -g $shir_rg --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/101-vms-with-selfhost-integration-runtime/azuredeploy.json --parameters existingDataFactoryName=$adfv2 existingDataFactoryResourceGroup=$rg existingDataFactoryVersion=V2 IntegrationRuntimeName=$shir_name NodeCount=2 adminUserName=$shir_admin adminPassword=$shir_pass existingVirtualNetworkName=$vnet existingVnetLocation=$loc existingVnetResourceGroupName=$rg existingSubnetInYourVnet="shir"
The scripts can take 15 minutes before it is finished. After the script is run, you can verify in your ADFv2 instance whether it is succesfully deployed, see also below.
In this part, the ADLS gen2 account will be secured. This is done as follows:
Add RBAC rule that only MI of ADFv2 can access ADLS gen 2
Add firewall rule that only VNET of SHIR can access ADLS gen 2 container
In this case, Azure PowerShell is used to configure the rules using the same variable
# Get params, PowerShell is used here, can be done in VSC terminalSet-AzDataFactoryV2 -ResourceGroupName $rg -Name $adfv2 -Location $l$adfv2_id = Get-AzDataFactoryV2 -ResourceGroupName $rg -Name $adfv2$sub_id = (Get-AzContext).Subscription.id# Give ADFv2 MI RBAC role to ADLS gen 2 accountNew-AzRoleAssignment -ObjectId $adfv2_id.Identity.PrincipalId -RoleDefinitionName "Reader" -Scope  "/subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.Storage/storageAccounts/$adls/blobServices/default"New-AzRoleAssignment -ObjectId $adfv2_id.Identity.PrincipalId -RoleDefinitionName "Storage Blob Data Contributor" -Scope  "/subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.Storage/storageAccounts/$adls/blobServices/default/containers/sqldbdata"# Turn on firewallUpdate-AzStorageAccountNetworkRuleSet -ResourceGroupName $rg -Name $adls -DefaultAction Deny# Set service endpoints for storage and SQL to subnetGet-AzVirtualNetwork -ResourceGroupName $rg -Name $vnet | Set-AzVirtualNetworkSubnetConfig -Name "shir" -AddressPrefix "10.100.0.0/24" -ServiceEndpoint "Microsoft.Storage", "Microsoft.SQL" | Set-AzVirtualNetwork# Add firewall rules$subnet = Get-AzVirtualNetwork -ResourceGroupName $rg -Name $vnet | Get-AzVirtualNetworkSubnetConfig -Name "shir"Add-AzStorageAccountNetworkRule -ResourceGroupName $rg -Name $adls -VirtualNetworkResourceId $subnet.Id
After the last step, access to your storage account from the Azure portal is refused (since it is not part of the VNET). To enable access from the portal, you you need to whitelist your IP address.
In this part, the SQLDB will be secured. This is done as follows:
Add Database rule that only MI of ADFv2 can access SQLDB
Add firewall rule that only VNET of SHIR can access SQLDB
Azure PowerShell is used to configure the rules.
# Configure AAD access to logical SQL serverConnect-AzureAD$aaduser = "<<your aad user email address>>"Set-AzSqlServerActiveDirectoryAdministrator -ResourceGroupName $rg -ServerName $sqlserver -DisplayName $aaduser# log in SQL with AAD (e.g. via portal query editor, SSMS or VSC)# Execute following SQL statementCREATE USER [<<your Data Factory name>>] FROM EXTERNAL PROVIDER;EXEC sp_addrolemember [db_datareader], [<<your Data Factory name>>];# Add firewall rules$subnet = Get-AzVirtualNetwork -ResourceGroupName $rg -Name $vnet | Get-AzVirtualNetworkSubnetConfig -Name "shir"New-AzSqlServerVirtualNetworkRule -ResourceGroupName $rg -ServerName $sqlserver -VirtualNetworkRuleName "shirvnet" -VirtualNetworkSubnetId $subnet.Id
To enable access from the computer for test purposes, you you need to white your IP address.
In this part, the Azure Function will be deployed. This is done as follows:
Deploy source to function
Add firewall rule that only VNET of SHIR can access Azure Function
Add App Registration that only MI of ADFv2 can access Azure Function
First step is to create an Azure Function in python using this quickstart. Subsequently, update the requirement.txt, __init__.py and add cdmschema.py from this git repository. Then publish your function to the $funname that was created in step 3a. To enhance security, add a firewall rule using the following Azure CLI script:
az webapp config access-restriction add -g $rg -n $funname --rule-name "shirvnet" --action Allow --vnet-name $vnet --subnet "shir" --priority 300
Finally, AAD access to the Azure Function can be configured by added an app registration to the Azure Function and only allow MI of ADFv2 to access Azure Function. Refer to this tutorial and this PowerShell script how to do this.
In this part, the ADFv2 pipeline will be configured and run. Go to your Azure Data Factory Instance, select to set up a code repository and import the following GitHub repository rebremer and project adfv2_cdm_metadata, see below.
Only the pipeline BlogADFv2sec is relevant for this blog. Customize the related linked services of this pipeline for ADLS gen2, Azure Function and SQLDB, see also below.
See this link how to be enabled logging in your Azure Function using Applications Insights. An example of logging can be found below.
In the final step, the pipeline can be run in ADFv2. Go to your pipeline and click debug. When everything goes well, all green checks will appear in the output, see below.
In this blog, an architecture is created that deploys a secure ADFv2 pipeline in which data is read from SQLDB, transformed using an Azure Function and written to an ADLS gen2 account. In this, AAD access control, Managed Identities, VNETs and firewall rules are used to secure the pipeline, see also architecture below.
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.