How to deploy Azure machine learning models as a secure endpoint | by René Bremer | Towards Data Science

Machine Learning Operations (MLOps) aims to deploy and maintain machine learning models in production. A common artifact of an MLOps pipeline is a realtime scoring endpoint that can be consumed by end user applications. Key is to secure this endpoint using network isolation and authentication. In this blogpost and git repo blog-mlopsapim-git, an MLOps pipeline in Azure is discussed that does the following:
1. Create Azure DevOps project and create Azure ML, AKS and API Management infrastructure
2. Train and create model in Azure ML
3a. Deploy model as docker image on isolated Azure Kubernetes cluster
3b. Expose secure endpoint with Azure API management and Azure AD
See also picture below:
This project is based on great work done by Clemens Siebler which can be found here. In the remaining of this blogpost, the project will be explained in more detail. In the next chapter, it is described how the MLOps project is setup.
In this chapter, an MLOps project will be created. In this, the following needs to be done:
2.1 Prerequisites
2.2 Create Azure DevOps project
2.3 Create Service connection
2.4 Substitute variables
The following resources are required in this tutorial:
Azure Account
Azure DevOps
Azure CLI (recommended, also for troubleshooting)
Subsequently, go to the Azure portal and create a resource group in which all Azure resources will be deployed. This can also be done using the following Azure CLI command:
az group create -n <<your resource group>> -l <<your location>>
Azure DevOps is the tool to continuously build, test, and deploy your code to any platform and cloud. Once you created a new project, click on the repository folder and select to import the following repository:
https://github.com/rebremer/blog-mlopsapim-git
See also the picture below.
A Service connection is needed to access the resources in the resource group from Azure DevOps. Go to project settings, service connection and then select Azure Resource Manager, see also picture below.
Select Service Principal Authentication and limit scope to your resource group which you created earlier, see also picture below.
Go to your repo, find pipelines/german-credit-config.yml and adapt the values to point to your workspace, see also below
variables:   #  ml_workspace_connection: '<<service connection created in 2.3>>'    ...  # subscription   ml_subscription_id: '<<your subscription>>'  ...  # apim  ml_apim_name: '<<your apim name'  ml_apim_email: '<<your email address>>'  ml_location: 'westeurope'  ml_tenant_id: '<<your tenant id>>'
All other variables can be substituted, but it this is not required for a succesfull build. The MLOps project is now ready to run.
In this chapter, the project comes to live and an MLOps pipelines will be depoyed generating secure endpoints as the main artifact. In this, the following needs to be done:
3.1 MLOps pipeline description
3.2 Run pipelines
3.3 Test secure endpoint
In this git repo, four pipelines are created that will be run in the next chapter. Pipelines can be described as follows:
pipelines/1-german-credit-infrastructure.yml - Deploys Azure ML workspace with a dataset, private AKS cluster in VNET and Azure Api management
pipelines/2-german-credit-train-and-register.yml - Trains and registers the model automatically
pipelines/3a-german-credit-deploy.yml - Deploys the trained model to AKS cluster creating a private endpoint and key authentication. Key rollover is part of the deployment process.
pipelines/3b-german-credit-apimoperation.yml - Deploys an APIM endpoint exposing the private AKS endpoint. User authentication to APIM is based on Azure AD. On its turn, APIM backend is part of AKS private endpoint and key authentication is used to authenticate to AKS endpoint. Key rollover is part of the deployment process.
In the next chapter, it is described how pipeline 1 can be deployed. Similar process can be followed for the other three pipelines.
Go to your Azure DevOps project, select Pipelines and then click “New pipeline”. Go to the wizard, select the Azure Repos Git and the git repo you created earlier. In the tab configure, choose “Existing Azure Pipelines YAML file” and then pipelines/1-german-credit-infrastructure.ymlthat can be found in the git repo, see also below.
Once the pipeline is created, it is run immediatelly, see below.
After the job is run, all resources are deployed and tests are executed. Subsequently, run all other pipelines that are part of the pipelines folder and that are described in 3.1.
In case all pipelines are run successful, an API is created in you API management pointing to the URL of the private AKS cluster
Subsequently, the score operation validates incoming Azure AD request using JWT token validation and the uses the named value with key to authenticate to AKS private endpoint, see also api_policy_template.xml .
After all pipelines are run, an endpoint is deployed that is exposed by API manager. Authentition requires an Azure bearer token. Typically, a managed identity or a services principal is used to create a bearer token. In the github, an example can be found to test with a bearer token generated from a service principal.
As a quick test, it can also be decided to create a bearer token from a user that is already logged in using the following CLI command.
az account get-access-token --query accessToken --output tsv
The token can then be copied and and used in the following script (or Postman)
token = resp.json()['access_token']#url = 'https://<<your apim>>.azure-api.net/testprivv2/score'#test_data = {  'data': [{    "Age": 20,    "Sex": "male",    "Job": 0,    "Housing": "own",    "Saving accounts": "little",    "Checking account": "little",    "Credit amount": 100,    "Duration": 48,    "Purpose": "radio/TV"  }]}headers = {'Content-Type':'application/json', 'Authorization': 'bearer ' + token}resp = requests.post(url, json=test_data, headers=headers)print("Prediction (good, bad):", resp.text)
Machine Learning Operations (MLOps) aims to deploy and maintain machine learning models in production. A common artifact of an MLOps pipeline is an REST endpoint that is consumed by an end user application. In this blogpost and git repo blog-mlopsapim-git, a project is discussed that 1) all the infrastructure, 2) build and trains a model, 3a) deploys model as endpoint and 3b) secures endpoint, see also architecture below.