Machine Learning on Azure with automated predictions | by Saurabh Maheshwari | Towards Data Science

I had a recent opportunity to work as a Data Science Intern at a company that used Microsoft Azure as their AI platform. For one of my projects, I had to recommend an optimal strategy (set-points) to run multiple steam boilers in parallel at a manufacturing plant. For example, given certain conditions, run boiler1 at 100% (full capacity), boiler2 at 50% and so on. The goal was to meet the plant steam demand with minimal cumulative fuel input.
Due to the nature of the problem, the boiler set-points were to be updated minute-by-minute, 24/7.
To achieve this, I created an end-to-end machine learning (ML) pipeline on Azure. By pipeline, I mean :
Getting the data from the on-premises server to cloudCreating predictive models using cloud computing and deploying them as an API endpointAutomating predictions every minute, 24/7 and storing them back to the on-premises server (this proved to be tricky)Creating a dashboard for monitoring the performance
Getting the data from the on-premises server to cloud
Creating predictive models using cloud computing and deploying them as an API endpoint
Automating predictions every minute, 24/7 and storing them back to the on-premises server (this proved to be tricky)
Creating a dashboard for monitoring the performance
While building this pipeline, I found Microsoft’s documentation to be spread across several products and less than straightforward, so I thought it would be helpful to share my workflow and provide links to appropriate resources.
The workflow can be summarized in the figure below.
The intent of this blog is not to discuss the ML methodology, but to understand how can one leverage different Microsoft products to create an end-to-end production pipeline.
So, let’s get started.
Before I start building my ML model, I needed to query relevant historical data from the on-premises SQL server and store it on the cloud. Here, Azure Data Factory v2 (ADF) comes in handy. ADF lets you connect to the on-premises database and write t-SQL queries to store data in the Azure Blob storage (blobs). The copy tool (under Move and Transform activity) of ADF lets you copy the required data.
Once you have the required data stored in blob storage, you can use the Machine Learning Service Workspace (ML Services) platform that Azure provides for developing ML models. If you’re familiar with Jupyter Notebook environment, then you’ll be happy to know that ML Services lets you create Notebooks with important libraries pre-installed. So you do not need to worry about setting up the environment.
Building a predictive model requires the following steps:
Create your Azure Machine Learning workspace, designate the size of Virtual Machine (VM) depending on your data size and computation requirements. I worked with a dataset of dimension 3M by 13 and needed to create multiple ML models. So, I decided to use 28 GB VM with 8 cores (STANDARD_D4_V2)Next, create a Jupyter Notebook and connect to the blob storage. Once you connect, you can easily convert blob data into a pandas dataframeCreate an Azure Machine Learning compute target that will run multiple jobs in parallel to speed up training and hyperparameter tuningCreate a training script that would import libraries, take user-defined arguments, perform data transformation (if any), tune hyperparameters, log metric values, etc. Basically, it is the code that runs on the compute target to train your modelCreate an Estimator that contains the location of training script, user-defined argument values, required conda and/or pip libraries, and compute targetConfigure your hyperparameter tuning experiment using a defined hyperparameter search space, early termination policy, primary metric, maximum hyperparameter runs and maximum concurrent jobs. Additionally, provide the estimator (created in above step) that will be called with the sampled hyperparametersSubmit the experiment and visualize the hyperparameter tuning progress in real-timeLastly, the cluster gives back the best model. Retrain it using the complete dataset and register the final model to be used for deploying
Create your Azure Machine Learning workspace, designate the size of Virtual Machine (VM) depending on your data size and computation requirements. I worked with a dataset of dimension 3M by 13 and needed to create multiple ML models. So, I decided to use 28 GB VM with 8 cores (STANDARD_D4_V2)
Next, create a Jupyter Notebook and connect to the blob storage. Once you connect, you can easily convert blob data into a pandas dataframe
Create an Azure Machine Learning compute target that will run multiple jobs in parallel to speed up training and hyperparameter tuning
Create a training script that would import libraries, take user-defined arguments, perform data transformation (if any), tune hyperparameters, log metric values, etc. Basically, it is the code that runs on the compute target to train your model
Create an Estimator that contains the location of training script, user-defined argument values, required conda and/or pip libraries, and compute target
Configure your hyperparameter tuning experiment using a defined hyperparameter search space, early termination policy, primary metric, maximum hyperparameter runs and maximum concurrent jobs. Additionally, provide the estimator (created in above step) that will be called with the sampled hyperparameters
Submit the experiment and visualize the hyperparameter tuning progress in real-time
Lastly, the cluster gives back the best model. Retrain it using the complete dataset and register the final model to be used for deploying
A good tutorial can be found here.
Once the final model is registered with the workspace, we need to deploy it as an API endpoint, which we can query to get predictions almost instantaneously. To create the API, I used Azure Container Instances (ACI).
In addition to the final model, you would need the following to successfully deploy models using ACI:
Entry script: The script passes the data submitted through the deployed web service to the final model. Then, it returns the output to the clientDependencies: Create a .yml file that contains the packages your model needs to make inferencesDeployment configuration: This describes the compute resources for the API, for example, number of cores and memory
Entry script: The script passes the data submitted through the deployed web service to the final model. Then, it returns the output to the client
Dependencies: Create a .yml file that contains the packages your model needs to make inferences
Deployment configuration: This describes the compute resources for the API, for example, number of cores and memory
Once you have the final model, entry script, dependencies, and deployment configuration, you can create a docker image and deploy the model.
This is the interesting part. I needed to submit the predictions to the on-premises server database minute-by-minute, 24/7, so I had to automate the prediction part. For this purpose, I created a Databricks notebook (see code examples below) that fetches the input, posts requests to the API and submits the predictions. The flow can be summarized as follows:
Copy the data for prediction from on-premises server to cloud using ADFRun the databricks notebook and store predictions in blob storage or SQL database, depending on your requirements (I discuss both below)Copy the predictions from blob to on-premises SQL database using ADF
Copy the data for prediction from on-premises server to cloud using ADF
Run the databricks notebook and store predictions in blob storage or SQL database, depending on your requirements (I discuss both below)
Copy the predictions from blob to on-premises SQL database using ADF
To automate this flow every minute, you can schedule a trigger (frequency of triggering, when to start and stop, etc.) in ADF. Follow this for more details on how to add Databricks to ADF.
My Databricks notebook looks like this:
## Connect to the storage account containing input datastorage_account_name = 'storage_account_name' storage_account_access_key = 'access_key' spark.conf.set("fs.azure.account.key."+storage_account_name+".blob.core.windows.net", storage_account_access_key)
This will let you connect to the blob storage. Next, I connect to the specific blob and convert it to a pandas dataframe.
## To read in the blob as spark dataframedf = spark.read.csv(“wasbs://folder@storage_account_name.blob.core.windows.net/blob_name”, header = True)df_panda = df.toPandas()
Now, you have your data in a pandas dataframe. Do the required data transformations, if any. Then, call the API endpoint to get predictions.
## Post requests to API end-point to get predictionsuri = ‘API_end_point’test_samples = json.dumps({‘input data feature’: values})test_samples = bytes(test_samples, encoding=’utf8')fitness_ga = requests.post(uri, test_samples, headers = {‘Content-Type’: ‘application/json’})ng_pred = [fitness_ga.json()]pd_df = spark.createDataFrame(pd.DataFrame(ng_pred))
Here pd_df is the spark dataframe containing the predictions. Finally, you can save these predictions to Azure SQL Server Database or a blob as follows.
Connecting Databricks to Azure SQL Server Database
## Connecting to SQL server database from databricksjdbcHostname = ‘your_sql_server_name.database.windows.net’jdbcDatabase = ‘sql_server_database_name’ jdbcPort = 1433jdbcUsername=’your_sql_server_username’jdbcPassword=’your_sql_server_password’jdbcUrl = “jdbc:sqlserver://{0}:{1};database={2}”.format(jdbcHostname, jdbcPort, jdbcDatabase)connectionProperties = { “user” : jdbcUsername, “password” : jdbcPassword, “driver” : “com.microsoft.sqlserver.jdbc.SQLServerDriver”} ## append spark dataframe to SQL databasemode=’append’pd_df.write.jdbc(url=jdbcUrl, table=”sql_table_name”, mode=mode, properties=connectionProperties)
Connecting Databricks to Azure Blob Storage
## Connecting Databricks to Azure blob storageoutput_container_path = “wasbs://folder@storage_account_name.blob.core.windows.net”output_blob_folder = output_container_path+”/temporary_blob_name” ## overwrite the blob, this creates a bunch of filespd_df.coalesce(1).write.mode(“overwrite”).option(‘header’, ‘true’).format(‘com.databricks.spark.csv’).save(output_blob_folder) ## move the required data to blob where you want predictions finallyfiles = dbutils.fs.ls(output_blob_folder)output_file = [x for x in        files if x.name.startswith(‘part-’)]dbutils.fs.mv(output_file[0].path, output_container_path+’/blob_name’)
You can connect the on-premises database with PowerBI once you start populating it with predictions. PowerBI lets you create awesome visualizations just by dragging and dropping fields from the database table. Once you have the required data in PowerBI, you can calculate and visualize key performance indicators (KPI) to gauge the performance of your model. For example, one of the KPI in my case could be output to input (efficiency) ratio over time. Once created, you can automatically update your dashboard as your database gets updated.
In fact, while analyzing your data, PowerBI could come really handy, saving you time and coding efforts while creating publication-quality interactive graphs.
Cost: There are 2 major costs involved, Virtual Machine (VM) for ML Services and Databricks cluster. Depending on the size of data and complexity of modeling, you must decide the optimum number of VM cores and memory. Similarly, choosing a larger than required Databricks cluster can result in higher costs. Also, make sure to stop your ML Services workspace and Databricks cluster when not in use to save undue additional costsLatency vs complexity of model: In my case, I had multiple ML models and predictions were needed every minute (1 min for getting the data from on-premises server to submitting the predictions back to on-premises server). Thus, I needed to choose a model with minimal prediction time without compromising the prediction accuracy. LightGBM worked well for meWhy Databricks?: I knew that I could automatically trigger a pipeline using ADF, and ADF allowed me to use Databricks notebook as an activity. Since I can write python and spark codes in Databricks, that was a natural path for me to followFrom on-premises to the cloud: It could be time-consuming to query large amounts of data from on-premises SQL database to cloud. Thus, make sure to save the queried data somewhere on the cloud. For example, if you want to retrain your model after 3 months, you would just need to query the last 3 months of additional dataDatabricks to database: I was not able to write predictions to the on-premises database directly from Databricks. But, was able to write the predictions from azure blob storage to on-premises database using ADFOutput error logs: Azure outputs error logs if something goes wrong. For example, if you encounter an error while hyperparameter tuning or deploying the model, reading the error logs could save you a lot of debugging timeXGBoost fans: XGBoost does not come pre-installed with Jupyter Notebooks in ML Services. To get XGBoost, copy-paste the following code in the jupyter notebook and it should work
Cost: There are 2 major costs involved, Virtual Machine (VM) for ML Services and Databricks cluster. Depending on the size of data and complexity of modeling, you must decide the optimum number of VM cores and memory. Similarly, choosing a larger than required Databricks cluster can result in higher costs. Also, make sure to stop your ML Services workspace and Databricks cluster when not in use to save undue additional costs
Latency vs complexity of model: In my case, I had multiple ML models and predictions were needed every minute (1 min for getting the data from on-premises server to submitting the predictions back to on-premises server). Thus, I needed to choose a model with minimal prediction time without compromising the prediction accuracy. LightGBM worked well for me
Why Databricks?: I knew that I could automatically trigger a pipeline using ADF, and ADF allowed me to use Databricks notebook as an activity. Since I can write python and spark codes in Databricks, that was a natural path for me to follow
From on-premises to the cloud: It could be time-consuming to query large amounts of data from on-premises SQL database to cloud. Thus, make sure to save the queried data somewhere on the cloud. For example, if you want to retrain your model after 3 months, you would just need to query the last 3 months of additional data
Databricks to database: I was not able to write predictions to the on-premises database directly from Databricks. But, was able to write the predictions from azure blob storage to on-premises database using ADF
Output error logs: Azure outputs error logs if something goes wrong. For example, if you encounter an error while hyperparameter tuning or deploying the model, reading the error logs could save you a lot of debugging time
XGBoost fans: XGBoost does not come pre-installed with Jupyter Notebooks in ML Services. To get XGBoost, copy-paste the following code in the jupyter notebook and it should work
import sys!conda install — yes — prefix {sys.prefix} py-xgboost!conda install -y anaconda py-xgboost
It could be tough to design such a complicated pipeline from scratch, especially if you are new to Azure. Hopefully, this blog would help you get started. You can create a Microsoft Azure free account and start using most of their services. For more details on how to get started, follow this.
Lastly, I would like to sincerely thank my collaborator Radek Rybicki, who helped me define the project and guided me through different stages of it.
This is my first data science blog, and I am really excited to continue blogging. Please let me know if you have any comments and/or suggestions. We can discuss each aspect of the pipeline in more detail later.