Automating Machine Learning Workflows with Amazon Glue, Amazon SageMaker and AWS Step Functions Data Science SDK | by Olalekan Elesin | Towards Data Science

Automating machine learning workflows helps to build repeatable and reproducible machine learning models. It is a key step of in putting machine learning projects in production as we want to make sure our models are up-to-date and performant on new data. Amazon Glue, Amazon SageMaker and AWS Step Functions can help automate machine learning workflows from data processing to model deployment in a managed environment.
In this post, I will use the AWS services mentioned above to develop and automate a machine learning workflow with PySpark on AWS Glue for data preparation and processing, and Amazon SageMaker for model training and batch predictions. The purpose of this is to show how engineers, and data scientists can quickly and easily create automated machine learning workflows.
AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics.
Amazon SageMaker is a fully managed service that enables data scientists to build, train, tune, and deploy machine learning models at any scale. This service provides a powerful and scalable compute environment that is also easy to use.
Amazon Step Functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly.
AWS Step Functions Data Science SDK is an open source library that allows data scientists to easily create workflows that process and publish machine learning models using Amazon SageMaker and Amazon Step Functions.
Orchestrating machine learning workflows is pivotal to productionizing ML. To this, I created a a simple end-to-end tutorial with AWS Step Functions and Amazon SageMaker using the AWS Data Science Step Functions SDK. You can find the full examples on the Amazon SageMaker Samples GitHub repository.
In this post, we will follow the steps below to create a machine learning workflow:
Write PySpark script for data processing on AWS Glue
Train a model, using Amazon SageMaker XGboost Algorithm
Deploy the model
Make batch predictions with Amazon SageMaker Batch Transform
To complete this example, I recommend that you launch an Amazon SageMaker Notebook instance by following the steps on the Amazon SageMaker workshop website. We also recommend you read how to create IAM roles and permissions required for running Amazon Glue Jobs.
I assume you are already familiar with writing PySpark jobs. This example touches on the Glue basics, for more complex data transformations kindly read up on Amazon Glue and PySpark. The code snippet below shows simple data transformations in AWS Glue.
After creating the PySpark script, the script has to be uploaded to an S3 location which is accessible to Amazon Glue:
$ aws s3 cp glue-etl-processing.py s3://my-code-bucket/glue/glue-etl-processing.py
Once the above are completed successfully, we will use the AWS Python SDK, Boto3, to create a Glue job. See example below:
In this post, we will use the Amazon SageMaker built-in XGBoost algorithm to train and host a regression model. The dataset is from the Video Game Sales Prediction Amazon SageMaker Workshop example. However, we will not delve into the complete details, since they are available on the Amazon SageMaker Workshop Website.
To prepare the workflow, I will use the AWS Step Functions Data Science SDK which makes it easier and quicker to create step function state machines on AWS. In the steps below, I will show how to create a Glue step for data processing, Amazon SageMaker model training and deployment steps, and Amazon SageMaker Batch Transform step. Finally, we will chain these steps together to create a workflow which is then executed with AWS Step Functions.
Glue Job Step
Amazon SageMaker Estimator and Train Step
Finally, let’s chain the steps together to create a workflow
In this tutorial, we demonstrated how run orchestrate batch inference machine learning learning pipeline with AWS Step Functions SDK, starting from data processing with Amazon Glue for PySpark to model creation and batch inference on Amazon SageMaker.
In this example, I demonstrated how you can create a machine learning workflow using Amazon Step Functions. You could automate model retraining with Amazon CloudWatch Scheduled events. Since, you could be making batch predictions daily, you could prepare your data with AWS Glue, run batch predictions and chain the entire step together with the Step Functions SDK. You could also add an SNSStep so that you get notified via email, slack or SMS on the status of your ML workflows. Many possibilities...
Amazon Step Functions
Amazon Step Functions Developer Guide
AWS Step Functions Data Science SDK
Amazon Glue
Kindly share your thoughts and comments — looking forward to your feedback. You can reach me via email, follow me on Twitter or connect with me on LinkedIn. Can’t wait to hear from you!!