I built a Machine Learning Platform on AWS after passing SAP-C01 exam: Use Cases Layer
By: Salah Rekik
Tags: Machine Learning, Data Science, Technology, AWS, Artificial Intelligence

Let us discuss every step of the Machine Learning workflow and see some challenges that could be faced when dealing with these steps.

4.1 | Data cleansing and preparation

From S3, and using Jupyter Notebooks spawned and managed by Kubeflow, the raw data is loaded and being studied.

Data Cleansing Pattern, by the author

During this phase, a lot could happen to data. For example:

Balancing the data : in some cases, the data is imbalanced. For instance, when dealing with a use case like email classification into SPAM and not SPAM, we could have a non-significant number of SPAMs compared to real emails.

Multiple techniques exist to balance the data like Oversampling , Undersampling , and SMOTE .

: in some cases, the data is imbalanced. For instance, when dealing with a use case like email classification into SPAM and not SPAM, we could have a non-significant number of SPAMs compared to real emails. Multiple techniques exist to balance the data like , , and . Labeling more data: in supervised learning, sometimes the number of labeled examples is just insufficient to train a model. In this case, it is possible to label more data.

Amazon Mechanical Turk could be used to accomplish this task. It is “a web service that provides an on-demand, scalable, human workforce to complete jobs that humans can do better than computers, such as recognizing objects in photographs.“ ⁵ .

in supervised learning, sometimes the number of labeled examples is just insufficient to train a model. In this case, it is possible to label more data. could be used to accomplish this task. It is “a web service that provides an on-demand, scalable, human workforce to complete jobs that humans can do better than computers, such as recognizing objects in photographs.“ . Data Augmentation: when we don’t have enough data to properly train a machine learning model, data augmentation could be a solution as well. It is a technique used to generate more labeled data from the existing examples. When used for images, for example, new images could be generated from one image by flipping that image, applying some filters on that image, or cutting some parts of that image to introduce loss.

The resulting data is stored in the cleansed data storage.

4.2 | Feature engineering

This is one of the most challenging steps. Identifying features needs a lot of brainstorming and imagination. Examples of Uber’s features: ‘restaurant’s average meal preparation time over the last one hour.’, ‘restaurant’s average meal preparation time over the last seven days.’⁶

Feature Engineering Pattern, by the author

As explained in the previous article, two types of features can be computed:

Online features : computed on streaming data and should be accessed with very low latency. These features are stored in DynamoDB. Storing a feature definition means storing its properties: feature’s name, feature’s type(numerical, categorical), min and max value of the feature, etc.

To compute online features, the ML platform uses the stream processing capability of the Data Platform. In other words, this will be a job running on the Data Platform with respect to the integration protocols between the two platforms.

: computed on streaming data and should be accessed with very low latency. These features are stored in DynamoDB. Storing a feature definition means storing its properties: To compute online features, the ML platform uses the capability of the Data Platform. In other words, this will be a job running on the Data Platform with respect to the integration protocols between the two platforms. Offline features: computed on historical data and have no constraint of low latency access. These features will be stored in S3. Like the online features, offline features are computed with the batch processing capability of the Data Platform.

Offline features store must be synchronized with the online feature store regularly to “guarantee that the same data and batch pipeline is used for both training and serving”⁶.

4.3 | Model training

The first step of the training phase consists of splitting the data into three sets: training set for training the model, validation set for hyperparameter optimization, and test set to evaluate the model’s performances.

Model Training Pattern, by the author

Cleansed data along with the features are fed into the Model Training capability of the Data & Model preparation layer. Several models are then applied to data aiming to shortlist a performant model.

As explained in the infrastructure layer article, FSx for Luster is used as a storage to host training data copied from S3. With this storage, the training phase is accelerated as worker nodes could have much better data access performances than accessing the data directly from S3.

The final step of this pattern is to store the trained model in a versioned repository of the trained models’ storage along with its versioned metadata. Model’s metadata could be the name and version of the model, the version of data used to build the model, features used by the model, the model’s output formats, etc.

4.4 | Model evaluation

Using Katib, along with the validation set, different hyperparameter combinations are tested on the chosen ML model to finally select the most performant combination.

The ML model is then tested using the test set to evaluate its performances.

Finally, the resulted model is stored in the trained Models repository along with its metadata.

Model Evaluation Pattern, by the author

4.5 | Model packaging and deployment

Two steps compose this pattern: Model packaging and Model deployment. Each step has its own specifications.

Model Operations Pattern, by the author

First, the trained model should be packaged with the feature extractor and a sample data set which will help validate the model once deployed.

One tricky thing to pay attention to though: in case the ML model is served in real-time, the model’s code must be efficient to respect SLAs. Sometimes, it should be reimplemented in a compiled language like C++ or Java instead of python.

Second, the resulting package is a docker image stored in Elastic Container Registry (ECR).

Finally, the model is deployed as a docker container. Different deployment strategies could be considered:

Single deployment : The already deployed model is simply replaced by the new one. Only one model is deployed at the same time.

This strategy is risky because:

- It will cause a certain downtime during the replacement of the existing model.

- The new model version could contain some bugs and a rollback strategy should be ready to be executed.

: The already deployed model is simply replaced by the new one. Only one model is deployed at the same time. This strategy is risky because: - It will cause a certain downtime during the replacement of the existing model. - The new model version could contain some bugs and a rollback strategy should be ready to be executed. Blue/Green deployment : This technique keeps two versions of the model alive: a new version (Green) and another already running version (Blue). The traffic is then progressively shifted to the new version depending on the new model’s performances.

With this strategy, downtime is minimized, and it is possible to do some A/B testing.

: This technique keeps two versions of the model alive: a new version (Green) and another already running version (Blue). The traffic is then progressively shifted to the new version depending on the new model’s performances. With this strategy, downtime is minimized, and it is possible to do some A/B testing. Multi-Armed Bandit: with this “smart” method, the traffic is gradually directed to the most optimal model.

4.6 | Model serving

Two possible patterns for serving the model:

Real-time serving: When the model is deployed online, it is exposed to the end-user in the form of a REST API. Seldon Core is well integrated with Kubeflow and its Ambassador API Gateway to manage ingress traffic and the API endpoint exposition.

The serving pattern is as follows:

1. As explained earlier, online features store must be synchronized with the offline feature store on a regular basis to guarantee that “the same data is used for training and serving”⁶

2. The user sends its contextualized data to the model using the API endpoint: in addition to the core data, some important information for traceability could be included in the request, like the sending time, the IP address of the sender, etc.

3. The features extractor processes the received data and uses the model’s metadata to identify the right features.

4. The features vector is then constructed and served to the model.

5. In the last step, a prediction is sent back to the user as well as to the prediction store. This prediction store is a must for monitoring the model’s performance.

Real-Time Serving Pattern, by the author

Batch serving: As explained in the previous article, this mode of serving is used when the ML model is applied to a large input size like a week history of songs chosen by the user to recommend the right songs for the next week.

The pattern is as follows:

1. Like in the real-time serving, offline and online are synched.

2. For the batch serving, features are pulled from the S3 storage as there is no constraint for fast access.

3. Finally, the predictions of this model do not go straight to the user but instead, go to another storage for future use.

Batch Serving Pattern, by the author

4.7 | Model Monitoring

As human behavior is unpredictable, the ML model’s performances are prone to degradation. That is why a monitoring pattern should be considered when deploying an ML model in production.

Changes in data are not the only performance degradation origin, other causes could be the unavailability of some worker nodes or a certain service, which could result in the non-respect of SLAs.

Model Monitoring Pattern, by the author

To monitor the model, some metrics should be continuously calculated and injected into the performance metrics storage.

A common practice is to:

Get the data from the predictions store and consider it unlabeled

Construct the right predictions for this data: the right labels could be recovered from the user for example, or by using Amazon Mechanical Turk

Run a job to compare the model’s prediction with these labels.

Prometheus and Grafana can be used to respectively collect and visualize these metrics as well. These two solutions are well integrated with Seldon Core.

Conclusion

In this article, I tried to explain how to land a use case on the machine learning platform.

I started by discussing the challenges faced when a new use case is selected by stakeholders like studying the real nature of the use case and identifying the data sources needed to solve the use case.

Then, with the Framework layer as a foundation, I studied the different patterns used during the lifecycle of a Machine Learning model and gave some known best practices.

I believe a lot of topics encountered during this journey of building a Machine Learning Platform on AWS should be more detailed. One example could be to dive deep into the API Gateway layer to efficiently serve a real-time model. This will be the topic of my upcoming articles.

If you have any questions, please reach out to me on LinkedIn.

[1] https://freeandopenmachinelearning.readthedocs.io/en/latest/ml-business-use.html

[2] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation

[3) https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act

[4] https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard

[5] https://docs.aws.amazon.com/mturk/index.html

[6] https://eng.uber.com/michelangelo-machine-learning-platform/