Building a CI Pipeline with Jupyter Notebooks on GCP
By: Gonzalo Gasca Meza
Tags: Github, TensorFlow, Gcp, Machine Learning, Jupyter Notebook

This week Slava and I presented at Big Things Conference Spain. The title of the talk was the same as the Blog Post, we didn’t want to use the term MLOps with Notebooks, or MLOps with ____ as we think MLOps term is currently an overloaded term. Yes, I say it, is overloaded and hyped, I will explain you why.

Working at Google I have access to very sophisticated software, tools and infrastructure which any Data Scientist, Data Engineer, ML Researcher would definitely love. But not every company is Google, ____, etc…and if you recently started implementing Machine Learning into your business you have realized that doing ML is not an easy task. Let’s talk specifically about the journey from taking a notebook from experimentation to model deployment.

Most of the Machine Learning journeys start with a Jupyter notebook. Here Data Scientists, explore their data with Python and develop models using different Machine Learning Frameworks such as TensorFlow, PyTorch, xgboost, etc. We will define 3 levels in terms of how you can get from experimentation to a model deployed and monitored in a production environment.

Level 0. All you have is a notebook and the rest of the steps, model training, serving and monitoring are done manually.

MLOps Level 0

Level 1. If you add a Git repository and automate the task of executing your notebook after a PR or a commit is submitted, we call this Level 1. Your git repository can be configured with GitHub actions, Jenkins, Cloud Build, etc. to effectively execute your notebook after submission. Some of the challenges here, are the following:

Guarantee that dependencies in your notebooks are the same as the new environment that notebook will be executing. Example: You are using same numpy, pandas, operating system, TensorFlow version, etc. Guarantee that Hardware is also the same or similar to the one you are using. Verify the status of each commit after execution is completed. Is the commit green or red?

MLOps Level 1

When you use AI Platform Notebooks, every time you save a notebook, the environment information in the notebook metadata is saved.

In this case you can see that user was using AI Platform Notebooks TensorFlow 2.3 with GPU support in DLVM version 59.

Notebooks Metadata environment

The above URI corresponds to a Google Cloud Deep Learning Containers. This means that you can create a new notebook using the same environment or execute the notebook file (.ipynb) and effectively forget about using the same versions of libraries as the original environment.

Level 2. Once you are able to automate the release of your Model automatically to production, you will be achieving this level.

MLOps Level 2

Level 3. If you are able to monitor your deployed model via Continuous Monitoring, we will be talking about Level 3. Think about monitoring your predictions, prediction accuracy, response time and model health checks.

MLOps Level 3

But wait…this doesn’t look at all what you have seen in many White papers, books, documentation, etc:

MLOps

Am I giving you a very simplistic view of the world? Am I over simplifying a very complex concept? Maybe…maybe not. The above diagram is definitely needed as a North Star and reference, but with new teams starting to build their Machine Learning infrastructure where you have nothing but some patched and hacky implementation, you need to follow a modular and agile approach, not a Waterfall one and this architecture is definitely a Waterfall approach. Why? Because until you are getting Predictions, you can actually start making or saving money.

Note: If you already have been building MLOps and infra for quite sometime, this will be close to how your actual infrastructure looks like:

MLOps + KubeFlow pipeline

You probably have seen KubeFlow pipelines before, KubeFlow Pipelines have an SDK that can be used from the notebooks therefore, in theory, since you have notebooks, you should easily move from the Level 0 to Level 3 easily, right?

So, I can move from Level 0 to level 3 just by using KubeFlow pipelines? Here is where second assumption often come to play. This assumption is simple: “If tool X can be used from the Notebook it means tool X can be used with the Notebooks”.

Does this make any sense??? I still need to re-write my code completely from scratch, I will need to learn new APIs and probably re-design a lot of code…

So you see the problem now, right?

In order to implement a CI Pipeline with Jupyter Notebooks on GCP let’s define a set of principles:

Follow established software development best practices.

Version control your Notebooks.

Reproducible Notebooks.

Continuous Integration (CI)

Parameterized Notebooks

Continuous Delivery (CD)

Log all experiments automatically

In this blog post I will show you how to build something like this:

The solution involves the following products:

CI Deployment with Notebooks

The flow is the following:

User saves Notebook in AI Platform Notebooks Using Git extension in AI Platform Notebooks, user commits Notebooks changes to GitHub GitHub is configured with a GitHub action workflow GitHub actions workflow uses https://github.com/gclouduniverse/notebook_training to run changed .ipynb notebooks and create a new AI Platform Training job for each notebook

In GitHub we connect to Google Cloud Platform by using this plugin: GoogleCloudPlatform/github-actions/setup-gcloud@master

We define in GitHub Secrets the following Environment variables:

PROJECT_ID: Your Google Cloud Project ID

GCP_SERVICE_ACCOUNT: Your Service Account Key in json format, saved as base64. Example:

base64 service-account.json

5. By using Notebook metadata we are able to extract the correct URI and use the Deep Learning Container version when we start AI Platform Training Job.

6. Notebook executes and when completed, the notebook result is saved in a GCS bucket.

7. If Execution completes successfully, user will see a Green checkbox in GitHub repo.

You can find a sample repo here: https://github.com/gogasca/bigthingsconference2020

As you can see, we have focused on the journey, focus on Levels 0–3 and eventually get to the North Star architecture using the building blocks and principles discussed before.

I will be working the next few weeks to be able to provide a solution to perform Model Deployment directly from the notebook, AI Platform Prediction currently supports Model prediction using custom Containers.

Please let us know what you think and what is what you would like to see.