Machine Learning Pipelines with Google Cloud Platform (Part 1)
By: Samet Ozturk
Tags: Vertex Ai, Machine Learning, Google Cloud Platform, Pipeline

On May 18, Google Cloud announced Vertex AI; a managed machine learning (ML) platform. They claim it requires %80 less code compared to competitive platforms(Google Cloud internal research, May, 2021).

Vertex has couple of components to ease the life of a data scientist/ML engineer like feature store, endpoints to deploy your model easily, managed dataset, notebooks(practically same old Jupyter notebooks) and my favorite part: Pipelines!

First of all, let's look at what a machine learning pipeline is for newcomers. Practically it is a workflow as whole, encapsulate all small steps you do when developing and deploying ML models. It is useful for couple reasons like Training-serving skew, Schema Validation, Data drift or Concept Drift, Continuous training, Model monitoring etc. but this topics deserve their own blog posts. TLDR, you can divide your training process into smaller pieces therefore you can easily debug it when something goes wrong, but also you can run it as a single piece of program. Then you can come back and see what is going on.

Okay, let's stop beating around the bush and write some code. I'll show you a toy example just enough to wrap your head around the whole idea of ML pipeline. Then you can write your own code and do most complicated task like importing data, feature engineering, training and serving.

Firstly, let's import neccessary libraries:

Import necessary libraries

Those modules are looks like our old grumpy pal Kubeflow! Right, but with one simple variation: We have version 2 which allows us create components right out of naked Python functions.

Before we get into fun part, let's set up our GCP environment to ditch the headache we can have later on.

Set up Google Cloud Environment

Even the begining looks complicated but fear not; I'll explain all of it.

In the first line I have typed a simple shell command to retrieve your google cloud default project id. It returns a list so we just get the [0] index.

Next we defined our region and bucket name to store and retrive objects. If you don't know what a bucket is, you can think of it as a directory on your local file system(but remember it is not a directory, google uses GFS which is the ancestor of Hadoop's HDFS).

Then we need some environment variables therefore we use a Jupyter magic. Simply, %env: lists all environment variables/values and we need PATH variable here.

Lastly, we define our pipeline root -a simple bucket.

Let's instantiate kfp client API:

Instantiate KFP Client API

After you deployed a Kubeflow from marketplace(which is a managed kubernetes cluster actually), you can get the link KFP(Kubeflow Pipelines) API needs.

Let's get into the fun part and define our first component.

We use component decorator and define simple python function. Before we get into component's parameters(all of them are optional, so feel free to skip this part) I need to mention each component is actually a container; like docker containers. Therefore we may have component's yaml file(or you can say dockerfile for multiple containers) which is a simple manuscript how to get up and running each container.

Armed with that, "add_component.yaml" is there just in case we need to rebuild the same component. And we specify which python image we will use for this so-called container.

Now, 2nd component:

This time I specify output as a NamedTuple. It comes in handy when you want other components reach those as a metadata handle which is beyond the scope of this post. Think of it as an attribute of the component, we can reach it like we reach a numpy array's shape(my_array.shape). It is not important right now.

Now that we have two components, we can build a simple hello-world pipeline. All we need is a name and a root path to store artifacts. Then you can define a function(we will use it when compiling) to use components we have just created above.

Building pipeline

All steps are pretty much self explanatory; we defined our pipelines name(we will see it on vertex ai platform), a description which is optional and a pipeline root. We have PIPELINE_ROOT variable but to make your life easier, I defined it again. Next, we need to compile the pipeline. Feels like a Keras model, right?

Compiling pipeline

Compile method needs pipeline function we defined above. This function consumes components as you might have recall. Compile method also needs a path to save json object. We can use this object in Vertex pipelines UI to import our pipeline our we can call GCP API make it run for us. Since you are reading this post, I deduce you are cruisin’ for a bruisin’ like myself.

We instantiated GCP AI platform(older version of Vertex, so to speak), we give her what she needs, our project id and region. Then we call "create_run_from_job_spec" function, we used json object as mentioned above.

If we head to GCP Vertex AI dashboard and click Pipelines, select our "hello-world" pipeline we will see the runtime graph. Notice the second component's outputs are there for you as we defined "before" and "after"?

Vertex Pipelines Dashboard UI

That's it, we have just runned our first toy pipeline. If we have trained a model we could have seen it in models and could create an endpoint to use it in production with a simple REST API call.