How to create and deploy a Kubeflow Machine Learning Pipeline (Part 1) | by Lak Lakshmanan | Towards Data Science

Google Cloud recently announced an open-source project to simplify the operationalization of machine learning pipelines. In this article, I will walk you through the process of taking an existing real-world TensorFlow model and operationalizing the training, evaluation, deployment, and retraining of that model using Kubeflow Pipelines (KFP in this article).
The complete code for this article is on GitHub.
Note: I wrote this article before Hosted Pipelines were available. A much easier way to create a Kubernetes cluster with pipelines installed is to follow the steps in this README file. If you do that, continue reading at Step #2.
KFP naturally requires a Kubernetes cluster to run on. Use this command to start a Google Kubernetes Engine (GKE) cluster and allow KFP to manage the cluster (create_cluster.sh in the repo):
#!/bin/bash  CLUSTERNAME=mykfpZONE=us-central1-bgcloud config set compute/zone $ZONEgcloud beta container clusters create $CLUSTERNAME \  --cluster-version 1.11.2-gke.18 --enable-autoupgrade \  --zone $ZONE \  --scopes cloud-platform \  --enable-cloud-logging \  --enable-cloud-monitoring \  --machine-type n1-standard-2 \  --num-nodes 4kubectl create clusterrolebinding ml-pipeline-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)
Creating a GKE cluster takes about 3 minutes, so navigate to the GKE section of the GCP console and make sure that the cluster is started and ready.
Once the cluster is up, install ML pipelines on the GKE cluster (2_deploy_kubeflow_pipelines.sh in the repo):
#!/bin/bashPIPELINE_VERSION=0.1.3kubectl create -f https://storage.googleapis.com/ml-pipeline/release/$PIPELINE_VERSION/bootstrapper.yaml
Change the version above to the latest version on the release page. Deploying this package will take several minutes. You can look at the status of the job and wait for the number of successful runs to change to 1. Or you can tell kubectl to wait for the create to finish:
#!/bin/bashjobname=$(kubectl get job | tail -1 | awk '{print $1}')kubectl wait --for=condition=complete --timeout=5m $jobname
While waiting for the software installation to complete, you can continue reading!
There are a couple of ways to author pipelines. The “dev-ops” way is to use Python3 and Docker. A more data-scientist-friendly way is to use Jupyter notebooks. In a later post, I will show you the Jupyter notebook method, but in this post, I will show you the Python3-Docker mechanism. It can be helpful to understand this so that you know what’s happening behind the scenes when you go the Jupyter route.
I will illustrate on a machine learning model to predict the weight of a baby. This model has the following steps: (a) extract data from BigQuery, transform it and write the transformed data to Cloud Storage. (b) Train a TensorFlow Estimator API model and do hyperparameter tuning of the model (c) Once the best learning rate, batch size, etc. are determined, train the model longer and on more data using those parameters (d) Deploy the trained model to Cloud ML Engine.
Here is the Python code that describes the above pipeline (mlp_babyweight.py in the repo):
preprocess = dsl.ContainerOp(  name='preprocess',  image='gcr.io/cloud-training-demos/babyweight-pipeline-bqtocsv:latest',  arguments=[    '--project', project,    '--mode', 'cloud',    '--bucket', bucket  ],  file_outputs={'bucket': '/output.txt'})hparam_train = dsl.ContainerOp(  name='hypertrain',  image='gcr.io/cloud-training-demos/babyweight-pipeline-hypertrain:latest',  arguments=[    preprocess.outputs['bucket']  ],  file_outputs={'jobname': '/output.txt'})train_tuned = dsl.ContainerOp(  name='traintuned',  image='gcr.io/cloud-training-demos/babyweight-pipeline-traintuned-trainer:latest',  arguments=[    hparam_train.outputs['jobname'],    bucket  ],  file_outputs={'train': '/output.txt'})train_tuned.set_memory_request('2G')train_tuned.set_cpu_request('1')deploy_cmle = dsl.ContainerOp(  name='deploycmle',  image='gcr.io/cloud-training-demos/babyweight-pipeline-deploycmle:latest',  arguments=[    train_tuned.outputs['train'],  # modeldir    'babyweight',    'mlp'  ],  file_outputs={    'model': '/model.txt',    'version': '/version.txt'  })
Each of the steps above is a Docker container. The output of a Docker container is the input to another, later step — it’s a directed acyclic graph, that when loaded into the pipelines UI, will look like this:
(The full end-to-end application also includes a fifth step, to deploy a web application that provides a user interface to the model.)
Take a look at the preprocess step:
preprocess = dsl.ContainerOp(  name='preprocess',  image='gcr.io/cloud-training-demos/babyweight-pipeline-bqtocsv:latest',  arguments=[    '--project', project,    '--mode', 'cloud',    '--bucket', bucket  ],  file_outputs={'bucket': '/output.txt'})
Notice that it uses two input parameters — the project and the bucket — and puts its output (the bucket containing the preprocessed data) in /output.txt.
Thus, the second step gets this bucket name using preprocess.outputs[‘bucket’] and puts its output (the jobname of the most performant hyperparameter tuning trial) that is then used by the third step of the pipeline as hparam_train.outputs[‘jobname’].
Pretty straightforward!
Of course, this begs two questions: how does the first step get the project and bucket it needs? Second, how are the individual steps implemented?
The project and bucket are obtained through pipeline parameters:
def train_and_deploy(    project='cloud-training-demos',    bucket='cloud-training-demos-ml'):
Essentially, the project and bucket will be provided by the end-user when the pipeline is run. The UI will be pre-populated with the default values I provided above:
The rest of this section is about how the individual steps are implemented. Again, as I mentioned, I am going to explain the Python3-Docker way of implementing things. In a later post, I will explain the Jupyter way. So, if you are confident that you will never go the Docker route, just skim this section or skip ahead to Section 3.
In every step, you need to create a Docker container. This is essentially a standalone program (bash, Python, C++, whatever) with all its dependencies nicely specified so that it can be run by KFP on the cluster.
In my case, my preprocessing code is a standalone Python program that uses Apache Beam and expects to be run on Cloud Dataflow. All the code is in a single command-line program called transform.py:
My Dockerfile has to specify all the dependencies. Fortunately, KFP has a bunch of sample containers and one of them has all the dependencies I’d need. So, I simply inherit from it. My Dockerfile is, therefore, all of 4 lines:
FROM gcr.io/ml-pipeline/ml-pipeline-dataflow-tft:latestRUN mkdir /babyweightCOPY transform.py /babyweightENTRYPOINT ["python", "/babyweight/transform.py"]
Essentially, I copy transform.py into the Docker container and say that the entrypoint is to execute that file.
I can then build my Docker container and publish it in gcr.io in my project in build.sh:
CONTAINER_NAME=babyweight-pipeline-bqtocsvdocker build -t ${CONTAINER_NAME} .docker tag ${CONTAINER_NAME} gcr.io/${PROJECT_ID}/${CONTAINER_NAME}:${TAG_NAME}docker push gcr.io/${PROJECT_ID}/${CONTAINER_NAME}:${TAG_NAME}
This is, of course, the image_name that I specified for the preprocessing step.
I will do training and hyperparameter tuning on Cloud ML Engine using bash to submit the job using gcloud:
gcloud ml-engine jobs submit training $JOBNAME \  --region=$REGION \  --module-name=trainer.task \  --package-path=${CODEDIR}/babyweight/trainer \  --job-dir=$OUTDIR \  --staging-bucket=gs://$BUCKET \  --scale-tier=STANDARD_1 \  --config=hyperparam.yaml \  --runtime-version=$TFVERSION \  --stream-logs \  -- \  --bucket=${BUCKET} \  --output_dir=${OUTDIR} \  --eval_steps=10 \  --train_examples=20000# write output file for next step in pipelineecho $JOBNAME > /output.txt
Note that I use — stream-logs that the gcloud command waits for completion, and note that I write out the jobname which is part of the contract with the following step in the pipeline code.
This time, I will create my Dockerfile by inheriting from a container that already has gcloud installed, and git clone the repository that contains the actual trainer code:
FROM google/cloud-sdk:latestRUN mkdir -p /babyweight/src && \    cd /babyweight/src && \    git clone https://github.com/GoogleCloudPlatform/training-data-analystCOPY train.sh hyperparam.yaml ./ENTRYPOINT ["bash", "./train.sh"]
The preprocessing and hyperparameter tuning steps above took advantage of managed services. All that KFP was doing was submitting the jobs, and allowing the managed services to do their thing. How about doing something on the Kubernetes cluster that KFP is running on?
Just to mix things up a bit, let me do the next step (training) locally on the GKE cluster. To do that, I can simply run a Docker container that invokes python directly:
NEMBEDS=$(gcloud ml-engine jobs describe $HYPERJOB --format 'value(trainingOutput.trials.hyperparameters.nembeds.slice(0))')TRIALID=$(gcloud ml-engine jobs describe $HYPERJOB --format 'value(trainingOutput.trials.trialId.slice(0))')...OUTDIR=gs://${BUCKET}/babyweight/hyperparam/$TRIALIDpython3 -m trainer.task \  --job-dir=$OUTDIR \  --bucket=${BUCKET} \  --output_dir=${OUTDIR} \  --eval_steps=10 \  --nnsize=$NNSIZE \  --batch_size=$BATCHSIZE \  --nembeds=$NEMBEDS \  --train_examples=200000
There is one wrinkle with performing a job on the cluster itself. How do you know the cluster is not already performing some task that is using up all available memory? The traintuned container operation “reserves” the necessary amount of memory and CPUs. kubeflow will take of scheduling the job only when the necessary resources are available:
train_tuned.set_memory_request('2G')train_tuned.set_cpu_request('1')
Deploying to Cloud ML Engine also uses gcloud, so deploy.sh and build.sh in that directory should look quite familiar:
gcloud ml-engine versions create ${MODEL_VERSION} \       --model ${MODEL_NAME} --origin ${MODEL_LOCATION} \       --runtime-version $TFVERSIONecho $MODEL_NAME > /model.txtecho $MODEL_VERSION > /version.txt
and
FROM google/cloud-sdk:latestRUN mkdir -p /babyweight/src && \    cd /babyweight/src && \    git clone https://github.com/GoogleCloudPlatform/training-data-analystCOPY deploy.sh ./ENTRYPOINT ["bash", "./deploy.sh"]
Just as we trained on the Kubernetes cluster, we can also deploy on Kubernetes itself (see this component for an example).
Now that all the Docker container for all the steps are built, we can submit the pipeline to KFP except ... that KFP asks us to compile our pipeline Python3 file into a domain-specific-language. We do that with a tool called dsl-compile that comes with the Python3 SDK. So, first install that SDK (3_install_sdk.sh):
pip3 install python-dateutil https://storage.googleapis.com/ml-pipeline/release/0.1.2/kfp.tar.gz --upgrade
Then, compile the DSL, using:
python3 mlp_babyweight.py mlp_babyweight.tar.gz
Alternately, you can add the directory containing dsl-compile to your PATH by
export PATH=”$PATH:`python -m site --user-base`/bin
and then invoke the compiler:
dsl-compile --py mlp_babyweight.py --output mlp_babyweight.tar.gz
It is this tar file that you will upload to the ML pipelines UI in Section 3.
The user interface server runs on the GKE cluster in port 80. Do a port-forward so that you can access it from your laptop as port 8085 (4_start_ui.sh):
export NAMESPACE=kubeflowkubectl port-forward -n ${NAMESPACE} $(kubectl get pods -n ${NAMESPACE} --selector=service=ambassador -o jsonpath='{.items[0].metadata.name}') 8085:80
Now, open your browser to http://localhost:8085/pipeline and switch to the Pipelines tab. Then, upload the tar.gz file created in Section 2e as the pipeline.
This simply makes the graph available. You will presumably run this with various combinations of settings. So, start an Experiment to hold all these runs. I named my experiment “blog” and then created a run. I named the run ‘try1’ and set it up with the pipeline that was just uploaded:
The pipeline now starts to run. I can see each step being executed and the logs for each:
The full source code includes a few things that I glossed over. Obviously, the full code didn’t appear fully formed and didn’t work the first time around. Rather than start the pipeline from scratch each time, I set up the pipeline so that I could start at any step (the step I was working on).
However, each step depends on the output of the previous step. How to do this? This was my solution:
 if start_step <= 2:    hparam_train = dsl.ContainerOp(      name='hypertrain',      # image needs to be a compile-time string      image='gcr.io/cloud-training-demos/babyweight-pipeline-hypertrain:latest',      arguments=[        preprocess.outputs['bucket']      ],      file_outputs={'jobname': '/output.txt'}    )  else:    hparam_train = ObjectDict({      'outputs': {        'jobname': 'babyweight_181008_210829'      }    })
Essentially, the container op gets created if the start_step ≤ 2. Otherwise, I simply create a dictionary with a “known” output of the previous step. This way, I can simply set the start_step to be 4 to start with the 4th step, skipping the previous steps. However, later steps will have the inputs they expect from any previous steps.
My experiment consisted of several runs:
For example, try1 failed after successfully running 3 steps, and I could start from step 4. I needed two tries to get step 4 right. Then, I added step 5 (which deploys an AppEngine app to front the web service). Then, I went back and tried different variations of stage 3.
Of course, you are not going to train a model once and let it go forever (are you?). You’ll want to retrain the model once you have more data. In our babyweight model, we might imagine retraining the model once we have one more year of data. The way I handle this in the pipeline is to ask for a startYear as an input parameter:
def train_and_deploy(    project=dsl.PipelineParam(name='project', value='cloud-training-demos'),    bucket=dsl.PipelineParam(name='bucket', value='cloud-training-demos-ml'),    startYear=dsl.PipelineParam(name='startYear', value='2000')):
The preprocessing code in the bqtocsv step extracts only rows at/after the start year:
WHERE year >= start_year
and names the output files so that they don’t clobber earlier output:
os.path.join(OUTPUT_DIR, 'train_{}.csv', start_year)
and the training now happens on a larger dataset because the pattern match is for train*. That’s it!
Of course, there are other ways to handle retraining. We might take the previous model and train only on the newer data. Semantics like this are not hard to incorporate — use naming conventions to help the pipeline do the right thing.
Normally, of course, you don’t push a trained model to production immediately. Instead, you will do some evaluation and maybe A/B testing before switching over all the traffic.
Also, you tend not develop in Docker containers. I lucked out here because my babyweight model was pretty much a Python package and rather convenient to Dockerize. What if you do all your model development in a Jupyter notebook?
How to do evaluation, and how to convert a notebook to a pipeline are topics for another blog post. Watch this space.
The complete code for this article is on GitHub. Clone the repo and follow the steps in the README.
Also, read Amy Unruh’s blog post on getting started with Kubeflow pipelines.