Kubeflow: An MLOps Perspective. ML Pipelines and ML Components | by Alex Punnen | Towards Data Science

When designing new technologies, almost always choices are made that would sacrifice some characteristic for some others. And it is important to understand those choices and how they may or may not affect your use case. Usually, such a study is never done when technology or framework is chosen over others; or sometimes there is no choice.
For example, we have well-designed platforms like Kubernetes, that can’t be faulted by either the most technical; or the least. Though of course it is still evolving and there could be better solutions for things like hard multi-tenancy, these do not concern a general user very much now. It has become the de-facto execution engine for any cloud-native application.
This has induced a lot of changes in traditional software development. Instead of compiling the source code for one or more Operating Systems and creating an installer or packager for it; developers need to containerize their code and create the related artefacts — like Services, Config-maps, Secrets; Ingress and similar in a Kubernetes specific markup language (YAML). This has become common practice and routine now.
For a developer; wrapping your micro-service in a container and deploying it in a pod; guarantees that it will perform the same; on any Kubernetes cluster with enough resources. One need not worry about the internal intricacy of pod networking or scheduling or Linux internals related to Control Groups or Namespaces or the like. Of course, that is a simplification; as if you are mounting a host volume and making an assumption of the underlying Linux OS and such, then such could break; but again these are outliers and not the way the technology has to be used.
And that’s a good thing. You wish similar abstraction was there for other technologies; for example, databases; That without having an understanding of the internals of data storage; one can efficiently store and efficiently retrieve data. For example, a DB stored for fast access may not be good at fast updates; and it would be surprising to know that many developers will not be aware; till the feature requirements come mid-way exposing the hidden characteristic of the systems. Basically, the system starts behaving in an unexpected way, when it is used for a use-case it was designed for.
How does all this factor in what we want to discuss?
What we are looking for in Kubeflow, is how non-leaky the abstraction is — to borrow a term from Joel Spolsky.
What we are looking for in Kubeflow is how ‘Cloud Native’ it is — to borrow a widely used or misused cliche. Basically, execution is scheduled in Kubernetes cluster Edge Cluster or Data Center. This box is ticked as Kubeflow is designed for Kubernetes. Unlike say MLFlow — https://mlflow.org/. Other contenders TFX , ZenML as of now
Before we go into Kubeflow, let's check the environment where it fits.
MLOps process should help in ensuring production quality is present and maintained for ML projects.
Is it enough to make a container out of your ML project and deploy in on Kubernetes? Or are there other things that need to be taken care of ? How is CI done? How are tests executed? and test evaluated?
From a heavily over-quoted paper; it seems the ML code has the least significance when deployed in production.
Example from this excellent blog → https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
The authors may have exaggerated to get our attention to the rest of the system that we tend to miss; or maybe not.
And the authors may be taking more about structured data and classical ML methodology that depends on Features; than unstructured data like say for Audio or Video or Image Analytics and Deep Neural Networks
MLOps - like DevOps, seems to be again a term that is used to mean a lot of different things to a lot of different people in a lot of different ways. We need to define what we mean when we say MLOps.
DevOps generally have settled to represent the CI/CD part more -https://aws.amazon.com/devops/what-is-devops/; than the original concept of developers doing end to end development, deployment and operations of their components — the principle of more feedback between developers and their deployments. This could be like proactively checking the error logs from customer deployments for possible problems even before bugs are raised. It is surely more than just CI/CD; though it is also tied more to development and management culture than a technical process.
It is not strange that you have a ‘DevOps’ team now in many companies that does this role, providing the tools and services for CI/CD, and tools and infrastructure for better feedback from deployments — monitoring, logging etc.
MLOps — is similar to DevOps for micro-services. But this has more ML related aspects to it, over just the algorithm, like data and model management, model versioning, model drift etc. At least that’s the promise, and we have so many tools and services in this space, commercial and open-source that it is really easy to get confused and bogged down unless we have some selection criteria.
Some idea you would get by browsing here or here; some authors referring to the proliferation as Cambrian explosion of MLOps projects. However, we could simplify this a bit to a few main stages and a lot of similar actors. Some actors like the Hadoop ecosystem are quite old and ready to step off the stage and we can safely ignore it. Also, similarly simpler technologies like S3 trumps over more complex ones like Ceph.
What Characteristics are we looking for in MLOps specific tools?
Cloud-Native — The system should be designed for container and pod-based execution; and not retrofitted into working with Kubernetes. Example XCom of Airflow may not be the best for pipeline components (containers) to pass data -https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4. Similar MLFlow could be containerized but may not be the best fit for Kubernetes based system.
Open Source — We don’t want to be locked to a particular vendor if that can be helped. This takes out a lot of tools — Data management tools like Pachyderm
Easy for ML Engineers to incorporate — There is a lot of work after creating a prototype Jupyter notebook for an ML problem to production. Still, we do not want our ML Engineers to know more about Kubernetes, Docker files and the like, than their business domain — Data Engineering, Model development and tuning.
Scalable — This is a characteristic of Cloud-Native and Kubernetes based design. Things like distributed training and state-full load-balancing are very complex and we leave such things out for now.
Before we go further it would be good to glance over the main phases of MLOps. The tools change with time and it is more subjective https://valohai.com/blog/the-mlops-stack/.
There needs to be a good Data analysis tool or OLAP database. Something like Apache Presto; that can ingest data from multiple data stores or traditional databases. Something similar to Google BigQuery but in Opensource space. The data scientist or ML engineer can use a simple SQL type query to analyze vast amounts of data; this will be important for algorithm selection and further ML work.
This is a whole infrastructure in itself, but actually can be kept out of MLOps as it comes more in the preview of (Big)Data Infrastructure. We can put this in the Data Engineering side and revisit this later in a future post. Sometimes the ML project could start with fewer data; but as data is collected and analyzed, and used for better training of the models, a scale-able data analytic system should be chosen. A snapshot of the eco-system here https://raw.githubusercontent.com/alexcpn/learnEvolve/master/mlflow.png
Then there is Model Development Environment and related — Languages and Frameworks. For example python libraries like Pandas, Scikit Learn, and Programming languages -Python, R. ML frameworks like TensorFlow or PyTorch and data formats like the simple CSV or more efficient ones like Apache Parquet. This part ties itself to the development environment which is usually Jupyter Notebook. Whereas most data analytics are based on Python and its libraries, for things like Video analytics, it could be C++, Go and related libraries — GStreamer, OpenCV etc.
This much would be clear to every practitioner. Now comes a part that does not seem so obvious at the beginning. Modular components tied through data flow graphs or Pipelines. This is where projects like Kubeflow fit in. Why is this needed? From Kubeflow site
https://www.kubeflow.org/docs/about/kubeflow/#the-kubeflow-mission
Our goal is to make scaling machine learning (ML) models and deploying them to production as simple as possible, by letting Kubernetes do what it’s great at:
Easy, repeatable, portable deployments on a diverse infrastructure (for example, experimenting on a laptop, then moving to an on-premises cluster or to the cloud)
Deploying and managing loosely-coupled microservices
Scaling based on demand
The first sentence says it all — Easy, repeatable and portable. So instead of creating a monolithic python code or notebook, we are splitting it into discreet components; and each task does one part, and these steps are portable -reusable in other projects. We are creating ML microservices and using a workflow scheduler to orchestrate the microservices together.
This is one important part of the MLOps picture. Via Pipelines you are ensuring that a monolith is not formed. Here is a similar opinion from one of the maintainers of a similar stack https://towardsdatascience.com/why-ml-should-be-written-as-pipelines-from-the-get-go-b2d95003f998
Whether Kubeflow makes it Easy and Efficient to create such modular ML components we will see later. But yes it makes it Easy; though - Efficiency not so much, as all data between components are serialized and de-serialized via Python pickle mechanism to disk. So if you are writing out plain CSV instead of Apache Parquet format; or not using TFData or similar efficient data representation in your code; you could take a hit. It will not be too difficult to pass this through an interface mechanism like GRPC or REST, but Kubelow has to persist the In/Out data between components as artefacts. So it needs data to be stored as well.
What else we need as essential for MLOps.
Model Training — And tacking the training parameters. The real training is done by Model frameworks like Tensorflow, Pytorch etc; and there are more complex frameworks here like Horvod that does distributed training https://github.com/horovod/horovod using parallel programming frameworks like MPI or distributed Tensorflow based training using https://developer.nvidia.com/nccl. Note that for very Deep Neural Networks like the one used for Image or Video analytics; training will need an NVIDIA GPU with CUDA cores, while for data analytics CPU may be enough.
Experiment Tracking — This could be useful to check historically the accuracy, precision, confusion matrix or similar metrics with the data and model and connected to Model Validation phase with the test set. Maybe more useful in the CI-CD pipeline for ML than in development.
Model Serving — TFServing, Seldon, KFServing or the like. The main thing is to make sure that the model is separated from the application and it can be versioned and independently updated via CI/CD. Another thing to note is that usually lot of data needs to be passed to the model in case of use-cases such as media analytics. So the model and the business microservice should be colocated in the same Kubernetes node and scaled via a load balancing layer.
Model Monitoring — In traditional DevOps, the feedback to Ops or Developers was via matrices collected from the deployment via Prometheus or a similar tool and displayed in Grafana dashboards. Similar could be used for Model monitoring also. However, to check if the model is accurate in prediction, there should be some manual checks or operator involvement or customer feedback incorporated in the predict — analyze loop. Or more advanced A/B test with other models and compare results using complex algorithms like Multi-Arm Bandit. Also validating the Model with newer data that is available is needed to detect Model Drift. Model monitoring is surely a higher level of MLOps proficiency, that would need very strong infrastructure to implement.
Model, Data and Experimentation Sharing — You need to have a central, access-controlled sharing between different teams; For this, some model or data registry is needed.
There are other parts which I have missed. Model Tuning could be one, where we have popular software like Optuna or Katib. There could be other important ones too, which I am not aware of now. Please indicate in the comments.
Kubeflow is an umbrella project; There are multiple projects that are integrated with it, some for Visualization like Tensor Board, others for Optimization like Katib and then ML operators for training and serving etc. But what is primarily meant is the Kubeflow Pipeline. This is the core of the Kubeflow project and the Pipeline tasks integrate into various other Projects. Note that everything in Kubeflow is running on Kubernetes Cluster. Kubernetes is the resource supplier and execution engine.
Kubeflow is based on Kubernetes. This is its strength. However, this means that everything internally has to run as Containers.
However, exposing a ML Scientist or ML Engineer to the intricacies of creating a container image via Docker file and K8s deployment files are not optimal. Companies would have teams of engineers to do this. But is there a way that ML scientists or ML Engineer can be nudged to develop their ML program as discreet components without them having to understand too much Kubernetes or Container technology? As we described before, this is one main raison d’etre of Kubeflow.
So, what we are going to do is to take a regular Jupyter Notebook based ML development and try to convert the same to a Kubeflow system but in a modular way.
Here is the Colab project that we use for reference [1]https://colab.research.google.com/gist/alexcpn/fa8b3207fbc72f471bdb72433102344c/heartattack-prediction-tfdata.ipynb
We will make this Jupyter Notebook, Kubeflow compliant.
Installing Kubeflow
Kubeflow uses the service mesh Istio. This means that it is a bit of a challenge to set this up. And this will be surely done by the ‘MLOps Team’. If you have a Linux based modern laptop; you can set this up using Kind cluster.
Please follow this; if you already do not have a Kubeflow cluster ready https://gist.github.com/alexcpn/f7068ba5b7205e75b955404f2fc24427.
Note that if you are using a GKE cluster do make sure your nodes have sufficient CPU and memory as there are resource requests set in the deployments. Also if you are planning to use GKE AutoPilot mode, at least for now it does not work as Istio is not supported.
Note that we are using the manifest method of installing Kubeflow from the Kubeflow Manifests Repo -https://github.com/kubeflow/manifests
Create a Jupyter Notebook
If you have successfully installed Kubeflow, you will get a Jupyter Notebook in the Kubeflow dashboard -where most of the default ‘components’ can be viewed
Let’s create a Jupyter Notebook instance; The following is used to connect to the Kubeflow Pipeline Server.
We will use the `create_component_from_func_v2` SDK helper method to create a deployment out of our Python component
Listing the python component here;
Let’s see this component a bit more in detail.
Creating a Component from Python function and Publishing it
What we are doing above is pretty simple; Take in a URL and read some data using the Pandas library. If you see the original Colab Jupyter Notebook [1] you can see that we are splitting the reading part into a different function.
This function has an Input that is a URL of type string. Since this is an in-built data structure in Python it can be provided as-is. We are going to output a Pandas Data-frame as output.
Once we make this a component, it becomes a container Image and runs as a container in Docker. So all function input-output need to be via some interface. Kubeflow uses file interface -K8s persistent volume. Kubeflow (v1) has this InputPath and OuputPath components helper functions that take in the data and uses the Python pickle function to serialize to disk.
Note that you can attach a PVC to the component and write to that. But this is against the Kubeflow usage principle; because of the moment you add a PVC the component no longer becomes non-re-usable. You may need to use such for external visualizations like Tensor-board visualization.
Full code here https://colab.research.google.com/drive/1PyfLflTftWFSHkhWo5quFLonHcKeXc_c#scrollTo=d19ebe08-eb0d-4917-b3d1-8a7ddf448280
Snippet below:
Let’s see how this component is created. As you can see the base image and the packages to install are given in the create_component_from_func method Rest is Kubeflow magic to create a Pod deployment out of it.
You can see how the Input and Output are represented in this Kubeflow dashboard snapshot
Note that I can also write this to a component and say put this in a component store and load from the component from the store
I have just uploaded this to gist, and you can see that in the second snippet I am reading from the URL. Anyone who has access can re-use the same component for their flow, by changing the input data to their own.
A similar flow is followed for other components.
Wiring Components Together
Now let’s see how the components are wired together.
In the second snippet, you can see that read_data output is piped to the input of process_data
Let’s see the magic that’s happening in the process_data function
#Lets use output or target as preiction labeldef process_data( pandas_parquet: InputPath("PandasDataFrame"),                 out_train_path: OutputPath("TF_DataSet"),                 out_test_path: OutputPath("TF_DataSet"),                out_validate_path:OutputPath("TF_DataSet")                ):    import pandas as pd    import sklearn as sk    import numpy as np    import tensorflow as tf        from tensorflow import keras    from tensorflow.keras import layers    from tensorflow.keras.layers.experimental import preprocessing    from sklearn.model_selection import train_test_split        df = pd.read_parquet(pandas_parquet)
We are reading in the Parquet file as input and converting to Pandas DataFrame. So yes — no magic; Similar you can see where we out a h5 Keras model in one component and read that in another.
This is the trade-off that you make in having to split into different components. Maybe future versions of Kubeflow will give GRPC interfaces for some of these, which could speed things a bit. But then also this example is a toy example and you don’t have to slice everything into components.
Visualization of the Data Flow
Kubeflow provides an interactive drill-down visualization of the Data flow. If the Pods are still in Kubernetes you could see the Pod logs. T
The Input/Output artefacts are persisted in the S3. Kubeflow by default runs a Minio based S3 bucket.
You can access the Minio directly by port forwarding
kubectl port-forward svc/minio-service -n kubeflow  9000:9000
default — minio/minio123
Sort of like a poor man's Artifact store. Note that you can also use this to output Tensor-board matrices; though I have not been able to connect the Tensorboard Pod running in the namespace to reach the S3 URL properly from the Dashboard UI.
Visualization of Experiments
The output of the Component in Kubeflow V1 Pipeline can be visualized from the Dashboard. However, it is not a very strong GUI as of now. In V2 version there seems to be better support of this; however, the version I have tested seems buggy (Kubeflow V2 is in Beta)
And the output
You can also compare between runs; on the matrices that you exposed
Optimization
There is an integration of Katib within Kubeflow; But one needs to specify the full YAML file -https://www.kubeflow.org/docs/components/katib/experiment/ with docker container and such. Unlike Optuna I guess Katib is Kubernetes first and hence the complexity.