Kubeflow: How to Install and Launch Kubeflow on your Local Machine | by Fernando López | Towards Data Science

Kubeflow [1] is an open source platform developed by google to contain the machine learning model development life cycle. Kubeflow is made up of a set of tools that address each of the stages which compound the machine learning life cycle, such as: data exploration, feature engineering, feature transformation, model experimentation, model training, model evaluation, model tuning, model serving and model versioning. The main attribute of kubeflow is that it is designed to work on top of kubernetes [2], that is, kubeflow takes advantage of the benefits that a kubernetes cluster provides such as container orchestration and auto-scaling.
While kubernetes is an standard technology for container orchestration, it can be a complex and time-consuming process for data scientists or machine learning engineers to configure and coordinate each stage of the machine learning life cycle directly in a kubernetes cluster. Thus kubeflow comes as the platform that provides the tools to configure, develop, automate and deploy each stage of the machine learning life cycle in a kubernetes cluster, avoiding data scientists or machine learning engineers from having to spend time on configure and apply changes directly in the kubernetes cluster.
In this blog we are going to see how to install and configure kubeflow on your local machine in order to be able to start using kubeflow locally without the need for a cloud provider. Therefore, this blog will be divided into:
Requirements
Minikube installation
Kubeflow installation
Creating a Kubeflow project
Accessing to Kubeflow UI
To install and run kubeflow on our local machine we will need a set of essential components. First of all, we are going to require a kubernetes cluster which is where the kubeflow service will be installed and deployed. Since it is not recommended to install kubernetes as such on our local machine due to the amount of resources it demands, a lighter alternative oriented to local tests is suggested for emulating a kubernetes cluster, in this case we will use minikube [3] (although there are other alternatives like: microk8s, kind and k3s). In addition to minikube, we are going to need a hypervisor which will allow us to generate the virtual environments within our mini-cluster, in this case we are going to use hyperkit [4] (although you can use other alternatives such as VirtualBox, kvm2 for linux and Hyper-V for windows), and finally the kubeflow package.
Disclaimer: since I am working on MacOS, the commands used are oriented to that operating system, however in the documentation of each component you can find its equivalent for Linux or Windows
To install minikube on MacOS, just type:
$ brew install minikube
When the installation is complete, the kubectl command will be added to $PATH . kubectl is the command for interacting with the kubernetes mini-cluster. Then, we will need a hypervisor to create the virtual environments within our kubernetes mini-cluster, for this we are going to use hyperkit which can be installed as follows:
$ brew install hyperkit
Great, to start our kubernetes mini-cluster, just type:
$ minikube start --driver=hyperkit
The above command will initialize a kubernetes mini-cluster using hyperkit as the hypervisor for virtual environments. To verify the information of our cluster, let’s type the following command:
$ kubectl cluster-info
you will get something like:
> Kubernetes control plane is running at https://192.168.64.5:8443
Great, at this point we already have the necessary components to start with the installation and configuration of kubeflow, let’s go for it!
For the installation of kubeflow we are going to define some environment variables, we are going to download the kubeflow project and then we will proceed with the installation. So, let’s define some variables:
$ PLATFORM=$(uname)$ export PLATFORM$ mkdir -p ~/bin$ export KF_TAG=1.0.1$ KF_BASE="https://api.github.com/repos/kubeflow/kfctl/releases"
The KF_TAG variable refers to the image version that we are going to use, you can use the latest one, in this case I will use version 1.2.0 . Then we will define the URL, download and unpack the project and then we will move the kfctlbinary to our binaries directory created in the previous step.
$ KFCTL_URL=$(curl -s ${KF_BASE} |\	      grep http |\	      grep "${KF_TAG}" |\	      grep -i "${PLATFORM}" |\	      cut -d : -f 2,3 |\	      tr -d '\" ' )$ wget "${KFCTL_URL}"$ KFCTL_FILE=${KFCTL_URL##*/}$ tar -xvf "${KFCTL_FILE}"$ mv ./kfctl ~/bin/
Finally, we are going to remove the kubeflow package (it is no longer necessary to keep it) and we are going to add the kubeflow path ~/bin (which contains the command kfctl) to our $PATH :
$ rm "${KFCTL_FILE}"$ export PATH=$PATH:~/bin
The kfctlcommand should be ready, to test it we are going to type:
$ kfctl versionkfctl v1.2.0-0-gbc038f9
The kfctlcommand will allow us to interact with the kubeflow components, specifically to define and apply manifests to our kubernetes cluster. So far we have our kuberentes mini-cluster as well as kubeflow installation ready, what proceeds is to generate our first kubeflow project using the kfctlcommand, let’s go for it!
As we already know, kubeflow works on top of kubernetes, therefore the interaction between both platforms will be through manifests or yaml files. So for generating a kubeflow project we have to do it through a yaml file which defines all the services that must be deployed into the kubernetes mini-cluster.
First we’ll generate some environment variables that define the kubeflow version that we’ll install, then we’ll define the directory where all the configuration and yaml files of each service deployed by the kubeflow project will be allocated and finally we will get the yaml for deploying a kubeflow project in our kubernetes mini-cluster.
So, let’s define some environment variables corresponding to the version of kubeflow that we installed:
$ MANIFEST_BRANCH=${MANIFEST_BRANCH:-v1.2-branch}$ export MANIFEST_BRANCH$ MANIFEST_VERSION=${MANIFEST_VERSION:-v1.2.0}$ export MANIFEST_VERSION
Then we define the project name and create the directory where the configuration and yaml files for each service will be allocated:
$ KF_PROJECT_NAME=${KF_PROJECT_NAME:-hello-kf-${PLATFORM}}$ export KF_PROJECT_NAME$ mkdir "${KF_PROJECT_NAME}"$ pushd "${KF_PROJECT_NAME}"
Then we define the repository from where we will take the manifest and assign it to an environment variable:
$ manifest_root=https://raw.githubusercontent.com/kubeflow/manifests$ FILE_NAME=kfctl_k8s_istio.${MANIFEST_VERSION}.yaml$ KFDEF=${manifest_root}${MANIFEST_BRANCH}/kfdef/${FILE_NAME}
As you can see, we are using a manifest for ISTIO (a service mesh implementation for kubernetes). Finally we apply the manifest for deploying our kubeflow project.
$ kfctl apply -f $KFDEF -V
The deployment process may take around 30 minutes.
To verify that the installation was successful, with the following command you can verify that all services are in RUNNING or COMPLETED mode.
$ kubectl get pods --all-namespaces -w
In case some services have an error status, you may need to allocate more memory to your kubernetes mini-cluster (minikube) and reapply the deployment manifest. If everything proceeds successfully, we are now ready to view our kubeflow project in the browser. Let’s go for it!
Since we are installing on a local server, to access kubeflow UI it is only required to do a port-forward as follows:
$ kubectl port-forward svc/istio-ingressgateway -n istio-system 7777:80
The above command is executed with kubectl , which exposes port 7777 of the istio-ingressgateway service with namespace istio-system which allows us access to kubeflow UI. Finally, if you go to: http://localhost:7777 you will be redirected to kubeflow UI as shown in Figure 2.
In this blog we saw how to download, configure and install kubeflow as well as its requirements (on a MacOS operating system).
With this tutorial you will be able to start interacting with kubeflow from the UI. As we could see, kubeflow provides a set of tools to develop the life cycle of a machine learning model, in a future blog-tutorial we will learn about each of the components of kubeflow as well as the generation of pipelines!