Deploying Kubeflow on a local Kubernetes cluster (minikube)
By: Shubhasmita Roy
Tags: Kubeflow, Dashboard, Kubernetes, Docker, Minikube

Intern, Canspirit.ai(Mentor — Mr. Arunkumar Nair Canspirit AI Arun Nair)

Student, School of ECE, MIT-WPU (Mentor — Dr. Vinaya Gohokar)

This is a tutorial on deploying Kubeflow on a local Kubernetes cluster from scratch. While trying to deploy it myself using the documentation on the official websites, I came across a lot of errors, mostly compatibility issues so I am documenting the process here, for reference.

We will create an Ubuntu VM first. I have created an instance on GCP.

VM Requirements:

VM OS — Ubuntu 20.04 LTS 16 GB RAM 4 vCPUs The official site recommends machine with 250 GB storage, I have made a 150 GB one. Ensure that the root is allocated at least 50GB.

Prerequisites :

Docker — version 1.19 kubectl — version 1.15 minikube — version 1.15

I am using NoMachine as a GUI for VM instance. This is required since we need to access Kubeflow which cannot be accessed from host machine’s browser. Please feel free to use other methods, eg.PuTTY, to access the Kubeflow Dashboard.

1. Installing Docker CE-

Run the following commands to install Docker CE:

apt-get update apt-get install -y apt-transport-https ca-certificates curl software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \$(lsb_release -cs) \stable" apt-get update apt-get install docker-ce docker-ce-cli containerd.io

To verify your installation, run:

sudo docker run hello-world

The expected output looks like this:

Checking Docker installation

Feel free to check the official website of Dockers: https://docs.docker.com/install/linux/docker-ce/ubuntu/

2. Installing kubectl-

Checking installation:

Checking for kubectl installation

3. Installing MiniKube –

We will created a local Kubernetes cluster using MiniKube.

Install minikube version 1.15, since this is compatible with Kubeflow.

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 sudo install minikube-linux-amd64 /usr/local/bin/minikube

Checking installation:

Checking minikube installation

4. Starting minikube-

minikube start --cpus 6 --memory 12288 --disk-size=120g --extra-config=apiserver.service-account-issuer=api --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/apiserver.key --extra-config=apiserver.service-account-api-audiences=api

We want to specify 6 CPUs, 12288 memory, 120G disk size.

5. Installing Kubeflow-

We want to install Kubeflow version 0.7 since this is the stable and compatible version with Kubernetes. First we go to root by entering sudo su in the terminal.

mkdir -p /root/kubeflow/v0.7 cd /root/kubeflow/v0.7 wget https://github.com/kubeflow/kfctl/releases/download/v0.7.1/kfctl_v0.7.1-2-g55f9b2a_linux.tar.gz tar -xvf kfctl_v0.7.1-2-g55f9b2a_linux.tar.gz export PATH=$PATH:/root/kubeflow/v0.7 export KF_NAME=my-kubeflow export BASE_DIR=/root/kubeflow/v0.7 export KF_DIR=${BASE_DIR}/${KF_NAME} mkdir -p ${KF_DIR} cd ${KF_DIR} kfctl apply -V -f https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml

If you get an error message like follows:

Then you must open the kfctl_k8s_istio.0.7.1.yaml file in nano, do not make any changes, just save (Ctrl+S) and exit (Ctrl+X) it. Then execute the following statememt:

kfctl apply -V -f kfctl_k8s_istio.0.7.1.yaml

You should get a long list of something running, it will take a few minutes to execute. Once done you can safely move on to the next step.

If this method does not work, manually creating .yaml file should work. The following method is inspired by this blog Setting up a Kubeflow deployment on a Kubernetes cluster | by Andre Benea | Medium. Please look through this tutorial as well, hopefully it give you a good insight.

Enter the following and open the nano editor:

nano kfctl_k8s_istio.yaml

Enter the following in the file:



kind: KfDef

metadata:

creationTimestamp: null

namespace: kubeflow

spec:

applications:

- kustomizeConfig:

parameters:

- name: namespace

value: istio-system

repoRef:

name: manifests

path: istio/istio-crds

name: istio-crds

- kustomizeConfig:

parameters:

- name: namespace

value: istio-system

repoRef:

name: manifests

path: istio/istio-install

name: istio-install

- kustomizeConfig:

parameters:

- name: clusterRbacConfig

value: "OFF"

repoRef:

name: manifests

path: istio/istio

name: istio

- kustomizeConfig:

repoRef:

name: manifests

path: application/application-crds

name: application-crds

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: application/application

name: application

- kustomizeConfig:

repoRef:

name: manifests

path: metacontroller

name: metacontroller

- kustomizeConfig:

overlays:

- istio

- application

repoRef:

name: manifests

path: argo

name: argo

- kustomizeConfig:

repoRef:

name: manifests

path: kubeflow-roles

name: kubeflow-roles

- kustomizeConfig:

overlays:

- istio

- application

repoRef:

name: manifests

path: common/centraldashboard

name: centraldashboard

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: admission-webhook/bootstrap

name: bootstrap

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: admission-webhook/webhook

name: webhook

- kustomizeConfig:

overlays:

- istio

- application

repoRef:

name: manifests

path: jupyter/jupyter-web-app

name: jupyter-web-app

- kustomizeConfig:

overlays:

- istio

repoRef:

name: manifests

path: metadata

name: metadata

- kustomizeConfig:

overlays:

- istio

- application

repoRef:

name: manifests

path: jupyter/notebook-controller

name: notebook-controller

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pytorch-job/pytorch-job-crds

name: pytorch-job-crds

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pytorch-job/pytorch-operator

name: pytorch-operator

- kustomizeConfig:

overlays:

- application

parameters:

- name: namespace

value: knative-serving

repoRef:

name: manifests

path: knative/knative-serving-crds

name: knative-crds

- kustomizeConfig:

overlays:

- application

parameters:

- name: namespace

value: knative-serving

repoRef:

name: manifests

path: knative/knative-serving-install

name: knative-install

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: kfserving/kfserving-crds

name: kfserving-crds

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: kfserving/kfserving-install

name: kfserving-install

- kustomizeConfig:

overlays:

- application

parameters:

- name: usageId

value: <randomly-generated-id>

- name: reportUsage

value: "true"

repoRef:

name: manifests

path: common/spartakus

name: spartakus

- kustomizeConfig:

overlays:

- istio

repoRef:

name: manifests

path: tensorboard

name: tensorboard

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: tf-training/tf-job-crds

name: tf-job-crds

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: tf-training/tf-job-operator

name: tf-job-operator

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: katib/katib-crds

name: katib-crds

- kustomizeConfig:

overlays:

- application

- istio

repoRef:

name: manifests

path: katib/katib-controller

name: katib-controller

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pipeline/api-service

name: api-service

- kustomizeConfig:

overlays:

- application

parameters:

- name: minioPvcName

value: minio-pv-claim

repoRef:

name: manifests

path: pipeline/minio

name: minio

- kustomizeConfig:

overlays:

- application

parameters:

- name: mysqlPvcName

value: mysql-pv-claim

repoRef:

name: manifests

path: pipeline/mysql

name: mysql

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pipeline/persistent-agent

name: persistent-agent

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pipeline/pipelines-runner

name: pipelines-runner

- kustomizeConfig:

overlays:

- istio

- application

repoRef:

name: manifests

path: pipeline/pipelines-ui

name: pipelines-ui

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pipeline/pipelines-viewer

name: pipelines-viewer

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pipeline/scheduledworkflow

name: scheduledworkflow

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: pipeline/pipeline-visualization-service

name: pipeline-visualization-service

- kustomizeConfig:

overlays:

- application

- istio

parameters:

- name: admin

value:

repoRef:

name: manifests

path: profiles

name: profiles

- kustomizeConfig:

overlays:

- application

repoRef:

name: manifests

path: seldon/seldon-core-operator

name: seldon-core-operator

repos:

- name: manifests

uri:

version: master

status: {} apiVersion: kfdef.apps.kubeflow.org/v1beta1kind: KfDefmetadata:creationTimestamp: nullnamespace: kubeflowspec:applications:- kustomizeConfig:parameters:- name: namespacevalue: istio-systemrepoRef:name: manifestspath: istio/istio-crdsname: istio-crds- kustomizeConfig:parameters:- name: namespacevalue: istio-systemrepoRef:name: manifestspath: istio/istio-installname: istio-install- kustomizeConfig:parameters:- name: clusterRbacConfigvalue: "OFF"repoRef:name: manifestspath: istio/istioname: istio- kustomizeConfig:repoRef:name: manifestspath: application/application-crdsname: application-crds- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: application/applicationname: application- kustomizeConfig:repoRef:name: manifestspath: metacontrollername: metacontroller- kustomizeConfig:overlays:- istio- applicationrepoRef:name: manifestspath: argoname: argo- kustomizeConfig:repoRef:name: manifestspath: kubeflow-rolesname: kubeflow-roles- kustomizeConfig:overlays:- istio- applicationrepoRef:name: manifestspath: common/centraldashboardname: centraldashboard- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: admission-webhook/bootstrapname: bootstrap- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: admission-webhook/webhookname: webhook- kustomizeConfig:overlays:- istio- applicationrepoRef:name: manifestspath: jupyter/jupyter-web-appname: jupyter-web-app- kustomizeConfig:overlays:- istiorepoRef:name: manifestspath: metadataname: metadata- kustomizeConfig:overlays:- istio- applicationrepoRef:name: manifestspath: jupyter/notebook-controllername: notebook-controller- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pytorch-job/pytorch-job-crdsname: pytorch-job-crds- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pytorch-job/pytorch-operatorname: pytorch-operator- kustomizeConfig:overlays:- applicationparameters:- name: namespacevalue: knative-servingrepoRef:name: manifestspath: knative/knative-serving-crdsname: knative-crds- kustomizeConfig:overlays:- applicationparameters:- name: namespacevalue: knative-servingrepoRef:name: manifestspath: knative/knative-serving-installname: knative-install- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: kfserving/kfserving-crdsname: kfserving-crds- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: kfserving/kfserving-installname: kfserving-install- kustomizeConfig:overlays:- applicationparameters:- name: usageIdvalue: - name: reportUsagevalue: "true"repoRef:name: manifestspath: common/spartakusname: spartakus- kustomizeConfig:overlays:- istiorepoRef:name: manifestspath: tensorboardname: tensorboard- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: tf-training/tf-job-crdsname: tf-job-crds- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: tf-training/tf-job-operatorname: tf-job-operator- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: katib/katib-crdsname: katib-crds- kustomizeConfig:overlays:- application- istiorepoRef:name: manifestspath: katib/katib-controllername: katib-controller- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pipeline/api-servicename: api-service- kustomizeConfig:overlays:- applicationparameters:- name: minioPvcNamevalue: minio-pv-claimrepoRef:name: manifestspath: pipeline/minioname: minio- kustomizeConfig:overlays:- applicationparameters:- name: mysqlPvcNamevalue: mysql-pv-claimrepoRef:name: manifestspath: pipeline/mysqlname: mysql- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pipeline/persistent-agentname: persistent-agent- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pipeline/pipelines-runnername: pipelines-runner- kustomizeConfig:overlays:- istio- applicationrepoRef:name: manifestspath: pipeline/pipelines-uiname: pipelines-ui- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pipeline/pipelines-viewername: pipelines-viewer- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pipeline/scheduledworkflowname: scheduledworkflow- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: pipeline/pipeline-visualization-servicename: pipeline-visualization-service- kustomizeConfig:overlays:- application- istioparameters:- name: adminvalue: johnDoe@acme.com repoRef:name: manifestspath: profilesname: profiles- kustomizeConfig:overlays:- applicationrepoRef:name: manifestspath: seldon/seldon-core-operatorname: seldon-core-operatorrepos:- name: manifestsuri: https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz version: masterstatus: {}

Ctrl + S to save and Ctrl +X to exit.

Enter:

kfctl apply -V -f kfctl_k8s_istio.yaml

Even if it shows failed after this, execute the next step to see if pods are running. If they aren’t then retry installing kfctl. Another thing we can try is to open kfctl_k8s_istio.0.7.1.yaml file file on nano, save and exit it without making changes and execute the next statements to check if pods are running.

6. Checking the status of pods-

After the installation, run the following command to check whether all the pods are in running status:

kubectl get pod -n kubeflow

Output should be like follows:

Checking status of pods

I had created 2 instances of Kubeflow by mistake, so we can see 2/2 in the status of pods. Yours should show 1/1 on majority of these pods.

7. Launching Kubeflow dashboard-

export INGRESS_HOST=$(minikube ip) export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

Obtain the variable values:

echo $INGRESS_HOST echo $INGRESS_PORT

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

Alternatively, you could save all the export commands in the bashrc file so that these variables are global and aren’t lost later on.

sudo nano ~/.bashrc

Within the file, type this and save:

export INGRESS_HOST=$(minikube ip) export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

To execute the commands execute:

source ~/.bashrc

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

8. Kubeflow Dashboard:

Type the web address as follows on a web browser and access Kubeflow.

http://<INGRESS_HOST>:<INGRESS_PORT>

Kubeflow Dashboard

Troubleshooting after installation

Sometimes I have observed that I am not able to access my Kubeflow Dashboard after executing minikube start command.

What I have done until now is used the rm command to remove the directories within my-kubeflow directory and re-executed all the statements from the .yaml file statement. The IP address of the Kubeflow Dashboard doesn’t change and I am able to access it.

— — — — — — — — — — — — — — *** — — — — — — — — — — — — — — —

But I still need to figure out solution for this. I will keep updating this blog as I am figuring out the best solution for this. Comments and suggestions are very much welcome.

I am an intern at the moment, currently in my 3rd year in ECE. So I am still new to these technologies. I hope I could help you out with this installation. Please feel free to comment down below and ask for help. I will try to get back at the earliest.

Some useful websites:

Kubeflow Install Error · Issue #4642 · kubeflow/kubeflow · GitHub