Machine Learning with Docker and Kubernetes: Training models
By: Xavier Vasques
Tags: Scikit Learn, Kubernetes, Machine Learning, Docker

Kubernetes Jobs: model training and batch inference

In this chapter, we will work on Kubernetes Jobs and how we can use these Jobs to train a machine learning model. A job creates one or more Pods. It is a Kubernetes controller making sure that the Pods successfully terminate their workload. The Job is considered complete when a specified number of Pods terminate. If a Pod fails, the Job will create a new Pod to replace it.

Our goal will be to create a Kubernetes Job, let’s call it “training Job”, that loads training data stored in GitHub, train a machine learning model, serialize the model, and store it outside the cluster. To store the data outside our cluster we could use different locations such as a public cloud, private cloud or on premises. In our example, we will use SSH between our Kubernetes cluster and an external server. The reason we are using external storage, is because Jobs will create Pods running containers that are independent from one to another. We need to persist our model object with the help of external storage.

You can find all the files used in this chapter on GitHub

Create and Prepare the Virtual Machines

To install our Kubernetes Cluster, let’s create virtual machines or use bare metal servers. All instructions in this article were tested using Virtual Machines create with an Ubuntu version 20.04:

- kubmaster: 2vCPUs, 4096MB of RAM, 20GB of drive space

- kubenode1: 2vCPUs, 4096MB of RAM, 20GB of drive space

We enabled traffic between the VM and the host machine.

Change to root and make sure you turn off the swap and comment out the reference swap in /etc/fstab:

swapoff -a vim /etc/fstab

Kubeadm installation

First, we will install docker engine in each Virtual Machines (In our case: kubmaster and kubenode1). To install it, you can choose your preferred methodology (https://docs.docker.com/engine/install). We will use the repository. The different steps below will update the apt package index, install the packages to allow apt to use a repository over https, add Docker’s official GPG key, set up the stable repository, and install the latest version of Docker Engine and containerd.

sudo apt-get update sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg \ lsb-release curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg —-dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo \

"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \

$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io

Then, after changing to root (sudo -s) we perform a curl to get the gpg key that will allow us to use the Kubernetes binaries: kubectl, kubeadm, kubelet

We add the access to the Google repository (http://apt.kubernetes.io) that will allow us to download and install the binaries:

To install binaries, we do the following:

apt-get install -y kubelet kubeadm kubectl kubernetes-cni systemctl enable kubelet

All these steps must be done in all nodes of your cluster (master and nodes).

Create a Kubernetes Cluster

Now that we installed the necessary packages in all our nodes, we will work on the initialization and network to join the different parts of the Kubernetes cluster.

To initiate the master node, connect to the master node and type:

root@kubmaster:~# kubeadm init --apiserver-advertise-address=192.168.1.55 --node-name $HOSTNAME --pod-network-cidr=10.244.0.0/16

192.168.1.55 is the IP address of our master node (kubmaster) we defined previously, and 10.244.0.0/16 is for the Kubernetes internal network defining the range that Kubernetes will use to attribute IP addresses within its network.

We get the following output:

As you can read in the output, to start using our cluster, we need to create the configuration file to work with kubectl (as a regular user):

mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config

To put in place the internal network, we need to provide a network between nodes in the cluster. For this, we will use Flannel which is a very simple and easy way to configure a layer 3 network fabric designed for Kubernetes. We need to provide the possibility to manage the internal network (command to be launched in all nodes):

sysctl net.bridge.bridge-nf-call-iptables=1

We then install our Flannel network with the help of a configuration file (kube-flannel.yml) available online by typing the following command in the master node:

We check the status of the pods in the master node (Flannel network, kube-scheduler, kube-apiserver, kube-controller-manager, kube-proxy, pods managing internal DNS, a pod that stores configurations with etcd etc …):

kubectl get pods --all-namespaces

If everything is running, it is time to join the nodes to the master. For this, we copy the previously generated token and type the following command in the nodes (kubenode1):

kubeadm join 192.168.1.55:6443 --token 08hcql.zbbieukn1h96f1px \

--discovery-token-ca-cert-hash sha256:851d02642c9b2177dd89c8e7cf7178c36185d61799eaaec4fec99b172809373f

Come back to the master node, and type the following commands to check the status:

kubectl get pods --all-namespaces

If you type in the master node the command below, we can see that our master and kubnode1 is ready:

kubectl get nodes

Containerize our Python application that train models

As we did in the previous chapters, we need to create our Dockerfile to indicate to Docker a base image, the settings we need and the commands to execute. We will use the jupyter/scipy-notebook image as our base image.

As you can read, we set our environment variables and we install joblib to allow serialization and deserialization of our trained models and paramiko which is a Python implementation of the SSHv2 protocol (requirements.txt).

We set environment variables from the beginning to persist the trained model and add data and metadata. We copy the train.py file into the image. We also copy the id_rsa file we will generate using ssh-keygen to be able to connect to a remote server through ssh.

We have to setup an RSA key authentication to establish the connection between the cluster and the external server. We need to generate a public (id_rsa.pub) and private key (id_rsa ) that we can use to authenticate:

ssh-keygen -t rsa

We copy/paste the content of id_rsa.pub into ~/.ssh/authorized_keys file (on the target system):

vim /home/xavi/.ssh/id_rsa.pub

If we want, we can copy the id_rsa file into our current directory and modify the permissions with chmod:

sudo cp /home/xavi/.ssh/id_rsa . sudo chmod a+r id_rsa

We will then remove the id_rsa file with the docker run command.

For this step, we could use different methodologies such as writing a Dockerfile like this one by making sure we remove the id_rsa file at the end of the build process:

ARG SSH_PRIVATE_KEY RUN mkdir /root/.ssh/ RUN echo “${SSH_PRIVATE_KEY}” > /root/.ssh/id_rsa # […] RUN rm /root/.ssh/id_rsa

And build the docker image with something like:

sudo docker build -t kubernetes-models -f Dockerfile --build-arg SSH_PRIVATE_KEY=”$(cat ~/.ssh/id_rsa)” .

It’s not really the topic of this chapter, but we need to be careful regarding not leaving traces inside our Docker image. Even if you we are deleting a file, it still can be viewed in one of the layers of the image we will push. We can use the –squash parameter to reduce multiple layers, between the origin and the latest stage, to one. In addition, the –squash parameter can be used to reduce the size of an image by removing files which are no more present. We can also work with multi-stage builds in a single Dockerfile in which we build multiple Docker image. Only the last one will persist and leave traces.

# This is intermediate FROM ubuntu as intermediate # […] # Final image FROM ubuntu # […]

For security reasons, we can also use Kubernetes secret objects to store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it in hard in the definition of a Pod or in an image container: https://kubernetes.io/docs/concepts/configuration/secret/.

Now that we have our Dockerfile, let’s have a look to other files. In train.py we start by importing the necessary libraries, read the environment variables set in our Docker image for persisting models, load training data (train.csv) which are stored on Github, train two models (Linear discriminant analysis and a multi-layer perceptron neural network), serialize them, perform a cross validation, upload the trained models and cross validation results in a remote server (192.168.1.11) and specified directory (/home/xavi/output/). We could also define the URL, specified directory and the IP address as environment variables.

We can go to the next step by building the docker image, run it to test our application locally, tag it with the name of an image repository on the Docker Hub registry, and push it to the registry and be ready to use the image in our Kubernetes cluster:

docker build -t kubernetes-models -f Dockerfile .

To test if our code is ok, we will run your container locally and test our built image:

docker run kubernetes-models python3 train.py rm ./id_rsa

If everything is ok, we can push a new image to the repository using the CLI:

docker login docker tag kubernetes-models:latest xaviervasques/kubernetes-models:latest docker push xaviervasques/kubernetes-models:latest

Create configuration files for Kubernetes

Once the image is successfully uploaded to the registry, we go to our project directory (connect to kubmaster) and create a Job config that executes our python code: job.yaml.

As explained in the Kubernetes documentation, as with all other Kubernetes config, a Job needs apiVersion, kind, and metadatafields. A Job also needs a .spec section. apiVersion specifies the version of the Kubernetes API to use, kind the type of Kubernetes resource, and we can provide a label with metadata. The .spec.template is the only required field of the .spec and represents a pod template as it has the same schema except it is nested with no apiVersion or kind. In. spec.template.spec.containers, we give each container a name, the image we want to use and the command we want to run in the container. Here we want to run train.py and remove our id_rsa file.

The equivalent docker command would be

docker run kubernetes-models python3 train.py rm ./id_rsa

Kubernetes will look for the image from the registry, instead of using a cached image, thanks to imagePullPolicy. Finally, we set if containers should be restarted or not if they failed (Never or OnFailure) and decide the number of retries before considering a Job as failed (back-off limit is set by default to 6 minutes).

We are finally ready to get our application running on Kubernetes. Launch the following command:

kubectl create -f job.yaml

Check the Job objects by running:

kubectl get jobs kubectl get pods --selector=job-name=train-models-job

We can view the logs of the process:

kubectl logs train-models-job-vd5rq

And all the files were stored in the remote server.

Few commands to delete the cluster

There are several useful commands that we can find in the Kubernetes documentation. For instance:

Delete the job

kubectl delete job train-models-job

Reset kubeadm

sudo kubeadm reset

Delete the entire cluster

kubeadm reset sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube* sudo apt-get autoremove sudo rm -rf ~/.kube

Next Steps

The next steps will be to do batch inference/scoring by loading predictions from our trained models and also do real-time, online inference, using REST APIs. We explored these concepts here: