Kubernetes CKA hands-on challenge #4 Node Management
By: Kim Wuestkamp
Tags: Continuous Deployment, Continuous Integration, Continuous Delivery, Cka, Kubernetes

Content

Overview and Tips

Rules!

be fast, avoid creating yaml manually from scratch use only kubernetes.io/docs for help. check my solution after you did yours. You probably have a better one!

Notices

This challenge was tested on k8s 1.18. Please let us know should you encounter any issues in the comments

how to be fast with Kubectl ≥ 1.18

Scenario Setup

You will start a two node cluster on your machine, one master and one worker. For this you need to install VirtualBox and vagrant, then:



cd cka-example-environments/cluster1

./up.sh git clone git@github.com :wuestkamp/cka-example-environments.gitcd cka-example-environments/cluster1./up.sh vagrant ssh cluster1-master1

vagrant@cluster1-master1:~$ sudo -i

root@cluster1-master1:~# kubectl get node

You should be connected as root@cluster1-master1 . You can connect to other worker nodes using root, like ssh root@cluster1-worker1

If you want to destroy the environment again run ./down.sh . You should destroy the environment after usage so no more resources are used!

Todays Task: Node Management

Drain the node cluster1-worker1 . Save single pods manually to restore afterwards. Remove the node cluster1-worker1 temporarily from the cluster. Add the worker node back to the cluster and allow scheduling again. Restore the single pods. Create a pod of image httpd:2.4.41-alpine and confirm its scheduled on cluster1-worker1 . Expose the pod via a NodePort service and connect to it using curl. You should be able to connect to the internal IP of cluster1-worker1 from your local machine. Suspend the master node ( vagrant suspend cluster1-master1 ), then check: is the pod still reachable via the NodePort service? Resume the master node ( vagrant resume cluster1-master1 ) and confirm node status normal.

Solution

The following commands will be executed as root@cluster1-master1 :

alias k=kubectl

1.

k get node # one master one worker

k drain cluster1-worker1

An error is displayed. BUT the node is now already marked as SchedulingDisabled so new pods couldn’t get scheduled!

We can use --ignore-daemonsets to kill pods of DaemonSets, but there are two pods not controlled by controller resources:

error: cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): management/important-pod, management/web-server

So we export those:

k -n management get pod important-pod -o yaml --export > important-pod.yaml k -n management get pod web-server -o yaml --export > web-server.yaml

I know the --export flag has been marked deprecated, but I don’t see a good alternative as of now. Else we need to clean the yaml configs ourselves.

And run:

k drain cluster1-worker1 --ignore-daemonsets --force k get pod --all-namespaces # various PENDING states

2.

To remove the worker node stop the kubelet :

ssh cluster1-worker1

root@cluster1-worker1:~# service kubelet stop

root@cluster1-worker1:~# exit

Then we check the node status, give it a minute or two:

k get node

We should see cluster1-worker1 marked as NotReady,SchedulingDisabled .

3.

We add the node back to the cluster:

ssh cluster1-worker1

root@cluster1-worker1:~# service kubelet start

root@cluster1-worker1:~# exit

Then we check back:

k get node # should show Ready,SchedulingDisabled k uncordon cluster1-worker1 k get node # back to normal

Restore the single pods:

k -n management -f important-pod.yaml create

k -n management -f web-server.yaml create k get pod --all-namespaces -o wide

This should show all pods are RUNNING again. The pod web-server might still be in an ERROR state, this was handled in part 1.

4.

k run pod --image=httpd:2.4.41-alpine k get pod pod -o wide

Working, scheduled on cluster1-worker1 .

5.

k expose -h

k expose pod pod -h

k expose pod pod --port 80 --type=NodePort k get svc pod # find node port k get node -o wide # find node internal IP

And connect from your main machine:

➜ curl 192.168.101.201:30801 <html><body><h1>It works!</h1></body></html>

Awesome. Check my explanation of NodePort and the other Kubernetes Services if you need a refresh.

6.

Log out and run from your main machine:

➜ vagrant suspend cluster1-master1

vagrant suspend saves the current state of the virtual machine on disk, no more ram used.

Then check the NodePort service again:

➜ curl 192.168.101.201:30801 <html><body><h1>It works!</h1></body></html>

Still working!

That’s one great thing about Kubernetes, the current state is implemented on all worker nodes. Like the services are implemented via the kube-proxy which runs on all workers. If the master (or all masters) are down, like during an upgrade, the cluster still runs normally.

BUT controllers are running on master nodes which means resources like Deployment or HorizontalPodAutoscalers will not work any longer. Also the api-server is not reachable so we couldn’t apply changes.

7.

Let’s boot the master up again:

➜ vagrant resume cluster1-master1

➜ vagrant ssh cluster1-master1 sudo -i k get node

Wonderful.

Clean up

Run: ./down.sh .

All CKA challenges

More challenges on