Building Your Home Raspberry PI Kubernetes Cluster
By: Lukasz Raczylo
Tags: Linux, Raspberry Pi, Kubernetes, Development, Clustering

Are there any Kubernetes fans here? I’m running it in production both at work and for my private projects for quite a while now, but sometimes I need a place to quickly develop and test new features without incurring additional costs or simply — run my home entertainment, backups, file sharing and kettle.

Few more nodes on the way, cluster is almost ready.

Hardware used ( Black Friday prices 🥳 )

Why Raspberry Pi?

TL;DR: Cost vs processing power.

4 node cluster provides 4 x ( 1.5Ghz, 4 core ARM cpu and 4 GB of RAM ) — that’s 16 1.5 Ghz cores and 16 GB of RAM available. I’ve done the math and the overall cost of the cluster is still cheaper than similar processing power / nodes cloud offering. Say no more…

Memory card preparation

Start with downloading the operating system, which will do the heavy lifting. I spend most of my days working with docker and Kubernetes, any one of my hobbies is slimming down the docker images to the absolute minimum — most of that achieved thanks to the Alpine Linux — therefore I’ll base my cluster on this exact distribution.

Head to Alpine Linux Downloads section and pick the AARCH64 version for your Raspberry Pi 4 Model B.

In the meanwhile — let’s prepare the memory card starting with formatting it with FAT32 filesystem. I’m OSX person in so using the following command to obtain the disk identifier for your memory card

diskutil list

Follow it with to format whole memory card — I called my memory card RPI

sudo diskutil eraseDisk FAT32 RPI MBRFormat /dev/diskX

Unpack downloaded package with Alpine linux onto the memory card

sudo tar xf alpine-rpi-3.12.1-aarch64.tar.gz -C /Volumes/RPI

Base system configuration

Congratulations, you are one step closer to the world of kubernetes in your own living room. Slide memory card into your Raspberry Pi, monitor ( or TV ), together with keyboard and switch the power on. After system boots up and asks you for login — use root as both username and password.

Execute following command to initiate the setup process.

setup-alpine

I’m running my cluster at home, and because I’m running short on available router ports I decided to use WiFi network. There’s not too much of a difference in options you’ll pick — choose wisely though.

After you’ll be done with setup — there’s few more things which you should do. Alpine by default runs from the RAM, but we’d like our changes to persist on the disk and between potential restarts.

apk update

apk add cfdisk e2fsprogs # Install disk tools

cfdisk /dev/mmcblk0 # Run cfdisk on your memory card

In this step you need to:

Resize the FAT32 partition to reasonable minimum — I gave it 1GB in my case.

Use all remaining free space to create new bootable partition.

Don’t forget to write the changes you just made.

Helpful tutorial: How to use cfdisk.

There’s few more commands you need to execute now to complete the whole process.

mkfs.ext4 /dev/mmcblk0p2 # Format newly created partition as EXT4

mount /dev/mmcblk0p2 /mnt # Mount it

setup-disk -m sys /mnt # Install system files

mount -o remount,rw /media/mmcblk0p1 # Remount old partition in RW # Let's do some housekeeping

rm -f /media/mmcblk0p1/boot/*

cd /mnt

rm boot/boot

mv boot/* /media/mmcblk0p1/boot/

rm -Rf boot

mkdir media/mmcblk0p1

ln -s media/mmcblk0p1/boot boot

Update /etc/fstab records

echo "/dev/mmcblk0p1 /media/mmcblk0p1 vfat defaults 0 0" >> etc/fstab

sed -i '/cdrom/d' etc/fstab

sed -i '/floppy/d' etc/fstab

cd /media/mmcblk0p1

Last touches after system restart — keep in mind that without appropriate cgroups enabled the kubeadm step will fail.

# Enable edge repository for Alpine

sed -i '/edge/s/^#//' /mnt/etc/apk/repositories

# Force use of new partition as the root one

sed -i 's/^/root=\/dev\/mmcblk0p2 /' /media/mmcblk0p1/cmdline.txt

# Make sure that appropriate cgroups are enabled

echo "cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1" >> /media/mmcblk0p1/cmdline.txt

sed -i ':a;N;$!ba;s/

/ /g' /media/mmcblk0p1/cmdline.txt rc-update add wpa_supplicant boot # Make sure your wifi will come back up after restart

Finally, really important thing after all those steps above — backup the changes you just made and reboot the system

lbu_commit -d

reboot

Rest of the system configuration

I’ve mentioned that I intend to use this kubernetes cluster in home environment — before you’ll ask — it will work in the office network as well, although — there’s few extra touches we should add

Announce hostname in local network — avahi daemon

Why this step? It’s much easier to ssh pi0.local than trying to find the correct IP address. It will also make your connectivity and clustering configuration much easier afterwards, especially when you can’t count on static IPs.

apk add dbus avahi

rc-update add dbus boot # avahi won't start without dbus

rc-update add avahi-daemon boot

Allow ssh access as root

Edit /etc/ssh/sshd_config file — add following line to it to enable access to ssh as a root.

PermitRootLogin yes

Install docker, kubernetes and rest of remaining packages you will need later

apk update

apk add kubernetes docker cni-plugins kubelet kubeadm

rc-update add docker default

rc-update add kubelet default

Little helper says: You can save your efforts

At this stage you should have everything ready for your journey. I used this opportunity to switch off my Raspberry Pi, take the memory card back to my laptop and create an image of the card so I can restore it on the remaining three cards and save time.

Remember to change the /etc/hostname content for every newly re-created machine to avoid conflicts. I’ve picked pi0, pi1, pi2 to make it easier to remember ( and use in my local ssh config ) although there’s no limitation.

Create your kubernetes master

service docker start

kubeadm config images pull # Get the necessary images

kubeadm init --pod-network-cidr=10.244.0.0/16

If you see any cgroups related errors which stop the initialisation process — you’ve missed one of the steps before, otherwise you should see message starting with `Your Kubernetes control-plane has initialised successfully!`.

SAVE the command from the output ( the one starting with kubeadm join ) in safe place. You will need it to allow remaining nodes join your cluster.

Run the following commands to store credentials in your home directory

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

How to access the node?

I’ve copied content of $HOME/.kube/config from the node onto my local machine, changing few default values to avoid potential clashes. Thanks to this I can use tools like kubectl or k9s from my laptop and make sure I’ll always reach the right server.

Master is running, what’s else to do?

Networking between the pods — without it your node will remain tainted and in NotReady state forever.

You can’t deploy anything on master node by default and node shows as tainted but don’t worry — we can change it with this command

kubectl taint nodes --all node-role.kubernetes.io/master-

Dashboard. Have you ever met anyone saying “I don’t like dashboards”? Kubernetes have their own, quite universal dashboard which can give you a great overview of your cluster ( and anything within ).

# Add kubernetes-dashboard repository

helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/

# Deploy a Helm Release named "kubernetes-dashboard" using the kubernetes-dashboard chart

helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --set protocolHttp=true,ingress.enabled=true,rbac.create=true,serviceAccount.create=true,service.externalPort=9090,networkPolicy.enabled=true,podLabels.app=dashboard

As you’ve noticed I have added quite a few extras to the helm chart settings but there’s a reason for that.

Your dashboard will be up and running but.. It won’t show anything. Missing permissions.

kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=default:kubernetes-dashboard

Okay. We are almost there. We have master node, we have dashboard but we can’t access it at the moment. We could potentially use the nodePort to expose dashboard, but it’s not the way we roll — to do it the kubernetes way we need a loadBalancer. Node runs in our local network so we can’t count on any goodies provided by AWS or GoogleCloud but fear not — there’s way to deal with it.

Load balancing in home network

Follow installation steps on MetalLLB till end of Installation By Manifest.

ifconfig wlan0 promisc # Set PROMISC mode for WiFi - for ARP

Command above have its effect as long as you keep the Pi running. To avoid creation of the startup scripts I decided to edit /etc/network/if-up.d/dad to set the promisc mode on when interface comes up.

Create following manifest as my-dashboard.yaml

apiVersion: v1

kind: ConfigMap

metadata:

namespace: metallb-system

name: config

data:

config: |

address-pools:

- name: default

protocol: layer2

addresses:

- 192.168.50.200-192.168.50.250

---

apiVersion: v1

kind: Service

metadata:

name: k8s-dashboard

annotations:

metallb.universe.tf/address-pool: default

spec:

ports:

- port: 80

targetPort: 9090

selector:

app: dashboard

type: LoadBalancer

Don’t forget to change addresses section to correspond with your local network settings.

kubectl apply -f my-dashboard.yaml

kubectl get svc k8s-dashboard

From now on your dashboard can be accessed at ( in my case ) http://192.168.50.200/

Raspberry Pi powered k8s cluster.

Article powered by the Saturday struggle, RedBull and love for experiments.

Cluster PODS overview provided by K9S

Adding additional nodes

As we DRY ( Don’t Repeat Yourself ) KISS ( Keep It Stupid Simple ) — scroll up to the beginning of article and follow all the steps until “Create your kubernetes master” on newly created nodes, then execute following ( remember to replace ip address with your master node IP — or it’s hostname ( pi0.local, thanks to avahi daemon )

service docker start

kubeadm config images pull

kubeadm join 192.168.50.132:6443 --token dugwjt.0k3n \--discovery-token-ca-cert-hash sha256:55cfadHelloSuperSecretHashbf4970f49dcadf533f86e3dba

Tip: If you forgot to copy kubeadm command during master node creation, I got your back — following command on the master node will print it out for you again