How to use NFS in Kubernetes Cluster — Configuring the NFS Server.
By: Fabio Fernandes
Tags: Raspberry Pi, Persistent Volume, Nfs Server, Kubernetes, Configuration

Kubernetes environment, by definition, should be stateless. That way, you can scale your applications, rolling upgrades, and restart crashed apps continuously without any problems.

Unfortunately, except for some serverless applications, we need to persist the data almost all the time.

Data must be retained beyond the life-cycle of a given application, pod, or even cluster in most cases. To make it possible Kubernetes users will need to utilize a persistent volume.

We have several options to do it in Kubernetes, from local storage in the node to cloud volumes.

In this series, I’m describing 3 different ways to create these persistent volumes using NFS.

This subject is super extensive, so I divided the content between the following articles:

General Notes

Source Code

All the files used in this article are published in this GitHub repository: https://github.com/fabiofernandesx/k8s-volumes. Feel free to clone it or even open a pull request if you find something that needs to be fixed or added.

Home Lab

For this article series, I’ll use my raspberry pi Kubernetes cluster, but the commands should work in any environment, including cloud or virtual machines. If you are interested in seeing how to create a raspberry pi k8s cluster, you can find it here: Yet Another Raspberry Pi k8’s Cluster, or if you interested in how to create this cluster using the virtual box, you can see this one: Preparing Virtual Box VM’s to run Kubernetes.

Configuring NFS Server

The first thing to do is to configure the NFS server. We want to export 5 endpoints:

/ssd/host-mapped — It will be a folder into the local disk and used in the cluster to demonstrate how to create a persistent volume using a “mapped” folder on each node.

/ssd/direct — it will also be a folder and will be used to demonstrate how to create a PV accessing the NFS server directly without mapping the folder into the node first.

/ssd/dynamic — Another folder into the NFS server's local disk, and it will be used as our main “fastest” source to the cluster. It will use a dynamic PV creation via storage class inside the cluster.

/mnt/usb3/dynamic — This one will be an external drive plugged into the usb3 port of the NFS server, mounted, and exported to be used by the cluster. It will be another dynamic PV creation (via storage class) and will provide a “medium-speed” for our cluster purposes, with a bigger space tho.

/mnt/usb2/dynamic — Similar to the last one, with the difference that it will be connected via usb2, providing a slower speed and 1TB of space. I’m not sure what I’ll allocate here, some not frequently used data, maybe the docker images or backup.

So, less talk more action, let’s start connecting via ssh to the NFS server $ ssh pi@192.168.2.30

Then, let’s mount our external drives. Running: $ sudo fdisk -l we can check where our drives are installed. As you can see, I have the usb3 3TB drive connected to /dev/sdb and the usb2 1TB to /dev/sdc

If your external drive is new, you can create a partition there with the command $ sudo mkfs.ext4 /dev/sdb and $ sudo mkfs.ext4 /dev/sdc .

Mine ones are not new, but I want to start from scratch and don’t care about losing the data there.

Now, let’s mount it to the path we chose in the plan. In my case /mnt/usb3 the drive on /dev/sdb and /mnt/usb2 the drive on /dev/sdc

We are going to 1. Create the folder — $ sudo mkdir /mnt/usb2 , 2. set the pi user as the owner of the folder $ sudo chown -R pi:pi /mnt/usb2 , 3 mount it $ sudo mount /dev/sdc /mnt/usb2

You can verify if the mount was succeeded with the command $ df -ha /dev/sdc

Ok, now, we need to mount it on startup automatically. We will configure “fstab” to do it, but first, we need to find the unique id of our disks using blkid $ sudo blkid

Dig in the middle of the results, and you will find the ids for sdb and sdc. Now we need to configure the mounts in the fstab file referencing these disk ids.

Edit the fstab with the text editor of your choice, ex. $ sudo nano /etc/fstab and add the new mounts at the end of the file (with the IDs you found with blkid). In my case, the lines are these:

UUID=1a594e47-fcd5–41d5–8c55–174469983711 /mnt/usb3 ext4 defaults 0 0

UUID=2fd11a1f-4c26-4631-87b0-5b9b0d2b4c5a /mnt/usb2 ext4 defaults 0 0

Let’s then reboot the system to verify if the automatic mount works. We can check it with the command. $ df -ha /dev/sd*

PS.: Note that our external drives changed to sda and sdb, respectively, but as we are mounting it based on UUID, it’s not a problem, and the drives are being mounted in the right places

Finally, we can configure the NFS server to share all of it. First of all, let’s install the NFS server packages:

$ sudo apt-get install nfs-kernel-server -y

And edit the file /etc/exports to export all the endpoints we want to share. Just a quick refresh of our memory, we want to share (NFS export) these folders:

/ssd/host-mapped

/ssd/direct

/ssd/dynamic

/mnt/usb3/dynamic

/mnt/usb2/dynamic

Edit the exports file: $ sudo nano /etc/exports and include the following lines:

/ssd/host-mapped *(rw,no_root_squash,insecure,async,no_subtree_check,anonuid=1001,anongid=1001)

/ssd/direct *(rw,no_root_squash,insecure,async,no_subtree_check,anonuid=1002,anongid=1002)

/ssd/dynamic *(rw,no_root_squash,insecure,async,no_subtree_check,anonuid=1003,anongid=1003)

/mnt/usb3/dynamic *(rw,no_root_squash,insecure,async,no_subtree_check,anonuid=1004,anongid=1004)

/mnt/usb2/dynamic *(rw,no_root_squash,insecure,async,no_subtree_check,anonuid=1005,anongid=1005)

Then, start the NFS server $ sudo exportfs -ra

We will now install NFS in the cluster nodes and map the ssd/host-mapped on each one.

Connect via ssh to each node and install nfs-common $ sudo apt-get install nfs-common -y .

Create the folder, pass the ownership to the pi user, and edit fstab to mount it automatically:

sudo mkdir -p /ssd/host-mapped

sudo chown -R pi:pi /ssd/host-mapped

sudo nano /etc/fstab

Edit the exports file: $ sudo nano /etc/exports and include the following line (change the server IP to your NFS Server IP):

192.168.2.30:/ssd/host-mapped /ssd/host-mapped nfs rw 0 0

Reboot the node and check the mounts with: $ df -h /ssd/host-mapped

Repeat these steps into all nodes:

And that’s it for the NFS part. In the next article, we will start to configure our Persistent Volumes and publish a simple website using these NFS configurations to store the website files.