Increase GlusterFS volume size in Kubernetes
By: Aymen Lamara
Tags: Heketi, Glusterfs, Kubernetes, Storage

One of the common issues when using a persistent volume solution in Kubernetes, like GlusterFS, is to start running out of storage capacity in your cluster, and you have to find a smooth way to increase volume size without losing your data. In this short article I will show you two ways of doing it.

Before going deeper, let me introduce you the context and some terminologies that will be used throughout this article.

From official websites:

“Gluster is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace”.

“Heketi provides a RESTful management interface which can be used to manage the lifecycle of GlusterFS volumes”.

Topology.json, “heketi uses this configuration file to define cluster nodes topology for GlusterFS. It contains the nodes’ IP addresses and the storage devices to use in each node”.

It looks like:

{

“clusters”: [

{

“nodes”: [

{

“node”: {

“hostnames”: {

“manage”:

[“node4”],

“storage”: [“10.0.XX.XX”]

},

“zone”: 1

},

“devices”: [

{

“name”: “/dev/xvdb”,

“destroydata”: false

}

]

}

…

I use GlusterFS and Heketi inside a Kubernetes cluster, running on six EC2 machines; three masters/etcd and three nodes (workers).

I use the Kubespray project to setup the cluster on an AWS environment. It has the advantage of using Ansible playbooks to setup the necessary bricks for Kubernetes, whether on AWS or elsewhere.

In the same project, there are also additional Ansible playbooks that I have used to setup GlusterFS and Heketi storage topology. The latter is running on k8s nodes and uses EBS volumes attached to EC2 instances. You need to customize this topology to fit your needs.

The picture below shows roughly how the setup is made.

I started with a volume of 30GiB mounted on /dev/xvdb device. The k8s cluster was almost empty, only some monitoring Daemonsets were running and other light applications with 8GiB of PersistentVolumes for each; everything was good until I had touched the limits. At this point I figured out that PersistentVolumeClaims were stuck in pending state forever. Here is an example of stucking pvc for Minio S3 storage.

minio persistentvolumeclaim/minio-pvc Pending 1d

Looking in logs, I found a clear message saying that there was not enough storage capacity to run the pvc.

Warning ProvisioningFailed … Failed to provision volume with StorageClass “gluster”: failed to create volume: heketi block volume creation failed: [heketi] failed to create volume: Failed to allocate new block volume: No space

Let’s get our hands dirty

There are two ways I have used to increase volume capacity:

Extend the volume attached to VM.

Add another storage device (for example: /dev/xvdc) to heketi topology.

1. Extend the volume:

This method assumes that you have already a volume mounted in heketi cluster. It can be used on cloud providers which allow you to expand volumes on-the-fly, like AWS, GCP etc.

The first step is to expand your volume size (from web GUI, command line, terraform …) SSH into the said instance. Expand the physical volume, in my case I added another 30GiB

pvresize /dev/xvdb

Physical volume “/dev/xvdb” changed

1 physical volume(s) resized / 0 physical volume(s) not resized

Before:

vgdisplay

— — Volume group — -

VG Name vg_a1751200878b2bb0a64d89a15e3bc801

VG Size 29.87 GiB

After:

vgdisplay

— — Volume group — -

VG Name vg_a1751200878b2bb0a64d89a15e3bc801

VG Size 59.87 GiB

4. Now tell Heketi to take into account the new volume size with Resync device. The heketi command lines are available on this website.

5. Connect to a master node or any other machine that have a remote and configured kubectl and do the following steps:

# Get heketi pod name kubectl get pods -n kube-system | grep heketi

# Get heketi secret for Admin access in order to use API as Admin kubectl get secret heketi-config-secret -n kube-system -o jsonpath — template ‘{.data.heketi\.json}’ | base64 -d

# List heketi cluster Name kubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXXX cluster list ClusterId: Id:65ddc905b8a32dc49db721a004a08ead

# Get detailed informations about the cluster kubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX cluster info 65ddc905b8a32dc49db721a004a08ead Cluster id: 65ddc905b8a32dc49db721a004a08ead Nodes: 310744c2908338d7d229268a4445951b 9ac5b33d03759ba7d9b161b879a4ea39 e0fd9cbad89eac2989cff2b426a59288 Volumes: 29e2a3fd578bb98a7d44e9a538cc9e74 7c955fe8dff62136046fc5c794abb6a5 b22ad96e06a4bff70c16e9fe61a4ed50 Block: true File: true

# Get informations about one node, you can apply the command again with another node id kubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39 Node Id: 310744c2908338d7d229268a4445951b State: online Cluster Id: 65ddc905b8a32dc49db721a004a08ead Zone: 1 Management Hostname: node5 Storage Hostname: 10.0.XX.XX Devices: Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdb State:online Size (GiB):34 Used (GiB):22 Free (GiB):12 Bricks:3

# Resync device kubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX device resync a1751200878b2bb0a64d89a15e3bc801 Device updated

6. Once the resync is done, you can check again the node to see what has changed.

# Get updated informations about node5 kubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39 Node Id: 310744c2908338d7d229268a4445951b State: online Cluster Id: 65ddc905b8a32dc49db721a004a08ead Zone: 1 Management Hostname: node5 Storage Hostname: 10.0.XX.XX Devices: Id:Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdb State:online Size (GiB):64 Used (GiB):22 Free (GiB):42 Bricks:3

7. At last, do not forget to run the same resync command for all glusterFS nodes (node4, node6 in my case).

And now you have just increased volume capacity in your cluster.

2. Add another storage device to heketi topology

In this case, I’m going to add a new storage device to heketi topology. This method is most likely used when you cannot increase volume sizes on-the-fly, and you have to add another volume in order to expand storage capacity in your cluster. These steps are inspired from IBM’s website.

Please note that the volume must be unmounted and empty, no partition or filesystem must be created on it, so it can be used by heketi.

Create and attach the volume to instance, for example: /dev/xvdc You can make sure to clear up your volume by using:

wipefs — all — force /dev/xvdc

3. Connect to a master node or any machine that have a remote and configured kubectl.

4. This step from the previous procedure is optional, I am using it just to get more visibility about what is happening, and also a small reminder for whomever that uses the second procedure.

Get heketi pod name

Get heketi secret for admin access

List cluster Name

Get more informations about the cluster

Get informations about one node, you can apply the command again with another node id

5. Add this volume to heketi topology:

# To add a new device to heketi, you must use volume ID got by cluster info command or heketi-cli volume list kubectl -n kube-system exec heketi-d55cbd78c-7tj8c — heketi-cli — user admin — secret XXXXX device add — name=/dev/xvdc — node=e0fd9cbad89eac2989cff2b426a59288 Device added successfully

Before:

kubectl -n kube-system exec heketi-d55cbd78c-7tj8c — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39 Node Id: 310744c2908338d7d229268a4445951b State: online Cluster Id: 65ddc905b8a32dc49db721a004a08ead Zone: 1 Management Hostname: node5 Storage Hostname: 10.0.25.109 Devices: Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdb State:online Size (GiB):34 Used (GiB):22 Free (GiB):12 Bricks:3

After:

kubectl -n kube-system exec heketi-d55cbd78c-7tj8c — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39 Node Id: 310744c2908338d7d229268a4445951b State: online Cluster Id: 65ddc905b8a32dc49db721a004a08ead Zone: 1 Management Hostname: node5 Storage Hostname: 10.0.25.109 Devices: Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdb State:online Size (GiB):34 Used (GiB):22 Free (GiB):12 Bricks:3 Id:fcb295c112eb0a1bbfa9e6fca8a3f30a Name:/dev/xvdc State:online Size (GiB):59 Used (GiB):0 Free (GiB):59 Bricks:5

6. Do not forget to apply the previous step on all glusterFS nodes (node4 and node6 in my case).

Conclusion

GlusterFs is a serious project supported by RedHat, IBM and others. It is a good start to play with dynamic volumes in k8s. For now the project is still young and I didn’t test it in a production context.

I have used both methods shown in this article and they work fine. Now enjoy your storage :)

Special thanks to my friends from The WeTribu who have patiently reviewed this article.