Setting up OpenEBS with cStor Pools on Red Hat OpenShift 4
By: Jim D'Agostino
Tags: Openebs, Storage, Openshift, Kubernetes, Cstor

This page is a collection of install notes from multiple sources, that is intended to be the most straight forward way to get started with OpenEBS on OpenShift — or it was in my case — mileage may vary.

Sources

Prerequisites

This document assumes you already have OpenShift up and running. You will want to be running version 4.2 or later (I am running 4.2.10).

(3.10 is also supported, but this page has been wrote with 4.2 in mind)

(3.10 is also supported, but this page has been wrote with 4.2 in mind) Your “worker” nodes should have additional storage disks attached. I am attaching disks from VMware, but the same should apply for any cloud vendor. These disks will be consumed by OpenEBS as usable storage space.

The additional storage disks should NOT be mounted, and should NOT contain a filesystem if you plan to use cStor Storage Pools.

Verify iSCSI Client

Source: https://docs.openebs.io/docs/next/prerequisites.html#rhel

OpenEBS provides block volume support through the iSCSI protocol. Therefore, the iSCSI client (initiator) presence on all Kubernetes nodes is required.

On each of your nodes you plan to have disks attached, check that the iSCSI service is running using the following command:

$ systemctl status iscsid

If the status is showing as Inactive , then you may have to enable and start the iscsid service using the following command.

$ sudo systemctl enable iscsid && sudo systemctl start iscsid

Install OpenEBS from OperatorHub

Login as an admin user, or user with cluster-admin role/permissions.

oc login -u admin

Create an namespace (project) into which OpenEBS will be installed.

$ oc create namespace openebs

Ensure that the serviceaccount used for the OpenEBS control plane deployments is added into the privileged SCC.

$ oc adm policy add-scc-to-user privileged system:serviceaccount:openebs:openebs-maya-operator

Provide access to the host-volumes (which are required by the OpenEBS volume replicas) by updating the default security context (scc) using the following command. (Update allowHostDirVolumePlugin to be true )

$ oc edit scc restricted allowHostDirVolumePlugin: true

From the Openshift console, open the OperatorHub, and find OpenEBS.

Click Install and configure your Operator “subscription”. Make sure to install to the openebs namespace.

Validate there are no install issues. InstallSucceeded is expected.

Click into the Operator and change tabs to OpenEBS Install Template.

Click Create OpenEBS Install Template, Keep the defaults, and click Create.

Check that all the pods were created without issue.

$ oc get pods -n openebs



NAME READY STATUS RESTARTS AGE

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-admission-server-7c69sp4 1/1 Running 0 5m20s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-apiserver-f65969-pjq5w 1/1 Running 2 5m20s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-localpv-provisionert2b6s 1/1 Running 0 5m20s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-ndm-gltpf 1/1 Running 0 5m20s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-ndm-operator-64c4ffmpfkp 1/1 Running 1 5m21s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-ndm-sbbqh 1/1 Running 0 5m20s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-ndm-wwh89 1/1 Running 0 5m20s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-provisioner-69c46fbr9xf6 1/1 Running 0 5m21s

oebs-brz7zili4sd236m6pr4ibxt1d-openebs-snapshot-operator-d24kkh 2/2 Running 0 5m21s

openebs-operator-79d4d575fc-86bdt 1/1 Running 0 16m

Configure Storage Pool and Storage Class

OPTIONAL: Label all your worker nodes so we can limit where storage pods run. (In my case, these are the nodes that I have attached storage disks)

$ oc get nodes | grep worker

prd-os-w01.hooli.com Ready worker 40d v1.14.6+888f9c630

prd-os-w02.hooli.com Ready worker 40d v1.14.6+888f9c630

prd-os-w03.hooli.com Ready worker 40d v1.14.6+888f9c630 $ oc label node prd-os-w01.hooli.com node=openebs

node/ prd-os-w01.hooli.com labeled $ oc label node prd-os-w02.hooli.com node=openebs

node/ prd-os-w02.hooli.com labeled $ oc label node prd-os-w03.hooli.com node=openebs

node/ prd-os-w03.hooli.com labeled

Collect the list of available block devices that can be used in a Storage Pool Claim.

$ oc get blockdevice -n openebs

blockdevice-0a02e2359b8c4628c5471abbd7829795 node prd-os-w01.hooli.com

blockdevice-79718ef7cd72adfc930af04e3c47eeaa node prd-os-w01.hooli.com

blockdevice-88d9c387348bea4775f953e80395e345 node prd-os-w02.hooli.com

blockdevice-acda9397092f2b71ce1a6798b5bc3d3d node prd-os-w02.hooli.com

blockdevice-dd2a75b44de36336a582bbeee552e88c node prd-os-w03.hooli.com

blockdevice-eca89291e7b351805be83bd72c8044e5 node prd-os-w03.hooli.com NAME NODENAME SIZE CLAIMSTATE STATUS AGEblockdevice-0a02e2359b8c4628c5471abbd7829795 node prd-os-w01.hooli.com 536870912000 Unclaimed Active 22mblockdevice-79718ef7cd72adfc930af04e3c47eeaa node prd-os-w01.hooli.com 536870912000 Unclaimed Active 22mblockdevice-88d9c387348bea4775f953e80395e345 node prd-os-w02.hooli.com 536870912000 Unclaimed Active 22mblockdevice-acda9397092f2b71ce1a6798b5bc3d3d node prd-os-w02.hooli.com 536870912000 Unclaimed Active 22mblockdevice-dd2a75b44de36336a582bbeee552e88c node prd-os-w03.hooli.com 536870912000 Unclaimed Active 22mblockdevice-eca89291e7b351805be83bd72c8044e5 node prd-os-w03.hooli.com 536870912000 Unclaimed Active 22m

Build a new yaml file with your blockdevices that creates your StoragePoolClaim.

$ cat storagepoolclaim.yaml



apiVersion: openebs.io/v1alpha1

kind: StoragePoolClaim

metadata:

name: cstor-disk-pool

annotations:

cas.openebs.io/config: |

- name: PoolResourceRequests

value: |-

memory: 2Gi

- name: PoolResourceLimits

value: |-

memory: 4Gi

spec:

name: cstor-disk-pool

type: disk

poolSpec:

poolType: striped

blockDevices:

blockDeviceList:

- blockdevice-0a02e2359b8c4628c5471abbd7829795

- blockdevice-79718ef7cd72adfc930af04e3c47eeaa

- blockdevice-88d9c387348bea4775f953e80395e345

- blockdevice-acda9397092f2b71ce1a6798b5bc3d3d

- blockdevice-dd2a75b44de36336a582bbeee552e88c

- blockdevice-eca89291e7b351805be83bd72c8044e5

Create your StoragePoolClaim. All your BlockDevices should now be claimed.

$ oc create -f storagepoolclaim.yaml

storagepoolclaim.openebs.io/cstor-disk-pool created





$ oc get spc

NAME AGE

cstor-disk-pool 87s



$ kubectl get blockdevice -n openebs

NAME NODENAME SIZE CLAIMSTATE STATUS AGE

blockdevice-0a02e2359b8c4628c5471abbd7829795 prd-os-w01.hooli.com 536870912000 Claimed Active 34m

blockdevice-79718ef7cd72adfc930af04e3c47eeaa prd-os-w01.hooli.com 536870912000 Claimed Active 34m

blockdevice-88d9c387348bea4775f953e80395e345 prd-os-w02.hooli.com 536870912000 Claimed Active 34m

blockdevice-acda9397092f2b71ce1a6798b5bc3d3d prd-os-w02.hooli.com 536870912000 Claimed Active 34m

blockdevice-dd2a75b44de36336a582bbeee552e88c prd-os-w03.hooli.com 536870912000 Claimed Active 34m

blockdevice-eca89291e7b351805be83bd72c8044e5 prd-os-w03.hooli.com 536870912000 Claimed Active 34m

Since OpenEBS/cstor are a CAS (container attached storage) model, you will now have new pods on each of your “node=openebs” nodes.

These are the pods that interface with your storage disks.

$ oc get pods -n openebs | grep cstor-disk-pool



cstor-disk-pool-1lwq-77b76fd5b7-5hw5r 3/3 Running 0 5m51s

cstor-disk-pool-bvh8-6b8b6dc5d7-khnr7 3/3 Running 0 5m52s

cstor-disk-pool-o2yc-f8dcf6648-24kg8 3/3 Running 0 5m51s

Finally create a StorageClass that takes advantage of your StoragePoolClaim. This is what apps will take advantage of to request storage. (Update the storage pool to fit your application needs, we change ReplicaCount depending on the app)

$ cat openebs-sc-disk.yaml



apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

name: openebs-cstor-disk

annotations:

openebs.io/cas-type: cstor

cas.openebs.io/config: |

- name: StoragePoolClaim

value: "cstor-disk-pool"

- name: ReplicaCount

value: "1"

provisioner: openebs.io/provisioner-iscsi

reclaimPolicy: Delete $ oc apply -f openebs-sc-disk.yaml

storageclass.storage.k8s.io/openebs-cstor-disk created

At this point you can configure your app to use “openebs-cstor-disk” as it’s Storage Class, and you will have disks provisioned.