IBM Spectrum Scale Container Native Storage Access
By: Ole Kristian Myklebust
Tags: IBM, Openshift, Ibm Spectrum Storage, Csi, Spectrum Scale

Deploying IBM Spectrum Scale Container Storage Interface (CSI)

This step could also be done from the OpenShift GUI, but you need to do the prereqs.

Complete the following steps for storage cluster, CNSA cluster, and custom resource file for CSI:

Ensure the following pre-installation tasks on the storage cluster are completed.

Note: this is done on Storage Cluster GUI node

Create an IBM Spectrum Scale user group “CsiAdmin” using the following command:

/usr/lpp/mmfs/gui/cli/mkusergrp CsiAdmin --role csiadmin

Create an IBM Spectrum Scale user in the “CsiAdmin” group. This user must be used on IBM Spectrum Scale Container Storage Interface driver configuration. Issue this command on the GUI node to create the user.

/usr/lpp/mmfs/gui/cli/mkuser csi-storage-gui-user -p csi-storage-gui-password -g CsiAdmin

Perfileset quota on the file systems to be used by IBM Spectrum Scale Container Storage Interface driver is set to No

mmlsfs gpfs01 --perfileset-quota

flag value description

------------------- ------------------------ -------------------------------

--perfileset-quota No Per-fileset quota enforcement

Fileset Quota is enabled in the file systems to be used by IBM Spectrum Scale Container Storage Interface driver

mmchfs gpfs01 -Q yes

Verify that quota is enabled.

mmlsfs gpfs01 -Q



flag value description

------------------- ------------------------ -----------------------------------

-Q user;group;fileset Quotas accounting enabled

user;group;fileset Quotas enforced

none Default quotas enabled

Enable quota for root user by issuing the following command:

mmchconfig enforceFilesetQuotaOnRoot=yes -i

Ensure that the controlSetxattrImmutableSELinux parameter is set to “yes” by issuing the following command:

mmchconfig controlSetxattrImmutableSELinux=yes -i

Enable filesetdf of the file system by using the following command:

mmchfs gpfs01 --filesetdf

2. Ensure the following pre-installation tasks on the CNSA cluster are completed.

Apply the IBM Spectrum Scale Container Storage Interface driver node label to the worker nodes where CNSA pods are running. By default, these will be the Red Hat OpenShift Container Platform worker nodes.

For example, to label all Red Hat OpenShift Container Platform worker nodes to use node label “scale=true”, issue the following command:

oc label nodes -l node-role.kubernetes.io/worker= scale=true output:

node/ocp4-q2qf4-worker-2nkfk labeled

node/ocp4-q2qf4-worker-c7j2q labeled

node/ocp4-q2qf4-worker-ljbfk labeled

Create a CNSA GUI user for CSI:

oc exec -c liberty ibm-spectrum-scale-gui-0 -- /usr/lpp/mmfs/gui/cli/mkuser csi-cnsa-gui-user -p csi-cnsa-gui-password -g CsiAdmin EFSSG0019I The user csi-cnsa-gui-user has been successfully created.

EFSSG1000I The command completed successfully.

3. Gather information necessary for Configuring Custom Resource (CR) for IBM Spectrum Scale Container Storage Interface (CSI) driver from the CNSA cluster.

The GUI host for CNSA follows the format of ibm-spectrum-scale-gui.<namespace> . In this case, our GUI host would be named ibm-spectrum-scale-gui.ibm-spectrum-scale-ns.

. In this case, our GUI host would be named ibm-spectrum-scale-gui.ibm-spectrum-scale-ns. Identify node mapping:

You can use the following REST API call to validate the IBM Spectrum Scale node names under the mount:nodesMountedReadWrite section.

From the infrastructure node, issue the following command against an existing IBM Spectrum Scale pod:

oc exec ibm-spectrum-scale-core-4xlkx -- curl -k https://ibm-spectrum-scale-gui.ibm-spectrum-scale-ns/scalemgmt/v2/filesystems/fs1?fields=mount -u "csi-cnsa-gui-user:csi-cnsa-gui-password"

Retrieve OpenShift pod by issuing the oc get pods -n ibm-spectrum-scale-ns command

Output:

name” : “fs1”,

nodesMountedReadWrite” : [ “ocp4-q2qf4-worker-2nkfk”, “ocp4-q2qf4-worker-c7j2q”, “ocp4-q2qf4-worker-ljbfk”, “scale-dev-01.oslo.forum.com”, “scale-dev-02.oslo.forum.com”, “scale-dev-03.oslo.forum.com” ],

Retrieve and note down the CNSA cluster ID:

From the infrastructure node, issue the following command against an existing IBM Spectrum Scale pod:

oc exec ibm-spectrum-scale-core-4xlkx -- curl -s -k https://ibm-spectrum-scale-gui.ibm-spectrum-scale-ns/scalemgmt/v2/cluster -u "csi-cnsa-gui-user:csi-cnsa-gui-password" | grep clusterId

Retrieve and note down the CNSA cluster ID:

“clusterId” : 15871122664024915663,

Retrieve and note down the storage cluster ID:

My Storage Cluster is scale-dev-01

curl -s -k https://scale-dev-01/scalemgmt/v2/cluster -u "csi-storage-gui-user:csi-storage-gui-password" | grep clusterId Output: clusterId" : 14324309008622167364

The storage GUI host is the same host used in the Remote Mount section of CNSA Custom Resource

remoteClusters:

- name: storageCluster1

gui:

host: "example-gui.com" <----

Check your file: spectrumscale/deploy/crds/ibm_v1_scalecluster_cr.yaml

remoteClusters:

- name: storageCluster1

gui:

host: “scale-dev-01.domain.com”

Install CSI driver

The CSI have the following functional support

Static provisioning : Ability to use existing directories and filesets as persistent volumes

: Ability to use existing directories and filesets as persistent volumes Lightweight dynamic provisioning : Ability to create directory-based volumes dynamically

: Ability to create directory-based volumes dynamically Fileset-based dynamic provisioning: Ability to create fileset based volumes dynamically

Ability to create fileset based volumes dynamically Supported volume access modes: RWX (readWriteMany) and RWO (ReadWriteOnce)

Phase 1: Deploying the Operator

Create the namespace for the IBM Spectrum Scale CSI driver

oc new-project ibm-spectrum-scale-csi-driver Deploy the Operator.

oc create -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.1.0/generated/installer/ibm-spectrum-scale-csi-operator.yaml -n ibm-spectrum-scale-csi-driver

In the KC the URL had an /IBM®/ this should only be /IBM/

https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/master/generated/installer/ibm-spectrum-scale-csi-operator.yaml

Verify that the Operator is deployed, and the Operator pod is in running state.

oc get pod,deployment -n ibm-spectrum-scale-csi-driver

NAME READY STATUS RESTARTS AGE

pod/ibm-spectrum-scale-csi-operator-56955949c4-tvv6q 1/1 Running 0 28s

NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/ibm-spectrum-scale-csi-operator 1/1 1 1 29s

Phase 2: Deploying IBM Spectrum Scale Container Storage Interface driver — CSI

Create a secret for the CNSA GUI user created in step 2 of Performing pre-installation tasks for CSI Operator deployment by issuing the following command:

oc create secret generic secret-cnsa --from-literal=username=csi-cnsa-gui-user --from-literal=password=csi-cnsa-gui-password -n ibm-spectrum-scale-csi-driver Create a secret for the GUI user on the storage cluster created in Storage cluster configuration.

oc create secret generic secret-storage --from-literal=username=csi-storage-gui-user --from-literal=password=csi-storage-gui-password -n ibm-spectrum-scale-csi-driver



oc label secret secret-storage product=ibm-spectrum-scale-csi Download the sample custom resource file on your cluster.

curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.1.0/operator/deploy/crds/csiscaleoperators.csi.ibm.com_cr.yaml Modify the parameters in the file to suit your environment. For more information, see Configuring Custom Resource (CR) for CSI driver. Once finished modifying the CR, return to this page and continue to step 5.

I copied the table below, so that i could fill in the details after each of the steps.

Configuring Custom Resource (CR) for CSI driver

There was som jumping between multiple pages to be sure that i have created the file correctly.

After also checking this: csi_remote_cluster_support

so after after editing: csiscaleoperators.csi.ibm.com_cr.yaml

I ended up with this file. (Shorten down)

5. Create the CR file.

oc create -f csiscaleoperators.csi.ibm.com_cr.yaml -n ibm-spectrum-scale-csi-driver

Output: csiscaleoperator.csi.ibm.com/ibm-spectrum-scale-csi created

6. Verify that IBM Spectrum Scale Container Storage Interface driver is installed, Operator and driver resources are ready, and pods are in running state.

oc get pod,daemonset,statefulset -n ibm-spectrum-scale-csi-driver

My Pod crashed, so I needed to check the logs.

oc logs ibm-spectrum-scale-csi-mbdqz ibm-spectrum-scale-cs

I have typed in the wrong cluster ID (or more correctly I copied the ID but the last digit was missing…) As the error was saying: Failed to Initialize scale CSI Driver: ID doesnt match the cluster



rest_v2.go:586] rest_v2 doHTTP. endpoint:

I1217 20:51:52.048073 1 http_utils.go:74] http_utils HttpExecuteUserAuth. type: GET, url:

I1217 20:51:52.198898 1 http_utils.go:44] http_utils UnmarshalResponse. response: &{0x5a5c00 0xc00001e680 0x6d56a0}

I1217 20:51:52.199043 1 rest_v2.go:43] rest_v2 isStatusOK. statusCode: 200

E1217 20:51:52.199064 1 gpfs.go:202] Cluster ID 1432430900862216736 from scale config doesnt match the ID from cluster 14324309008622167364.

E1217 20:51:52.199073 1 gpfs.go:143] Error in plugin initialization: Cluster ID doesnt match the cluster

F1217 20:51:52.199083 1 main.go:65] Failed to initialize Scale CSI Driver: Cluster ID doesnt match the cluster

[root@ocp-admin scale-binary]# rest_v2.go:586] rest_v2 doHTTP. endpoint: https://scale-dev-01.oslo.forum.com:443/scalemgmt/v2/cluster , method: GET, param: I1217 20:51:52.048073 1 http_utils.go:74] http_utils HttpExecuteUserAuth. type: GET, url: https://scale-dev-01.oslo.forum.com:443/scalemgmt/v2/cluster , user: csi-storage-gui-userI1217 20:51:52.198898 1 http_utils.go:44] http_utils UnmarshalResponse. response: &{0x5a5c00 0xc00001e680 0x6d56a0}I1217 20:51:52.199043 1 rest_v2.go:43] rest_v2 isStatusOK. statusCode: 200E1217 20:51:52.199064 1 gpfs.go:202] Cluster ID 1432430900862216736 from scale config doesnt match the ID from cluster 14324309008622167364.E1217 20:51:52.199073 1 gpfs.go:143] Error in plugin initialization: Cluster ID doesnt match the clusterF1217 20:51:52.199083 1 main.go:65] Failed to initialize Scale CSI Driver: Cluster ID doesnt match the cluster[root@ocp-admin scale-binary]#

So deleted the operator and tried again.

oc delete -f csiscaleoperators.csi.ibm.com_cr.yaml Output: csiscaleoperator.csi.ibm.com “ibm-spectrum-scale-csi” deleted

Create the CR again.

oc create -f csiscaleoperators.csi.ibm.com_cr.yaml -n ibm-spectrum-scale-csi-driver

Checked again, and the pods were starting.

# oc get pod,daemonset,statefulset -n ibm-spectrum-scale-csi-driver NAME READY STATUS RESTARTS AGE

pod/ibm-spectrum-scale-csi-attacher-0 1/1 Running 0 9m53s

pod/ibm-spectrum-scale-csi-hvg6n 2/2 Running 0 9m35s

pod/ibm-spectrum-scale-csi-jtj2x 2/2 Running 0 9m35s

pod/ibm-spectrum-scale-csi-operator-56955949c4-2z77d 1/1 Running 0 7h9m

pod/ibm-spectrum-scale-csi-provisioner-0 1/1 Running 0 9m44s

pod/ibm-spectrum-scale-csi-rd6fd 2/2 Running 0 9m35s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

daemonset.apps/ibm-spectrum-scale-csi 3 3 3 3 3 scale=true 9m35s NAME READY AGE

statefulset.apps/ibm-spectrum-scale-csi-attacher 1/1 9m53s

statefulset.apps/ibm-spectrum-scale-csi-provisioner 1/1 9m44s

Are you still here? 😃 we are almost there now!

Create Storage Class and PVC for Spectrum Scale.

The CSI for CSNA have the following functional support

Static provisioning : Ability to use existing directories and filesets as persistent volumes

: Ability to use existing directories and filesets as persistent volumes Lightweight dynamic provisioning : Ability to create directory-based volumes dynamically

: Ability to create directory-based volumes dynamically Fileset-based dynamic provisioning: Ability to create fileset based volumes dynamically

Ability to create fileset based volumes dynamically Supported volume access modes: RWX (readWriteMany) and RWO (ReadWriteOnce)

Fileset: you are maybe asking yourself, what is fileset and what is different between independent and dependent fileset.

From the KC: GPFS utilizes a file system object called a fileset. A fileset is a subtree of a file system namespace that in many respects behaves like an independent file system. Filesets provide a means of partitioning the file system to allow administrative operations at a finer granularity than the entire file system:

Filesets can be used to define quotas on both data blocks and inodes.

The owning fileset is an attribute of each file and can be specified in a policy to control initial data placement, migration, and replication of the file’s data. See Policies for automating file management.

Fileset snapshots can be created instead of creating a snapshot of an entire file system.

Independent and dependent filesets:

An independent fileset is a fileset with its own inode space. An inode space is a collection of inode number ranges reserved for an independent fileset. An inode space enables more efficient per-fileset functions, such as fileset snapshots.

Dependent fileset shares the inode space of an existing, independent fileset. Files created in a dependent fileset are assigned inodes in the same collection of inode number ranges that were reserved for the independent fileset from which it was created.

When the file system is created, only one fileset, called the root fileset, exists. The root fileset is an independent fileset that cannot be deleted. It contains the root directory as well as any system files such as quota files. As new files and directories are created, they automatically become part of the parent directory’s fileset. The fileset to which a file belongs is largely transparent for ordinary file access, but the containing fileset can be displayed along with the other attributes of each file using the mmlsattr -L command.

So if we look into this some more.

We have 2 directories in my mount /mnt/gpfs01 on the Storage Cluster

pvc-9* is a test with independent fileset.

is a test with independent fileset. spectrum-scale-csi-volume-store is also a fileset

# ls -la

pvc-9b789e4c-744b-4520–9b04–487fa6ae29ca

spectrum-scale-csi-volume-store

If I list filesets in the filesystem gpfs01.

We can see 3 filesets, same as above but also root, all filesets are linked to /mnt/gpfs01/*

[root@scale-dev-01 gpfs01]# mmlsfileset gpfs01 Filesets in file system ‘gpfs01’:

Name Status Path

root Linked /mnt/gpfs01

spectrum-scale-csi-volume-store Linked /mnt/gpfs01/spectrum-scale-csi-volume-store

pvc-9b789e4c-744b-4520–9b04–487fa6ae29ca Linked /mnt/gpfs01/pvc-9b789e4c-744b-4520–9b04–487fa6ae29ca

Note: To check the attributes for each filset, use the command: mmlsattr -L <fileset>

mmlsattr -L spectrum-scale-csi-volume-store/ file name: spectrum-scale-csi-volume-store/

metadata replication: 1 max 2

immutable: no

appendOnly: no

flags:

storage pool name: system

fileset name: spectrum-scale-csi-volume-store

snapshot name:

creation time: Thu Dec 17 21:08:00 2020

Misc attributes: DIRECTORY

Encrypted: no

If we look into lwdir directory and pvc claim, the fileset is spectrum-scale-csi-volume-store.

mmlsattr -L spectrum-scale-csi-volume-store/lwdir/pvc-984b42ce-c9c4-4764-baa8-4765d227dccb/ file name: spectrum-scale-csi-volume-store/lwdir/pvc-984b42ce-c9c4-4764-baa8-4765d227dccb/

metadata replication: 1 max 2

immutable: no

appendOnly: no

flags:

storage pool name: system

fileset name: spectrum-scale-csi-volume-store

snapshot name:

creation time: Thu Dec 17 22:01:16 2020

Misc attributes: DIRECTORY

Encrypted: no

To check what filesets have quotas, use the command mmrepquouta

mmrepquota -j gpfs01 --block-size G

From the Spectrum Scale GUI on your Storage Server, you will also get a good overview. Example: Type, Limits/quota, inode, Snapshots,