OpenShift Data Foundation (ODF)  - 4.16 Installation

### OpenShift Data Foundation (ODF) 

Red Hat OpenShift Data Foundation とは

Red Hat® OpenShift® Data Foundation (旧 Red Hat OpenShift Container Storage) は、コンテナ用ソフトウェア・デファインド・ストレージであり、Red Hat OpenShift に緊密に統合されたデータサービスを提供します。Red Hat OpenShift Data Foundation は、クラウド全体でアプリケーションを迅速かつ効率的に開発し、デプロイするのに役立ちます。

https://www.redhat.com/ja/technologies/cloud-computing/openshift-data-foundation

ここでは、以下のドキュメントに基づいて、AWS 上の OpenShift Container Platform (OCP) IPI/UPI 環境に ODF 4.16 を導入・設定する手順をご紹介します。

https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html-single/deploying_openshift_data_foundation_using_amazon_web_services/index

### Installing Red Hat OpenShift Data Foundation Operator

#### Infrastructure Node
以下を参考にして Infrastructure Node を設定します。

https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html-single/managing_and_allocating_storage_resources/index?extIdCarryOver=true&percmp=7015Y0000048KZLQA2&sc_cid=701f2000001OH7JAAW#manual_creation_of_infrastructure_nodes

今回は `instance-type` が `r5.8xlarge` の Worker Node 6台を Infrastructure Node に設定してみます。
```shell
$ oc get node -o name -l node-role\.kubernetes\.io/worker -l node.kubernetes.io/instance-type="r5.8xlarge"
node/ip-33-555-14-118.us-westside-99.machine.external
node/ip-33-555-14-19.us-westside-99.machine.external
node/ip-33-555-14-72.us-westside-99.machine.external
node/ip-33-555-14-78.us-westside-99.machine.external
node/ip-33-555-14-94.us-westside-99.machine.external
node/ip-33-555-14-99.us-westside-99.machine.external
```

```shell
$ for N in $(oc get node -o name -l node-role\.kubernetes\.io/worker -l node.kubernetes.io/instance-type="r5.8xlarge")
do
  oc label $N node-role.kubernetes.io/infra=""
  oc label $N cluster.ocs.openshift.io/openshift-storage=""
  oc adm taint $N node.ocs.openshift.io/storage="true":NoSchedule
done
node/ip-33-555-14-19.us-westside-99.machine.external labeled
node/ip-33-555-14-19.us-westside-99.machine.external labeled
node/ip-33-555-14-19.us-westside-99.machine.external tainted
node/ip-33-555-14-72.us-westside-99.machine.external labeled
node/ip-33-555-14-72.us-westside-99.machine.external labeled
node/ip-33-555-14-72.us-westside-99.machine.external tainted
node/ip-33-555-14-78.us-westside-99.machine.external labeled
node/ip-33-555-14-78.us-westside-99.machine.external labeled
node/ip-33-555-14-78.us-westside-99.machine.external tainted
node/ip-33-555-14-94.us-westside-99.machine.external labeled
node/ip-33-555-14-94.us-westside-99.machine.external labeled
node/ip-33-555-14-94.us-westside-99.machine.external tainted
node/ip-33-555-14-99.us-westside-99.machine.external labeled
node/ip-33-555-14-99.us-westside-99.machine.external labeled
node/ip-33-555-14-99.us-westside-99.machine.external tainted
node/ip-33-555-14-118.us-westside-99.machine.external labeled
node/ip-33-555-14-118.us-westside-99.machine.external labeled
node/ip-33-555-14-118.us-westside-99.machine.external tainted
```
#### Install ODF Operator
Operator Hub から ODF Operator を導入します。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/1db53b64-ae7d-49c1-cab2-818e59e169a8.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/8eca2032-8775-45fa-f11e-ac7757da6555.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/975a63a8-ab29-a1e6-5d6e-093902a43381.png)

導入が完了すると、以下のようになります。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/78050f76-0b10-5662-51b0-9df387c6f920.png)

### Creating an OpenShift Data Foundation cluster

`Installed Operator` から `OpenShift Data Foundation` を選択し、`Create StorageSystem` Tab から `StorageSystem` を作成します。

最初の画面で選択可能な `Storage Class` は以下の2つで、Backing Storage として `Amazon Elastic Block Store (EBS)` を使用します。今回は `gp3-csi` を使用します。

https://github.com/kubernetes-sigs/aws-ebs-csi-driver

```shell
$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2-csi                       ebs.csi.aws.com                         Delete          WaitForFirstConsumer   true                   7d
gp3-csi (default)             ebs.csi.aws.com                         Delete          WaitForFirstConsumer   true                   7d

$ oc -o yaml get sc gp3-csi
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: gp3-csi
parameters:
  encrypted: "true"
  type: gp3
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
```

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/3d40ecfe-816c-4e46-7d2b-212eabc62fcc.png)

`Requested capacity` で実効容量を選択します。0.5 / 2 / 4 (TiB) から選択できますが、今回は `4 TiB` とします。
`Select nodes` では、先程 Label を設定した Node が選択された状態で表示されます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/0e3dc86b-22f7-ba7a-f187-1bf3df29f8b3.png)

暗号化の設定を行います。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/33ac39ff-b222-89c5-c060-a98915556706.png)

最終画面で設定一覧が表示され `Create StorageSystem` ボタンで実行されます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/f10cdc6c-edbd-e28f-6c47-e403d05f44eb.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/8bdbe9c4-d13c-cdf2-a66f-0ef6ddee927f.png)

### Verifying OpenShift Data Foundation deployment
#### Resources
ODF のリソースは `openshift-storage` Project に作成されるため、`oc get all` 等でリソースの状況を確認します。

#### PV
以下の PV が作成されます。`CAPACITY=4Ti` がデータ用 PV で、Backing Storage は EBS になります。暗号化を指定したため、AWS Console からも該当 EBS Volume の暗号化設定が確認できます。`.spec.csi.volumeHandle` が EBS の `Volume ID` に該当します。
StorageSystem を削除すると、これらの PV も自動的に削除されます。 

```shell
$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                   STORAGECLASS                  VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-0a4a38c0-f790-44b5-8113-ff2d7aa4d86a   50Gi       RWO            Delete           Bound    openshift-storage/db-noobaa-db-pg-0                     ocs-storagecluster-ceph-rbd   <unset>                          47h
pvc-26c5a955-bceb-4467-af2b-4359b3858ab9   4Ti        RWO            Delete           Bound    openshift-storage/ocs-deviceset-gp3-csi-2-data-0bxxc9   gp3-csi                       <unset>                          47h
pvc-6a628a05-004a-491f-a345-369140d3a3b1   50Gi       RWO            Delete           Bound    openshift-storage/rook-ceph-mon-b                       gp3-csi                       <unset>                          47h
pvc-8a83005b-61b0-4acd-baa0-909374c1b349   50Gi       RWO            Delete           Bound    openshift-storage/rook-ceph-mon-a                       gp3-csi                       <unset>                          47h
pvc-9b95f79c-10e0-4a02-8f1c-ae804015a8ae   4Ti        RWO            Delete           Bound    openshift-storage/ocs-deviceset-gp3-csi-0-data-086rvd   gp3-csi                       <unset>                          47h
pvc-e5226452-dc5d-4b2c-af70-9633e0958f41   4Ti        RWO            Delete           Bound    openshift-storage/ocs-deviceset-gp3-csi-1-data-0fn6zd   gp3-csi                       <unset>                          47h
pvc-f4b99572-f16c-4260-87d7-91dc9e38ed9b   50Gi       RWO            Delete           Bound    openshift-storage/rook-ceph-mon-c                       gp3-csi                       <unset>                          47h

$ oc -o yaml get pv pvc-26c5a955-bceb-4467-af2b-4359b3858ab9
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
    volume.kubernetes.io/provisioner-deletion-secret-name: ""
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/ebs-csi-aws-com
  name: pvc-26c5a955-bceb-1234-af2b-abe910d
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 4Ti
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: ocs-deviceset-gp3-csi-2-data-axyb
    namespace: openshift-storage
  csi:
    driver: ebs.csi.aws.com
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: xxxx-yyyy-ebs.csi.aws.com
    volumeHandle: vol-abcd0123syz
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.ebs.csi.aws.com/zone
          operator: In
          values:
          - ap-northeast-3b
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gp3-csi
  volumeMode: Block
```

#### StorageClass
以下の StorageClass が追加されます。
```shell
$ oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   4d18h
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   4d18h
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  4d18h


$ oc -o yaml get sc ocs-storagecluster-ceph-rbd ocs-storagecluster-cephfs openshift-storage.noobaa.io
apiVersion: v1
items:
- allowVolumeExpansion: true
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      description: Provides RWO Filesystem volumes, and RWO and RWX Block volumes
    name: ocs-storagecluster-ceph-rbd
  parameters:
    clusterID: openshift-storage
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
    csi.storage.k8s.io/fstype: ext4
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
    imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
    imageFormat: "2"
    pool: ocs-storagecluster-cephblockpool
  provisioner: openshift-storage.rbd.csi.ceph.com
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  
- allowVolumeExpansion: true
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      description: Provides RWO and RWX Filesystem volumes
    name: ocs-storagecluster-cephfs
  parameters:
    clusterID: openshift-storage
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
    csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
    fsName: ocs-storagecluster-cephfilesystem
  provisioner: openshift-storage.cephfs.csi.ceph.com
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      description: Provides Object Bucket Claims (OBCs)
    name: openshift-storage.noobaa.io
  parameters:
    bucketclass: noobaa-default-bucket-class
  provisioner: openshift-storage.noobaa.io/obc
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
```

#### Console
OCP Console の `Storage` 配下に `Data Foundation` が追加され、ODF の状況を確認することが出来ます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/8a275206-1a8c-b9ad-60df-5949ddd456da.png)