OpenShift AI - Computing Resources

#### OpenShift AI

Red Hat® OpenShift® AI とは、柔軟でスケーラブルな人工知能 (AI) および機械学習 (ML) プラットフォームです。このプラットフォームにより、企業はハイブリッドクラウド環境全体で AI 対応アプリケーションを大規模に作成および提供できるようになります。

OpenShift AI はオープンソース・テクノロジーを使用して構築されており、実験、モデル提供、革新的なアプリケーションの提供のための、信頼性と一貫性に優れた運用機能を提供します。

https://www.redhat.com/ja/technologies/cloud-computing/openshift/openshift-ai

#### Computing Resources
ここでは、OpenShift AI 2.9.1 / Red Hat OpenShift on IBM Cloud (ROKS) 4.14 の環境で、どの程度の Computing Resource が使用されているか確認します。

今回は、`mx2-8x64` Flavor の Worker Node x3 を使用します。

https://cloud.ibm.com/docs/vpc?topic=vpc-profiles&interface=ui#memory

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/9887f499-24a7-98b7-44aa-0c65c48aab94.png)

##### OpenShift AI Components
OpenShift AI の導入に必要な Computing Resource の要件は、以下に記載されています。

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.9/html-single/installing_and_uninstalling_openshift_ai_self-managed/index#requirements-for-openshift-ai-self-managed_install

- Use an existing cluster or create a new cluster by following the OpenShift Container Platform documentation: OpenShift Container Platform installation overview.

    Your cluster must have `at least 2 worker nodes with at least 8 CPUs and 32 GiB RAM available for OpenShift AI to use when you install the Operator.` To ensure that OpenShift AI is usable, additional cluster resources are required beyond the minimum requirements.
    <BR>
- A default storage class that can be dynamically provisioned must be configured.

    `Confirm that a default storage class is configured by running the oc get storageclass command.` If no storage classes are noted with (default) beside the name, follow the OpenShift Container Platform documentation to configure a default storage class: Changing the default storage class. For more information about dynamic provisioning, see Dynamic provisioning.
    <BR>
- Open Data Hub must not be installed on the cluster.

    For more information about managing the machines that make up an OpenShift cluster, see Overview of machine management. 

OpenShift AI では、導入・構成した `Component` の Pod が `redhat-ods-applications` Project で稼働します。
```shell
$ oc get pod -n redhat-ods-applications --sort-by=.spec.nodeName
NAME                                                              READY   STATUS    RESTARTS       AGE
notebook-controller-deployment-556969c7cf-jj5jj                   1/1     Running   0              15d
rhods-dashboard-6f64f58c89-pwhqh                                  2/2     Running   0              15d
kserve-controller-manager-849dcf98b6-xq6k8                        1/1     Running   0              15d
kuberay-operator-759464465c-xsl2j                                 1/1     Running   0              15d
kueue-controller-manager-56559d6db6-wkc7s                         1/1     Running   0              15d
modelmesh-controller-58cc8bd7c5-d57mg                             1/1     Running   0              15d
odh-model-controller-6d9b6db854-sbhvr                             1/1     Running   0              15d
data-science-pipelines-operator-controller-manager-bc4f57dc2xtk   1/1     Running   0              15d
modelmesh-controller-58cc8bd7c5-594jp                             1/1     Running   0              15d
codeflare-operator-manager-6cb5c976d5-zzrkw                       1/1     Running   0              15d
rhods-dashboard-6f64f58c89-9b9dt                                  2/2     Running   0              15d
odh-model-controller-6d9b6db854-5zp9l                             1/1     Running   0              15d
rhods-dashboard-6f64f58c89-khvd4                                  2/2     Running   0              15d
modelmesh-controller-58cc8bd7c5-xjkcm                             1/1     Running   0              15d
odh-notebook-controller-manager-66d759cbb4-6bdql                  1/1     Running   0              15d
odh-model-controller-6d9b6db854-wlxn5                             1/1     Running   0              15d
rhods-dashboard-6f64f58c89-mn9mq                                  2/2     Running   0              15d
rhods-dashboard-6f64f58c89-nbfkd                                  2/2     Running   0              15d
etcd-7ddbc959b8-5wf2m                                             1/1     Running   0              15d
```
今回は、デフォルトで構成しています。
```shell
$ oc get DataScienceCluster default-dsc -o json | jq -r '.status.installedComponents'
{
  "codeflare": true,
  "dashboard": true,
  "data-science-pipelines-operator": true,
  "kserve": true,
  "kueue": true,
  "model-mesh": true,
  "ray": true,
  "trustyai": false,
  "workbenches": true
}
```
導入直後に各 Pod が要求する Computing Resource `(Pod.spec.containers[].resources)` 等を確認すると、以下のようになっています。
```shell
$ oc get pod -n redhat-ods-applications --sort-by=.spec.nodeName -o custom-columns-file=POD_CUSTOM.txt
NAME                                                              NODE         OWNER_KIND   CONTAINERS                    CPU_LIMITS   MEM_LIMITS   CPU_REQUESTS   MEM_REQUESTS
notebook-controller-deployment-556969c7cf-jj5jj                   11.222.3.4   ReplicaSet   manager                       500m         4Gi          500m           256Mi
rhods-dashboard-6f64f58c89-pwhqh                                  11.222.3.4   ReplicaSet   rhods-dashboard,oauth-proxy   1,1          2Gi,2Gi      500m,500m      1Gi,1Gi
kserve-controller-manager-849dcf98b6-xq6k8                        11.222.3.4   ReplicaSet   manager                       500m         5Gi          100m           200Mi
kuberay-operator-759464465c-xsl2j                                 11.222.3.4   ReplicaSet   kuberay-operator              100m         512Mi        100m           512Mi
kueue-controller-manager-56559d6db6-wkc7s                         11.222.3.4   ReplicaSet   manager                       500m         512Mi        500m           512Mi
modelmesh-controller-58cc8bd7c5-d57mg                             11.222.3.4   ReplicaSet   manager                       1            2Gi          50m            96Mi
odh-model-controller-6d9b6db854-sbhvr                             11.222.3.4   ReplicaSet   manager                       500m         2Gi          10m            64Mi
data-science-pipelines-operator-controller-manager-bc4f57dc2xtk   11.222.3.5   ReplicaSet   manager                       1            4Gi          10m            64Mi
modelmesh-controller-58cc8bd7c5-594jp                             11.222.3.5   ReplicaSet   manager                       1            2Gi          50m            96Mi
codeflare-operator-manager-6cb5c976d5-zzrkw                       11.222.3.5   ReplicaSet   manager                       1            1Gi          1              1Gi
rhods-dashboard-6f64f58c89-9b9dt                                  11.222.3.5   ReplicaSet   rhods-dashboard,oauth-proxy   1,1          2Gi,2Gi      500m,500m      1Gi,1Gi
odh-model-controller-6d9b6db854-5zp9l                             11.222.3.5   ReplicaSet   manager                       500m         2Gi          10m            64Mi
rhods-dashboard-6f64f58c89-khvd4                                  11.222.3.5   ReplicaSet   rhods-dashboard,oauth-proxy   1,1          2Gi,2Gi      500m,500m      1Gi,1Gi
modelmesh-controller-58cc8bd7c5-xjkcm                             11.222.3.6   ReplicaSet   manager                       1            2Gi          50m            96Mi
odh-notebook-controller-manager-66d759cbb4-6bdql                  11.222.3.6   ReplicaSet   manager                       500m         4Gi          500m           256Mi
odh-model-controller-6d9b6db854-wlxn5                             11.222.3.6   ReplicaSet   manager                       500m         2Gi          10m            64Mi
rhods-dashboard-6f64f58c89-mn9mq                                  11.222.3.6   ReplicaSet   rhods-dashboard,oauth-proxy   1,1          2Gi,2Gi      500m,500m      1Gi,1Gi
rhods-dashboard-6f64f58c89-nbfkd                                  11.222.3.6   ReplicaSet   rhods-dashboard,oauth-proxy   1,1          2Gi,2Gi      500m,500m      1Gi,1Gi
etcd-7ddbc959b8-5wf2m                                             11.222.3.6   ReplicaSet   etcd                          800m         800Mi        400m           200Mi
```
`rhods-dashboard` や `codeflare-operator-manager` が、比較的多くのリソースを要求していることが分かります。PVC はありません。
```shell
$ oc get pvc -n redhat-ods-applications
No resources found in redhat-ods-applications namespace.
```
同様に、`redhat-ods-monitoring`、`redhat-ods-operator` Project も確認してみます。
```shell
$ oc get pod -n redhat-ods-monitoring --sort-by=.spec.nodeName -o custom-columns-file=POD_CUSTOM.txt
NAME   NODE   OWNER_KIND   CONTAINERS   CPU_LIMITS   MEM_LIMITS   CPU_REQUESTS   MEM_REQUESTS

$ oc get pvc -n redhat-ods-monitoring
No resources found in redhat-ods-monitoring namespace.
```
```shell
$ oc get pod -n redhat-ods-operator --sort-by=.spec.nodeName -o custom-columns-file=POD_CUSTOM.txt
NAME                              NODE         OWNER_KIND   CONTAINERS       CPU_LIMITS   MEM_LIMITS   CPU_REQUESTS   MEM_REQUESTS
rhods-operator-7666b99787-lsn6z   11.222.3.0.4   ReplicaSet   rhods-operator   500m         4Gi          500m           256Mi

$ oc get pvc -n redhat-ods-operator
No resources found in redhat-ods-operator namespace.
```

##### Jupyter Notebook
導入完了時点では Jupyter Notebook のみが使用可能になっています。 簡単な設定で構成できるようになっており、`Deployment size` で CPU/Memory サイズの選択が可能です。

![1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/2ff884a9-0d97-5bce-3109-39c5c1646534.png)

Settings -> Cluster settings にある `PVC size` で PVC のサイズ指定も可能です。

![6.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/18c78be0-a649-ab02-8d3f-f600ebf8851f.png)

Pod は `rhods-notebooks` Project に配置され、`Deployment size = Small`、`PVC size = 20GiB (Default)` の Computing Resource は、以下のようになっています。
```shell
$ oc get pod -n rhods-notebooks -o custom-columns-file=POD_CUSTOM.txt
NAME                                         NODE         OWNER_KIND    CONTAINERS                                             CPU_LIMITS   MEM_LIMITS   CPU_REQUESTS   MEM_REQUESTS
jupyter-nb-iam-xxue91db-40jp-2exyz-2eabc-0   11.222.3.4   StatefulSet   jupyter-nb-iam-xxue91db-40jp-2exyz-2eabc,oauth-proxy   2,100m       8Gi,64Mi     1,100m         8Gi,64Mi

$ oc get pvc -n rhods-notebooks
NAME                                                  STATUS   VOLUME                        CAPACITY   ACCESS MODES   STORAGECLASS                 AGE
jupyter-nb-iam-xxue91db-40jp-2exyz-2eabc-pvc          Bound    pvc-bb0ee0cc-2797-4dcb-a8ec   20Gi       RWO            ibmc-vpc-block-10iops-tier   13d
```

##### Data Science Project (OpenShift AI tutorial - Fraud detection example)
Data Science Project の例として、OpenShift AI tutorial - Fraud detection example を確認してみます。

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.9/html-single/openshift_ai_tutorial_-_fraud_detection_example/index

Tutorial を実行すると `fraud-detection` Project に Pod が配置されます。 Tutorial 実施時の設定変更の有無にもよりますが、Computing Resource は、概ね以下のようになります。
```shell
$ oc get pod -n fraud-detection --sort-by=.spec.nodeName -o custom-columns-file=POD_CUSTOM.txt
NAME                                                   NODE         OWNER_KIND    CONTAINERS                                    CPU_LIMITS     MEM_LIMITS                    CPU_REQUESTS          MEM_REQUESTS
ds-pipeline-metadata-grpc-dspa-57d47dd549-r5zds        11.222.3.4   ReplicaSet    container                                     100m           256Mi                         100m                  256Mi
create-minio-buckets-sqstp                             11.222.3.4   Job           create-buckets                                <none>         <none>                        <none>                <none>
fraud-detection-0                                      11.222.3.4   StatefulSet   fraud-detection,oauth-proxy                   2,100m         8Gi,64Mi                      1,100m                8Gi,64Mi
ds-pipeline-persistenceagent-dspa-6b96b9c8d8-sgwkj     11.222.3.4   ReplicaSet    ds-pipeline-persistenceagent                  250m           1Gi                           120m                  500Mi
ds-pipeline-workflow-controller-dspa-5b4cdb559-bzg6g   11.222.3.5   ReplicaSet    ds-pipeline-workflow-controller               <none>         <none>                        100m                  500Mi
ds-pipeline-scheduledworkflow-dspa-64c767c896-lv8d2    11.222.3.5   ReplicaSet    ds-pipeline-scheduledworkflow                 250m           250Mi                         120m                  100Mi
create-minio-root-user-gxcpj                           11.222.3.5   Job           create-minio-root-user                        <none>         <none>                        <none>                <none>
minio-6bff775986-hh2kz                                 11.222.3.5   ReplicaSet    minio                                         2              2Gi                           200m                  1Gi
modelmesh-serving-model-server-778b47cc76-487x2        11.222.3.5   ReplicaSet    rest-proxy,oauth-proxy,ovms,ovms-adapter,mm   1,100m,2,2,3   512Mi,256Mi,8Gi,512Mi,448Mi   50m,100m,1,50m,300m   96Mi,256Mi,4Gi,96Mi,448Mi
ds-pipeline-metadata-envoy-dspa-9c745bb5d-bphwb        11.222.3.6   ReplicaSet    container,oauth-proxy                         100m,100m      256Mi,256Mi                   100m,100m             256Mi,256Mi
ds-pipeline-dspa-7ffd8ff549-dhtnx                      11.222.3.6   ReplicaSet    ds-pipeline-api-server,oauth-proxy            500m,100m      1Gi,256Mi                     250m,100m             500Mi,256Mi
create-s3-storage-hjqr2                                11.222.3.6   Job           create-s3-storage                             <none>         <none>                        <none>                <none>
create-ds-connections-j62fq                            11.222.3.6   Job           create-ds-connections                         <none>         <none>                        <none>                <none>
mariadb-dspa-8685d54888-8qhhj

$ oc get pvc -n fraud-detection
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                 AGE
fraud-detection   Bound    pvc-32db862a-b2bb-45c7-8218-6462b18a2472   20Gi       RWO            ibmc-vpc-block-10iops-tier   13d
mariadb-dspa      Bound    pvc-a1be0870-cf1e-49fe-a564-b0c6d6e23fd0   10Gi       RWO            ibmc-vpc-block-10iops-tier   13d
minio             Bound    pvc-1ec0b7a6-9d38-4d0c-b76b-2fdd5599ba9b   10Gi       RWO            ibmc-vpc-block-10iops-tier   13d
```