OpenShift AI - Data Science Project


#### OpenShift AI

Red Hat® OpenShift® AI とは、柔軟でスケーラブルな人工知能 (AI) および機械学習 (ML) プラットフォームです。このプラットフォームにより、企業はハイブリッドクラウド環境全体で AI 対応アプリケーションを大規模に作成および提供できるようになります。

OpenShift AI はオープンソース・テクノロジーを使用して構築されており、実験、モデル提供、革新的なアプリケーションの提供のための、信頼性と一貫性に優れた運用機能を提供します。

https://www.redhat.com/ja/technologies/cloud-computing/openshift/openshift-ai

#### Data Science Project
ここでは、OpenShift AI 2.9.1 / Red Hat OpenShift on IBM Cloud (ROKS) 4.14 の環境で、以下の Tutorial を用いて Data Science Project の実装を確認します。

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.9/html-single/openshift_ai_tutorial_-_fraud_detection_example/index

##### OpenShift AI UI
Data Science Project は、OpenShift AI UI の `Data Science Projects` から作成します。Tutorial に従って一連のリソースを作成すると、以下のようになります。ここから `Edit project` や `Delete project` なども可能です。

![1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/60594832-bfe9-5699-33fe-aa4b95aedf9f.png)

##### Data Science Project
作成した Data Science Project を開くと幾つかの Tab があり、それれぞれの内容は以下のとおりです。

Overview
![2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/2d6172b6-4093-dcb7-d27f-f255321b1db5.png)

Workbenches
![3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/43e9da8a-ac42-5a64-06e7-b8691eafcef0.png)

Pipelines
![4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/9efb579d-8ce7-4319-9fa5-02b70a167b36.png)

Models
![5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/2ec0a262-98bf-fc81-b0f3-8dabba1192ba.png)

Cluster storage
![6.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/5aede16a-3e7c-edd5-5b21-8f57fc45ed31.png)

Data connections
![7.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/97a1404a-0c6f-8b82-6bfe-2c62ef40d655.png)

Permissions
![8.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/bfc9a4f2-6964-4916-3887-abe5406c08fd.png)

##### fraud-detection Project
Tutorial の手順に従うと、Data Science Project のリソースは `fraud-detection` Project に配置されます。
```shell
$ oc get all,pvc -n fraud-detection -o name
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
pod/create-ds-connections-j62fq
pod/create-minio-buckets-sqstp
pod/create-minio-root-user-gxcpj
pod/create-s3-storage-hjqr2
pod/ds-pipeline-dspa-7ffd8ff549-dhtnx
pod/ds-pipeline-metadata-envoy-dspa-9c745bb5d-bphwb
pod/ds-pipeline-metadata-grpc-dspa-57d47dd549-r5zds
pod/ds-pipeline-persistenceagent-dspa-6b96b9c8d8-sgwkj
pod/ds-pipeline-scheduledworkflow-dspa-64c767c896-lv8d2
pod/ds-pipeline-workflow-controller-dspa-5b4cdb559-bzg6g
pod/fraud-detection-0
pod/mariadb-dspa-8685d54888-8qhhj
pod/minio-6bff775986-hh2kz
pod/modelmesh-serving-model-server-778b47cc76-487x2
service/ds-pipeline-dspa
service/ds-pipeline-metadata-envoy-dspa
service/ds-pipeline-metadata-grpc-dspa
service/ds-pipeline-workflow-controller-metrics-dspa
service/fraud-detection
service/fraud-detection-tls
service/mariadb-dspa
service/metadata-grpc-service
service/minio
service/ml-pipeline
service/modelmesh-serving
deployment.apps/ds-pipeline-dspa
deployment.apps/ds-pipeline-metadata-envoy-dspa
deployment.apps/ds-pipeline-metadata-grpc-dspa
deployment.apps/ds-pipeline-persistenceagent-dspa
deployment.apps/ds-pipeline-scheduledworkflow-dspa
deployment.apps/ds-pipeline-workflow-controller-dspa
deployment.apps/mariadb-dspa
deployment.apps/minio
deployment.apps/modelmesh-serving-model-server
replicaset.apps/ds-pipeline-dspa-7ffd8ff549
replicaset.apps/ds-pipeline-metadata-envoy-dspa-9c745bb5d
replicaset.apps/ds-pipeline-metadata-grpc-dspa-57d47dd549
replicaset.apps/ds-pipeline-persistenceagent-dspa-6b96b9c8d8
replicaset.apps/ds-pipeline-scheduledworkflow-dspa-64c767c896
replicaset.apps/ds-pipeline-workflow-controller-dspa-5b4cdb559
replicaset.apps/mariadb-dspa-8685d54888
replicaset.apps/minio-6bff775986
replicaset.apps/modelmesh-serving-model-server-778b47cc76
statefulset.apps/fraud-detection
job.batch/create-ds-connections
job.batch/create-minio-buckets
job.batch/create-minio-root-user
job.batch/create-s3-storage
route.route.openshift.io/ds-pipeline-dspa
route.route.openshift.io/ds-pipeline-metadata-envoy-dspa
route.route.openshift.io/fraud-detection
route.route.openshift.io/minio-console
route.route.openshift.io/minio-s3
persistentvolumeclaim/fraud-detection
persistentvolumeclaim/mariadb-dspa
persistentvolumeclaim/minio
```

詳細を確認すると、以下の CR から実装されたことが分かります。
```shell
$ oc get svc,deploy,sts,job,route,pvc -n fraud-detection -o custom-columns-file=RES_OWNER.txt
NAMESPACE         KIND                    NAME                                           NODE     OWNER_KIND                        OWNER_NAME        OWNER_APIV
fraud-detection   Service                 ds-pipeline-dspa                               <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Service                 ds-pipeline-metadata-envoy-dspa                <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Service                 ds-pipeline-metadata-grpc-dspa                 <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Service                 ds-pipeline-workflow-controller-metrics-dspa   <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Service                 fraud-detection                                <none>   Notebook                          fraud-detection   kubeflow.org/v1beta1
fraud-detection   Service                 fraud-detection-tls                            <none>   Notebook                          fraud-detection   kubeflow.org/v1
fraud-detection   Service                 mariadb-dspa                                   <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Service                 metadata-grpc-service                          <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Service                 minio                                          <none>   <none>                            <none>            <none>
fraud-detection   Service                 ml-pipeline                                    <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Service                 modelmesh-serving                              <none>   Namespace                         fraud-detection   v1
fraud-detection   Deployment              ds-pipeline-dspa                               <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Deployment              ds-pipeline-metadata-envoy-dspa                <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Deployment              ds-pipeline-metadata-grpc-dspa                 <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Deployment              ds-pipeline-persistenceagent-dspa              <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Deployment              ds-pipeline-scheduledworkflow-dspa             <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Deployment              ds-pipeline-workflow-controller-dspa           <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Deployment              mariadb-dspa                                   <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Deployment              minio                                          <none>   <none>                            <none>            <none>
fraud-detection   Deployment              modelmesh-serving-model-server                 <none>   ServingRuntime                    model-server      serving.kserve.io/v1alpha1
fraud-detection   StatefulSet             fraud-detection                                <none>   Notebook                          fraud-detection   kubeflow.org/v1beta1
fraud-detection   Job                     create-ds-connections                          <none>   <none>                            <none>            <none>
fraud-detection   Job                     create-minio-buckets                           <none>   <none>                            <none>            <none>
fraud-detection   Job                     create-minio-root-user                         <none>   <none>                            <none>            <none>
fraud-detection   Job                     create-s3-storage                              <none>   <none>                            <none>            <none>
fraud-detection   Route                   ds-pipeline-dspa                               <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Route                   ds-pipeline-metadata-envoy-dspa                <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   Route                   fraud-detection                                <none>   Notebook                          fraud-detection   kubeflow.org/v1
fraud-detection   Route                   minio-console                                  <none>   <none>                            <none>            <none>
fraud-detection   Route                   minio-s3                                       <none>   <none>                            <none>            <none>
fraud-detection   PersistentVolumeClaim   fraud-detection                                <none>   <none>                            <none>            <none>
fraud-detection   PersistentVolumeClaim   mariadb-dspa                                   <none>   DataSciencePipelinesApplication   dspa              datasciencepipelinesapplications.opendatahub.io/v1alpha1
fraud-detection   PersistentVolumeClaim   minio                                          <none>   <none>                            <none>            <none>
```
```shell
$ oc get DataSciencePipelinesApplication,Notebook,ServingRuntime -n fraud-detection
NAME                                                                                   AGE
datasciencepipelinesapplication.datasciencepipelinesapplications.opendatahub.io/dspa   15d

NAME                                    AGE
notebook.kubeflow.org/fraud-detection   15d

NAME                                            DISABLED   MODELTYPE     CONTAINERS   AGE
servingruntime.serving.kserve.io/model-server              openvino_ir   ovms         15d
```

DataSciencePipelinesApplication

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.9/html/working_on_data_science_projects/working-with-data-science-pipelines_ds-pipelines

https://github.com/opendatahub-io/data-science-pipelines-operator

```shell
$ oc explain DataSciencePipelinesApplication.spec
GROUP:      datasciencepipelinesapplications.opendatahub.io
KIND:       DataSciencePipelinesApplication
VERSION:    v1alpha1

FIELD: spec <Object>

DESCRIPTION:
    <empty>
FIELDS:
  apiServer     <Object>
    DS Pipelines API Server configuration.

  database      <Object>
    Database specifies database configurations, used for DS Pipelines metadata
    tracking. Specify either the default MariaDB deployment, or configure your
    own External SQL DB.

  dspVersion    <string>
    <no description>

  mlmd  <Object>
    <no description>

  mlpipelineUI  <Object>
    Deploy the KFP UI with DS Pipelines UI. This feature is unsupported, and
    primarily used for exploration, testing, and development purposes.

  objectStorage <Object> -required-
    ObjectStorage specifies Object Store configurations, used for DS Pipelines
    artifact passing and storage. Specify either the your own External Storage
    (e.g. AWS S3), or use the default Minio deployment (unsupported, primarily
    for development, and testing) .

  persistenceAgent      <Object>
    DS Pipelines PersistenceAgent configuration.

  scheduledWorkflow     <Object>
    DS Pipelines Scheduled Workflow configuration.

  workflowController    <Object>
    WorkflowController is an argo-specific component that manages a DSPA's
    Workflow objects and handles the orchestration of them with the central Argo
    server
```

Notebook

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.9/html-single/working_on_data_science_projects/index#creating-a-new-notebook_notebooks

https://www.kubeflow.org/docs/components/notebooks/api-reference/notebook-v1/

```shell
$ oc explain notebook.spec.template.spec
GROUP:      kubeflow.org
KIND:       Notebook
VERSION:    v1

FIELD: spec <Object>

DESCRIPTION:
    <empty>
FIELDS:
  activeDeadlineSeconds <integer>
    <no description>

  affinity      <Object>
    <no description>

  automountServiceAccountToken  <boolean>
    <no description>

  containers    <[]Object> -required-
    <no description>

  dnsConfig     <Object>
    <no description>

  dnsPolicy     <string>
    <no description>

  enableServiceLinks    <boolean>
    <no description>

  ephemeralContainers   <[]Object>
    <no description>

  hostAliases   <[]Object>
    <no description>

  hostIPC       <boolean>
    <no description>

  hostNetwork   <boolean>
    <no description>

  hostPID       <boolean>
    <no description>

  hostname      <string>
    <no description>

  imagePullSecrets      <[]Object>
    <no description>

  initContainers        <[]Object>
    <no description>

  nodeName      <string>
    <no description>

  nodeSelector  <map[string]string>
    <no description>

  os    <Object>
    <no description>

  overhead      <map[string]Object>
    <no description>

  preemptionPolicy      <string>
    <no description>

  priority      <integer>
    <no description>

  priorityClassName     <string>
    <no description>

  readinessGates        <[]Object>
    <no description>

  restartPolicy <string>
    <no description>

  runtimeClassName      <string>
    <no description>

  schedulerName <string>
    <no description>

  securityContext       <Object>
    <no description>

  serviceAccount        <string>
    <no description>

  serviceAccountName    <string>
    <no description>

  setHostnameAsFQDN     <boolean>
    <no description>

  shareProcessNamespace <boolean>
    <no description>

  subdomain     <string>
    <no description>

  terminationGracePeriodSeconds <integer>
    <no description>

  tolerations   <[]Object>
    <no description>

  topologySpreadConstraints     <[]Object>
    <no description>

  volumes       <[]Object>
    <no description>
```

ServingRuntime

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.9/html-single/serving_models/index

https://kserve.github.io/website/master/sdk_docs/docs/V1alpha1ServingRuntime/#properties

```shell
$ oc explain servingruntime.spec
GROUP:      serving.kserve.io
KIND:       ServingRuntime
VERSION:    v1alpha1

FIELD: spec <Object>

DESCRIPTION:
    <empty>
FIELDS:
  affinity      <Object>
    <no description>

  annotations   <map[string]string>
    <no description>

  builtInAdapter        <Object>
    <no description>

  containers    <[]Object> -required-
    <no description>

  disabled      <boolean>
    <no description>

  grpcDataEndpoint      <string>
    <no description>

  grpcEndpoint  <string>
    <no description>

  httpDataEndpoint      <string>
    <no description>

  imagePullSecrets      <[]Object>
    <no description>

  labels        <map[string]string>
    <no description>

  multiModel    <boolean>
    <no description>

  nodeSelector  <map[string]string>
    <no description>

  protocolVersions      <[]string>
    <no description>

  replicas      <integer>
    <no description>

  storageHelper <Object>
    <no description>

  supportedModelFormats <[]Object>
    <no description>

  tolerations   <[]Object>
    <no description>

  volumes       <[]Object>
    <no description>
```

Tutorial に従うと PVC は3つ作成され、それぞれ以下の Pod に割り当てられます。
```shell
$ oc get pvc -n fraud-detection
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                 AGE
fraud-detection   Bound    pvc-32db862a-b2bb-45c7-8218-6462b18a2472   20Gi       RWO            ibmc-vpc-block-10iops-tier   15d
mariadb-dspa      Bound    pvc-a1be0870-cf1e-49fe-a564-b0c6d6e23fd0   10Gi       RWO            ibmc-vpc-block-10iops-tier   15d
minio             Bound    pvc-1ec0b7a6-9d38-4d0c-b76b-2fdd5599ba9b   10Gi       RWO            ibmc-vpc-block-10iops-tier   15d

$ for P in $(oc get pvc -n fraud-detection -o custom-columns=:.metadata.name --no-headers); do echo "PVC: $P -> POD(s) [ $(oc get pod -o json | jq --arg P $P -rc '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName == $P) | .metadata.name') ]"; done
PVC: fraud-detection -> POD(s) [ fraud-detection-0 ]
PVC: mariadb-dspa -> POD(s) [ mariadb-dspa-8685d54888-8qhhj ]
PVC: minio -> POD(s) [ minio-6bff775986-hh2kz ]
```