MLOpsを実現するKubeflowを前提のKubernetesも含めて構築する(後編)

初版: 2021/4/9
著者: 高久 隆史, 株式会社日立製作所

# はじめに
[Kubeflow](https://www.kubeflow.org/)はKubernetes上でMLOpsを実現するためのOSSのツールキットです。2020/3にバージョン1.0がリリースされ、2021/4現在も活発に開発が行われており、機械学習を利用したシステムの開発/運用のライフサイクルを回すための有効な手段の１つとして注目されています。

本連載では、Kubeflowのバージョン1.2(連載開始時点の最新版、2020年11月リリース)について、構築手順、Kubeflow Pipelines、KFServingの基礎的な利用手順についての情報を紹介いたします。

第2回の本稿では、[第1回の投稿](https://qiita.com/t-takahisa/items/5eefa9cf6be783cfe170)のKubeflow環境構築手順の続きを紹介いたします。

投稿一覧：
1. [MLOpsを実現するKubeflowを前提のKubernetesも含めて構築する(前編)](https://qiita.com/t-takahisa/items/5eefa9cf6be783cfe170)
2. MLOpsを実現するKubeflowを前提のKubernetesも含めて構築する(後編) (本投稿)
3. [Kubeflow Pipelinesでscikit-learnの機械学習モデルを訓練・評価してみる](https://qiita.com/t-takahisa/items/0a409478cb502f4d3c5f)
4. [Kubeflow KFServingでscikit-learnの学習済みモデルを用いた推論サービスを公開する](https://qiita.com/a-yanagimura/items/2fd5a7cb51ced76ed4c6)


# 構築手順(第1回の投稿の続き)
第2回の本稿では、[第1回の投稿](https://qiita.com/t-takahisa/items/5eefa9cf6be783cfe170)の続きを記載します。これより前の手順や前提環境などの情報は、[第1回の投稿](https://qiita.com/t-takahisa/items/5eefa9cf6be783cfe170)を参照してください。


## 10. Dynamic Volume Provisionerのインストール
Dynamic Volume Provisionerとして、Local Path Provisionerをインストールします。
https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/#provisioning-of-persistent-volumes-in-kubernetes
https://github.com/rancher/local-path-provisioner#deployment

```shell
# Local Path Provisionerをデプロイ
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
# --- 実行結果例 ここから -----------------------------------------------------------
# namespace/local-path-storage created
# serviceaccount/local-path-provisioner-service-account created
# clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
# clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
# deployment.apps/local-path-provisioner created
# storageclass.storage.k8s.io/local-path created
# configmap/local-path-config created
# --- 実行結果例 ここまで -----------------------------------------------------------

# ステータスを確認
kubectl -n local-path-storage get pod
# --- 実行結果例 ここから -----------------------------------------------------------
# NAME                                      READY   STATUS    RESTARTS   AGE
# local-path-provisioner-5696dbb894-tzc4m   1/1     Running   0          27s
# --- 実行結果例 ここまで -----------------------------------------------------------

# ログを確認(※確認したら、Ctrl+Cで終了させます。)
kubectl -n local-path-storage logs -f -l app=local-path-provisioner
# --- 実行結果例 ここから -----------------------------------------------------------
# time="2021-01-07T05:11:45Z" level=debug msg="Applied config: {\"nodePathMap\":[{\"node\":\"DEFAULT_PATH_FOR_NON_LISTED_NODES\",\"paths\":[\"/opt/local-path-provisioner\"]}]}"
# time="2021-01-07T05:11:45Z" level=debug msg="Provisioner started"
# I0107 05:11:45.549926       1 leaderelection.go:242] attempting to acquire leader lease  local-path-storage/rancher.io-local-path...
# I0107 05:11:45.557442       1 leaderelection.go:252] successfully acquired lease local-path-storage/rancher.io-local-path
# I0107 05:11:45.557684       1 controller.go:773] Starting provisioner controller rancher.io/local-path_local-path-provisioner-5696dbb894-tzc4m_1829333c-3f91-4182-a685-8ae811112c52!
# I0107 05:11:45.557699       1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"local-path-storage", Name:"rancher.io-local-path", UID:"3c02588b-99af-4e2a-8f6c-62c025552ef2", APIVersion:"v1", ResourceVersion:"12777", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' local-path-provisioner-5696dbb894-tzc4m_1829333c-3f91-4182-a685-8ae811112c52 became leader
# I0107 05:11:45.657940       1 controller.go:822] Started provisioner controller rancher.io/local-path_local-path-provisioner-5696dbb894-tzc4m_1829333c-3f91-4182-a685-8ae811112c52!
# ^C
# --- 実行結果例 ここまで -----------------------------------------------------------

# Local Path Provisionerの動作確認
# PersistentVolumeClaim(PVC)を作成
kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pvc.yaml
# --- 実行結果例 ここから -----------------------------------------------------------
# persistentvolumeclaim/local-path-pvc created
# --- 実行結果例 ここまで -----------------------------------------------------------

# Pod作成時にPVCを使用してPersistentVolume(PV)を割り当て
kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml
# --- 実行結果例 ここから -----------------------------------------------------------
pod/volume-test created
# --- 実行結果例 ここまで -----------------------------------------------------------

# しばらく待つ。

# 作成されたPVを確認(ステータスがBoundならOK)
kubectl get pv
# --- 実行結果例 ここから -----------------------------------------------------------
# NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
# pvc-1a64b85b-a9d7-4551-be89-d8c9980a7011   2Gi        RWO            Delete           Bound    default/local-path-pvc   local-path              5m39s
# --- 実行結果例 ここまで -----------------------------------------------------------

# PVCを確認(ステータスがBoundならOK)
kubectl get pvc
# --- 実行結果例 ここから -----------------------------------------------------------
# NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
# local-path-pvc   Bound    pvc-1a64b85b-a9d7-4551-be89-d8c9980a7011   2Gi        RWO            local-path     6m27s
# --- 実行結果例 ここまで -----------------------------------------------------------

# describeで警告やエラーが出ていないことを確認
kubectl describe pvc local-path-pvc
# --- 実行結果例 ここから -----------------------------------------------------------
# Name:          local-path-pvc
# 
# Namespace:     default
# StorageClass:  local-path
# Status:        Bound
# Volume:        pvc-1a64b85b-a9d7-4551-be89-d8c9980a7011
# Labels:        <none>
# Annotations:   pv.kubernetes.io/bind-completed: yes
#                pv.kubernetes.io/bound-by-controller: yes
#                volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
#                volume.kubernetes.io/selected-node: ip-x-x-x-x
# Finalizers:    [kubernetes.io/pvc-protection]
# Capacity:      2Gi
# Access Modes:  RWO
# VolumeMode:    Filesystem
# Used By:       volume-test
# Events:
#   Type    Reason                 Age                    From                                                                                                Message
#   ----    ------                 ----                   ----                                                                                                -------
#   Normal  WaitForFirstConsumer   7m10s (x2 over 7m21s)  persistentvolume-controller                                                                         waiting for first consumer to be created before binding
#   Normal  ExternalProvisioning   7m6s                   persistentvolume-controller                                                                         waiting for a volume to be created, either by external provisioner "rancher.io/local-path" or manually created by system administrator
#   Normal  Provisioning           7m6s                   rancher.io/local-path_local-path-provisioner-5696dbb894-tzc4m_1829333c-3f91-4182-a685-8ae811112c52  External provisioner is provisioning volume for claim "default/local-path-pvc"
#   Normal  ProvisioningSucceeded  7m                     rancher.io/local-path_local-path-provisioner-5696dbb894-tzc4m_1829333c-3f91-4182-a685-8ae811112c52  Successfully provisioned volume pvc-1a64b85b-a9d7-4551-be89-d8c9980a7011
# --- 実行結果例 ここまで -----------------------------------------------------------

# 作成されたPodを確認(ステータスがRunningならOK)
kubectl get pod
# --- 実行結果例 ここから -----------------------------------------------------------
# NAME          READY   STATUS    RESTARTS   AGE
# volume-test   1/1     Running   0          9m
# --- 実行結果例 ここまで -----------------------------------------------------------

# Podに書き込み
kubectl exec volume-test -- sh -c "echo local-path-test > /data/test"

# PVのデータを確認
kubectl exec volume-test -- cat /data/test
# --- 実行結果例 ここから -----------------------------------------------------------
# local-path-test
# --- 実行結果例 ここまで -----------------------------------------------------------

# Podを削除
kubectl delete -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml
# --- 実行結果例 ここから -----------------------------------------------------------
# pod "volume-test" deleted
# --- 実行結果例 ここまで -----------------------------------------------------------

# Podを再作成
kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml
# --- 実行結果例 ここから -----------------------------------------------------------
# pod/volume-test created
# --- 実行結果例 ここまで -----------------------------------------------------------

# しばらく待つ

# PVにデータが残っていることを確認
kubectl exec volume-test -- cat /data/test
# --- 実行結果例 ここから -----------------------------------------------------------
# local-path-test
# --- 実行結果例 ここまで -----------------------------------------------------------

# 動作確認に使用したPodを削除
kubectl delete -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml
# --- 実行結果例 ここから -----------------------------------------------------------
# pod "volume-test" deleted
# --- 実行結果例 ここまで -----------------------------------------------------------

# 動作確認に使用したPVCを削除
kubectl delete -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pvc.yaml
# --- 実行結果例 ここから -----------------------------------------------------------
# persistentvolumeclaim "local-path-pvc" deleted
# --- 実行結果例 ここまで -----------------------------------------------------------

# local-pathをデフォルトのstorageClassに設定
https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

# StorageClass一覧を確認。local-pathがあればOK
kubectl get storageclass
# --- 実行結果例 ここから -----------------------------------------------------------
# NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
# local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  15m
# --- 実行結果例 ここまで -----------------------------------------------------------

# local-pathをデフォルトのstorageClassに設定
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
# --- 実行結果例 ここから -----------------------------------------------------------
# storageclass.storage.k8s.io/local-path patched
# --- 実行結果例 ここまで -----------------------------------------------------------

# デフォルトになったことを確認
kubectl get storageclass
# --- 実行結果例 ここから -----------------------------------------------------------
# NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
# local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  16m
# --- 実行結果例 ここまで -----------------------------------------------------------
```

## 11. Kubeflowのデプロイ
Kubeflowのデプロイを行います。
https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/

```shell
# ダウンロード
wget https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_v1.2.0-0-gbc038f9_linux.tar.gz

# ダウンロードしたファイルを解凍
tar -xvf kfctl_v1.2.0-0-gbc038f9_linux.tar.gz

# 解凍したファイル(kfctlコマンド)を移動
mkdir ./cmd
mv kfctl ./cmd

# 環境変数を設定
# kfctlコマンドをパスに追加
export PATH=$PATH:"/home/ubuntu/cmd"
# Deployment名
export KF_NAME=my-kubeflow
# Kubeflow deployments用ディレクトリ
export BASE_DIR=/home/ubuntu/k8s
export KF_DIR=${BASE_DIR}/${KF_NAME}
# Kubeflowのデプロイ用設定ファイル
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml"

# Kubeflow deployments用ディレクトリを作成
rm -rf ${KF_DIR}
mkdir -p ${KF_DIR}
cd ${KF_DIR}

# Kubeflowをデプロイ
kfctl apply -V -f ${CONFIG_URI}
# --- 実行結果例 ここから -------------------------------------------------------------------------------
# INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml to /tmp/085975924/tmp.yaml  filename="utils/k8utils.go:178"
# (中略)
# INFO[0110] Successfully applied application spartakus    filename="kustomize/kustomize.go:291"
# INFO[0110] Applied the configuration Successfully!       filename="cmd/apply.go:75"
# --- 実行結果例 ここまで -------------------------------------------------------------------------------

# kubeflowネームスペースのリソースのステータスを確認(しばらく待ってすべてRunningになったらOK)
kubectl -n kubeflow get all
# --- 実行結果例 ここから -------------------------------------------------------------------------------
# NAME                                                         READY   STATUS    RESTARTS   AGE
# pod/admission-webhook-bootstrap-stateful-set-0               1/1     Running   0          26m
# pod/admission-webhook-deployment-5cd7dc96f5-5hmj4            1/1     Running   0          26m
# pod/application-controller-stateful-set-0                    1/1     Running   0          28m
# pod/argo-ui-65df8c7c84-l8stp                                 1/1     Running   0          26m
# pod/cache-deployer-deployment-5f4979f45-7pnkj                2/2     Running   1          26m
# pod/cache-server-7859fd67f5-kprjk                            2/2     Running   0          26m
# pod/centraldashboard-67767584dc-6czbb                        1/1     Running   0          26m
# pod/jupyter-web-app-deployment-8486d5ffff-pmt5v              1/1     Running   0          26m
# pod/katib-controller-7fcc95676b-925k2                        1/1     Running   0          26m
# pod/katib-db-manager-85db457c64-p44kz                        1/1     Running   3          26m
# pod/katib-mysql-6c7f7fb869-bn4rv                             1/1     Running   0          26m
# pod/katib-ui-65dc4cf6f5-jkxlg                                1/1     Running   0          26m
# pod/kfserving-controller-manager-0                           2/2     Running   0          26m
# pod/kubeflow-pipelines-profile-controller-797fb44db9-vgdd7   1/1     Running   0          26m
# pod/metacontroller-0                                         1/1     Running   0          26m
# pod/metadata-db-6dd978c5b-dp7p2                              1/1     Running   0          26m
# pod/metadata-envoy-deployment-67bd5954c-m4jg9                1/1     Running   0          26m
# pod/metadata-grpc-deployment-577c67c96f-mzhgv                1/1     Running   5          26m
# pod/metadata-writer-756dbdd478-8hbcl                         2/2     Running   0          26m
# pod/minio-54d995c97b-wcvbn                                   1/1     Running   0          26m
# pod/ml-pipeline-7c56db5db9-g6l2m                             2/2     Running   1          26m
# pod/ml-pipeline-persistenceagent-d984c9585-gj5th             2/2     Running   0          26m
# pod/ml-pipeline-scheduledworkflow-5ccf4c9fcc-7qkvt           2/2     Running   0          26m
# pod/ml-pipeline-ui-7ddcd74489-8rhq2                          2/2     Running   0          26m
# pod/ml-pipeline-viewer-crd-56c68f6c85-6npdk                  2/2     Running   1          26m
# pod/ml-pipeline-visualizationserver-5b9bd8f6bf-pfcrq         2/2     Running   0          26m
# pod/mpi-operator-d5bfb8489-hcjjd                             1/1     Running   0          26m
# pod/mxnet-operator-7576d697d6-zk9mr                          1/1     Running   0          26m
# pod/mysql-74f8f99bc8-b5xbp                                   2/2     Running   0          26m
# pod/notebook-controller-deployment-5bb6bdbd6d-4jvzf          1/1     Running   0          26m
# pod/profiles-deployment-56bc5d7dcb-hcb2p                     2/2     Running   0          26m
# pod/pytorch-operator-847c8d55d8-6sqqj                        1/1     Running   0          26m
# pod/seldon-controller-manager-6bf8b45656-cp8mq               1/1     Running   0          26m
# pod/spark-operatorsparkoperator-fdfbfd99-hpxcc               1/1     Running   0          26m
# pod/spartakus-volunteer-558f8bfd47-jxkn2                     1/1     Running   0          26m
# pod/tf-job-operator-58477797f8-zn8fq                         1/1     Running   0          26m
# pod/workflow-controller-64fd7cffc5-zmbt5                     1/1     Running   0          26m
# 
# NAME                                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
# service/admission-webhook-service                      ClusterIP   10.111.39.110    <none>        443/TCP             26m
# service/application-controller-service                 ClusterIP   10.105.238.230   <none>        443/TCP             28m
# service/argo-ui                                        NodePort    10.102.157.141   <none>        80:30871/TCP        26m
# service/cache-server                                   ClusterIP   10.96.221.195    <none>        443/TCP             26m
# service/centraldashboard                               ClusterIP   10.110.98.241    <none>        80/TCP              26m
# service/jupyter-web-app-service                        ClusterIP   10.110.208.28    <none>        80/TCP              26m
# service/katib-controller                               ClusterIP   10.103.147.6     <none>        443/TCP,8080/TCP    26m
# service/katib-db-manager                               ClusterIP   10.96.82.42      <none>        6789/TCP            26m
# service/katib-mysql                                    ClusterIP   10.111.194.254   <none>        3306/TCP            26m
# service/katib-ui                                       ClusterIP   10.103.207.79    <none>        80/TCP              26m
# service/kfserving-controller-manager-metrics-service   ClusterIP   10.110.16.255    <none>        8443/TCP            26m
# service/kfserving-controller-manager-service           ClusterIP   10.106.200.178   <none>        443/TCP             26m
# service/kfserving-webhook-server-service               ClusterIP   10.111.24.235    <none>        443/TCP             26m
# service/kubeflow-pipelines-profile-controller          ClusterIP   10.97.67.169     <none>        80/TCP              26m
# service/metadata-db                                    ClusterIP   10.101.121.148   <none>        3306/TCP            26m
# service/metadata-envoy-service                         ClusterIP   10.103.44.6      <none>        9090/TCP            26m
# service/metadata-grpc-service                          ClusterIP   10.110.241.47    <none>        8080/TCP            26m
# service/minio-service                                  ClusterIP   10.98.231.70     <none>        9000/TCP            26m
# service/ml-pipeline                                    ClusterIP   10.103.62.15     <none>        8888/TCP,8887/TCP   26m
# service/ml-pipeline-ui                                 ClusterIP   10.99.11.46      <none>        80/TCP              26m
# service/ml-pipeline-visualizationserver                ClusterIP   10.98.68.126     <none>        8888/TCP            26m
# service/mysql                                          ClusterIP   10.108.232.76    <none>        3306/TCP            26m
# service/notebook-controller-service                    ClusterIP   10.110.38.4      <none>        443/TCP             26m
# service/profiles-kfam                                  ClusterIP   10.104.19.220    <none>        8081/TCP            26m
# service/pytorch-operator                               ClusterIP   10.96.97.201     <none>        8443/TCP            26m
# service/seldon-webhook-service                         ClusterIP   10.103.206.64    <none>        443/TCP             26m
# service/tf-job-operator                                ClusterIP   10.107.223.153   <none>        8443/TCP            26m
# 
# NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
# deployment.apps/admission-webhook-deployment            1/1     1            1           26m
# deployment.apps/argo-ui                                 1/1     1            1           26m
# deployment.apps/cache-deployer-deployment               1/1     1            1           26m
# deployment.apps/cache-server                            1/1     1            1           26m
# deployment.apps/centraldashboard                        1/1     1            1           26m
# deployment.apps/jupyter-web-app-deployment              1/1     1            1           26m
# deployment.apps/katib-controller                        1/1     1            1           26m
# deployment.apps/katib-db-manager                        1/1     1            1           26m
# deployment.apps/katib-mysql                             1/1     1            1           26m
# deployment.apps/katib-ui                                1/1     1            1           26m
# deployment.apps/kubeflow-pipelines-profile-controller   1/1     1            1           26m
# deployment.apps/metadata-db                             1/1     1            1           26m
# deployment.apps/metadata-envoy-deployment               1/1     1            1           26m
# deployment.apps/metadata-grpc-deployment                1/1     1            1           26m
# deployment.apps/metadata-writer                         1/1     1            1           26m
# deployment.apps/minio                                   1/1     1            1           26m
# deployment.apps/ml-pipeline                             1/1     1            1           26m
# deployment.apps/ml-pipeline-persistenceagent            1/1     1            1           26m
# deployment.apps/ml-pipeline-scheduledworkflow           1/1     1            1           26m
# deployment.apps/ml-pipeline-ui                          1/1     1            1           26m
# deployment.apps/ml-pipeline-viewer-crd                  1/1     1            1           26m
# deployment.apps/ml-pipeline-visualizationserver         1/1     1            1           26m
# deployment.apps/mpi-operator                            1/1     1            1           26m
# deployment.apps/mxnet-operator                          1/1     1            1           26m
# deployment.apps/mysql                                   1/1     1            1           26m
# deployment.apps/notebook-controller-deployment          1/1     1            1           26m
# deployment.apps/profiles-deployment                     1/1     1            1           26m
# deployment.apps/pytorch-operator                        1/1     1            1           26m
# deployment.apps/seldon-controller-manager               1/1     1            1           26m
# deployment.apps/spark-operatorsparkoperator             1/1     1            1           26m
# deployment.apps/spartakus-volunteer                     1/1     1            1           26m
# deployment.apps/tf-job-operator                         1/1     1            1           26m
# deployment.apps/workflow-controller                     1/1     1            1           26m
# 
# NAME                                                               DESIRED   CURRENT   READY   AGE
# replicaset.apps/admission-webhook-deployment-5cd7dc96f5            1         1         1       26m
# replicaset.apps/argo-ui-65df8c7c84                                 1         1         1       26m
# replicaset.apps/cache-deployer-deployment-5f4979f45                1         1         1       26m
# replicaset.apps/cache-server-7859fd67f5                            1         1         1       26m
# replicaset.apps/centraldashboard-67767584dc                        1         1         1       26m
# replicaset.apps/jupyter-web-app-deployment-8486d5ffff              1         1         1       26m
# replicaset.apps/katib-controller-7fcc95676b                        1         1         1       26m
# replicaset.apps/katib-db-manager-85db457c64                        1         1         1       26m
# replicaset.apps/katib-mysql-6c7f7fb869                             1         1         1       26m
# replicaset.apps/katib-ui-65dc4cf6f5                                1         1         1       26m
# replicaset.apps/kubeflow-pipelines-profile-controller-797fb44db9   1         1         1       26m
# replicaset.apps/metadata-db-6dd978c5b                              1         1         1       26m
# replicaset.apps/metadata-envoy-deployment-67bd5954c                1         1         1       26m
# replicaset.apps/metadata-grpc-deployment-577c67c96f                1         1         1       26m
# replicaset.apps/metadata-writer-756dbdd478                         1         1         1       26m
# replicaset.apps/minio-54d995c97b                                   1         1         1       26m
# replicaset.apps/ml-pipeline-7c56db5db9                             1         1         1       26m
# replicaset.apps/ml-pipeline-persistenceagent-d984c9585             1         1         1       26m
# replicaset.apps/ml-pipeline-scheduledworkflow-5ccf4c9fcc           1         1         1       26m
# replicaset.apps/ml-pipeline-ui-7ddcd74489                          1         1         1       26m
# replicaset.apps/ml-pipeline-viewer-crd-56c68f6c85                  1         1         1       26m
# replicaset.apps/ml-pipeline-visualizationserver-5b9bd8f6bf         1         1         1       26m
# replicaset.apps/mpi-operator-d5bfb8489                             1         1         1       26m
# replicaset.apps/mxnet-operator-7576d697d6                          1         1         1       26m
# replicaset.apps/mysql-74f8f99bc8                                   1         1         1       26m
# replicaset.apps/notebook-controller-deployment-5bb6bdbd6d          1         1         1       26m
# replicaset.apps/profiles-deployment-56bc5d7dcb                     1         1         1       26m
# replicaset.apps/pytorch-operator-847c8d55d8                        1         1         1       26m
# replicaset.apps/seldon-controller-manager-6bf8b45656               1         1         1       26m
# replicaset.apps/spark-operatorsparkoperator-fdfbfd99               1         1         1       26m
# replicaset.apps/spartakus-volunteer-558f8bfd47                     1         1         1       26m
# replicaset.apps/tf-job-operator-58477797f8                         1         1         1       26m
# replicaset.apps/workflow-controller-64fd7cffc5                     1         1         1       26m
# 
# NAME                                                        READY   AGE
# statefulset.apps/admission-webhook-bootstrap-stateful-set   1/1     26m
# statefulset.apps/application-controller-stateful-set        1/1     28m
# statefulset.apps/kfserving-controller-manager               1/1     26m
# statefulset.apps/metacontroller                             1/1     26m
# --- 実行結果例 ここまで -------------------------------------------------------------------------------

# Kubernetesダッシュボードをデプロイしている場合は、「https://<IPアドレス>:30843/#/overview?namespace=kubeflow」でも確認できる。
```


## 12. Kubeflow Central Dashboardへのアクセス
* ブラウザからKubeflow Central Dashboardにアクセスします。<br/>
https://<構築したマシンのIPアドレス>:31380/

**Kubeflow Central Dashboard 初回アクセス画面1**
![Kubeflow_Dashboard_init1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/865b6c68-c420-4677-edfe-16db50b48876.png)

* Start Setupをクリックします。

**Kubeflow Central Dashboard 初回アクセス画面2**
![Kubeflow_Dashboard_init2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/56f54d43-2b33-13c3-19fa-d8a8ab3768e0.png)

* 「Namespace Name」に任意のネームスペース名を入力して、「Finish」をクリックします。

Kubeflow Central Dashboardのトップページが表示されます。

**Kubeflow Central Dashboard トップページ**
![Kubeflow_Dashboard_top.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/96b5d18b-a031-19a4-ba64-6685fed991bd.png)


## 13. Notebook Serverの作成、接続
Kubeflowには、Jupyter Notebooksが統合されています。ここでは、Kubeflowの構築確認も兼ねて、Notebooks Serverの作成、接続手順を記載します。
https://www.kubeflow.org/docs/components/notebooks/setup/

* Kubeflow Central Dashboardのトップページから左ペインの「Notebook Server」をクリックします。

**Kubeflow Central Dashboard トップページ**
![Kubeflow_Dashboard_top02.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/c549864e-da58-3e83-477c-bdbef536e021.png)

* 「＋NEW SERVER」をクリックします。

**Kubeflow Central DashboardのNotebook Serversページ(構築前)**
![Notebook_Server1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/71742ca6-4406-0c9d-e121-c432c212fadf.png)

* 設定情報を入力し、「LAUNCH」をクリックします。

**Kubeflow Central Dashboard Notebook Server構築の設定情報入力ページ**
![Notebook_Server2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/809d485c-7a96-99a9-1735-e36342c8870e.png)

Notebook Server構築の設定情報の指定例を次に示します。最初の「Name - Name」以外はデフォルト値のままでも構築できます。

**Notebook Server構築の設定情報の指定例**

<table>
<col width="3%">
<col width="22%">
<col width="22%">
<col width="22%">
<col width="31%">
 <tr>
  <th>#</th>
  <th>大項目</th>
  <th>小項目</th>
  <th>値の指定例</th>
  <th>補足</th>
 </tr>
 <tr>
  <td>1</td>
  <td rowspan="2">Name</td>
  <td>Name</td>
  <td>my-first-notebook</td>
  <td>任意の値。ここでは、Kubeflow公式サイトの例をそのまま指定。</td>
 </tr>
 <tr>
  <td>2</td>
  <td>Namespace</td>
  <td>ns-agent</td>
  <td>Kubeflow Central Dashboardの初回起動時に指定したNamespaceの値(デフォルト値)。</td>
 </tr>
 <tr>
  <td>3</td>
  <td rowspan="2">Image</td>
  <td>Custom Image</td>
  <td>チェックなし</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>4</td>
  <td>Image</td>
  <td>gcr.io/kubeflow-images-public/tensorflow-1.15.2-notebook-cpu:1.0.0</td>
  <td>デフォルト値。値はプルダウンから選択できます。</td>
 </tr>
 <tr>
  <td>5</td>
  <td rowspan="2">CPU/RAM</td>
  <td>CPU</td>
  <td>0.5</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>6</td>
  <td>Memory</td>
  <td>1.0Gi</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>7</td>
  <td rowspan="6">Workspace Volume</td>
  <td>Don’t use Persistent Storage for User’s home</td>
  <td>チェックなし</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>8</td>
  <td>Type</td>
  <td>New</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>9</td>
  <td>Name</td>
  <td>workspace-my-first-notebook</td>
  <td>デフォルト値。「workspace-&lt;Name - Nameの値&gt;」が自動設定されます。変更もできます。</td>
 </tr>
 <tr>
  <td>10</td>
  <td>Size</td>
  <td>10Gi</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>11</td>
  <td>Mode</td>
  <td>ReadWriteOnce</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>12</td>
  <td>Mount Point</td>
  <td>/home/jovyan</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>13</td>
  <td>Data Volumes</td>
  <td>+ ADD VOLUME</td>
  <td>追加せず。</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>14</td>
  <td>Configurations</td>
  <td>Configurations</td>
  <td>空欄</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>15</td>
  <td rowspan="2">GPUs</td>
  <td>Number of GPUs</td>
  <td>None</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>16</td>
  <td>GPU Vendor</td>
  <td>GPU Vendor</td>
  <td>デフォルト値。</td>
 </tr>
 <tr>
  <td>17</td>
  <td>Miscellaneous Setting</td>
  <td>Enable Shared Memory</td>
  <td>有効</td>
  <td>デフォルト値。</td>
 </tr>
</table>

「LANCH」ボタンをクリック後、少し待つとStatusがRunning(チェックマーク)となって構築が完了します。

**Kubeflow Central DashboardのNotebook Serversページ(構築後)**
![Notebook_Server3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/eff027db-1587-3d52-a171-87ce15a515e6.png)
※しばらく待ってもStatusがRunningにならない場合、そのステータスアイコン部分をマウスオーバーすると、エラーメッセージが出力されることがありますので、確認してみてください。また、Kubeflow公式サイトの「Troubleshooting Guide for Jupyter Notebooks」の情報も参考にしてください。
https://www.kubeflow.org/docs/components/notebooks/troubleshoot/

* Notebook Serversページで、「CONNECT」をクリックします。

**Kubeflow Central DashboardのNotebook Serversページ(構築後)**
![Notebook_Server4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/56dfb3ba-64b9-709f-46a3-5e022a781701.png)

Jupyterのページが開きます。

**Jupyterページ**
![jupyter.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/624448/4fac7906-f91d-d503-cf58-faee243db65d.png)


# おわりに
[第1回の投稿](https://qiita.com/t-takahisa/items/5eefa9cf6be783cfe170)と本稿で、MLOpsを実現するKubeflowを前提のKubernetesも含めて構築する手順を紹介しました。次回は、Kubeflow Pipelinesの利用手順を紹介します。

[第3回：Kubeflow Pipelinesでscikit-learnの機械学習モデルを訓練・評価してみる](https://qiita.com/t-takahisa/items/0a409478cb502f4d3c5f)