Wazi: OpenShift上でのメインフレーム開発環境構築 - (5)Sandboxインスタンスの作成

# はじめに
前回までの設定で下準備が整ったはずなので、ようやく本丸であるSandboxの作成をやってみます。

## 関連記事
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (1)事前準備](https://qiita.com/tomotagwork/items/4f254440959ff0a97e4b)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (2)ライセンス・サーバーの構成](https://qiita.com/tomotagwork/items/2017ef0fbf8777a3c95e)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (3)イメージ・ストレージ・サーバーの構成](https://qiita.com/tomotagwork/items/1577193ca59eb2ae1fa5)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (4)OpenShiftクラスターの構成](https://qiita.com/tomotagwork/items/d0b358db6b0bfd819ffe)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (5)Sandboxインスタンスの作成](https://qiita.com/tomotagwork/items/74193fb90e3a3acd9c9b)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (6)Sandboxインスタンスの確認](https://qiita.com/tomotagwork/items/c8370878c0fd549b0a44)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (7)Sandboxインスタンスへの接続](https://qiita.com/tomotagwork/items/cfeec0ccab50d72a48e2)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (8)Sandboxのカスタマイズ](https://qiita.com/tomotagwork/items/52f66ab9c78b9d8b3131)
[Wazi: OpenShift上でのメインフレーム開発環境構築 - (9)Wazi Developer for Workspacesの作成](https://qiita.com/tomotagwork/items/7de6f979dabe160175e3)

# 全体像
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/132331/ca33a87a-cf9e-7963-b918-77ca3441acf0.png)


# Sandboxインスタンスの作成
参考: [Creating a sandbox instance with WaziSandboxSystem](https://www.ibm.com/docs/en/wdfrhcw/1.2.0?topic=sandbox-creating-instance-wazisandboxsystem)

## 必要な情報の整理
インスタンス作成にあたり、ここまで事前準備してきた情報を含め整理します。

|項目|WaziSandboxSystemリソースのPath|値|補足|
|---|---|---|---|
|Sandboxインスタンスの名前|metadata.name|wazi-sandbox01|namespaceでユニークな名前をつけます|
|Sandboxインスタンスを作成するプロジェクト名(namespace)|metadata.namespace| wazi-test01 |[参考](https://qiita.com/tomotagwork/items/d0b358db6b0bfd819ffe#%E3%83%97%E3%83%AD%E3%82%B8%E3%82%A7%E3%82%AF%E3%83%88)|
|SandboxイメージPull用のSecret名|spec.imagePullSecrets|wazi-sandbox-pull-secret|[参考](https://qiita.com/tomotagwork/items/d0b358db6b0bfd819ffe#sandbox%E3%82%A4%E3%83%A1%E3%83%BC%E3%82%B8%E3%82%92pull%E3%81%99%E3%82%8B%E9%9A%9B%E3%81%AB%E4%BD%BF%E7%94%A8%E3%81%99%E3%82%8B%E3%83%91%E3%82%B9%E3%83%AF%E3%83%BC%E3%83%89%E7%94%A8%E3%81%AEsecret)|
|Sandboxインスタンスで使用するパスワード用Secret名|spec.zpdt.SecretName|wazi-sandbox-pw-secret|[参考](https://qiita.com/tomotagwork/items/d0b358db6b0bfd819ffe#sandbox%E3%82%A4%E3%83%B3%E3%82%B9%E3%82%BF%E3%83%B3%E3%82%B9%E3%81%A7%E4%BD%BF%E7%94%A8%E3%81%99%E3%82%8B%E3%83%91%E3%82%B9%E3%83%AF%E3%83%BC%E3%83%89%E7%94%A8%E3%81%AEsecret)|
|ライセンス・サーバーなどの情報を設定したConfigMap名|spec.zpdt.configMap.name|wazi-sandbox-config01|[参考](https://qiita.com/tomotagwork/items/d0b358db6b0bfd819ffe#configmap)|
|Sandboxインスタンスから使用する永続ストレージ用のPVC名|spec.zVolumes.existingClaimName|wazi-sandbox-pvc01|[参考](https://qiita.com/tomotagwork/items/d0b358db6b0bfd819ffe#%E6%B0%B8%E7%B6%9A%E3%82%B9%E3%83%88%E3%83%AC%E3%83%BC%E3%82%B8)|

## Volume Copy
マニュアルの手順だと、後述のSandboxインスタンス・デプロイ用のYAMLを直接適用すればVolume Copyも最初に行われるはずですが、不具合があるようでうまくいきませんでした。(詳細は末尾の参考に記載している[試行錯誤ログ-その2](https://qiita.com/tomotagwork/items/74193fb90e3a3acd9c9b#%E3%81%9D%E3%81%AE2)参照)
以下の情報に従って、Volume Copyを事前に行ってからSandboxインスタンスのデプロイを行います。
参考: [Unable to create Sandbox on IBM Cloud](https://www.ibm.com/support/pages/node/6375272)
※この不具合はWazi V1.2.0では解消されているようなので、V1.2.0ではVolume Copyを個別に実施しなくてもよいようです。

Volume Copy用に以下のYAMLファイルを作成します。

```yaml:sandbox-fix-init-copy.yaml
apiVersion: sandbox.wazi.ibm.com/v1alpha1
kind: WaziSandboxVolumeCopy
metadata:
  name: sandbox-fix-init-copy
spec:
  source:
    type: sftp
    existingConfigName: wazi-sandbox-config01
    secretName: wazi-sandbox-pw-secret
  destination:
    existingClaimName: wazi-sandbox-pvc01
  imagePullSecrets:
    - name: wazi-sandbox-pull-secret
```

適用！

```
[root@Test05 ~/openshift/Wazi]# oc apply -f sandbox-fix-init-copy.yaml
wazisandboxvolumecopy.sandbox.wazi.ibm.com/sandbox-fix-init-copy created
```

JobとしてPodが起動され、sftpサーバーから永続ボリュームにADCDのファイルが転送されて解凍されます。

```
[root@Test05 ~]# oc get all
NAME                                                           READY   STATUS    RESTARTS   AGE
pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-8dbc7   1/1     Running   0          94m

NAME                                                           COMPLETIONS   DURATION   AGE
job.batch/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job   0/1           94m        94m
```

ファイルの転送に約2時間くらいかかりました。
完了後の状態

```
[root@Test05 ~/openshift/Wazi]# oc get all
NAME                                                           READY   STATUS      RESTARTS   AGE
pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-8dbc7   0/1     Completed   0          116m

NAME                                                           COMPLETIONS   DURATION   AGE
job.batch/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job   1/1           114m       116m
```

<details><summary><font color="blue">参考: Volume Copy用Podのログ</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc logs -f sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-8dbc7
Jan 15, 2021 9:23:32 AM com.ibm.zdt.init.App main
INFO: Remote Copy Task:
Jan 15, 2021 9:23:34 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Inventory.txt file does not exists and will be downloaded to directory /zdt/zvolumes/
Jan 15, 2021 9:23:34 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: No. of threads:4
Jan 15, 2021 9:23:34 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: COPY_BLOCK_SIZE:524288
Jan 15, 2021 9:23:36 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: inventory.txt with thread id:28
Jan 15, 2021 9:23:36 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 9:23:36 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: inventory.txt with thread id:28 Total time to complete in seconds: 0
Jan 15, 2021 9:23:41 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: Transferring volumes complete.
Jan 15, 2021 9:23:41 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Reading inventory.txt file from directory /zdt/zvolumes/
Jan 15, 2021 9:23:41 AM com.ibm.zdt.constants.Constants checkVolumes
INFO: List of files to download.
B4BLZ1.gz
B4C541.gz
B4C551.gz
B4CFG1.gz
B4DBAR.gz
B4DBB1.gz
B4DBB2.gz
B4DBC1.gz
B4DBC2.gz
B4DIS1.gz
B4DIS2.gz
B4DIS3.gz
B4IME1.gz
B4IMF1.gz
B4INM1.gz
B4KAN1.gz
B4PAGA.gz
B4PAGB.gz
B4PAGC.gz
B4PRD1.gz
B4PRD2.gz
B4PRD3.gz
B4PRD4.gz
B4RES1.ZPD
B4RES2.gz
B4SYS1.gz
B4USR1.gz
B4USS1.gz
B4USS2.gz
B4W901.gz
B4W902.gz
B4ZCX1.gz
B4ZWE1.gz
devmap.txt
SARES1.ZPD

Jan 15, 2021 9:23:41 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: No. of threads:4
Jan 15, 2021 9:23:41 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: COPY_BLOCK_SIZE:524288
Jan 15, 2021 9:23:43 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4BLZ1.gz with thread id:36
Jan 15, 2021 9:23:44 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4C541.gz with thread id:38
Jan 15, 2021 9:23:46 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4C551.gz with thread id:40
Jan 15, 2021 9:23:47 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4CFG1.gz with thread id:42
Jan 15, 2021 9:26:10 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4CFG1.gz with thread id:42 Total time to complete in seconds: 143
Jan 15, 2021 9:26:10 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBAR.gz with thread id:42
Jan 15, 2021 9:37:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4BLZ1.gz with thread id:36 Total time to complete in seconds: 855
Jan 15, 2021 9:37:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBB1.gz with thread id:36
Jan 15, 2021 9:37:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4C541.gz with thread id:38 Total time to complete in seconds: 854
Jan 15, 2021 9:37:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBB2.gz with thread id:38
Jan 15, 2021 9:38:05 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4C551.gz with thread id:40 Total time to complete in seconds: 859
Jan 15, 2021 9:38:05 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBC1.gz with thread id:40
Jan 15, 2021 9:39:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBAR.gz with thread id:42 Total time to complete in seconds: 828
Jan 15, 2021 9:39:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBC2.gz with thread id:42
Jan 15, 2021 9:51:37 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBB2.gz with thread id:38 Total time to complete in seconds: 819
Jan 15, 2021 9:51:37 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DIS1.gz with thread id:38
Jan 15, 2021 9:52:33 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBB1.gz with thread id:36 Total time to complete in seconds: 875
Jan 15, 2021 9:52:33 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DIS2.gz with thread id:36
Jan 15, 2021 9:52:43 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBC1.gz with thread id:40 Total time to complete in seconds: 878
Jan 15, 2021 9:52:43 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DIS3.gz with thread id:40
Jan 15, 2021 9:53:22 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBC2.gz with thread id:42 Total time to complete in seconds: 804
Jan 15, 2021 9:53:22 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4IME1.gz with thread id:42
Jan 15, 2021 10:06:00 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DIS2.gz with thread id:36 Total time to complete in seconds: 807
Jan 15, 2021 10:06:00 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4IMF1.gz with thread id:36
Jan 15, 2021 10:06:05 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DIS3.gz with thread id:40 Total time to complete in seconds: 802
Jan 15, 2021 10:06:05 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4INM1.gz with thread id:40
Jan 15, 2021 10:06:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4IME1.gz with thread id:42 Total time to complete in seconds: 816
Jan 15, 2021 10:06:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4KAN1.gz with thread id:42
Jan 15, 2021 10:07:57 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DIS1.gz with thread id:38 Total time to complete in seconds: 980
Jan 15, 2021 10:07:57 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PAGA.gz with thread id:38
Jan 15, 2021 10:19:42 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4IMF1.gz with thread id:36 Total time to complete in seconds: 822
Jan 15, 2021 10:19:42 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PAGB.gz with thread id:36
Jan 15, 2021 10:19:54 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4KAN1.gz with thread id:42 Total time to complete in seconds: 776
Jan 15, 2021 10:19:54 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PAGC.gz with thread id:42
Jan 15, 2021 10:20:29 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PAGA.gz with thread id:38 Total time to complete in seconds: 752
Jan 15, 2021 10:20:29 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD1.gz with thread id:38
Jan 15, 2021 10:23:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4INM1.gz with thread id:40 Total time to complete in seconds: 1071
Jan 15, 2021 10:23:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD2.gz with thread id:40
Jan 15, 2021 10:31:44 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PAGB.gz with thread id:36 Total time to complete in seconds: 722
Jan 15, 2021 10:31:44 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD3.gz with thread id:36
Jan 15, 2021 10:31:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PAGC.gz with thread id:42 Total time to complete in seconds: 722
Jan 15, 2021 10:31:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD4.gz with thread id:42
Jan 15, 2021 10:36:49 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD1.gz with thread id:38 Total time to complete in seconds: 980
Jan 15, 2021 10:36:49 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4RES1.ZPD with thread id:38
Jan 15, 2021 10:36:49 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 10:39:21 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4RES1.ZPD with thread id:38 Total time to complete in seconds: 152
Jan 15, 2021 10:39:21 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4RES2.gz with thread id:38
Jan 15, 2021 10:41:36 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD2.gz with thread id:40 Total time to complete in seconds: 1060
Jan 15, 2021 10:41:36 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4SYS1.gz with thread id:40
Jan 15, 2021 10:43:05 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD4.gz with thread id:42 Total time to complete in seconds: 669
Jan 15, 2021 10:43:05 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4USR1.gz with thread id:42
Jan 15, 2021 10:48:36 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD3.gz with thread id:36 Total time to complete in seconds: 1012
Jan 15, 2021 10:48:36 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4USS1.gz with thread id:36
Jan 15, 2021 10:52:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4RES2.gz with thread id:38 Total time to complete in seconds: 768
Jan 15, 2021 10:52:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4USS2.gz with thread id:38
Jan 15, 2021 10:53:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4SYS1.gz with thread id:40 Total time to complete in seconds: 740
Jan 15, 2021 10:53:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4W901.gz with thread id:40
Jan 15, 2021 10:57:24 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4USR1.gz with thread id:42 Total time to complete in seconds: 859
Jan 15, 2021 10:57:24 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4W902.gz with thread id:42
Jan 15, 2021 11:02:30 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4USS1.gz with thread id:36 Total time to complete in seconds: 834
Jan 15, 2021 11:02:30 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4ZCX1.gz with thread id:36
Jan 15, 2021 11:05:31 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4USS2.gz with thread id:38 Total time to complete in seconds: 802
Jan 15, 2021 11:05:31 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4ZWE1.gz with thread id:38
Jan 15, 2021 11:09:18 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4W902.gz with thread id:42 Total time to complete in seconds: 714
Jan 15, 2021 11:09:18 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: devmap.txt with thread id:42
Jan 15, 2021 11:09:18 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 11:09:18 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: devmap.txt with thread id:42 Total time to complete in seconds: 0
Jan 15, 2021 11:09:18 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: SARES1.ZPD with thread id:42
Jan 15, 2021 11:09:18 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 11:10:54 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: SARES1.ZPD with thread id:42 Total time to complete in seconds: 96
Jan 15, 2021 11:11:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4W901.gz with thread id:40 Total time to complete in seconds: 1082
Jan 15, 2021 11:15:10 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4ZWE1.gz with thread id:38 Total time to complete in seconds: 579
Jan 15, 2021 11:17:39 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4ZCX1.gz with thread id:36 Total time to complete in seconds: 909
Jan 15, 2021 11:17:39 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: Transferring volumes complete.
Jan 15, 2021 11:17:39 AM com.ibm.zdt.init.App main
INFO: Total time to complete all the file transfers (in seconds): 6847
```

</div></details>


## Sandboxインスタンス・デプロイ

Sandboxインスタンス・デプロイ用のYAMLファイルを作成します(Wazi V1.1.0の例)。

```yaml:wazi-sandbox01.yaml
apiVersion: sandbox.wazi.ibm.com/v1alpha1
kind: WaziSandboxSystem
metadata:
  name: wazi-sandbox01
  namespace: wazi-test01
spec:
  imagePullSecrets:
    - name: wazi-sandbox-pull-secret
  zpdt:
    configMap:
      name: wazi-sandbox-config01
    secretName: wazi-sandbox-pw-secret
  zVolumes:
    existingClaimName: wazi-sandbox-pvc01
```

<details><summary><font color="blue">参考:Wazi V1.2.0の場合のyaml</font></summary><div>

```yaml:wazi-sandbox01.yaml
apiVersion: sandbox.wazi.ibm.com/v1alpha1
kind: WaziSandboxSystem
metadata:
  name: wazi-sandbox01
  namespace: wazi-test01
spec:
  license:
    accept: true
  imagePullSecrets:
    - name: wazi-sandbox-pull-secret
  zpdt:
    configMap:
      name: wazi-sandbox-config01
    secretName: wazi-sandbox-pw-secret
  zVolumes:
    existingClaimName: wazi-sandbox-pvc01
```

Wazi V1.2.0の場合、sandboxデプロイ時のyamlでspec.licesne.acceptを追加する必要があります。その他のyamlファイルはV1.1.0の例で示したものをそのまま利用できます。

</div></details>

適用！

```
[root@Test05 ~/openshift/Wazi]# oc apply -f wazi-sandbox01.yaml
wazisandboxsystem.sandbox.wazi.ibm.com/wazi-sandbox01 created
```

Podが起動しました！

```
[root@Test05 ~/openshift/Wazi]# oc get all
NAME                                                           READY   STATUS      RESTARTS   AGE
pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-8dbc7   0/1     Completed   0          121m
pod/wazi-sandbox01-wazi-sandbox-system-6679544c49-9d8xp        1/1     Running     0          2m58s

NAME                                         TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                                                               AGE
service/wazi-sandbox01-wazi-sandbox-system   NodePort   172.21.137.238   <none>        8443:30690/TCP,22:31175/TCP,2023:32103/TCP,7394:31125/TCP,7395:30146/TCP,7392:30837/TCP,4035:31510/TCP,4037:31771/TCP,7397:30884/TCP,9443:32443/TCP,10443:31917/TCP   2m56s

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/wazi-sandbox01-wazi-sandbox-system   1/1     1            1           2m59s

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/wazi-sandbox01-wazi-sandbox-system-6679544c49   1         1         1       2m58s

NAME                                                           COMPLETIONS   DURATION   AGE
job.batch/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job   1/1           114m       121m
```


<details><summary><font color="blue">参考: Pod起動直後にPodに入って覗いてみた</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc exec wazi-sandbox01-wazi-sandbox-system-6679544c49-9d8xp -c wazi-sandbox-system -it -- bash
[root@wazi-sandbox01-wazi-sandbox-system-6679544c49-9d8xp ibmsys1]# whoami
root

[root@wazi-sandbox01-wazi-sandbox-system-6679544c49-9d8xp ibmsys1]# top -d 1

top - 11:27:59 up 18 days,  5:00,  0 users,  load average: 6.00, 5.52, 5.94
Tasks:  14 total,   1 running,  13 sleeping,   0 stopped,   0 zombie
%Cpu0  : 11.7 us,  4.3 sy,  0.0 ni, 81.9 id,  0.0 wa,  0.0 hi,  1.1 si,  1.1 st
%Cpu1  : 16.3 us, 10.2 sy,  0.0 ni, 27.6 id, 44.9 wa,  0.0 hi,  0.0 si,  1.0 st
%Cpu2  :  7.3 us,  6.2 sy,  0.0 ni, 85.4 id,  0.0 wa,  0.0 hi,  0.0 si,  1.0 st
%Cpu3  : 31.6 us, 12.2 sy,  0.0 ni, 56.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  : 15.2 us,  8.1 sy,  0.0 ni, 35.4 id, 41.4 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  7.1 us, 10.2 sy,  0.0 ni,  0.0 id, 78.6 wa,  0.0 hi,  3.1 si,  1.0 st
%Cpu6  : 24.2 us,  5.3 sy,  0.0 ni, 53.7 id, 16.8 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  : 29.9 us,  4.1 sy,  0.0 ni, 51.5 id,  9.3 wa,  0.0 hi,  4.1 si,  1.0 st
MiB Mem :  32006.8 total,  13232.6 free,   6520.5 used,  12253.6 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  25035.4 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    836 ibmsys1   20   0   56760   3440   2852 D  23.8   0.0   0:19.03 Z1091_ADCD_inst
    223 ibmsys1   20   0 5377720 226840  53624 S   2.0   0.7   1:30.54 java
      1 root      20   0   94328   6812   4760 S   0.0   0.0   0:00.41 systemd
     29 root      20   0  127560  14148  13148 S   0.0   0.0   0:00.80 systemd-journal
     44 dbus      20   0   53972   2500   1964 S   0.0   0.0   0:00.09 dbus-daemon
     66 root      20   0   95404  33728   3512 S   0.0   0.1   0:01.66 haproxy
    172 haproxy   20   0  100392  34196    208 S   0.0   0.1   0:00.69 haproxy
    176 root      20   0   18864    316    212 S   0.0   0.0   0:00.00 aksusbd
    181 root      20   0   40880  11160   3524 S   0.0   0.0   0:00.63 hasplmd
    427 ibmsys1   20   0   16944   1456   1224 S   0.0   0.0   0:00.00 ipl_zOS.sh
    428 ibmsys1   20   0   16944   1636   1380 S   0.0   0.0   0:00.00 decrypt_volumes
    791 root      20   0   19304   2360   1784 S   0.0   0.0   0:00.01 bash
    837 ibmsys1   20   0   30204   1192    964 S   0.0   0.0   0:00.00 tee
    944 root      20   0   56264   2312   1664 R   0.0   0.0   0:00.05 top

[root@wazi-sandbox01-wazi-sandbox-system-6679544c49-9d8xp ibmsys1]# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 11:22 ?        00:00:00 /sbin/init
root          29       1  0 11:22 ?        00:00:00 /usr/lib/systemd/systemd-journald
dbus          44       1  0 11:22 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root          66       1  0 11:22 ?        00:00:01 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy      172      66  0 11:22 ?        00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
root         176       1  0 11:22 ?        00:00:00 /usr/sbin/aksusbd
root         181       1  0 11:22 ?        00:00:00 /usr/sbin/hasplmd -s
ibmsys1      223       1 23 11:22 ?        00:01:36 /zdt/java/jre/bin/java -javaagent:/zdt/wlp/bin/tools/ws-javaagent.jar -Djava.awt.headless=true -Djdk.attach.allowAttachSelf=true -jar /zdt/wlp/bin/tools/ws-server.jar zdt-instance
ibmsys1      427       1  0 11:23 ?        00:00:00 /bin/bash /zdt/bin/ipl_zOS.sh &
ibmsys1      428     427  0 11:23 ?        00:00:00 /bin/bash /zdt/bin/decrypt_volumes.sh
root         791       0  0 11:26 pts/0    00:00:00 bash
ibmsys1      836     428 24 11:26 ?        00:00:45 Z1091_ADCD_install /zdt/zvolumes/SARES1.ZPD /zdt/zvolumes/SARES1
ibmsys1      837     428  0 11:26 ?        00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee -a /zdt/zvolumes/log.txt
root        1226     791  0 11:29 pts/0    00:00:00 ps -ef
```

しばらくするとzPDTのプロセスが起動してきた。

```
[root@wazi-sandbox01-wazi-sandbox-system-6679544c49-9d8xp logs]# ps -ef | grep emily
ibmsys1     1413    1410 85 11:30 ?        00:03:23 emily -c 3,0,0,0 -u 44 -m 8192 -x 0 -l -p 1410
```

```txt:/home/ibmsys1/z1090/logs/log_console_xxx.txt
CMD : 011521 11:30:53: awsstart /zdt/zvolumes/devmap.txt --clean
LOG : 011521 11:30:53: AWSSTART:
z1091, version 1-10.55.05.01, build date - 09/15/20 for Linux on Redhat 64bit

LOG : 011521 11:30:53: AWSSTART: Intel-64 architecture
LOG : 011521 11:30:53: AWSSTART: Configuration File-/zdt/zvolumes/devmap.txt
INFO: 011521 11:30:53: STA: AWSSTA204I zPDT started in directory '/'.
INFO: 011521 11:30:53: STA: AWSSTA146I Starting independent 1090 instance 'ibmsys1'
CMD : 011521 11:30:53: awszpci -p 1410
CMD : 011521 11:30:53: awsap -p 1410
LOG : 011521 11:30:53: CPU: DEVMAP loaded for EMILY process, RC=69
LOG : 011521 11:30:53: CPU: Creating 3 CPUs
LOG : 011521 11:30:53: CPU: no_of_resources......0
LOG : 011521 11:30:53: CPU: config_tab_length....1E0
LOG : 011521 11:30:53: CPU: zpci_msg_q_name......AWSZPCI_message queue
LOG : 011521 11:30:53: CPU: zpci_sem_name........AWSZPCI_semaphore
LOG : 011521 11:30:53: CPU:
LOG : 011521 11:30:53: CPU: Starting CPUs.
LOG : 011521 11:30:53: CPU: ERAS: Starting RAS setup
CMD : 011521 11:30:53: sclp_process 0 0 valid
LOG : 011521 11:30:54: CPU: STP TIMING ALERT
LOG : 011521 11:30:54: CPU: IPR started for CPU 1
LOG : 011521 11:30:54: CPU: IPR started for CPU 2
LOG : 011521 11:30:54: CPU: CPU reset
LOG : 011521 11:30:54: CPU: CPU reset
LOG : 011521 11:30:54: CPU: IPR complete for CPU 1
LOG : 011521 11:30:54: CPU: IPR complete for CPU 2
LOG : 011521 11:30:54: CPU: IPR started for CPU 0
LOG : 011521 11:30:54: CPU: CPU reset
LOG : 011521 11:30:54: CPU: IPR complete for CPU 0
INFO: 011521 11:30:55: CPU:
LDK license obtained for CPU 1
LOG : 011521 11:30:55: EMILY_DEVMAP: Creating UIM service...
LOG : 011521 11:30:55: EMILY_DEVMAP: UIM service pid=1426...
INFO: 011521 11:30:55: CPU:
LDK license obtained for CPU 0
INFO: 011521 11:30:56: EMI: AWSEMI005I Waiting for 1090 license
LOG : 011521 11:30:56: EMILY_DEVMAP: UIM service created... 1426
LOG : 011521 11:30:56: CPU: ERAS: SIGCHLD from unknown PID
INFO: 011521 11:30:56: CPU:
LDK license obtained for CPU 2
INFO: 011521 11:30:57: OSA: OSA code level = 0x7617
INFO: 011521 11:30:58: DSA: AWSDSA010I AWSOSA is ready for chpid: 0xA0 device: 0x400
INFO: 011521 11:30:58: DSA: AWSDSA010I AWSOSA is ready for chpid: 0xA0 device: 0x401
INFO: 011521 11:30:58: DSA: AWSDSA010I AWSOSA is ready for chpid: 0xA0 device: 0x402
CMD : 011521 11:31:02: tnportl2 -r -l -n IBMSYS1
CMD : 011521 11:31:05: /usr/z1090/bin/ipl A80 parm 0A82NZ
LOG : 011521 11:31:08: CPU: CPU reset
LOG : 011521 11:31:08: CPU: Accepted code 7
LOG : 011521 11:31:08: CPU: CPU reset
LOG : 011521 11:31:08: CPU: Accepted code 7
LOG : 011521 11:31:08: CPU: CPU reset
LOG : 011521 11:31:08: CPU: Accepted code 7
INFO: 011521 11:31:09: MIP: AWSMIP008I zArchitecture IPL mode (ZARCH_ONLY=ON)
LOG : 011521 11:31:09: CPU: IPL started on CPU 0
LOG : 011521 11:31:09: CPU: IPR started for CPU 0
LOG : 011521 11:31:09: CPU: CPU reset
LOG : 011521 11:31:09: CPU: IPR complete for CPU 0
LOG : 011521 11:31:10: CPU: IPL bootstrap successful for device A80
LOG : 011521 11:31:10: CPU: Accepted code 0
LOG : 011521 11:31:10: CPU: Running CPU
LOG : 011521 11:31:10: CPU:  Short PSW Change 0008000080000652
INFO: 011521 11:31:11: STA: AWSSTA059I System initialization complete
INFO: 011521 11:31:11: STA: AWSSTA012I All configured subsystems started
LOG : 011521 11:31:15: DIAGNOSE: load device stored is 0A80 parm=F0C1F8F2D5E94040 F=80 VF=00
INFO: 011521 11:31:18:     : OPRMSG:   *IEA247I USING IEASYSNZ FOR z/OS 02.04.00 HBB77C0
INFO: 011521 11:31:18:     : OPRMSG: Beep!
INFO: 011521 11:31:18:     : OPRMSG:    ISG313I SYSTEM IS INITIALIZING IN GRS NONE MODE.  RING OR STAR CONFIGURATION KEYWORDS IN GRSCNF00 ARE IGNORED.
INFO: 011521 11:31:19:     : OPRMSG:    IAR040I REAL STORAGE AMOUNTS:
INFO: 011521 11:31:19:     : OPRMSG:      TOTAL AVAILABLE ONLINE: 8G
INFO: 011521 11:31:19:     : OPRMSG:        LFAREA LIMIT FOR xM, xG, OR xT      : 4505M
INFO: 011521 11:31:19:     : OPRMSG:        LFAREA LIMIT FOR SUM OF 1M= AND 2G= : 3276M
INFO: 011521 11:31:19:     : OPRMSG:        LFAREA LIMIT FOR 2GB PAGES FOR 2G=  : 1
INFO: 011521 11:31:19:     : OPRMSG:    IAR048I LFAREA=(1M=(15%,0%),NOPROMPT) WAS PROCESSED WHICH RESULTED IN 614 1MB PAGES AND 0 2GB PAGES.
INFO: 011521 11:32:03:     : OPRMSG:    IEA598I TIME ZONE = W.04.00.00
INFO: 011521 11:32:03:     : OPRMSG:    CNZ2600I AUTO-REPLY POLICY ACTIVATED.
INFO: 011521 11:32:04:     : OPRMSG:    IXC470I SYSTEM S0W1 EFFECTIVE VALUES: INTERVAL=165 OPNOTIFY=168
INFO: 011521 11:32:04:     : OPRMSG:    DEFAULT USER INTERVAL:    165
INFO: 011521 11:32:04:     : OPRMSG:    DERIVED SPIN INTERVAL:    165
INFO: 011521 11:32:04:     : OPRMSG:    DEFAULT USER OPNOTIFY: +    3
INFO: 011521 11:32:04:     : OPRMSG:    COMPUTED FOR:  XCF INITIALIZATION
INFO: 011521 11:32:05:     : OPRMSG:    IXC414I CANNOT JOIN SYSPLEX ADCDPL WHICH IS RUNNING IN MONOPLEX MODE: CONFIGURATION REQUIREMENT
INFO: 011521 11:32:05:     : OPRMSG:    IXC404I SYSTEM(S) ACTIVE OR IPLING: S0W1
INFO: 011521 11:32:05:     : OPRMSG:   *   IXC420D REPLY I TO INITIALIZE SYSPLEX ADCDPL, OR R TO REINITIALIZE XCF.  REPLYING I WILL IMPACT OTHER ACTIVE SYSTEMS.
INFO: 011521 11:32:05:     : OPRMSG: Beep!
INFO: 011521 11:32:05:     : OPRMSG: PROMPT:   R 00,
INFO: 011521 11:32:05:     : OPRMSG:    CNZ2602I REPLY TO 00 IS:I <- Auto replied
INFO: 011521 11:32:14:     : OPRMSG:    IXC413I MULTISYSTEM SYSPLEX CONFIGURATION PREVENTED BY SYSTEM COMPONENT
INFO: 011521 11:32:14:     : OPRMSG:    ISG150I GRS=NONE IS NOT SUPPORTED WHEN RUNNING IN A MULTISYSTEM SYSPLEX.
INFO: 011521 11:32:14:     : OPRMSG:    IXC418I SYSTEM S0W1 IS NOW ACTIVE IN SYSPLEX ADCDPL
INFO: 011521 11:32:42:     : OPRMSG:    IEE712I SET CNGRP PROCESSING COMPLETE
INFO: 011521 11:32:42:     : OPRMSG:    IEA549I SYSTEM CONSOLE FUNCTIONS AVAILABLE
INFO: 011521 11:32:42:     : OPRMSG:            SYSTEM CONSOLE NAME ASSIGNED HWCI
INFO: 011521 11:32:43:     : OPRMSG:    CEE3739I LANGUAGE ENVIRONMENT INITIALIZATION COMPLETE
LOG : 011521 11:32:46: CPU: IPR started for CPU 1
LOG : 011521 11:32:46: CPU: CPU reset
LOG : 011521 11:32:46: CPU: IPR complete for CPU 1
LOG : 011521 11:32:46: CPU: Accepted code 0
LOG : 011521 11:32:47: CPU: CPU restart
LOG : 011521 11:32:47: CPU: Restarting CPU 1
LOG : 011521 11:32:47: CPU: Accepted code 6
LOG : 011521 11:32:47: CPU: Running CPU
LOG : 011521 11:32:47: CPU: IPR started for CPU 2
LOG : 011521 11:32:47: CPU: CPU reset
LOG : 011521 11:32:47: CPU: IPR complete for CPU 2
LOG : 011521 11:32:47: CPU: Accepted code 0
LOG : 011521 11:32:48: CPU: CPU restart
LOG : 011521 11:32:48: CPU: Restarting CPU 2
LOG : 011521 11:32:48: CPU: Accepted code 6
LOG : 011521 11:32:48: CPU: Running CPU
LOG : 011521 11:32:48: CPU:  Returning from CPU 1 11
LOG : 011521 11:32:48: CPU: CPU 0 stop
LOG : 011521 11:32:48: CPU: Accepted code 1
LOG : 011521 11:32:49: CPU: IPR started for CPU 0
LOG : 011521 11:32:49: CPU: CPU reset
LOG : 011521 11:32:49: CPU: IPR complete for CPU 0
LOG : 011521 11:32:49: CPU: Accepted code 0
LOG : 011521 11:32:49: CPU: IPR started for CPU 0
LOG : 011521 11:32:49: CPU: CPU reset
LOG : 011521 11:32:49: CPU: IPR complete for CPU 0
LOG : 011521 11:32:49: CPU: Accepted code 0
LOG : 011521 11:32:50: CPU: CPU restart
LOG : 011521 11:32:50: CPU: Restarting CPU 0
LOG : 011521 11:32:50: CPU: Accepted code 6
LOG : 011521 11:32:50: CPU: Running CPU
INFO: 011521 11:32:58:     : OPRMSG:    IEE389I MVS COMMAND PROCESSING AVAILABLE
```

</div></details>

一応、Zのエミュレーターの起動は出来てるっぽい！

## Sandbox稼働確認後やっておいた方がよいこと
※この不具合もWazi V1.2.0では解消されているようなので、V1.2.0では以下の操作は不要です。
Sandbox PodのContainerに接続して、/zdt/zvolumes/inventory.txtを編集して、B4RES1.ZPD、SARES1.ZPDを削除しておく！

<details><summary><font color="blue">デフォルトのinventory.txt</font></summary><div>

```txt:/zdt/zvolumes/inventory.txt
B4BLZ1.gz
B4C541.gz
B4C551.gz
B4CFG1.gz
B4DBAR.gz
B4DBB1.gz
B4DBB2.gz
B4DBC1.gz
B4DBC2.gz
B4DIS1.gz
B4DIS2.gz
B4DIS3.gz
B4IME1.gz
B4IMF1.gz
B4INM1.gz
B4KAN1.gz
B4PAGA.gz
B4PAGB.gz
B4PAGC.gz
B4PRD1.gz
B4PRD2.gz
B4PRD3.gz
B4PRD4.gz
B4RES1.ZPD
B4RES2.gz
B4SYS1.gz
B4USR1.gz
B4USS1.gz
B4USS2.gz
B4W901.gz
B4W902.gz
B4ZCX1.gz
B4ZWE1.gz
devmap.txt
SARES1.ZPD
```

</div></details>

補足:
なぜ上の操作が必要なのか、ログ等から判断できるPodの挙動を解説しながら補足します。
SandboxのPodにはInit Containerが設定されていて、まず本体のContainerが動く前にこのInit Containerが実行されます。
Init ContainerではsftpサーバーからADCDのDASDイメージファイル(.gz)**等**を取得して永続ボリュームがマウントされたディレクトリ(/zdt/zvolumes)にコピーし、.gzファイルについては解凍処理を行います。これらが完了すると本体のコンテナが動いて、/zdt/zvolumes以下にあるDASDイメージを元にz/OSを起動する、という流れになります。これが基本的な動作です。
このInit Container部分の永続ボリュームに対する書き込みの所に不具合があったので、今回はその処理は個別にオペレーターで実行しておく、という手順を踏んでいます。
このInit Containerでは、sftpからDASDイメージファイルをコピーする際、既にファイルがあるかどうかをチェックし、存在していれば実際の転送はスキップしてくれるので、上の回避策が有効となっています。

Init Containerでのファイル転送の処理をもう少し詳しくみていきます。
sftpサーバー上に配置されているADCD関連のファイルには以下のものがあります。
(1) inventory.txt (転送対象ファイルのリストが記載されたテキストファイル)
(2) devmap.txt (DeviceMapファイル)
(3) DASDイメージファイル(.gz)
(4) DASDイメージファイル(.ZPD)
Init Containerでは、まずinventory.txtをsftpサーバーから入手します。そしてそのinventory.txtに記載されている(2)～(4)の各ファイル名のリストを見て、**もしファイルがローカルになかったら**それらのファイルをsftpサーバーから取得します。
実はこの判定ロジックに問題があり、後々影響してきます。.gzファイルは解凍処理も行うので解凍後のファイル(拡張子なし)が存在していれば転送はスキップされますが、.gz以外のファイルは解凍しないので.ZPD, .txtなどはそのまま拡張子付きファイル名の存在有無を判定し、存在していれば転送をスキップ、存在していなければ転送しようとします。

Volume CopyのOperator実行後、最初のSandboxデプロイのOperater実行時はこれで特に問題ありません。inventory.txtや解凍後の.gzファイルが全て永続ボリュームに保持されているので、Init Containerは実質何もせず、本体のContainerの実行に進みます。

次に本体のContainerでは何が行われるかというと、zPDT(z/OSエミュレーター)のセットアップと起動が行われます。SYSRES相当のDASDイメージファイル(.ZPD)はライセンス保護の観点から通常のunzipコマンド等では解凍できないようになっており、製品提供の専用のコマンドを使ってライセンスチェックを行いながら解凍を行います。すなわちこの段階で、B4RES1.ZDP => B4RES1 に解凍され、元のB4RES1.ZDPというファイルは無くなります。この状態でz/OSのIPLが走る訳です。結果として、Sandboxが稼働している状態では、永続ボリューム上には.ZPDというファイルは無くなっているということになります。

さて、ここでPod障害などでSandboxのPodが落ちてしまった場合を考えます。Podが新たに作成されて起動する時、Init Containerから処理が始まります。永続ボリュームは残っていますのでDASDイメージファイルなどはそのまま使われるはずですが、初期状態と違うのは、.ZPDファイルが無くなっているということです。Init Containerはinventory.txtにあるファイルを見て、無ければsftpサーバーから転送する、ということを行います。本来なら全てファイルがそろっているので何も転送する必要は無いのですが、先の本体のContainerの処理で.ZPDファイルが消えてしまっているので、.ZPDファイルを転送しようとしますが、先に挙げた内在している不具合のために永続ボリュームに書き込めずに失敗 => Pod起動に失敗ということになります。

つまり、一度Sandboxが起動したら、その後なんらかの状況でPodを再起動しないといけなくなった時に上がらなくなってしまうので、.ZPDファイルの転送を回避するためにinventory.txtから.ZPDファイルを削除しておく必要がある、ということです。永続ボリューム書き出しの不具合が解消されたとしても、.ZPDファイルの無駄な転送は避けた方がよいのでいずれにしてもこの対応はしておいた方がよいです。

※もしinventory.txtを書き換えるまえに誤ってPod落としてしまったら、ダミーのPod立ち上げてそこにボリュームマウントしてそこからinventory.txtの中身を書き換えてあげればOKです。



# 参考: 試行錯誤のログ
トライ&エラーを実行した時のログを参考までに載せておきます。ここまでの手順は試行錯誤した結果を反映したものなので、以下は読み飛ばしてもOKです。

## その1
上のYAMLを適用してSandboxインスタンスをデプロイします。

```
[root@Test05 ~/openshift/Wazi]# oc project
Using project "wazi-test01" on server "https://xxx.jp-tok.containers.cloud.ibm.com:xxx".

[root@Test05 ~/openshift/Wazi]# oc apply -f wazi-sandbox01.yaml
wazisandboxsystem.sandbox.wazi.ibm.com/wazi-sandbox01 created
```

イベント見るとエラーが出た。

```
[root@Test05 ~/openshift/Wazi]# oc get event
LAST SEEN   TYPE      REASON                  OBJECT                                                     MESSAGE
14m         Normal    ExternalProvisioning    persistentvolumeclaim/wazi-sandbox-pvc01                   waiting for a volume to be created, either by external provisioner "ibm.io/ibmc-block" or manually created by system administrator
15m         Normal    Provisioning            persistentvolumeclaim/wazi-sandbox-pvc01                   External provisioner is provisioning volume for claim "wazi-test01/wazi-sandbox-pvc01"
14m         Normal    ProvisioningSucceeded   persistentvolumeclaim/wazi-sandbox-pvc01                   Successfully provisioned volume pvc-14737ef3-5c7c-4d65-b1ef-288f9936193b
32s         Warning   FailedCreate            replicaset/wazi-sandbox01-wazi-sandbox-system-6679544c49   Error creating: pods "wazi-sandbox01-wazi-sandbox-system-6679544c49-" is forbidden: error looking up service account wazi-test01/wazi-sandbox-operator: serviceaccount "wazi-sandbox-operator" not found
2m35s       Normal    ScalingReplicaSet       deployment/wazi-sandbox01-wazi-sandbox-system              Scaled up replica set wazi-sandbox01-wazi-sandbox-system-6679544c49 to 1
```

wazi-sandbox-operatorというserviceaccountが無いよと言われているらしいが、マニュアル調べてみてもそのようなserviceaccount作る手順は特に記載されていない。うーむ。

一旦削除します。

```
[root@Test05 ~/openshift/Wazi]# oc delete -f wazi-sandbox01.yaml
wazisandboxsystem.sandbox.wazi.ibm.com "wazi-sandbox01" deleted
```

serviceaccount "wazi-sandbox-operator"を作成して、privileged 権限を付与してみます。

```
[root@Test05 ~/openshift/Wazi]# oc project
Using project "wazi-test01" on server "https://xxx.jp-tok.containers.cloud.ibm.com:xxx".

[root@Test05 ~/openshift/Wazi]# oc create serviceaccount wazi-sandbox-operator
serviceaccount/wazi-sandbox-operator created

[root@Test05 ~/openshift/Wazi]# oc get serviceaccounts
NAME                    SECRETS   AGE
builder                 2         7h50m
default                 2         7h50m
deployer                2         7h50m
wazi-sandbox-operator   2         9s

[root@Test05 ~/openshift/Wazi]# oc adm policy add-scc-to-user privileged -z wazi-sandbox-operator
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "wazi-sandbox-operator"
```

これで、再度Sandboxインスタンスをデプロイしてみます。

```
[root@Test05 ~/openshift/Wazi]# oc apply -f wazi-sandbox01.yaml
wazisandboxsystem.sandbox.wazi.ibm.com/wazi-sandbox01 created
```

少し進んだが、PodのInit Containerでエラー発生

```
[root@Test05 ~/openshift/Wazi]# oc get pod
NAME                                                  READY   STATUS                  RESTARTS   AGE
wazi-sandbox01-wazi-sandbox-system-6679544c49-z58nc   0/1     Init:CrashLoopBackOff   4          4m16s
```

Pod/Init ContainerのLog

```
Jan 14, 2021 10:00:10 AM com.ibm.zdt.init.App main
INFO: Remote Copy Task:
Jan 14, 2021 10:00:12 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Inventory.txt file does not exists and will be downloaded to directory /zdt/zvolumes/
Jan 14, 2021 10:00:12 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: No. of threads:4
Jan 14, 2021 10:00:12 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: COPY_BLOCK_SIZE:524288
Jan 14, 2021 10:00:14 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: inventory.txt with thread id:28
Jan 14, 2021 10:00:14 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 14, 2021 10:00:14 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
SEVERE: Failed with an exception:2: No such file
2: No such file
	at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2873)
	at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2225)
	at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2242)
	at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:928)
	at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:888)
	at com.ibm.zdt.zvolumes.GetZVolumesHandler.run(GetZVolumesHandler.java:91)
Jan 14, 2021 10:00:18 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
SEVERE: Terminating volume deploy threads: 

Jan 14, 2021 10:00:18 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: Transferring volumes complete.
Jan 14, 2021 10:00:18 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Reading inventory.txt file from directory /zdt/zvolumes/
Jan 14, 2021 10:00:18 AM com.ibm.zdt.init.App main
SEVERE: java.nio.file.NoSuchFileException: /zdt/zvolumes/inventory.txt
java.nio.file.NoSuchFileException: /zdt/zvolumes/inventory.txt
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:98)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:114)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:119)
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:226)
	at java.nio.file.Files.newByteChannel(Files.java:372)
	at java.nio.file.Files.newByteChannel(Files.java:418)
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:395)
	at java.nio.file.Files.newInputStream(Files.java:163)
	at java.nio.file.Files.newBufferedReader(Files.java:2795)
	at java.nio.file.Files.readAllLines(Files.java:3213)
	at java.nio.file.Files.readAllLines(Files.java:3253)
	at com.ibm.zdt.constants.Constants.readVolumesFromInventory(Constants.java:163)
	at com.ibm.zdt.init.App.remoteCopyTask(App.java:160)
	at com.ibm.zdt.init.App.main(App.java:76)
Jan 14, 2021 10:00:18 AM com.ibm.zdt.init.App main
INFO: Total time to complete all the file transfers (in seconds): 8
```

`SEVERE: Failed with an exception:2: No such file` => inventory.txtが無いって言われてる？

sftpPathが良くないのかも。configmapで`sftpPath: "/ftpuser1/adcd/zos24_2020may/volumes"`と変更して(最後のvolumesを追加指定)、リトライ。

```
[root@Test05 ~/openshift/Wazi]# oc apply -f wazi-sandbox01.yaml
wazisandboxsystem.sandbox.wazi.ibm.com/wazi-sandbox01 created
```

ちょっと違うエラーが出た
`Caused by: java.io.FileNotFoundException: /zdt/zvolumes/inventory.txt (Permission denied)`

```
Jan 14, 2021 10:17:07 AM com.ibm.zdt.init.App main
INFO: Remote Copy Task:
Jan 14, 2021 10:17:10 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Inventory.txt file does not exists and will be downloaded to directory /zdt/zvolumes/
Jan 14, 2021 10:17:10 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: No. of threads:4
Jan 14, 2021 10:17:10 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: COPY_BLOCK_SIZE:524288
Jan 14, 2021 10:17:12 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: inventory.txt with thread id:28
Jan 14, 2021 10:17:12 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 14, 2021 10:17:12 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
SEVERE: Failed with an exception:4: 
4: 
	at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1003)
	at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:888)
	at com.ibm.zdt.zvolumes.GetZVolumesHandler.run(GetZVolumesHandler.java:91)
Caused by: java.io.FileNotFoundException: /zdt/zvolumes/inventory.txt (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:286)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:226)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:112)
	at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:979)
	... 2 more
Jan 14, 2021 10:17:16 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
SEVERE: Terminating volume deploy threads: 

Jan 14, 2021 10:17:16 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: Transferring volumes complete.
Jan 14, 2021 10:17:16 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Reading inventory.txt file from directory /zdt/zvolumes/
Jan 14, 2021 10:17:16 AM com.ibm.zdt.init.App main
SEVERE: java.nio.file.NoSuchFileException: /zdt/zvolumes/inventory.txt
java.nio.file.NoSuchFileException: /zdt/zvolumes/inventory.txt
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:98)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:114)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:119)
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:226)
	at java.nio.file.Files.newByteChannel(Files.java:372)
	at java.nio.file.Files.newByteChannel(Files.java:418)
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:395)
	at java.nio.file.Files.newInputStream(Files.java:163)
	at java.nio.file.Files.newBufferedReader(Files.java:2795)
	at java.nio.file.Files.readAllLines(Files.java:3213)
	at java.nio.file.Files.readAllLines(Files.java:3253)
	at com.ibm.zdt.constants.Constants.readVolumesFromInventory(Constants.java:163)
	at com.ibm.zdt.init.App.remoteCopyTask(App.java:160)
	at com.ibm.zdt.init.App.main(App.java:76)
Jan 14, 2021 10:17:16 AM com.ibm.zdt.init.App main
INFO: Total time to complete all the file transfers (in seconds): 9
```

<details><summary><font color="blue">deploymentの詳細確認</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc describe deployment wazi-sandbox01-wazi-sandbox-system
Name:                   wazi-sandbox01-wazi-sandbox-system
Namespace:              wazi-test01
CreationTimestamp:      Thu, 14 Jan 2021 19:58:33 +0900
Labels:                 app.kubernetes.io/instance=wazi-sandbox01
                        app.kubernetes.io/managed-by=ansible
                        app.kubernetes.io/name=wazi-sandbox-system
                        app.kubernetes.io/part-of=ibm-wazi-developer-sandbox
                        app.kubernetes.io/version=v1.1.0
Annotations:            cloudpakId: 4df52d2cdc374ba09f631a650ad2b5bf
                        cloudpakName: IBM Cloud Pak for Applications
                        cloudpakVersion: 4.1.1
                        deployment.kubernetes.io/revision: 1
                        productChargedContainers: All
                        productCloudpakRatio: 1:1
                        productID: 9d41d2d8126f4200b62ba1acc0dffa2e
                        productMetric: VIRTUAL_PROCESSOR_CORE
                        productName: IBM Wazi Developer for Red Hat CodeReady Workspaces
                        productVersion: v1.1.0
Selector:               app.kubernetes.io/instance=wazi-sandbox01,app.kubernetes.io/name=wazi-sandbox-system
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=wazi-sandbox01
                    app.kubernetes.io/managed-by=ansible
                    app.kubernetes.io/name=wazi-sandbox-system
                    app.kubernetes.io/part-of=ibm-wazi-developer-sandbox
                    app.kubernetes.io/version=v1.1.0
  Annotations:      cloudpakId: 4df52d2cdc374ba09f631a650ad2b5bf
                    cloudpakName: IBM Cloud Pak for Applications
                    cloudpakVersion: 4.1.1
                    productChargedContainers: All
                    productCloudpakRatio: 1:1
                    productID: 9d41d2d8126f4200b62ba1acc0dffa2e
                    productMetric: VIRTUAL_PROCESSOR_CORE
                    productName: IBM Wazi Developer for Red Hat CodeReady Workspaces
                    productVersion: v1.1.0
  Service Account:  wazi-sandbox-operator
  Init Containers:
   wazi-sandbox-system-copy:
    Image:      cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1
    Port:       <none>
    Host Port:  <none>
    Environment:
      COPY_SOURCE_TYPE:          sftp
      COPY_DEST_TYPE:            pvc
      COPY_THREADS:              4
      COPY_BLOCK_SIZE:           524288
      FTP_HOST:                  <set to the key 'sftpHost' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_PORT:                  <set to the key 'sftpPort' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_ID:                    <set to the key 'sftpUser' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_DIR:                   <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_PASSWORD:              <set to the key 'sftpPassword' in secret 'wazi-sandbox-pw-secret'>  Optional: true
      INVENTORY_FILE_DIRECTORY:  <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>   Optional: true
    Mounts:
      /zdt/zvolumes from zpdt-volume (rw)
  Containers:
   wazi-sandbox-system:
    Image:      cp.icr.io/cp/wazi-sandbox@sha256:43aa9181e8ccb3205c1612e83dee0d431fcd447cce7ac1608ef7b36d087dbd4c
    Port:       <none>
    Host Port:  <none>
    Liveness:   exec [/zdt/bin/instcontroller_check.sh] delay=30s timeout=1s period=30s #success=1 #failure=20
    Readiness:  exec [/zdt/bin/instcontroller_check.sh] delay=30s timeout=1s period=10s #success=1 #failure=60
    Environment:
      LIC_SERVER:          <set to the key 'licenseServer' of config map 'wazi-sandbox-config01'>  Optional: false
      IPL_ZOS:             <set to the key 'iplZOS' of config map 'wazi-sandbox-config01'>         Optional: true
      FTP_HOST:            <set to the key 'sftpHost' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_PORT:            <set to the key 'sftpPort' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_ID:              <set to the key 'sftpUser' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_DIR:             <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_PASSWORD:        <set to the key 'sftpPassword' in secret 'wazi-sandbox-pw-secret'>      Optional: true
      ZDT_BASIC_AUTH_PWD:  <set to the key 'zdtAuthPassword' in secret 'wazi-sandbox-pw-secret'>   Optional: false
    Mounts:
      /sys/fs/cgroup from cgroup (ro)
      /zdt/zvolumes from zpdt-volume (rw)
  Volumes:
   zpdt-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wazi-sandbox-pvc01
    ReadOnly:   false
   cgroup:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/cgroup
    HostPathType:
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  wazi-sandbox01-wazi-sandbox-system-6679544c49 (1/1 replicas created)
NewReplicaSet:   <none>
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  73s   deployment-controller  Scaled up replica set wazi-sandbox01-wazi-sandbox-system-6679544c49 to 1
```

</div></details>

<details><summary><font color="blue">Podの詳細確認</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc describe pod wazi-sandbox01-wazi-sandbox-system-6679544c49-h6s95
Name:         wazi-sandbox01-wazi-sandbox-system-6679544c49-h6s95
Namespace:    wazi-test01
Priority:     0
Node:         10.129.176.11/10.129.176.11
Start Time:   Thu, 14 Jan 2021 20:40:22 +0900
Labels:       app.kubernetes.io/instance=wazi-sandbox01
              app.kubernetes.io/managed-by=ansible
              app.kubernetes.io/name=wazi-sandbox-system
              app.kubernetes.io/part-of=ibm-wazi-developer-sandbox
              app.kubernetes.io/version=v1.1.0
              pod-template-hash=6679544c49
Annotations:  cloudpakId: 4df52d2cdc374ba09f631a650ad2b5bf
              cloudpakName: IBM Cloud Pak for Applications
              cloudpakVersion: 4.1.1
              cni.projectcalico.org/podIP: 172.30.19.159/32
              cni.projectcalico.org/podIPs: 172.30.19.159/32
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "172.30.19.159"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "172.30.19.159"
                    ],
                    "default": true,
                    "dns": {}
                }]
              openshift.io/scc: privileged
              productChargedContainers: All
              productCloudpakRatio: 1:1
              productID: 9d41d2d8126f4200b62ba1acc0dffa2e
              productMetric: VIRTUAL_PROCESSOR_CORE
              productName: IBM Wazi Developer for Red Hat CodeReady Workspaces
              productVersion: v1.1.0
Status:       Pending
IP:           172.30.19.159
IPs:
  IP:           172.30.19.159
Controlled By:  ReplicaSet/wazi-sandbox01-wazi-sandbox-system-6679544c49
Init Containers:
  wazi-sandbox-system-copy:
    Container ID:   cri-o://d741260b7f22f75d5328eb3215a9ddb31a3ff85fc9ddc7fb1fe1cde6854d7a82
    Image:          cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1
    Image ID:       cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 14 Jan 2021 20:44:44 +0900
      Finished:     Thu, 14 Jan 2021 20:44:53 +0900
    Ready:          False
    Restart Count:  5
    Environment:
      COPY_SOURCE_TYPE:          sftp
      COPY_DEST_TYPE:            pvc
      COPY_THREADS:              4
      COPY_BLOCK_SIZE:           524288
      FTP_HOST:                  <set to the key 'sftpHost' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_PORT:                  <set to the key 'sftpPort' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_ID:                    <set to the key 'sftpUser' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_DIR:                   <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>   Optional: true
      FTP_PASSWORD:              <set to the key 'sftpPassword' in secret 'wazi-sandbox-pw-secret'>  Optional: true
      INVENTORY_FILE_DIRECTORY:  <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>   Optional: true
    Mounts:
      /zdt/zvolumes from zpdt-volume (rw)
Containers:
  wazi-sandbox-system:
    Container ID:
    Image:          cp.icr.io/cp/wazi-sandbox@sha256:43aa9181e8ccb3205c1612e83dee0d431fcd447cce7ac1608ef7b36d087dbd4c
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       exec [/zdt/bin/instcontroller_check.sh] delay=30s timeout=1s period=30s #success=1 #failure=20
    Readiness:      exec [/zdt/bin/instcontroller_check.sh] delay=30s timeout=1s period=10s #success=1 #failure=60
    Environment:
      LIC_SERVER:          <set to the key 'licenseServer' of config map 'wazi-sandbox-config01'>  Optional: false
      IPL_ZOS:             <set to the key 'iplZOS' of config map 'wazi-sandbox-config01'>         Optional: true
      FTP_HOST:            <set to the key 'sftpHost' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_PORT:            <set to the key 'sftpPort' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_ID:              <set to the key 'sftpUser' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_DIR:             <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>       Optional: true
      FTP_PASSWORD:        <set to the key 'sftpPassword' in secret 'wazi-sandbox-pw-secret'>      Optional: true
      ZDT_BASIC_AUTH_PWD:  <set to the key 'zdtAuthPassword' in secret 'wazi-sandbox-pw-secret'>   Optional: false
    Mounts:
      /sys/fs/cgroup from cgroup (ro)
      /zdt/zvolumes from zpdt-volume (rw)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  zpdt-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wazi-sandbox-pvc01
    ReadOnly:   false
  cgroup:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/cgroup
    HostPathType:
QoS Class:         BestEffort
Node-Selectors:    <none>
Tolerations:       node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       6m9s                   default-scheduler  Successfully assigned wazi-test01/wazi-sandbox01-wazi-sandbox-system-6679544c49-h6s95 to 10.129.176.11
  Normal   AddedInterface  5m53s                  multus             Add eth0 [172.30.19.159/32]
  Normal   Created         4m35s (x4 over 5m51s)  kubelet            Created container wazi-sandbox-system-copy
  Normal   Started         4m35s (x4 over 5m51s)  kubelet            Started container wazi-sandbox-system-copy
  Normal   Pulling         3m35s (x5 over 5m53s)  kubelet            Pulling image "cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1"
  Normal   Pulled          3m32s (x5 over 5m51s)  kubelet            Successfully pulled image "cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1"
  Warning  BackOff         46s (x19 over 5m29s)   kubelet            Back-off restarting failed container
```

</div></details>


`Caused by: java.io.FileNotFoundException: /zdt/zvolumes/inventory.txt (Permission denied)` => sftpでダウンロードしたinventory.txtを、Block Storageの永続ボリュームをマウントしたディレクトリ(/zdt/zvolumes/)に書き込もうとしたけど権限が無くて書き込みできないということか？

一旦、PVCも削除/再作成してからリトライしたけどやっぱりNG。


https://www.ibm.com/support/knowledgecenter/en/SSCH39_1.0.0/sandbox/configure_cluster.html
↑にはこういう記述があるんだがなぁ...

>When a sandbox instance is started, the storage is initialized with correct user and group file permissions by an init container. Therefore, no extra configuration of file system permissions is required.

## その2
参考: [IBM Support - Unable to create Sandbox on IBM Cloud](https://www.ibm.com/support/pages/node/6375272)
上のような問題が報告されていました。ドンピシャっぽい。
Volume Copyを行うOperatorがあるので、Copyの所は別で実施する回避策が提示されています。この手順に従ってやってみます。

以下のyamlを作成します。

```yaml:sandbox-fix-init-copy.yaml
apiVersion: sandbox.wazi.ibm.com/v1alpha1
kind: WaziSandboxVolumeCopy
metadata:
  name: sandbox-fix-init-copy
spec:
  source:
    type: sftp
    existingConfigName: wazi-sandbox-config01
    secretName: wazi-sandbox-pw-secret
  destination:
    existingClaimName: wazi-sandbox-pvc01
  imagePullSecrets:
    - name: wazi-sandbox-pull-secret
```

適用！

```
[root@Test05 ~/openshift/Wazi]# oc apply -f sandbox-fix-init-copy.yaml
wazisandboxvolumecopy.sandbox.wazi.ibm.com/sandbox-fix-init-copy created
```

エラー

```
[root@Test05 ~/openshift/Wazi]# oc get event
LAST SEEN   TYPE      REASON         OBJECT                                                   MESSAGE
11s         Warning   FailedCreate   job/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job   Error creating: pods "sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-" is forbidden: error looking up service account wazi-test01/wazi-sandbox-volume-copy: serviceaccount "wazi-sandbox-volume-copy" not found
```

前と同じようなエラーですね。こちらは"wazi-sandbox-volume-copy"というserviceaccountが使われるので作成してprivileged権限設定します。

```
[root@Test05 ~/openshift/Wazi]# oc create serviceaccount wazi-sandbox-volume-copy
serviceaccount/wazi-sandbox-volume-copy created

[root@Test05 ~/openshift/Wazi]# oc adm policy add-scc-to-user privileged -z wazi-sandbox-volume-copy
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "wazi-sandbox-volume-copy"
```

リトライ

少し待っていたら、Podが正常に稼働してどうやらVolのコピーが始まったらしい！

```
[root@Test05 ~/openshift/Wazi]# oc get event
LAST SEEN   TYPE      REASON             OBJECT                                                         MESSAGE
81s         Normal    Scheduled          pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   Successfully assigned wazi-test01/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9 to 10.129.176.11
64s         Normal    AddedInterface     pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   Add eth0 [172.30.19.172/32]
64s         Normal    Pulling            pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   Pulling image "cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1"
62s         Normal    Pulled             pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   Successfully pulled image "cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1"
62s         Normal    Created            pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   Created container wazi-sandbox-volume-copy
62s         Normal    Started            pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   Started container wazi-sandbox-volume-copy
15m         Warning   FailedCreate       job/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job         Error creating: pods "sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-" is forbidden: error looking up service account wazi-test01/wazi-sandbox-volume-copy: serviceaccount "wazi-sandbox-volume-copy" not found
81s         Normal    SuccessfulCreate   job/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job         Created pod: sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9
[root@Test05 ~/openshift/Wazi]# oc get pod
NAME                                                       READY   STATUS    RESTARTS   AGE
sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   1/1     Running   0          90s
```

こんな感じでCPU, Memory, ネットワークの使用率が上がってきた。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/132331/aa478ed9-b405-f6c4-f504-0a5e5f54f461.png)

さて、Volume Copy用のPodはJobとして稼働しているようです。

```
[root@Test05 ~/openshift/Wazi]# oc get all
NAME                                                           READY   STATUS    RESTARTS   AGE
pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   1/1     Running   0          5m29s

NAME                                                           COMPLETIONS   DURATION   AGE
job.batch/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job   0/1           5m29s      5m29s
```

<details><summary><font color="blue">jobの詳細確認</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc describe job.batch/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job
Name:           sandbox-fix-init-copy-wazi-sandbox-volume-copy-job
Namespace:      wazi-test01
Selector:       controller-uid=f8fceae1-9648-4d5d-b000-a470f45ad92d
Labels:         app.kubernetes.io/instance=sandbox-fix-init-copy
                app.kubernetes.io/managed-by=ansible
                app.kubernetes.io/name=wazi-sandbox-volume-copy
                app.kubernetes.io/part-of=ibm-wazi-developer-sandbox
                app.kubernetes.io/version=v1.1.0
Annotations:    cloudpakId: 4df52d2cdc374ba09f631a650ad2b5bf
                cloudpakName: IBM Cloud Pak for Applications
                cloudpakVersion: 4.1.1
                productChargedContainers: All
                productCloudpakRatio: 1:1
                productID: 9d41d2d8126f4200b62ba1acc0dffa2e
                productMetric: VIRTUAL_PROCESSOR_CORE
                productName: IBM Wazi Developer for Red Hat CodeReady Workspaces
                productVersion: v1.1.0
Parallelism:    1
Completions:    1
Start Time:     Fri, 15 Jan 2021 15:53:39 +0900
Pods Statuses:  1 Running / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/instance=sandbox-fix-init-copy
                    app.kubernetes.io/managed-by=ansible
                    app.kubernetes.io/name=wazi-sandbox-volume-copy
                    app.kubernetes.io/part-of=ibm-wazi-developer-sandbox
                    app.kubernetes.io/version=v1.1.0
                    controller-uid=f8fceae1-9648-4d5d-b000-a470f45ad92d
                    job-name=sandbox-fix-init-copy-wazi-sandbox-volume-copy-job
  Annotations:      cloudpakId: 4df52d2cdc374ba09f631a650ad2b5bf
                    cloudpakName: IBM Cloud Pak for Applications
                    cloudpakVersion: 4.1.1
                    productChargedContainers: All
                    productCloudpakRatio: 1:1
                    productID: 9d41d2d8126f4200b62ba1acc0dffa2e
                    productMetric: VIRTUAL_PROCESSOR_CORE
                    productName: IBM Wazi Developer for Red Hat CodeReady Workspaces
                    productVersion: v1.1.0
  Service Account:  wazi-sandbox-volume-copy
  Containers:
   wazi-sandbox-volume-copy:
    Image:      cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1
    Port:       <none>
    Host Port:  <none>
    Environment:
      COPY_SOURCE_TYPE:          sftp
      COPY_DEST_TYPE:            pvc
      COPY_THREADS:              4
      COPY_BLOCK_SIZE:           524288
      FTP_HOST:                  <set to the key 'sftpHost' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_PORT:                  <set to the key 'sftpPort' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_ID:                    <set to the key 'sftpUser' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_DIR:                   <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_PASSWORD:              <set to the key 'sftpPassword' in secret 'wazi-sandbox-pw-secret'>                                     Optional: true
      INVENTORY_FILE_DIRECTORY:  <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>                                      Optional: true
      DEST_PVC_PATH:             <set to the key 'pvcPath' of config map 'sandbox-fix-init-copy-wazi-sandbox-volume-copy-dest-config'>  Optional: true
    Mounts:
      /zdt/zvolumes from dest-volume (rw)
  Volumes:
   dest-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wazi-sandbox-pvc01
    ReadOnly:   false
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  5m44s  job-controller  Created pod: sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9
```

</div></details>

<details><summary><font color="blue">podの詳細確認</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc describe pod/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9
Name:         sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9
Namespace:    wazi-test01
Priority:     0
Node:         10.129.176.11/10.129.176.11
Start Time:   Fri, 15 Jan 2021 15:53:39 +0900
Labels:       app.kubernetes.io/instance=sandbox-fix-init-copy
              app.kubernetes.io/managed-by=ansible
              app.kubernetes.io/name=wazi-sandbox-volume-copy
              app.kubernetes.io/part-of=ibm-wazi-developer-sandbox
              app.kubernetes.io/version=v1.1.0
              controller-uid=f8fceae1-9648-4d5d-b000-a470f45ad92d
              job-name=sandbox-fix-init-copy-wazi-sandbox-volume-copy-job
Annotations:  cloudpakId: 4df52d2cdc374ba09f631a650ad2b5bf
              cloudpakName: IBM Cloud Pak for Applications
              cloudpakVersion: 4.1.1
              cni.projectcalico.org/podIP: 172.30.19.172/32
              cni.projectcalico.org/podIPs: 172.30.19.172/32
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "172.30.19.172"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "172.30.19.172"
                    ],
                    "default": true,
                    "dns": {}
                }]
              openshift.io/scc: privileged
              productChargedContainers: All
              productCloudpakRatio: 1:1
              productID: 9d41d2d8126f4200b62ba1acc0dffa2e
              productMetric: VIRTUAL_PROCESSOR_CORE
              productName: IBM Wazi Developer for Red Hat CodeReady Workspaces
              productVersion: v1.1.0
Status:       Running
IP:           172.30.19.172
IPs:
  IP:           172.30.19.172
Controlled By:  Job/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job
Containers:
  wazi-sandbox-volume-copy:
    Container ID:   cri-o://600a56b47595a3234a0436852a94f0a45407ae9bf421eaa0519af7b74e0f56cd
    Image:          cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1
    Image ID:       cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 15 Jan 2021 15:53:58 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      COPY_SOURCE_TYPE:          sftp
      COPY_DEST_TYPE:            pvc
      COPY_THREADS:              4
      COPY_BLOCK_SIZE:           524288
      FTP_HOST:                  <set to the key 'sftpHost' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_PORT:                  <set to the key 'sftpPort' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_ID:                    <set to the key 'sftpUser' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_DIR:                   <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>                                      Optional: true
      FTP_PASSWORD:              <set to the key 'sftpPassword' in secret 'wazi-sandbox-pw-secret'>                                     Optional: true
      INVENTORY_FILE_DIRECTORY:  <set to the key 'sftpPath' of config map 'wazi-sandbox-config01'>                                      Optional: true
      DEST_PVC_PATH:             <set to the key 'pvcPath' of config map 'sandbox-fix-init-copy-wazi-sandbox-volume-copy-dest-config'>  Optional: true
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from wazi-sandbox-volume-copy-token-xdvk4 (ro)
      /zdt/zvolumes from dest-volume (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  dest-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wazi-sandbox-pvc01
    ReadOnly:   false
  wazi-sandbox-volume-copy-token-xdvk4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  wazi-sandbox-volume-copy-token-xdvk4
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  Scheduled       9m14s  default-scheduler  Successfully assigned wazi-test01/sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9 to 10.129.176.11
  Normal  AddedInterface  8m57s  multus             Add eth0 [172.30.19.172/32]
  Normal  Pulling         8m57s  kubelet            Pulling image "cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1"
  Normal  Pulled          8m55s  kubelet            Successfully pulled image "cp.icr.io/cp/wazi-sandbox-volume-copy@sha256:806ab8c94b9a903d94684ff60761373960c140971d8ce43d40562d211646f5f1"
  Normal  Created         8m55s  kubelet            Created container wazi-sandbox-volume-copy
  Normal  Started         8m55s  kubelet            Started container wazi-sandbox-volume-copy
```

</div></details>

<details><summary><font color="blue">Copy中にPodの中を覗いてみると...</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc exec sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9 -it -- bash
bash-4.4$ whoami
ibmsys1
bash-4.4$ id
uid=2105(ibmsys1) gid=2105(ibmsys) groups=2105(ibmsys)
bash-4.4$ pwd
/zdt/volume-copy
bash-4.4$ df -m
Filesystem                                    1M-blocks  Used Available Use% Mounted on
overlay                                          100663 20859     74669  22% /
tmpfs                                                64     0        64   0% /dev
tmpfs                                             16004     0     16004   0% /sys/fs/cgroup
shm                                                  64     0        64   0% /dev/shm
/dev/mapper/docker_data                          100663 20859     74669  22% /etc/hosts
/dev/mapper/3600a09803830565a682b515065437048    302251 45107    257129  15% /zdt/zvolumes
tmpfs                                             16004     1     16004   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                             16004     0     16004   0% /proc/acpi
tmpfs                                             16004     0     16004   0% /proc/scsi
tmpfs                                             16004     0     16004   0% /sys/firmware
bash-4.4$ cd /zdt/z^C
bash-4.4$ ls -laR /zdt/zvolumes
/zdt/zvolumes:
total 46687580
drwxrwsr-x. 3 root    ibmsys       4096 Jan 15 07:10 .
drwxr-xr-x. 1 ibmsys1 ibmsys       4096 Jan 15 06:53 ..
-rw-r--r--. 1 ibmsys1 ibmsys 8539292672 Jan 15 07:08 B4BLZ1
-rw-r--r--. 1 ibmsys1 ibmsys 8539292672 Jan 15 07:08 B4C541
-rw-r--r--. 1 ibmsys1 ibmsys 8539292672 Jan 15 07:08 B4C551
-rw-r--r--. 1 ibmsys1 ibmsys 8539292672 Jan 15 06:56 B4CFG1
-rw-r--r--. 1 ibmsys1 ibmsys 8539292672 Jan 15 07:10 B4DBAR
-rw-r--r--. 1 ibmsys1 ibmsys 1654939648 Jan 15 07:10 B4DBB1
-rw-r--r--. 1 ibmsys1 ibmsys 1668517888 Jan 15 07:10 B4DBB2
-rw-r--r--. 1 ibmsys1 ibmsys 1585192960 Jan 15 07:10 B4DBC1
-rw-r--r--. 1 ibmsys1 ibmsys  202887168 Jan 15 07:10 B4DBC2
-rw-r--r--. 1 ibmsys1 ibmsys        353 Jan 15 06:54 inventory.txt
drwxrws---. 2 root    ibmsys      16384 Jan 14 11:22 lost+found

/zdt/zvolumes/lost+found:
total 20
drwxrws---. 2 root ibmsys 16384 Jan 14 11:22 .
drwxrwsr-x. 3 root ibmsys  4096 Jan 15 07:10 ..

bash-4.4$ ps -ef
UID         PID   PPID  C STIME TTY          TIME CMD
ibmsys1       1      0  0 06:53 ?        00:00:00 /bin/sh /zdt/volume-copy/wazi-volume-copy.sh
ibmsys1       8      1 37 06:53 ?        00:07:11 /zdt/java/jre/bin/java -jar wazi-volume-copy.jar
ibmsys1      51      0  0 07:10 pts/0    00:00:00 bash
ibmsys1      61     51  0 07:13 pts/0    00:00:00 ps -ef

bash-4.4$ cat /zdt/volume-copy/wazi-volume-copy.sh
#!/bin/sh
##  COPYRIGHT --                                                          ##
## Licensed Material - Property of IBM                                    ##
## (C) Copyright IBM Corp. 2020 - All Rights Reserved.                    ##
## US Government Users Restricted Rights - Use, duplication or disclosure ##
## restricted by GSA ADP Schedule Contract with IBM Corp.                 ##
##                                                                        ##
##========================================================================##

chmod -R g+w /zdt/zvolumes > /dev/null 2>&1
/zdt/java/jre/bin/java -jar wazi-volume-copy.jar
status=$?
chmod -R g+w /zdt/zvolumes > /dev/null 2>&1
exit $status
```

</div></details>

<details><summary><font color="blue">Podのログ</font></summary><div>

```
[root@Test05 ~/openshift/Wazi]# oc get pod
NAME                                                       READY   STATUS    RESTARTS   AGE
sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9   1/1     Running   0          70m
[root@Test05 ~/openshift/Wazi]# oc logs sandbox-fix-init-copy-wazi-sandbox-volume-copy-job-fkpm9
Jan 15, 2021 6:53:58 AM com.ibm.zdt.init.App main
INFO: Remote Copy Task:
Jan 15, 2021 6:54:01 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Inventory.txt file does not exists and will be downloaded to directory /zdt/zvolumes/
Jan 15, 2021 6:54:01 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: No. of threads:4
Jan 15, 2021 6:54:01 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: COPY_BLOCK_SIZE:524288
Jan 15, 2021 6:54:03 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: inventory.txt with thread id:28
Jan 15, 2021 6:54:03 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 6:54:03 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: inventory.txt with thread id:28 Total time to complete in seconds: 0
Jan 15, 2021 6:54:07 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: Transferring volumes complete.
Jan 15, 2021 6:54:07 AM com.ibm.zdt.init.App remoteCopyTask
INFO: Reading inventory.txt file from directory /zdt/zvolumes/
Jan 15, 2021 6:54:07 AM com.ibm.zdt.constants.Constants checkVolumes
INFO: List of files to download.
B4BLZ1.gz
B4C541.gz
B4C551.gz
B4CFG1.gz
B4DBAR.gz
B4DBB1.gz
B4DBB2.gz
B4DBC1.gz
B4DBC2.gz
B4DIS1.gz
B4DIS2.gz
B4DIS3.gz
B4IME1.gz
B4IMF1.gz
B4INM1.gz
B4KAN1.gz
B4PAGA.gz
B4PAGB.gz
B4PAGC.gz
B4PRD1.gz
B4PRD2.gz
B4PRD3.gz
B4PRD4.gz
B4RES1.ZPD
B4RES2.gz
B4SYS1.gz
B4USR1.gz
B4USS1.gz
B4USS2.gz
B4W901.gz
B4W902.gz
B4ZCX1.gz
B4ZWE1.gz
devmap.txt
SARES1.ZPD

Jan 15, 2021 6:54:07 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: No. of threads:4
Jan 15, 2021 6:54:07 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: COPY_BLOCK_SIZE:524288
Jan 15, 2021 6:54:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4BLZ1.gz with thread id:36
Jan 15, 2021 6:54:10 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4C541.gz with thread id:38
Jan 15, 2021 6:54:12 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4C551.gz with thread id:40
Jan 15, 2021 6:54:13 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4CFG1.gz with thread id:42
Jan 15, 2021 6:56:59 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4CFG1.gz with thread id:42 Total time to complete in seconds: 166
Jan 15, 2021 6:56:59 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBAR.gz with thread id:42
Jan 15, 2021 7:08:10 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4C541.gz with thread id:38 Total time to complete in seconds: 840
Jan 15, 2021 7:08:10 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBB1.gz with thread id:38
Jan 15, 2021 7:08:13 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4BLZ1.gz with thread id:36 Total time to complete in seconds: 844
Jan 15, 2021 7:08:13 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBB2.gz with thread id:36
Jan 15, 2021 7:08:17 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4C551.gz with thread id:40 Total time to complete in seconds: 845
Jan 15, 2021 7:08:17 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBC1.gz with thread id:40
Jan 15, 2021 7:10:38 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBAR.gz with thread id:42 Total time to complete in seconds: 819
Jan 15, 2021 7:10:38 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DBC2.gz with thread id:42
Jan 15, 2021 7:21:48 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBB2.gz with thread id:36 Total time to complete in seconds: 815
Jan 15, 2021 7:21:48 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DIS1.gz with thread id:36
Jan 15, 2021 7:22:46 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBB1.gz with thread id:38 Total time to complete in seconds: 876
Jan 15, 2021 7:22:46 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DIS2.gz with thread id:38
Jan 15, 2021 7:22:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBC1.gz with thread id:40 Total time to complete in seconds: 881
Jan 15, 2021 7:22:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4DIS3.gz with thread id:40
Jan 15, 2021 7:23:59 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DBC2.gz with thread id:42 Total time to complete in seconds: 801
Jan 15, 2021 7:23:59 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4IME1.gz with thread id:42
Jan 15, 2021 7:36:16 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DIS2.gz with thread id:38 Total time to complete in seconds: 810
Jan 15, 2021 7:36:16 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4IMF1.gz with thread id:38
Jan 15, 2021 7:36:19 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DIS3.gz with thread id:40 Total time to complete in seconds: 801
Jan 15, 2021 7:36:19 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4INM1.gz with thread id:40
Jan 15, 2021 7:37:37 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4IME1.gz with thread id:42 Total time to complete in seconds: 818
Jan 15, 2021 7:37:37 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4KAN1.gz with thread id:42
Jan 15, 2021 7:38:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4DIS1.gz with thread id:36 Total time to complete in seconds: 981
Jan 15, 2021 7:38:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PAGA.gz with thread id:36
Jan 15, 2021 7:49:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4IMF1.gz with thread id:38 Total time to complete in seconds: 820
Jan 15, 2021 7:49:56 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PAGB.gz with thread id:38
Jan 15, 2021 7:50:31 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4KAN1.gz with thread id:42 Total time to complete in seconds: 774
Jan 15, 2021 7:50:31 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PAGC.gz with thread id:42
Jan 15, 2021 7:50:43 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PAGA.gz with thread id:36 Total time to complete in seconds: 754
Jan 15, 2021 7:50:43 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD1.gz with thread id:36
Jan 15, 2021 7:54:07 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4INM1.gz with thread id:40 Total time to complete in seconds: 1068
Jan 15, 2021 7:54:07 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD2.gz with thread id:40
Jan 15, 2021 8:01:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PAGB.gz with thread id:38 Total time to complete in seconds: 722
Jan 15, 2021 8:01:58 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD3.gz with thread id:38
Jan 15, 2021 8:02:22 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PAGC.gz with thread id:42 Total time to complete in seconds: 711
Jan 15, 2021 8:02:22 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4PRD4.gz with thread id:42
Jan 15, 2021 8:07:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD1.gz with thread id:36 Total time to complete in seconds: 986
Jan 15, 2021 8:07:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4RES1.ZPD with thread id:36
Jan 15, 2021 8:07:09 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 8:09:52 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4RES1.ZPD with thread id:36 Total time to complete in seconds: 163
Jan 15, 2021 8:09:52 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4RES2.gz with thread id:36
Jan 15, 2021 8:11:44 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD2.gz with thread id:40 Total time to complete in seconds: 1057
Jan 15, 2021 8:11:44 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4SYS1.gz with thread id:40
Jan 15, 2021 8:13:29 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD4.gz with thread id:42 Total time to complete in seconds: 667
Jan 15, 2021 8:13:29 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4USR1.gz with thread id:42
Jan 15, 2021 8:18:47 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4PRD3.gz with thread id:38 Total time to complete in seconds: 1009
Jan 15, 2021 8:18:47 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4USS1.gz with thread id:38
Jan 15, 2021 8:22:46 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4RES2.gz with thread id:36 Total time to complete in seconds: 774
Jan 15, 2021 8:22:46 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4USS2.gz with thread id:36
Jan 15, 2021 8:23:59 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4SYS1.gz with thread id:40 Total time to complete in seconds: 735
Jan 15, 2021 8:23:59 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4W901.gz with thread id:40
Jan 15, 2021 8:27:51 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4USR1.gz with thread id:42 Total time to complete in seconds: 862
Jan 15, 2021 8:27:51 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4W902.gz with thread id:42
Jan 15, 2021 8:32:28 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4USS1.gz with thread id:38 Total time to complete in seconds: 821
Jan 15, 2021 8:32:28 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4ZCX1.gz with thread id:38
Jan 15, 2021 8:36:39 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4USS2.gz with thread id:36 Total time to complete in seconds: 833
Jan 15, 2021 8:36:39 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: B4ZWE1.gz with thread id:36
Jan 15, 2021 8:39:47 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: B4W902.gz with thread id:42 Total time to complete in seconds: 716
Jan 15, 2021 8:39:47 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: devmap.txt with thread id:42
Jan 15, 2021 8:39:47 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Finished transferring file: devmap.txt with thread id:42 Total time to complete in seconds: 1
Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: Transferring file: SARES1.ZPD with thread id:42
Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
INFO: The filename does not end with .gz or .GZ extension, therefore it will only be downloaded and not extracted.
Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumesHandler run
SEVERE: Failed with an exception:2: No such file
2: No such file
        at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2873)
        at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2225)
        at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2242)
        at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:928)
        at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:888)
        at com.ibm.zdt.zvolumes.GetZVolumesHandler.run(GetZVolumesHandler.java:91)
Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
SEVERE: Terminating volume deploy threads:

Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
SEVERE: Terminating Sub-thread to track the volume transfer status for B4ZWE1
Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
SEVERE: Terminating Sub-thread to track the volume transfer status for B4ZCX1
Jan 15, 2021 8:39:48 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
SEVERE: Terminating Sub-thread to track the volume transfer status for B4W901
Jan 15, 2021 8:39:49 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
SEVERE: Terminating volume deploy threads:

Jan 15, 2021 8:39:49 AM com.ibm.zdt.zvolumes.GetZVolumes getVolumes
INFO: Transferring volumes complete.
Jan 15, 2021 8:39:49 AM com.ibm.zdt.init.App main
INFO: Total time to complete all the file transfers (in seconds): 6351
```

</div></details>

エラー: SARES1.ZPDが無いって言われてる。しまった、sftpサーバーに転送する時のファイルが漏れてた。
sftpサーバーにSAREA1.ZPDファイルを配置しなおしてリトライ。
※ちなみにファイル転送は4スレッド並行で処理されているらしく、途中のファイル転送で失敗するとその時並行して実行されていたファイル転送もキャンセルされるようです。すなわち中途半端に転送された状態のファイルが残ってしまいます。その後、足りなかったファイルをsftpサーバーに追加してそのままリトライしてしまうと、途中のファイルは既に転送済みと判断されて再転送されないので、中途半端な状態のファイルが残ってしまうことになります。ファイルサイズがでかいので転送に時間がかかってやっかいなのですが、一旦PVC/PVを削除して一からやり直しました。