Kubernetes1.11 iscsiボリュームの利用検証

# はじめに

Kubernetesには、永続データを扱うために、PVやPVCといったストレージ概念が存在します。
PV, PVC を利用する際には、Backendで稼働するストレージを Plugin という形で選択します。
今回は、iscsiボリュームを利用する方法を技術調査します。

- iscsi には Dynamic Provisioner が、基本的には存在しないため、Dynamic Provisioner を使用しない連携方法を確認します



# targetdを構築

Kubernetesと連携するために、iscsi のターゲットとして、targerd を構築します。
前提として、CentOSを利用します。

```
yum -y install targetcli
```



backstoreの定義

```
targetcli /backstores/block create name=block_backend_vdb dev=/dev/vdb
targetcli /backstores/block create name=block_backend_vdc dev=/dev/vdc
```



確認

```
[root@k8s-targetd01 dev]# targetcli ls /backstores/block
o- block ...................................................................................................... [Storage Objects: 2]
  o- block_backend_vdb ................................................................. [/dev/vdb (10.0GiB) write-thru deactivated]
  | o- alua ....................................................................................................... [ALUA Groups: 1]
  |   o- default_tg_pt_gp ........................................................................... [ALUA state: Active/optimized]
  o- block_backend_vdc ................................................................. [/dev/vdc (15.0GiB) write-thru deactivated]
    o- alua ....................................................................................................... [ALUA Groups: 1]
      o- default_tg_pt_gp ........................................................................... [ALUA state: Active/optimized]
```



target の iqn を定義

```
targetcli /iscsi create iqn.2018-08.localdomain.hoge:k8s-targetd01
```





確認

```
[root@k8s-targetd01 dev]# targetcli ls /
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 2]
  | | o- block_backend_vdb ............................................................. [/dev/vdb (10.0GiB) write-thru deactivated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- block_backend_vdc ............................................................. [/dev/vdc (15.0GiB) write-thru deactivated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2018-08.localdomain.hoge:k8s-targetd01 ........................................................................ [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 0]
  |     o- luns .......................................................................................................... [LUNs: 0]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
```



backstoreとiSCSI targetの紐付け設定

```
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/luns create /backstores/block/block_backend_vdb
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/luns create /backstores/block/block_backend_vdc
```



確認

lun 0, 1 が作成されています

```
[root@k8s-targetd01 dev]# targetcli ls /
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 2]
  | | o- block_backend_vdb ............................................................... [/dev/vdb (10.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- block_backend_vdc ............................................................... [/dev/vdc (15.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2018-08.localdomain.hoge:k8s-targetd01 ........................................................................ [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 0]
  |     o- luns .......................................................................................................... [LUNs: 2]
  |     | o- lun0 .......................................................... [block/block_backend_vdb (/dev/vdb) (default_tg_pt_gp)]
  |     | o- lun1 .......................................................... [block/block_backend_vdc (/dev/vdc) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
```



aclの設定

Kubernetesの全Nodeを設定します

- iqn.2018-08.localdomain.hoge:k8s-master01
- iqn.2018-08.localdomain.hoge:k8s-nodevm01
- iqn.2018-08.localdomain.hoge:k8s-nodegpu01

```
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/acls create iqn.2018-08.localdomain.hoge:k8s-master01
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/acls create iqn.2018-08.localdomain.hoge:k8s-nodevm01
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/acls create iqn.2018-08.localdomain.hoge:k8s-nodegpu01
```



確認します

```
[root@k8s-targetd01 dev]# targetcli ls /
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 2]
  | | o- block_backend_vdb ............................................................... [/dev/vdb (10.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- block_backend_vdc ............................................................... [/dev/vdc (15.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2018-08.localdomain.hoge:k8s-targetd01 ........................................................................ [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 3]
  |     | o- iqn.2018-08.localdomain.hoge:k8s-master01 ........................................................ [Mapped LUNs: 2]
  |     | | o- mapped_lun0 ..................................................................... [lun0 block/block_backend_vdb (rw)]
  |     | | o- mapped_lun1 ..................................................................... [lun1 block/block_backend_vdc (rw)]
  |     | o- iqn.2018-08.localdomain.hoge:k8s-nodegpu01 ....................................................... [Mapped LUNs: 2]
  |     | | o- mapped_lun0 ..................................................................... [lun0 block/block_backend_vdb (rw)]
  |     | | o- mapped_lun1 ..................................................................... [lun1 block/block_backend_vdc (rw)]
  |     | o- iqn.2018-08.localdomain.hoge:k8s-nodevm01 ........................................................ [Mapped LUNs: 2]
  |     |   o- mapped_lun0 ..................................................................... [lun0 block/block_backend_vdb (rw)]
  |     |   o- mapped_lun1 ..................................................................... [lun1 block/block_backend_vdc (rw)]
  |     o- luns .......................................................................................................... [LUNs: 2]
  |     | o- lun0 .......................................................... [block/block_backend_vdb (/dev/vdb) (default_tg_pt_gp)]
  |     | o- lun1 .......................................................... [block/block_backend_vdc (/dev/vdc) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
```



設定を保存します

```
targetcli saveconfig
```



serviceの起動

```
systemctl enable target
systemctl start target
systemctl status target
```



# Kubernetes クラスタの iqn 設定

initiatorとして必要なものがinstallされているか確認

```
root@k8s-nodevm01:~# apt list | grep -i open-iscsi
open-iscsi/xenial-updates,now 2.0.873+git0.3b4b4500-14ubuntu3.4 amd64 [installed]
```



iqnの設定 (Master)

```
sed -i -e "s/InitiatorName=.*/InitiatorName=iqn.2018-08.localdomain.hoge:k8s-master01/g" /etc/iscsi/initiatorname.iscsi
```



iqnの設定 (k8s-nodevm01)

```
sed -i -e "s/InitiatorName=.*/InitiatorName=iqn.2018-08.localdomain.hoge:k8s-nodevm01/g" /etc/iscsi/initiatorname.iscsi
```



iqnの設定 (k8s-nodegpu01)

```
sed -i -e "s/InitiatorName=.*/InitiatorName=iqn.2018-08.localdomain.hoge:k8s-nodegpu01/g" /etc/iscsi/initiatorname.iscsi
```



serviceの再起動

```
systemctl restart iscsid open-iscsi
```



target を discoveryして、正常にtargetdが構成されているか確認します

```
iscsiadm -m discovery -t sendtargets -p 10.44.194.72
```



実行例

```
root@k8s-master01(default kubernetes-admin):~# iscsiadm -m discovery -t sendtargets -p 10.44.194.72
10.44.194.72:3260,1 iqn.2018-08.localdomain.hoge:k8s-targetd01
```



Snapshotを取得し、Loginしてみます。 (Kubernetesクラスタの各Nodeで手動実行する必要はありません。Kubelet等が自動的に行ってくれます)

```
root@k8s-master01(default kubernetes-admin):~# iscsiadm -m node --login 
Logging in to [iface: default, target: iqn.2018-08.localdomain.hoge:k8s-targetd01, portal: 10.44.194.72,3260] (multiple)
Login to [iface: default, target: iqn.2018-08.localdomain.hoge:k8s-targetd01, portal: 10.44.194.72,3260] successful.
```



セッションの確認

```
root@k8s-master01(default kubernetes-admin):~# iscsiadm -m session -o show 
tcp: [1] 10.44.194.72:3260,1 iqn.2018-08.localdomain.hoge:k8s-targetd01 (non-flash)
```



block storage を確認

```
root@k8s-master01(default kubernetes-admin):~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   10G  0 disk <---- here
sdb      8:16   0   15G  0 disk <---- here
vda    253:0    0  100G  0 disk 
└─vda1 253:1    0  100G  0 part /
```



#Kubernetes上でPersistentVolumeとして扱う

## Volumeの定義

PVを定義します。

**spec.capacity.storage の容量指定は、はあくまで自己申告制。実体と異なっても定義できます。**

**実際に使用されるのは、`spec.iscsi.lun`で指定しているLUNが使用されます。**

```
cat <<'EOF' > /root/manifests/persistent_volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: iscsi-15gb
spec:
  capacity:
    storage: 15Gi
  volumeMode: BlockVolume
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: slow
  iscsi:
    targetPortal: 10.44.194.72:3260
    portals: ['0.0.0.0:3260']
    iqn: iqn.2018-08.localdomain.hoge:k8s-targetd01
    lun: 0
    fsType: xfs
    readOnly: false
EOF
kubectl apply -f /root/manifests/persistent_volume.yaml
```



## PVCの定義

requests で指定する容量は、自己申告制で定義したPVの数値を基に、要求を満たしているPVがboundされます。

```
cat <<'EOF' > /root/manifests/persistent_volume_claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: iscsi-15gb-claim
  annotations:
    volume.kubernetes.io/storage-class: "iscsi"
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: BlockVolume
  storageClassName: slow
  resources:
    requests:
      storage: 11Gi
EOF
kubectl create -f /root/manifests/persistent_volume_claims.yaml
```



PVCにPVが紐づけられたことが分かります

```
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                      STORAGECLASS   REASON    AGE
iscsi-15gb   15Gi       RWO            Recycle          Bound     default/iscsi-15gb-claim   slow                     4m
root@k8s-master01(default kubernetes-admin):~# 
root@k8s-master01(default kubernetes-admin):~# kubectl get persistentvolumeclaims -o wide
NAME               STATUS    VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
iscsi-15gb-claim   Bound     iscsi-15gb   15Gi       RWO            slow           40s

```



詳細を確認します

[Volume:        iscsi-15gb]を表示されており、紐づけが確認できます

```
root@k8s-master01(default kubernetes-admin):~# kubectl describe persistentvolumeclaims iscsi-15gb-claim 
Name:          iscsi-15gb-claim
Namespace:     default
StorageClass:  slow
Status:        Bound
Volume:        iscsi-15gb
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.kubernetes.io/storage-class=iscsi
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      15Gi
Access Modes:  RWO
Events:        <none>
```



## Deploymentの作成

```
cat <<'EOF' > /root/manifests/persistent_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: persistent-deployment
spec:
  selector:
    matchLabels:
      app: persistest_test
  replicas: 1
  template:
    metadata:
      labels:
        app: persistest_test
    spec:
      containers:
      - name: master
        image: nginx
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /persistent-volume
          name: pvc-volume
      volumes:
      - name: pvc-volume
        persistentVolumeClaim:
          claimName: iscsi-15gb-claim
      restartPolicy: Always
EOF
kubectl apply -f /root/manifests/persistent_deployment.yaml
```



Podが作成されたことを確認

```
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME                                     READY     STATUS    RESTARTS   AGE       IP              NODE
nginx-test-58586b9f9c-4qqk7              1/1       Running   2          8d        192.168.15.23   k8s-nodegpu01
nginx-test-58586b9f9c-9kf7v              1/1       Running   2          9d        192.168.15.16   k8s-nodegpu01
persistent-deployment-769b6dbb7c-bp7kv   1/1       Running   0          42s       192.168.15.28   k8s-nodegpu01 <--- here
private-reg                              1/1       Running   0          4d        192.168.15.26   k8s-nodegpu01
private-reg2                             1/1       Running   0          4d        192.168.15.27   k8s-nodegpu01
```



Node(k8s-nodegpu01)で自動的にtargetdのPortalへLoginし、Mountしていることが分かります

```
root@k8s-nodegpu01:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 464.7G  0 disk 
└─sda1   8:1    0 464.7G  0 part /
sdb      8:16   1   7.2G  0 disk 
sdf      8:80   0    10G  0 disk /var/lib/kubelet/pods/f8079bf7-9994-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb
sdg      8:96   0    15G  0 disk 
sr0     11:0    1   4.2G  0 rom  
sr1     11:1    1  1024M  0 rom  
```



docker inspect で確認すると、dockerレイヤーでmountしていることを確認できます

```
snip
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/f8079bf7-9994-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb",
                "Destination": "/persistent-volume",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
snip
```



また、df で確認すると、iscsiデバイスをxfsとしてMountしていることが分かります

```
root@k8s-nodegpu01:~# df -hT | grep iscsi-15gb
/dev/sdf       xfs        10G   33M   10G   1% /var/lib/kubelet/pods/f8079bf7-9994-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb
```



# データのライフサイクルについて確認

Podの障害やNodeの障害時に、正常にPersistentVolume上にデータを退避出来るか確認します。

**結論から記載すると、現段階の検証内容では、PV(iscsi)を使用しているPodは、正常にセルフヒーリングされなかった。Nodeの障害時に、ハングアップしているように見えました**

## テストデータの作成

PodへLogin

```
kubectl exec -it persistent-deployment-769b6dbb7c-bp7kv bash
```



テストデータの作成

```
echo '2018-08-27 mieru?1' >> /persistent-volume/test.txt
```



確認

```
root@persistent-deployment-769b6dbb7c-bp7kv:~# cat /persistent-volume/test.txt
2018-08-27 mieru?1
```



Podから離脱

```
exit
```



## Podの削除

Podの削除を行います

```
kubectl delete pod persistent-deployment-769b6dbb7c-bp7kv
```



同一のNodeで再作成されました

```
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME                                     READY     STATUS    RESTARTS   AGE       IP              NODE
nginx-test-58586b9f9c-4qqk7              1/1       Running   2          8d        192.168.15.23   k8s-nodegpu01
nginx-test-58586b9f9c-9kf7v              1/1       Running   2          9d        192.168.15.16   k8s-nodegpu01
persistent-deployment-769b6dbb7c-m7p9z   1/1       Running   0          15s       192.168.15.29   k8s-nodegpu01
private-reg                              1/1       Running   0          4d        192.168.15.26   k8s-nodegpu01
private-reg2                             1/1       Running   0          4d        192.168.15.27   k8s-nodegpu01
```



PodにLoginして、persistentvolumeを確認すると、データが確認できました

```
root@persistent-deployment-769b6dbb7c-m7p9z:~# cat /persistent-volume/test.txt
2018-08-27 mieru?1
```





## Nodeのシャットダウン

k8s-nodegpu01 をシャットダウンすると、PV(iscsi)も含めて他のNodeへ移動するかと思ったが、移動しませんでした。iscsiのpvがずっと使用中ということで、移動されなかった(20時間放置しても変わらず)

**PVをマウントしていないPodの場合：Nodeの障害時には、NodeのStatusがNotReadyとなってから約5分後にPodの移動が発生した。パラメータで短く出来るのかも**

```
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME                                     READY     STATUS              RESTARTS   AGE       IP               NODE
nginx-test-58586b9f9c-4qqk7              1/1       Unknown             2          8d        192.168.15.23    k8s-nodegpu01
nginx-test-58586b9f9c-9kf7v              1/1       Unknown             2          9d        192.168.15.16    k8s-nodegpu01
nginx-test-58586b9f9c-wwvnb              1/1       Running             0          5m        192.168.125.79   k8s-nodevm01
nginx-test-58586b9f9c-x42qp              1/1       Running             0          5m        192.168.125.78   k8s-nodevm01
persistent-deployment-769b6dbb7c-d98dv   0/1       ContainerCreating   0          9m        <none>           k8s-nodevm01
persistent-deployment-769b6dbb7c-m7p9z   1/1       Unknown             0          12m       192.168.15.29    k8s-nodegpu01
private-reg                              1/1       Unknown             0          4d        192.168.15.26    k8s-nodegpu01
private-reg2                             1/1       Unknown             0          4d        192.168.15.27    k8s-nodegpu01
```



その後、nodegpu01を起動し直すと、nodevm01へ移動された (iscsiは、Node障害に対してまだ機能が不足しているのかも)

```
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME                                     READY     STATUS    RESTARTS   AGE       IP               NODE
nginx-test-58586b9f9c-wwvnb              1/1       Running   0          7m        192.168.125.79   k8s-nodevm01
nginx-test-58586b9f9c-x42qp              1/1       Running   0          7m        192.168.125.78   k8s-nodevm01
persistent-deployment-769b6dbb7c-d98dv   1/1       Running   0          10m       192.168.125.83   k8s-nodevm01
```



移動先の nodevm01 でも、正常に iscsi を mount している

```
root@k8s-nodevm01:~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   10G  0 disk /var/lib/kubelet/pods/2b056bdb-9997-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb
sdb      8:16   0   15G  0 disk 
vda    253:0    0  100G  0 disk 
└─vda1 253:1    0  100G  0 part /
```

```
root@k8s-nodevm01:~# df -hT | grep -i iscsi-15gb
/dev/sda       xfs        10G   33M   10G   1% /var/lib/kubelet/pods/2b056bdb-9997-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb

```



PodにLogin

```
kubectl exec -it persistent-deployment-769b6dbb7c-d98dv bash
```



Podが再作成されても、テストデータを確認することが出来る

```
root@persistent-deployment-769b6dbb7c-d98dv:/# cat /persistent-volume/test.txt 
2018-08-27 mieru?1
```



## Deploymentを削除して、再作成してみる

```
kubectl delete deployments persistent-deployment
```

```
kubectl apply -f /root/manifests/persistent_deployment.yaml
```



PodにLogin

```
kubectl exec -it persistent-deployment-769b6dbb7c-hvqkg bash
```



Deploymentを再作成しても、Persistent Volume 上のデータは正しく引き継がれる

```
root@persistent-deployment-769b6dbb7c-hvqkg:/# cat /persistent-volume/test.txt 
2018-08-27 mieru?1
```



# まとめ

iscsiを使用して、PV, PVCを利用することが出来たが、Node障害時の挙動が怪しい部分がありました。

商用利用の際には、iscsi利用は厳しいと思われるので、他のストレージを検討した方が良いと思います。



# 参考URL

kubernetes docs
https://kubernetes.io/docs/concepts/storage/storage-classes/



Targetd の構築
https://qiita.com/sugimount/items/60a7ac50247f6bf7130e



iscsi initiatorの構築
https://www.server-world.info/query?os=Ubuntu_16.04&p=iscsi&f=3



kubernetes iscsi
https://github.com/kubernetes/examples/blob/master/staging/volumes/iscsi/iscsi.yaml