[IBMCloud] IKS(Kubernetes) Cordon/Drain時の挙動確認

## テスト環境
東京リージョンの各ZoneにVSI for VPCのLinuxサーバを配置
* VSI for VPC(CentOS) TOK01 x 1台
* VSI for VPC(CentOS) TOK02 x 1台
* VSI for VPC(CentOS) TOK03 x 1台

サーバ側はIKS(Kubernetes)にてNginxのPodを各ゾーンに配置し、前段にIngressALBを配置
* Application load balancer for VPC x 1(TOK02/TOK03)
* IBM Cloud Kubernetes Service x 1Cluster(TOK02/TOK03)
    * Nginx Pod(TOK02) x 1台
    * Nginx Pod(TOK03) x 1台
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/19ba4bb5-1827-accc-1408-add382bfeb7e.png)

## テスト内容
今回はWorkerNodeのcordonとPodのdrain実施時の通信状態について確認する

### 事前確認
PODの稼働状態、各WorkerNodeで1台づつNginxPodが動いていることを確認
```
% kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
iks-nginx-7b789b9b4-29bnv   1/1     Running   0          29s   172.17.4.111    10.244.64.28    <none>           <none>
iks-nginx-7b789b9b4-6rvcn   1/1     Running   0          29s   172.17.22.245   10.244.128.21   <none>           <none>
```

## スケジューリングの停止(Cordon)
「10.244.128.21」のノードに対して新しいポッドのスケジューリングを停止する
作業手順は[こちら](https://cloud.ibm.com/docs/containers?topic=containers-host-maintenance&locale=ja#worker-maintenance-vpc)

### スケジュールが有効になっていることを確認
STATUSが**Ready**となっていることを確認する
STATUSは**SchedulingDisabled**が表示されている場合、スケジューリング対象から外れている
```
% kubectl get nodes

NAME            STATUS   ROLES    AGE   VERSION
10.244.128.21   Ready    <none>   17h   v1.28.11+IKS
10.244.64.28    Ready    <none>   17h   v1.28.11+IKS
```

### Cordonの実行
Cordonの実施
```
% kubectl cordon 10.244.128.21
node/10.244.128.21 cordoned
```

スケジュールが無効になっていることを確認
```
% kubectl get node
NAME            STATUS                     ROLES    AGE   VERSION
10.244.128.21   Ready,SchedulingDisabled   <none>   17h   v1.28.11+IKS
10.244.64.28    Ready                      <none>   17h   v1.28.11+IKS
```

Cordon時のPODの状態を確認
Cordon前と同様に、各ノードでそれぞれPODが１つづつ動いている状態
```
% kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
iks-nginx-7b789b9b4-29bnv   1/1     Running   0          29s   172.17.4.111    10.244.64.28    <none>           <none>
iks-nginx-7b789b9b4-6rvcn   1/1     Running   0          29s   172.17.22.245   10.244.128.21   <none>           <none>
```

VPCのサーバからPODへの通信の停止なし、POD分散も正常の状態
```
[root@tok01-sv ~]# while true; do curl --max-time 1 -s http://bf9b5d7d-jp-tok.lb.appdomain.cloud:8080 || echo "Request timed out or failed."; sleep 1; done
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:00
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:02
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:03

[root@tok02-sv ~]# while true; do curl --max-time 1 -s http://bf9b5d7d-jp-tok.lb.appdomain.cloud:8080 || echo "Request timed out or failed."; sleep 1; done
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:02
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:03
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:04
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:05

[root@tok03-sv ~]# while true; do curl --max-time 1 -s http://bf9b5d7d-jp-tok.lb.appdomain.cloud:8080 || echo "Request timed out or failed."; sleep 1; done
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:04
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:05
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:06
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:07
```

uncordonにする場合は下記の通り実施する
```
kubectl uncordon 10.244.128.21   
```

## drainの実行
ノード上のすべてのPODは停止され、他のノードに再スケジュールされる
今回は、「10.244.128.21」にあるPOD「iks-nginx-7fbc7dfc94-6x4s9」が「10.244.64.28」で再スケジュールされる

kubectl drain を実行すると、ノード上のポッドは停止されるため、その間は一時的にポッドへの通信が中断される。ただし、ポッドが再スケジュールされ、新しいノードで稼働を開始すると、通信は再び確立される。

```
%  kubectl drain 10.244.128.21 --ignore-daemonsets --delete-emptydir-data
node/10.244.128.21 cordoned

Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-mm4jx, ibm-object-s3fs/ibmcloud-object-storage-driver-q4gbh, ibm-observe/sysdig-agent-node-analyzer-bmhpj, kube-system/ibm-vpc-block-csi-node-cx9fb, kube-system/ibmcloud-iks-debug-daemonset-769d2, kube-system/konnectivity-agent-g9zxm
evicting pod kube-system/coredns-jp-tok-3-6fb86b959c-rwhhc
evicting pod kube-system/ibm-vpc-block-csi-controller-0
evicting pod kube-system/private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw
evicting pod kube-system/coredns-jp-tok-3-6fb86b959c-2d6nt
evicting pod kube-system/coredns-jp-tok-3-6fb86b959c-j5mnv
evicting pod default/iks-nginx-7b789b9b4-6rvcn
evicting pod kube-system/coredns-autoscaler-jp-tok-3-5c77b4959c-z8jf2
evicting pod kube-system/public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28
error when evicting pods/"private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/coredns-autoscaler-jp-tok-3-5c77b4959c-z8jf2 evicted
pod/ibm-vpc-block-csi-controller-0 evicted
evicting pod kube-system/private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw
evicting pod kube-system/public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28
error when evicting pods/"private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod kube-system/private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw
evicting pod kube-system/public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28
error when evicting pods/"private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/coredns-jp-tok-3-6fb86b959c-2d6nt evicted
pod/coredns-jp-tok-3-6fb86b959c-j5mnv evicted
pod/coredns-jp-tok-3-6fb86b959c-rwhhc evicted
```

## Drain時の通信状態

Drain開始時間
```
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:25
```

PodがDrainされ、方系のPodのみとなり続けて新しいPodが作られたことがわかる

DrainされたPod:<font color="blue">6rvcn</font>
新しく作成されたPod:<font color="green">76wp7</font>
方系で動き続けていたPod:**29bnv**

sv1 -> IngressALB -> Node -> Pod
```
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:54
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:55
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:56
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:57
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:58
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:59
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:00
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:02
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:03
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:04
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:05
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:06
```

sv2 -> IngressALB -> Node -> Pod
```
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:54
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:55
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:56
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:57
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:58
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:59
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:00
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:02
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:03
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:04
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:05
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:06
```

sv3 -> IngressALB -> Node -> Pod
```
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:54
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:56
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:57
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:58
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:59
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:00
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:02
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:03
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:04
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:05
```

**今回の検証ではPodへの通信の停止は見られなかった**