EKSのManaged Worker NodeのUpdateの挙動を確認する

# はじめに
これはEKSのManaged Worker Nodeのアップデートの挙動を確認する記事です。

Managed Worker Node自体のドキュメントはこちらです
https://aws.amazon.com/jp/about-aws/whats-new/2019/11/amazon-eks-adds-support-for-provisioning-and-managing-kubernetes-worker-nodes/
https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/managed-node-groups.html

# 前提条件
Managed Worker Node自体は現時点でAMIを指定して作成することはできません。
つまり、その次点での最新AMIが自動的に選択されるため、アップデートの挙動確認はすぐにはできません。

今回検証する環境は予め別の検証目的で作っていたものです。
たまたまAMIの更新が降ってきたので、アップデート検証記事を書くことにしました。

# アップデート前の状態
## EKSのGUI
こんな感じで表示されます
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/d8e088f5-26ea-a587-4030-8b6ea1378a7a.png)

## Updateの動作について
https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/managed-node-update-behavior.html
ここに記載されている通りです。

簡単に書くと、

- AutoScalingの起動テンプレートを最新AMIにして更新し、EC2インスタンスの数を１つ増やす。
増やすことで、EC2インスタンスが１つ新規で起動する、かつ、それは新しいAMIで起動する。
- ```eks.amazonaws.com/nodegroup-image``` labelを見て最新のAMIになっていない、Worker Node すべてを　cordon して、Podがスケジュールされないようにする。
- EKSが、そのcordonされたNodeの中から１つ選び、drainを実行する(drainのtimeoutは15分、その15分後の挙動については、エラーを発生させるか、nodeを強制deleteさせるか選ぶことが可能) drainが正常に完了した場合は、nodeを削除する
- そうするとAutoScalingにより新規でNodeが起動する（かつ新しいAMIで）ので、上記の処理を繰り返して、すべてのNodeが更新されたら、EC2インスタンス数を１つ減らして元の起動テンプレートの状態に戻す

この動作を検証したいと思います。

# 現在の状態

```sh
k get node
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-111-0-241.eu-west-1.compute.internal   Ready    <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready    <none>   18d   v1.14.7-eks-1861c5
```

こんな感じでnodeが２台あります。
ラベルはこんな感じです。

```sh
k describe node ip-10-111-0-241.eu-west-1.compute.internal  
Name:               ip-10-111-0-241.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=t3.medium
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/nodegroup=managed1
                    eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1a
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-111-0-241.eu-west-1.compute.internal
                    kubernetes.io/os=linux
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 17 Feb 2020 14:43:40 +0000
Taints:             <none>
Unschedulable:      false
```

```eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9``` の部分でAMIが記載されていることが分かります。
Taintsもついていません


こんな感じでPodもdaemonsetも動かしています。
pod名やその役割には特に意味はないです。

```sh
k get all --all-namespaces
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
default       pod/awscli-7bc86c4488-2qj2d    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-7xvcn    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-868ln    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-bd5bl    1/1     Running   0          17d
default       pod/awscli-7bc86c4488-jh6v4    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-q2l7m    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-qnfg5    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-qtvcq    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-t5zkl    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-z9tps    1/1     Running   0          85s
default       pod/awscli2-654c7776dd-zrmxh   1/1     Running   0          17d
kube-system   pod/aws-node-w57dk             1/1     Running   0          18d
kube-system   pod/aws-node-z8jpf             1/1     Running   0          20d
kube-system   pod/coredns-9c7df9cf9-2gzdg    1/1     Running   0          25m
kube-system   pod/coredns-9c7df9cf9-rmpn9    1/1     Running   0          25m
kube-system   pod/kube-proxy-69mw4           1/1     Running   0          20d
kube-system   pod/kube-proxy-96kb2           1/1     Running   0          18d

NAMESPACE     NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   172.20.0.1    <none>        443/TCP         23d
kube-system   service/kube-dns     ClusterIP   172.20.0.10   <none>        53/UDP,53/TCP   23d

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/aws-node     2         2         2       2            2           <none>          23d
kube-system   daemonset.apps/kube-proxy   2         2         2       2            2           <none>          23d

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
default       deployment.apps/awscli    10/10   10           10          17d
default       deployment.apps/awscli2   1/1     1            1           17d
kube-system   deployment.apps/coredns   2/2     2            2           23d

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
default       replicaset.apps/awscli-7bc86c4488    10        10        10      17d
default       replicaset.apps/awscli2-654c7776dd   1         1         1       17d
kube-system   replicaset.apps/coredns-9c7df9cf9    2         2         2       23d
```

# 確認したいこと
- 起動テンプレートの設定が代わること
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/bff7caa3-637a-974b-cacb-9d654b451882.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/77d78997-b371-0865-4897-1ca4e3b3f51b.png)

- 新規で起動したEC2 nodeのラベルが代わること
- 古いEC2たちが、cordonされること
- そのうちの一台がdrainされること
- そしてそれが、terminaleされること
- そしてそれが、繰り返されることと
- アップデートが完了すること

# アップデートの実行
GUIで実行。ポチッと
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/e8551bb7-e7bd-6edd-5349-1b1ff5a7cbed.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/9292042e-a686-ff60-510a-b8ef7fe7f955.png)

## 起動テンプレートのAMIが更新された
これは想定通り
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/f13fcb42-7b33-fb92-bce1-c7808438b125.png)

## Nodeの状態

だいぶ分かりにくいが

- 最初は2台 => Ready,SchedulingDisabledになった。つまり、新規NodeがReadyになることを待たずに、cordonが実行されていることが分かる
- そしてそのあと、なぜが5台以上のEC2インスタンスが起動されていることが分かる



```
k get node -w
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-111-0-241.eu-west-1.compute.internal   Ready    <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready    <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   2s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   4s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   11s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   11s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   14s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   12s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   9s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   14s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   31s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   61s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   2m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   11s   v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   3m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   3m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   4m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   4m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   5m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   5m30s   v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   6m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   5m59s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   5m59s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m4s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m4s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m8s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m8s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m12s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m12s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m30s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   90s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m1s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   2m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m45s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m45s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m46s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m46s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m50s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m51s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m47s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m53s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m53s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m56s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m54s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m7s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m5s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m5s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m7s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   3m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   4m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   7m1s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   5m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   8m1s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   6m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   9m1s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   7m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   10m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   8m31s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   11m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   9m31s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   12m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   13m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   11m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   14m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   12m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   15m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   13m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   16m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   14m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   17m   v1.14.8-eks-b8860f
```

最終的に一時的に9台のEC2が起動する状態となり、最後に初期状態の2台に落ち着いた


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/2386fcc0-bde1-42d9-989f-c29288c9b0d6.png)

8台のときに取得したスクショ。その後一台増えて、9台となった
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/107953/2389309d-59be-26ab-31d7-58afa03ffa21.png)

daemonsetもこんな感じで推移

```
k get daemonset -n kube-system -w
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aws-node     2         2         2       2            2           <none>          23d
kube-proxy   2         2         2       2            2           <none>          23d
kube-proxy   3     3     2     3     2     <none>   23d
aws-node   3     3     2     3     2     <none>   23d
aws-node   4     4     2     4     2     <none>   23d
kube-proxy   4     4     2     4     2     <none>   23d
kube-proxy   5     4     2     4     2     <none>   23d
aws-node   5     5     2     5     2     <none>   23d
kube-proxy   5     5     2     5     2     <none>   23d
kube-proxy   6     6     2     6     2     <none>   23d
aws-node   6     5     2     5     2     <none>   23d
aws-node   6     6     2     6     2     <none>   23d
aws-node   7     7     2     7     2     <none>   23d
kube-proxy   7     6     2     6     2     <none>   23d
kube-proxy   7     7     2     7     2     <none>   23d
aws-node   8     7     2     7     2     <none>   23d
kube-proxy   8     7     2     7     2     <none>   23d
aws-node   8     8     2     8     2     <none>   23d
kube-proxy   8     8     2     8     2     <none>   23d
kube-proxy   8     8     3     8     3     <none>   23d
kube-proxy   8     8     4     8     4     <none>   23d
kube-proxy   8     8     5     8     5     <none>   23d
aws-node   8     8     3     8     3     <none>   23d
aws-node   8     8     4     8     4     <none>   23d
aws-node   8     8     5     8     5     <none>   23d
kube-proxy   8     8     6     8     6     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
aws-node   8     8     6     8     6     <none>   23d
kube-proxy   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
aws-node   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
kube-proxy   9     9     7     9     7     <none>   23d
aws-node   9     9     7     9     7     <none>   23d
kube-proxy   9     9     8     9     8     <none>   23d
aws-node   9     9     8     9     8     <none>   23d
aws-node   8     8     8     8     8     <none>   23d
kube-proxy   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
aws-node   9     9     7     9     7     <none>   23d
kube-proxy   9     9     7     9     7     <none>   23d
kube-proxy   9     9     8     9     8     <none>   23d
aws-node   9     9     8     9     8     <none>   23d
aws-node   8     8     8     8     8     <none>   23d
kube-proxy   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
aws-node   8     8     6     8     6     <none>   23d
kube-proxy   8     8     6     8     6     <none>   23d
aws-node   6     6     6     6     6     <none>   23d
kube-proxy   6     6     6     6     6     <none>   23d
aws-node   6     6     5     6     5     <none>   23d
kube-proxy   6     6     5     6     5     <none>   23d
aws-node   6     6     4     6     4     <none>   23d
kube-proxy   6     6     4     6     4     <none>   23d
aws-node   6     6     3     6     3     <none>   23d
kube-proxy   6     6     3     6     3     <none>   23d
aws-node   3     3     2     3     2     <none>   23d
kube-proxy   3     3     2     3     2     <none>   23d
kube-proxy   2     2     2     2     2     <none>   23d
aws-node   2     2     2     2     2     <none>   23d
```



podもこんな感じで推移

```
k get pod -w
NAME                       READY   STATUS    RESTARTS   AGE
awscli-7bc86c4488-2qj2d    1/1     Running   0          3m15s
awscli-7bc86c4488-7xvcn    1/1     Running   0          3m15s
awscli-7bc86c4488-868ln    1/1     Running   0          3m15s
awscli-7bc86c4488-bd5bl    1/1     Running   0          17d
awscli-7bc86c4488-jh6v4    1/1     Running   0          3m15s
awscli-7bc86c4488-q2l7m    1/1     Running   0          3m15s
awscli-7bc86c4488-qnfg5    1/1     Running   0          3m15s
awscli-7bc86c4488-qtvcq    1/1     Running   0          3m15s
awscli-7bc86c4488-t5zkl    1/1     Running   0          3m15s
awscli-7bc86c4488-z9tps    1/1     Running   0          3m15s
awscli2-654c7776dd-zrmxh   1/1     Running   0          17d
awscli-7bc86c4488-2qj2d   1/1   Terminating   0     12m
awscli-7bc86c4488-tv2f6   0/1   Pending   0     0s
awscli-7bc86c4488-tv2f6   0/1   Pending   0     0s
awscli-7bc86c4488-tv2f6   0/1   ContainerCreating   0     1s
awscli-7bc86c4488-7xvcn   1/1   Terminating   0     12m
awscli-7bc86c4488-nss6s   0/1   Pending   0     0s
awscli-7bc86c4488-nss6s   0/1   Pending   0     0s
awscli-7bc86c4488-nss6s   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-868ln   1/1   Terminating   0     12m
awscli-7bc86c4488-5vxdk   0/1   Pending   0     0s
awscli-7bc86c4488-5vxdk   0/1   Pending   0     0s
awscli-7bc86c4488-5vxdk   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-bd5bl   1/1   Terminating   0     17d
awscli-7bc86c4488-wb259   0/1   Pending   0     0s
awscli-7bc86c4488-wb259   0/1   Pending   0     0s
awscli-7bc86c4488-wb259   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-qtvcq   1/1   Terminating   0     12m
awscli-7bc86c4488-dwtk8   0/1   Pending   0     0s
awscli-7bc86c4488-dwtk8   0/1   Pending   0     0s
awscli-7bc86c4488-dwtk8   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-wb259   1/1   Running   0     10s
awscli-7bc86c4488-5vxdk   1/1   Running   0     11s
awscli-7bc86c4488-tv2f6   1/1   Running   0     12s
awscli-7bc86c4488-nss6s   1/1   Running   0     11s
awscli-7bc86c4488-dwtk8   1/1   Running   0     11s
awscli-7bc86c4488-2qj2d   0/1   Terminating   0     12m
awscli-7bc86c4488-7xvcn   0/1   Terminating   0     12m
awscli-7bc86c4488-bd5bl   0/1   Terminating   0     17d
awscli-7bc86c4488-868ln   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-7xvcn   0/1   Terminating   0     12m
awscli-7bc86c4488-7xvcn   0/1   Terminating   0     12m
awscli-7bc86c4488-2qj2d   0/1   Terminating   0     12m
awscli-7bc86c4488-2qj2d   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-bd5bl   0/1   Terminating   0     17d
awscli-7bc86c4488-bd5bl   0/1   Terminating   0     17d
awscli-7bc86c4488-868ln   0/1   Terminating   0     12m
awscli-7bc86c4488-868ln   0/1   Terminating   0     12m
awscli-7bc86c4488-jh6v4   1/1   Terminating   0     14m
awscli-7bc86c4488-dd679   0/1   Pending   0     0s
awscli-7bc86c4488-dd679   0/1   Pending   0     0s
awscli-7bc86c4488-dd679   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-q2l7m   1/1   Terminating   0     14m
awscli-7bc86c4488-5jvch   0/1   Pending   0     0s
awscli-7bc86c4488-5jvch   0/1   Pending   0     0s
awscli-7bc86c4488-5jvch   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-qnfg5   1/1   Terminating   0     14m
awscli-7bc86c4488-5knbq   0/1   Pending   0     0s
awscli-7bc86c4488-5knbq   0/1   Pending   0     0s
awscli-7bc86c4488-5knbq   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-t5zkl   1/1   Terminating   0     14m
awscli-7bc86c4488-zfhns   0/1   Pending   0     0s
awscli-7bc86c4488-zfhns   0/1   Pending   0     0s
awscli-7bc86c4488-zfhns   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-z9tps   1/1   Terminating   0     14m
awscli-7bc86c4488-jffnr   0/1   Pending   0     0s
awscli-7bc86c4488-jffnr   0/1   Pending   0     0s
awscli-7bc86c4488-jffnr   0/1   ContainerCreating   0     0s
awscli2-654c7776dd-zrmxh   1/1   Terminating   0     17d
awscli2-654c7776dd-8p7mx   0/1   Pending   0     0s
awscli2-654c7776dd-8p7mx   0/1   Pending   0     0s
awscli2-654c7776dd-8p7mx   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-zfhns   1/1   Running   0     1s
awscli-7bc86c4488-5knbq   1/1   Running   0     2s
awscli-7bc86c4488-jffnr   1/1   Running   0     2s
awscli2-654c7776dd-8p7mx   1/1   Running   0     2s
awscli-7bc86c4488-dd679   1/1   Running   0     10s
awscli-7bc86c4488-5jvch   1/1   Running   0     10s
awscli-7bc86c4488-jh6v4   0/1   Terminating   0     15m
awscli-7bc86c4488-qnfg5   0/1   Terminating   0     15m
awscli-7bc86c4488-q2l7m   0/1   Terminating   0     15m
awscli-7bc86c4488-jh6v4   0/1   Terminating   0     15m
awscli-7bc86c4488-jh6v4   0/1   Terminating   0     15m
awscli-7bc86c4488-z9tps   0/1   Terminating   0     15m
awscli-7bc86c4488-t5zkl   0/1   Terminating   0     15m
awscli2-654c7776dd-zrmxh   0/1   Terminating   0     17d
awscli-7bc86c4488-t5zkl   0/1   Terminating   0     15m
awscli-7bc86c4488-t5zkl   0/1   Terminating   0     15m
awscli-7bc86c4488-q2l7m   0/1   Terminating   0     15m
awscli-7bc86c4488-q2l7m   0/1   Terminating   0     15m
awscli2-654c7776dd-zrmxh   0/1   Terminating   0     17d
awscli2-654c7776dd-zrmxh   0/1   Terminating   0     17d
awscli-7bc86c4488-z9tps   0/1   Terminating   0     15m
awscli-7bc86c4488-z9tps   0/1   Terminating   0     15m
awscli-7bc86c4488-qnfg5   0/1   Terminating   0     15m
awscli-7bc86c4488-qnfg5   0/1   Terminating   0     15m
awscli-7bc86c4488-dwtk8   1/1   Terminating   0     5m23s
awscli-7bc86c4488-pd5c9   0/1   Pending   0     0s
awscli-7bc86c4488-pd5c9   0/1   Pending   0     0s
awscli-7bc86c4488-pd5c9   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-5knbq   1/1   Terminating   0     2m57s
awscli-7bc86c4488-ddx8r   0/1   Pending   0     0s
awscli-7bc86c4488-ddx8r   0/1   Pending   0     0s
awscli2-654c7776dd-8p7mx   1/1   Terminating   0     2m55s
awscli-7bc86c4488-ddx8r   0/1   ContainerCreating   0     0s
awscli2-654c7776dd-rh59c   0/1   Pending   0     0s
awscli2-654c7776dd-rh59c   0/1   Pending   0     0s
awscli2-654c7776dd-rh59c   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-nss6s   1/1   Terminating   0     5m25s
awscli-7bc86c4488-n2thk   0/1   Pending   0     0s
awscli-7bc86c4488-n2thk   0/1   Pending   0     0s
awscli-7bc86c4488-n2thk   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-pd5c9   1/1   Running   0     2s
awscli2-654c7776dd-rh59c   1/1   Running   0     2s
awscli-7bc86c4488-n2thk   1/1   Running   0     2s
awscli-7bc86c4488-jffnr   1/1   Terminating   0     3m3s
awscli-7bc86c4488-9x9ws   0/1   Pending   0     1s
awscli-7bc86c4488-9x9ws   0/1   Pending   0     1s
awscli-7bc86c4488-9x9ws   0/1   ContainerCreating   0     1s
awscli-7bc86c4488-wb259   1/1   Terminating   0     5m31s
awscli-7bc86c4488-5tthr   0/1   Pending   0     0s
awscli-7bc86c4488-5tthr   0/1   Pending   0     0s
awscli-7bc86c4488-5tthr   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-5vxdk   1/1   Terminating   0     5m32s
awscli-7bc86c4488-5vrhj   0/1   Pending   0     0s
awscli-7bc86c4488-5vrhj   0/1   Pending   0     0s
awscli-7bc86c4488-5vrhj   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-5jvch   1/1   Terminating   0     3m4s
awscli-7bc86c4488-md55k   0/1   Pending   0     0s
awscli-7bc86c4488-md55k   0/1   Pending   0     0s
awscli-7bc86c4488-md55k   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-zfhns   1/1   Terminating   0     3m3s
awscli-7bc86c4488-2plnt   0/1   Pending   0     0s
awscli-7bc86c4488-2plnt   0/1   Pending   0     0s
awscli-7bc86c4488-2plnt   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-pd5c9   1/1   Terminating   0     8s
awscli-7bc86c4488-8mbgr   0/1   Pending   0     0s
awscli-7bc86c4488-8mbgr   0/1   Pending   0     0s
awscli-7bc86c4488-8mbgr   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-9x9ws   1/1   Running   0     2s
awscli-7bc86c4488-5vrhj   1/1   Running   0     1s
awscli-7bc86c4488-md55k   1/1   Running   0     2s
awscli-7bc86c4488-8mbgr   1/1   Running   0     2s
awscli-7bc86c4488-ddx8r   1/1   Running   0     10s
awscli-7bc86c4488-2plnt   1/1   Running   0     4s
awscli-7bc86c4488-5tthr   1/1   Running   0     4s
awscli-7bc86c4488-5vrhj   1/1   Terminating   0     7s
awscli-7bc86c4488-cldgf   0/1   Pending   0     0s
awscli-7bc86c4488-cldgf   0/1   Pending   0     0s
awscli-7bc86c4488-cldgf   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-8mbgr   1/1   Terminating   0     7s
awscli-7bc86c4488-cx62z   0/1   Pending   0     0s
awscli-7bc86c4488-cx62z   0/1   Pending   0     0s
awscli-7bc86c4488-cx62z   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-n2thk   1/1   Terminating   0     15s
awscli-7bc86c4488-c4zdj   0/1   Pending   0     0s
awscli-7bc86c4488-c4zdj   0/1   Pending   0     0s
awscli-7bc86c4488-c4zdj   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-tv2f6   1/1   Terminating   0     5m41s
awscli-7bc86c4488-cm54v   0/1   Pending   0     0s
awscli-7bc86c4488-cm54v   0/1   Pending   0     0s
awscli-7bc86c4488-cm54v   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-cldgf   1/1   Running   0     2s
awscli-7bc86c4488-cx62z   1/1   Running   0     2s
awscli-7bc86c4488-cm54v   1/1   Running   0     1s
awscli-7bc86c4488-c4zdj   1/1   Running   0     2s
awscli-7bc86c4488-5knbq   0/1   Terminating   0     3m27s
awscli-7bc86c4488-nss6s   0/1   Terminating   0     5m55s
awscli-7bc86c4488-dwtk8   0/1   Terminating   0     5m55s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m26s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m26s
awscli-7bc86c4488-nss6s   0/1   Terminating   0     5m59s
awscli-7bc86c4488-nss6s   0/1   Terminating   0     5m59s
awscli-7bc86c4488-5knbq   0/1   Terminating   0     3m31s
awscli-7bc86c4488-5knbq   0/1   Terminating   0     3m31s
awscli-7bc86c4488-jffnr   0/1   Terminating   0     3m33s
awscli-7bc86c4488-wb259   0/1   Terminating   0     6m2s
awscli-7bc86c4488-5vxdk   0/1   Terminating   0     6m3s
awscli-7bc86c4488-zfhns   0/1   Terminating   0     3m34s
awscli-7bc86c4488-5jvch   0/1   Terminating   0     3m35s
awscli-7bc86c4488-pd5c9   0/1   Terminating   0     39s
awscli-7bc86c4488-dwtk8   0/1   Terminating   0     6m2s
awscli-7bc86c4488-dwtk8   0/1   Terminating   0     6m3s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m34s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m34s
awscli-7bc86c4488-5jvch   0/1   Terminating   0     3m36s
awscli-7bc86c4488-5jvch   0/1   Terminating   0     3m36s
awscli-7bc86c4488-pd5c9   0/1   Terminating   0     40s
awscli-7bc86c4488-pd5c9   0/1   Terminating   0     41s
awscli-7bc86c4488-5vrhj   0/1   Terminating   0     37s
awscli-7bc86c4488-wb259   0/1   Terminating   0     6m9s
awscli-7bc86c4488-wb259   0/1   Terminating   0     6m9s
awscli-7bc86c4488-jffnr   0/1   Terminating   0     3m41s
awscli-7bc86c4488-jffnr   0/1   Terminating   0     3m41s
awscli-7bc86c4488-tv2f6   0/1   Terminating   0     6m11s
awscli-7bc86c4488-n2thk   0/1   Terminating   0     46s
awscli-7bc86c4488-8mbgr   0/1   Terminating   0     39s
awscli-7bc86c4488-5vrhj   0/1   Terminating   0     39s
awscli-7bc86c4488-5vrhj   0/1   Terminating   0     39s
awscli-7bc86c4488-5vxdk   0/1   Terminating   0     6m11s
awscli-7bc86c4488-5vxdk   0/1   Terminating   0     6m11s
awscli-7bc86c4488-zfhns   0/1   Terminating   0     3m42s
awscli-7bc86c4488-zfhns   0/1   Terminating   0     3m42s
awscli-7bc86c4488-tv2f6   0/1   Terminating   0     6m15s
awscli-7bc86c4488-tv2f6   0/1   Terminating   0     6m15s
awscli-7bc86c4488-n2thk   0/1   Terminating   0     49s
awscli-7bc86c4488-n2thk   0/1   Terminating   0     49s
awscli-7bc86c4488-8mbgr   0/1   Terminating   0     42s
awscli-7bc86c4488-8mbgr   0/1   Terminating   0     42s
```

古いNodeは全部Cordonが実行されて、こんな状態になった

```
k describe node ip-10-111-1-170.eu-west-1.compute.internal 
Name:               ip-10-111-1-170.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=t3.medium
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/nodegroup=managed1
                    eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1b
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-111-1-170.eu-west-1.compute.internal
                    kubernetes.io/os=linux
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 19 Feb 2020 05:25:17 +0000
Taints:             node.kubernetes.io/unschedulable:NoSchedule
Unschedulable:      true
```


新しいNodeは tainsはなく、nodegroup-imageが更新されていることが分かる

```
k describe node ip-10-111-1-43.eu-west-1.compute.internal 
                                                                                    Name:               ip-10-111-1-43.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=t3.medium
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/nodegroup=managed1
                    eks.amazonaws.com/nodegroup-image=ami-0b9d2c11b47bd8264
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1b
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-111-1-43.eu-west-1.compute.internal
                    kubernetes.io/os=linux
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 08 Mar 2020 16:19:10 +0000
Taints:             <none>
Unschedulable:      false
```
 

# まとめ
最後はすごい散らかってしまいましたが、

- AutoScalingの起動テンプレートを最新AMIにして更新し、EC2インスタンスの数を１つ増やす。
増やすことで、EC2インスタンスが１つ新規で起動する、かつ、それは新しいAMIで起動する
 - １つ増えるのではなく、一気に増えるようです。今回の検証では古いNodeは２つで、Max9Nodeになったので、7つ増えた？この原因は、ドキュメントの理解が足りない可能性があるので時間があるときにもう一度検証してみようと思います。
- ```eks.amazonaws.com/nodegroup-image``` labelを見て最新のAMIになっていない、Worker Node すべてを　cordon して、Podがスケジュールされないようにする。
 - これはそのとおりでした 
- EKSが、そのcordonされたNodeの中から１つ選び、drainを実行する(drainのtimeoutは15分、その15分後の挙動については、エラーを発生させるか、nodeを強制deleteさせるか選ぶことが可能) drainが正常に完了した場合は、nodeを削除する
 - 異常系は試していないのですが、drainされたnodeが削除されたのは事実です。
- そうするとAutoScalingにより新規でNodeが起動する（かつ新しいAMIで）ので、上記の処理を繰り返して、すべてのNodeが更新されたら、EC2インスタンス数を１つ減らして元の起動テンプレートの状態に戻す
 - １つずつ減らしたかどうかは見ていませんが、もとのEC2インスタンス数に戻ったのは確かです。


# まとめ
damonsetがあっても問題なくdrainしてくれました。本番環境だと、PDBが必須だと思うので、これも時間があるときに（次回のアップデートが降ってきたときに）PDBと組み合わせて検証してみようと思います。