EKSで機械学習 #5  Cluster AutoScaler設定編

# はじめに
このシリーズでは Amazon EKSで機械学習を行っていきたいと思います

# シリーズ目次
EKSで機械学習 #1 準備編
EKSで機械学習 #2 クラスター作成編
EKSで機械学習 #3 Managed Worker Node作成編
EKSで機械学習 #4 GPU Managed Worker Node作成編
EKSで機械学習 #5 Cluster AutoScaler設定編 (この記事)

# この記事の目的
今回は、Worker Node自体をAutoScalingさせるためのClusterAutoScalerを設定します

# Cluster AutoScaler用のIAM Policyを作成
https://aws.amazon.com/jp/premiumsupport/knowledge-center/eks-cluster-autoscaler-setup/
こちらを参考にして「ClusterAutoScaler」というIAM policyを作成します。

Policyの中身はこんな感じ

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}
```

# IAM OICD Providerを有効化
本記事では、よく見られるEC2インスタンスにIAM roleをアタッチする方法ではなく、Service Accountを利用します。
以下のコマンドをまず実行して、IAM OICD Providerを有効化します

```shell
eksctl utils associate-iam-oidc-provider --cluster=ml --region=us-west-2 --approve
[ℹ]  eksctl version 0.13.0
[ℹ]  using region us-west-2
[ℹ]  will create IAM Open ID Connect provider for cluster "ml" in "us-west-2"
[✔]  created IAM Open ID Connect provider for cluster "ml" in "us-west-2"
```

# ClusterAutoScaler用の service account を作成します

XXXXXXXXX の部分は各自のAWS Account IDで置き換えてください

```shell
eksctl create iamserviceaccount --cluster=ml --name=cluster-autoscaler --namespace=kube-system --attach-policy-arn=arn:aws:iam::XXXXXXXXX:policy/ClusterAutoScaler --region=us-west-2 --approve
[ℹ]  eksctl version 0.13.0
[ℹ]  using region us-west-2
[ℹ]  1 iamserviceaccount (kube-system/cluster-autoscaler) was included (based on the include/exclude rules)
[!]  serviceaccounts that exists in Kubernetes will be excluded, use --override-existing-serviceaccounts to override
[ℹ]  1 task: { 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/cluster-autoscaler", create serviceaccount "kube-system/cluster-autoscaler" } }
[ℹ]  building iamserviceaccount stack "eksctl-ml-addon-iamserviceaccount-kube-system-cluster-autoscaler"
[ℹ]  deploying stack "eksctl-ml-addon-iamserviceaccount-kube-system-cluster-autoscaler"
[ℹ]  created serviceaccount "kube-system/cluster-autoscaler"
```

## service accountが作成されたことを念の為確認

```shell
k get sa -n kube-system | grep cluster-autoscaler
cluster-autoscaler                   1         119s
```

# Cluster AutoScalerのデプロイします

以下のymlを作成して実行します

```yml
#---
#apiVersion: v1
#kind: ServiceAccount
#metadata:
#  labels:
#    k8s-addon: cluster-autoscaler.addons.k8s.io
#    k8s-app: cluster-autoscaler
#  name: cluster-autoscaler
#  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-autoscaler
  labels:
    k8s-addon: cluster-autoscaler.addons.k8s.io
    k8s-app: cluster-autoscaler
rules:
  - apiGroups: [""]
    resources: ["events", "endpoints"]
    verbs: ["create", "patch"]
  - apiGroups: [""]
    resources: ["pods/eviction"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["pods/status"]
    verbs: ["update"]
  - apiGroups: [""]
    resources: ["endpoints"]
    resourceNames: ["cluster-autoscaler"]
    verbs: ["get", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["watch", "list", "get", "update"]
  - apiGroups: [""]
    resources:
      - "pods"
      - "services"
      - "replicationcontrollers"
      - "persistentvolumeclaims"
      - "persistentvolumes"
    verbs: ["watch", "list", "get"]
  - apiGroups: ["extensions"]
    resources: ["replicasets", "daemonsets"]
    verbs: ["watch", "list", "get"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["watch", "list"]
  - apiGroups: ["apps"]
    resources: ["statefulsets", "replicasets", "daemonsets"]
    verbs: ["watch", "list", "get"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses", "csinodes"]
    verbs: ["watch", "list", "get"]
  - apiGroups: ["batch", "extensions"]
    resources: ["jobs"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["create"]
  - apiGroups: ["coordination.k8s.io"]
    resourceNames: ["cluster-autoscaler"]
    resources: ["leases"]
    verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    k8s-addon: cluster-autoscaler.addons.k8s.io
    k8s-app: cluster-autoscaler
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create","list","watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
    verbs: ["delete", "get", "update", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-autoscaler
  labels:
    k8s-addon: cluster-autoscaler.addons.k8s.io
    k8s-app: cluster-autoscaler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-autoscaler
subjects:
  - kind: ServiceAccount
    name: cluster-autoscaler
    namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    k8s-addon: cluster-autoscaler.addons.k8s.io
    k8s-app: cluster-autoscaler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cluster-autoscaler
subjects:
  - kind: ServiceAccount
    name: cluster-autoscaler
    namespace: kube-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8085'
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
        - image: k8s.gcr.io/cluster-autoscaler:v1.14.7
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled
            #- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs/ca-certificates.crt
              readOnly: true
          imagePullPolicy: "Always"
      volumes:
        - name: ssl-certs
          hostPath:
            path: "/etc/ssl/certs/ca-bundle.crt"

```

ポイントは
- service accountはこのymlでは作成しない（コメントアウトしている)
- 以下のように、 cluster名を設定に入れないようにしている。よりymlを汎用的にするため。推奨設定ではないらしいが。

```
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled
            #- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>
```

# 確認

以下のymlを実行して、auto scalingが発動することを確認する

```gpu-app-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-app-deployment
  labels:
    app: gpu-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpu-app
  template:
    metadata:
      labels:
        app: gpu-app
    spec:
      containers:
        - name: digits-container
          image: nvidia/digits:6.0
          resources:
            limits:
              nvidia.com/gpu: 1 
```

```shell
k apply -f gpu-app-deployment.yml 
deployment.apps/gpu-app-deployment created

# リソース(GPU)が不足しているため、2つ目のPodが Pending のままになっているが、しばらくすると
# Cluster AutoScalerが発動して、EC2ノードが一台増えて(2台 -> 3台)、podが作成されたことがわかる
k get pod -w
NAME                                  READY   STATUS    RESTARTS   AGE
gpu-app-deployment-5998b859d9-cfn66   1/1     Running   0          42s
gpu-app-deployment-5998b859d9-w6w6c   0/1     Pending   0          42s
gpu-app-deployment-5998b859d9-w6w6c   0/1     Pending   0          107s
gpu-app-deployment-5998b859d9-w6w6c   0/1     Pending   0          2m17s
gpu-app-deployment-5998b859d9-w6w6c   0/1     ContainerCreating   0          2m17s
gpu-app-deployment-5998b859d9-w6w6c   1/1     Running             0          3m23s


k get node
NAME                                            STATUS   ROLES    AGE   VERSION
ip-192-168-109-12.us-west-2.compute.internal    Ready    <none>   19h   v1.14.8-eks-b8860f
ip-192-168-134-74.us-west-2.compute.internal    Ready    <none>   72m   v1.14.8-eks-b8860f
ip-192-168-178-187.us-west-2.compute.internal   Ready    <none>   86s   v1.14.8-eks-b8860f
PowerUser:~/environment $ 

```

最後に後始末

```shell

k delete -f gpu-app-deployment.yml 
deployment.apps "gpu-app-deployment" deleted
```

# まとめ
今回は、Cluster AutoScalerの設定を行い、実際にGPUインスタンス数が増えたことを確認しました。