[Kubernetes]LimitRangeの動作を確認する 2

#はじめに
[前回](https://qiita.com/dingtianhongjie/items/06367432e17d6412c187#maxlimitrequestratio)、ContainerでのLimitRangeの動作を確認しました。今回はPodとPVCでの動作を確認したいと思います。

前回も記載しましたが、LimitRangeの設定項目とContainer/Pod/PVCの対応は以下のようになります。

|設定項目|概要|Container|Pod|PVC
|:---|:---|:--:|:--:|:--:
|max|最大リソース|レ|レ|レ
|min|最小リソース|レ|レ|レ
|default|defaultのLimits|レ||
|defaultRequest|defaultのRequests|レ
|maxLimitRequestRatio|Limits / Requestsの割合|レ|レ|

#Pod
まずはPodから確認します。

##max/min
以下のマニフェストのLimitRangeを作成します。

```limitrange-max_min-pod.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: limit-range-max-min-pod
  namespace: stg
spec:
  limits:
  - max:
      cpu: "500m"
    min:
      cpu: "200m"
    type: Pod
```

```
$ kubectl apply -f limitrange-max_min-pod.yaml
limitrange/limit-range-max-min-pod created
$ kubectl -n stg describe limitranges
Name:       limit-range-max-min-pod
Namespace:  stg
Type        Resource  Min   Max   Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---   ---   ---------------  -------------  -----------------------
Pod         cpu       200m  500m  -                -              -
```
ContainerにMax/Minを設定したときは、Default Request/Default Limitも自動で設定されましたが、Podの場合は設定されていないですね。

このMax/Minは、そのPod内にあるContainer合計のMax/Minになります。そのため、Podに複数のContainerがある場合には注意が必要です。
以下のPodをデプロイして確認します。

```nginx_redis1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-redis1
  namespace: stg
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          resources:
            requests:
              cpu: 100m
            limits:
              cpu: 200m
        - name: redis
          image: redis:latest
          resources:
            requests:
              cpu: 100m
            limits:
              cpu: 300m
```

```
$ kubectl apply -f nginx_redis1.yaml
deployment.apps/nginx-redis1 created
$ kubectl -n stg get pod -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP               NODE           NOMINATED NODE   READINESS GATES
nginx-redis1-cd7f9697d-4csm2   2/2     Running   0          18s   192.168.79.85    k8s-worker01   <none>           <none>
nginx-redis1-cd7f9697d-jwbqh   2/2     Running   0          18s   192.168.69.238   k8s-worker02   <none>           <none>
```
デプロイできました。
このPodの設定値を図にすると、以下になります。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/492530/7ff9d62a-7cac-84ce-b6b1-b8e375ad6880.png)

Containerのrequestsの合計をLimitRangeのmin以上、limitsの合計をmax以下にしています。

---

なお、Requestsを設定しない場合や上記の条件を満たさない場合、Podのデプロイに失敗します。また、limitsのみを指定した場合は、requestsにはlimitsと同じ値が設定されます。

* Requests/Limitsを設定しない場合

```
$ kubectl -n stg describe replicasets.apps
Name:           nginx-redis1-d856469b4
Namespace:      stg
・・・
Events:
  Type     Reason        Age               From                   Message
  ----     ------        ----              ----                   -------
  Warning  FailedCreate  43s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-48f89" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  43s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-rzzh7" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  43s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-n2nt2" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  43s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-gvsqr" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  43s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-94cjr" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  43s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-6g2w4" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  42s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-67kbk" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  42s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-455gx" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  41s               replicaset-controller  Error creating: pods "nginx-redis1-d856469b4-fz9xz" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
  Warning  FailedCreate  2s (x5 over 40s)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-redis1-d856469b4-gmbsz" is forbidden: [minimum cpu usage per Pod is 200m.  No request is specified, maximum cpu usage per Pod is 500m.  No limit is specified]
```

* Limitsの合計がLimitRangeのmaxを超える場合

```
$ kubectl -n stg describe replicasets.apps nginx-redis1-74d95f5b7d
Name:           nginx-redis1-74d95f5b7d
Namespace:      stg
・・・
Events:
  Type     Reason        Age               From                   Message
  ----     ------        ----              ----                   -------
  Warning  FailedCreate  22s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-4md4c" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  22s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-w4c4z" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  22s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-27g5j" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  22s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-7sxdz" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  22s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-hvz9d" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  21s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-jj45j" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  21s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-n658t" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  21s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-9rz4l" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  20s               replicaset-controller  Error creating: pods "nginx-redis1-74d95f5b7d-8mm77" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
  Warning  FailedCreate  1s (x4 over 19s)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-redis1-74d95f5b7d-gl5jq" is forbidden: maximum cpu usage per Pod is 500m, but limit is 700m
```

##maxLimitRequestRatio
次にmaxLimitRequestRatioの動作を確認します。

以下のLimitRangeを作成します。

```limitrange-ratio-pod.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: limitlange-ratio-pod
  namespace: prd
spec:
  limits:
  - maxLimitRequestRatio:
      cpu: 2
    type: Pod
```
```
$ kubectl apply -f limitrange-ratio-pod.yaml
limitrange/limitlange-ratio-pod created
$ kubectl -n prd describe limitranges
Name:       limitlange-ratio-pod
Namespace:  prd
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Pod         cpu       -    -    -                -              2
```

Podをデプロイして確認します。

```nginx_redis2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-redis2
  namespace: prd
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          resources:
            requests:
              cpu: 100m
            limits:
              cpu: 100m
        - name: redis
          image: redis:latest
          resources:
            requests:
              cpu: 100m
            limits:
              cpu: 300m
```

```
$ kubectl apply -f nginx_redis2.yaml
deployment.apps/nginx-redis2 created
$ kubectl -n prd get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-redis2-6547498d-2xkbk   2/2     Running   0          15s
nginx-redis2-6547498d-dwwdw   2/2     Running   0          15s
```

maxLimitRequestRatioを「2」に設定しました。
各コンテナの設定値を見ると、nginxのLimit/Requestは「1」ですが、redisは「3」です。maxLimitRequestRatioもmax/minと同様に判断基準はコンテナ一つ一つではなく、Pod内に含まれるすべてのコンテナの合計値になります。
合計値だとLimit/Requestは「2」となりますので、満たしていますね。

#PVC
最後にPVCの時の動作を確認したいと思います。
PVCのときは、Max/Minのみ指定可能です。

以下のLimitRangeを作成します。

```limitrange-pvc.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: limit-range-pvc
  namespace: stg
spec:
  limits:
  - max:
      storage: 100Mi
    min:
      storage: 50Mi
    type: PersistentVolumeClaim
```

```
$ kubectl apply -f limitrange-pvc.yaml
limitrange/limit-range-pvc created
$ kubectl describe -n stg limitranges
Name:                  limit-range-pvc
Namespace:             stg
Type                   Resource  Min   Max    Default Request  Default Limit  Max Limit/Request Ratio
----                   --------  ---   ---    ---------------  -------------  -----------------------
PersistentVolumeClaim  storage   50Mi  100Mi  -                -              -
```

PVCの動作を確認するために、StorageClassを作成します。
StorageClassとDynamicProvisioningについての詳細は、こちらをご確認ください。

>[[Kubernetes]Dynamic Provisioningの動作を確認する](https://qiita.com/dingtianhongjie/items/6142bfdddbbf627da055#zfs-provisioner-service%E3%81%AE%E8%B5%B7%E5%8B%95)

```storageclass.yml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: zfs-sc
  namespace: stg
provisioner: gentics.com/zfs
reclaimPolicy: Retain
```

```
$ kubectl apply -f storageclass.yml
storageclass.storage.k8s.io/zfs-sc created
$ kubectl get storageclasses
NAME     PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
zfs-sc   gentics.com/zfs   Retain          Immediate           false                  16s
```
PVCを作成して、LimitRangeの動作を確認します。

```pvc.yaml
apiVersion: v1
metadata:
  name: example-pvc
  namespace: stg
  annotations:
    volume.beta.kubernetes.io/storage-class: "zfs-sc"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Mi
```

```
$ kubectl apply -f pvc.yaml
persistentvolumeclaim/example-pvc created
$ kubectl -n stg get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
example-pvc   Bound    pvc-a4b904b0-84bc-4bb8-b481-d7c1c13174da   100Mi      RWX            zfs-sc         3m22s
$ kubectl -n stg get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
pvc-a4b904b0-84bc-4bb8-b481-d7c1c13174da   100Mi      RWX            Retain           Bound    stg/example-pvc   zfs-sc                  15s
```
PVCとPVが作成されていますね。

なお、Max/Minを超えた/満たない値を指定した場合は、以下のようにエラーになります。

```
$ kubectl apply -f pvc.yaml
Error from server (Forbidden): error when creating "pvc.yaml": persistentvolumeclaims "example-pvc" is forbidden: maximum storage usage per PersistentVolumeClaim is 100Mi, but request is 110Mi
$ kubectl apply -f pvc.yaml
Error from server (Forbidden): error when creating "pvc.yaml": persistentvolumeclaims "example-pvc" is forbidden: minimum storage usage per PersistentVolumeClaim is 50Mi, but request is 10Mi
```

#まとめ
今回と前回でLimitRangeの動作を確認しました。
コンテナはリソース制限を設定しないと、CPUやメモリリソースをあるだけ使いますので、環境やシステムごとにLimitRangeを適切に設定したいですね。また、DynamicProvisioningだとストレージ（PV）も簡単に作れてしまいますので、PVCに関してもLimitRangeを設定しておきたいですね。