KubernetesでMulti-container Podの1つのコンテナがRestartされつづけても別のコンテナには影響しない (Serviceには影響する)

# 初めにPodとContainerのStatusについて

## Overview

PodのStatusは、`PodStatus`というObjectで、その中に以下のFieldがある (公式: [Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/))

- conditions
- containerStatuses
- hostIP
- phase
- podIP
- qosClass
- startTime

おおまかに以下のようになっている。 Documentには `phase`, `conditions`, `container status`についての記載がある。

```yaml
status:
  conditions:
  - <conditionがArrayで入っている>
  containerStatuses:
  - <containerのStatusesが入っている>
  hostIP: <hostIp>
  phase: <phase>
  podIP: <podIp>
  qosClass: <qosClass>
  startTime: "2019-12-17T11:51:17Z"
```

重要な部分は、

- podの`phase`と`status` (`condition`の中のfield) は異なる
- containerには`state`と`ready`という状態を表すfieldがある 

## 1. Phase

Phaseは以下の5種類のみ

- Pending
- Running
- Succeeded
- Failed
- Unknown

## 2. Conditions

Arrayで以下の項目で構成される

- lastProbeTime
- lastTransitionTime
- status
- message
- reason
- type

例: 

```yaml

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-12-17T11:51:17Z"
    status: "True"
    type: Initialized
...
  - lastProbeTime: null
    lastTransitionTime: "2020-01-24T09:58:21Z"
    message: 'containers with unready status: [web-app]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
```


## 3. ContainerStatuses

ContainerのstatusもArrayである

Stateは以下の3つのみ

- Waiting
- Running
- Terminated

Stateの中に `message`と`reason`がある



```yaml
  containerStatuses:
  - containerID: docker://<hash>
    image: /path/to/image/web-app:tag
    imageID: docker-pullable:///path/to/image/web-app:tag@sha256:<hash>
    lastState:
      terminated:
        containerID: docker://<hash>
        exitCode: 0
        finishedAt: "2020-01-24T09:58:20Z"
        reason: Completed
        startedAt: "2020-01-24T09:58:19Z"
    name: web-app
    ready: false
    restartCount: 1148
    state:
      waiting:
        message: Back-off 5m0s restarting failed container=web-app pod=web-app-<hash>_<namespace>(<hash>)
        reason: CrashLoopBackOff
```

## 4. get podした時に出てくる STATUSは？ 

普段良く使う `kubectl get pod` で見るSTATUSは、単純に上記のどれかのFieldと対応していない！ (詳細は、[Kubernetes: kubectl 上の Pod のステータス表記について
](https://qiita.com/tkusumi/items/825ccde31fdc3d0b8425) のページがとても詳しいので是非)

```
kubectl get pod <pod_name> -n <namespace>
NAME         READY   STATUS             RESTARTS   AGE
<pod_name>   0/1     CrashLoopBackOff   1152       37d
```

- `phase`は`Running`で, ContainerStatusは `waiting`で `reason`が `CrashLoopBackOff`という状態
- `READY`が `0/1`なのは、 containerが一つで、readyが`false`のため

# readinessProbeとlivenessProbe

[Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)


- containerがスタートした時に、readyかどうかをチェックして、readyの状態になってからtrafficを流すようにしてくれる (readinessProbe)
- containerが動いているとき、定期的にチェック (health check) して、もしもfalseになった場合には、containerをrestartする (livenessProbe)

## チェック: Multi-Containerで一つのcontainerのReadinessProbe とLivenessProbeがずっとReadyにならないケース

`kubectl get pod` で状態を観察した

Status:Running Ready:1/2 ↔ Status:CrashLoopBackOff Ready:1/2 このふたつの状態を繰り返していた。Readyとならないcontainerはずっとrestartされていたが、もう一つのコンテナ(今回nginxを使用)は問題なく動いていた

```yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "10M", "--vm-hang", "1"]
```

```yaml
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-dsdk2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dsdk2
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                                                       Message
  ----     ------     ----                  ----                                                       -------
  Normal   Scheduled  39m                   default-scheduler                                          Successfully
assigned naka/nginx-deployment-669b897679-msndc to ip-192-168-4-159.ap-northeast-1.compute.internal
  Normal   Pulled     39m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Container ima
ge "nginx:1.7.9" already present on machine
  Normal   Created    39m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Created conta
iner nginx
  Normal   Started    39m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Started conta
iner nginx
  Normal   Killing    38m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Container mem
ory-demo-ctr failed liveness probe, will be restarted
  Normal   Pulling    38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Pulling image
 "polinux/stress"
  Normal   Pulled     38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Successfully
pulled image "polinux/stress"
  Normal   Created    38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Created conta
iner memory-demo-ctr
  Normal   Started    38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Started conta
iner memory-demo-ctr
  Warning  Unhealthy  37m (x8 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Readiness pro
be failed: dial tcp 192.168.4.201:8080: connect: connection refused
  Warning  Unhealthy  9m56s (x32 over 39m)  kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Liveness prob
e failed: dial tcp 192.168.4.201:8080: connect: connection refused
  Warning  BackOff    4m52s (x79 over 28m)  kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Back-off rest
arting failed container
```

```yaml
NAME                                READY   STATUS             RESTARTS   AG
E
nginx-deployment-669b897679-kq84s   1/2     CrashLoopBackOff   13         42
m
nginx-deployment-669b897679-msndc   1/2     CrashLoopBackOff   13         42
m

```

# OOMKilled


次にOOMKilledとは、Containerが `ResourceLimit` で設定されたMemoryの値以上にメモリを使おうとした時に、killされたということ

[Configure Out of Resource Handling](https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/)


> if a Pod container is OOM killed, it may be restarted by the kubelet based on its RestartPolicy.



https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/

> Pod is running and has one Container. Container runs out of memory.

> Container terminates in failure.
> Log OOM event.
> If restartPolicy is:
> Always: Restart Container; Pod phase stays Running.
> OnFailure: Restart Container; Pod phase stays Running.
> Never: Log failure event; Pod phase becomes Failed.

# チェック: Multi-Containerで一つのcontainerが常にOOMKilledされるケース

ContainerCreating -> OOMKilled　-> CrashLoopBackOff -> OOMKilled ... STAUSは繰り返し (exponential back-off) が、もう一つのcontainerは問題なく動いている

```yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "15M", "--vm-hang", "1"]
```

get

```
kubectl get pod -n <namespace>
NAME                                READY   STATUS             RESTARTS   AG
E
nginx-deployment-6bd9c88968-dssp4   1/2     CrashLoopBackOff   11         35
m
nginx-deployment-6bd9c88968-xfmx6   1/2     CrashLoopBackOff   11         35
m
```

一つのPodをdescribeしてみる

```
kubectl describe pod <pod-name> -n <namespace>

...

Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-dsdk2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dsdk2
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                                                       Message
  ----     ------     ----                   ----                                                       -------
  Normal   Scheduled  33m                    default-scheduler                                          Successfully assigned naka/nginx-deployment-6bd9c88968-xfmx6 to ip-192-168-5-113.ap-northeast-1.compute.internal
  Normal   Pulled     33m                    kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Container image "nginx:1.7.9" already present on machine
  Normal   Created    33m                    kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Created container nginx
  Normal   Started    33m                    kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Started container nginx
  Normal   Pulling    32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Pulling image "polinux/stress"
  Normal   Pulled     32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Successfully pulled image "polinux/stress"
  Normal   Created    32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Created container memory-demo-ctr
  Normal   Started    32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Started container memory-demo-ctr
  Warning  BackOff    3m52s (x145 over 33m)  kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Back-off restarting failed container
```

# Service

## 概要

コンテナごとにReadinessProbeが設定できるので、Multi-containerのPodで一つContainerがサービスが続くかと思っていた。が、違う。

> With the introduction of new Pod conditions, a Pod is evaluated to be ready only when both the following statements are true:

>  - All containers in the Pod are ready.
>  - All conditions specified in ReadinessGates are “True”.

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate

Serviceを利用する場合は、すべてのコンテナがreadyでないといけない。上記の例では、nginxコンテナは問題なく動作していたとしても、serviceを介してアクセスしている場合には、 同居しているcontainerが死ぬと、 そのpodがreadyではなくなり、serviceのendpointから外されてしまう。


## 検証

以下のように二組 Deploymentと Serviceを用意する.

```yaml

tree practice/resource/02
practice/resource/02
├── README.md
├── mulit-container-oom.yaml
├── mulit-container.yaml
├── service-oom.yaml
└── service.yaml
```

- nginxとOOMされるコンテナを同居させたdeploymentとnginxのservice
- nginxとReadyになるコンテナを同居させたdeploymentとnginxのservice

```yaml:practice/resource/02/mulit-container-oom.yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-oom
  namespace: naka
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx-oom
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-oom
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "15M", "--vm-hang", "1"]
```

```yaml:practice/resource/02/service-oom.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-oom
  labels:
    app: nginx-oom
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx-oom
```

```yaml:practice/resource/02/mulit-container.yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx
  namespace: naka
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "1M", "--vm-hang", "1"]
```

```yaml:practice/resource/02/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx
```

## DeploymentとServiceの作成

```
kubectl create namespace naka
kubectl apply -n naka -f practice/resource/02/
deployment.apps/nginx-oom created
deployment.apps/nginx created
service/nginx-oom created
service/nginx created
```

## Podを確認

```
kubectl get pod -n naka
NAME                         READY   STATUS             RESTARTS   AGE
nginx-7fb468f99f-b252k       2/2     Running            0          13m
nginx-7fb468f99f-rf7x2       2/2     Running            0          13m
nginx-oom-6655d46664-khbr6   1/2     CrashLoopBackOff   7          14m
nginx-oom-6655d46664-v5d6r   1/2     CrashLoopBackOff   7          14m
```

## Serviceの確認

```
kubectl get svc -n naka
NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
nginx       ClusterIP   10.100.104.244   <none>        80/TCP    15m
nginx-oom   ClusterIP   10.100.133.104   <none>        80/TCP    15m
```

## port-forwardで確認

### 正常に動いている方をport-forward

```
kubectl port-forward svc/nginx -n naka 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
```

別windowでcurlを叩いてみる

```
curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```

正常に動いている


### OOMKilledされるcontainerと同居させたほう

```
kubectl port-forward svc/nginx-oom -n naka 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
```

```
curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```

こちらもnginxは動いている

## Endpointの確認


```
kubectl get endpoints -n naka
NAME        ENDPOINTS                         AGE
nginx       192.168.5.22:80,192.168.6.60:80   18m
nginx-oom                                     18m
```

ドキュメント通り2つのコンテナがReadyとなってない `nginx-oom` では EndpointsにPodのIPがふられていないことがわかる!

## 別のPodからcurlでサービス通してアクセス

別のPodをたてる

```
kubectl run curl --image=radial/busyboxplus:curl -i --tty
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-66bdcf564-g7cl8:/ ]$
```

### 正常に動いている方を確認

サービス名を使って名前解決してアクセスできる

```
[ root@curl-66bdcf564-g7cl8:/ ]$ nslookup nginx
Server:    10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx
Address 1: 10.100.104.244 nginx.naka.svc.cluster.local
[ root@curl-66bdcf564-g7cl8:/ ]$ curl nginx.naka.svc.cluster.local:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```


### OOMのほうを確認

EndpointがServiceから削除されてるので、サービスからアクセスが出来ずTimeoutするのが確認できた

```
nslookup nginx-oom
Server:    10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx-oom
Address 1: 10.100.133.104 nginx-oom.naka.svc.cluster.local
curl --connect-timeout 10 nginx-oom.naka.svc.cluster.local:80
curl: (28) Connection timed out after 10001 milliseconds
```