k8s Router Nodeの追加

---
title: k8s Router Nodeの追加
tags: k8s kubernetes nginx-ingress
author: murata-tomohide
slide: false
---
前回はオンプレのサーバでk8sのクラスタを組みました。このクラスタの1台にラベルを付けてネットワーク周りのpodsを集約させてみようと思います。

# TL;DR

1台のノードにrouterというラベルとTaintを付けてnginx-ingressが特定のノードにプロビジョニングされるよう設定します。
特定のノードに配置されるようにする方法としては、Node Selector, Node Affinity, Taints Tolerationsとあるようです。それぞれ目的が違うようです。(Taint: 汚れという意味。)



# nodeにラベルをつける


k8s-node02.deroris.localだけrouter Roleにしてみませう。

```
[root@k8s-node01 ~]# kubectl get node -o wide
NAME                     STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-node01.deroris.local   Ready    master   81m   v1.15.3   172.31.3.101   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.1
k8s-node02.deroris.local   Ready    <none>   78m   v1.15.3   172.31.3.102   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.1
k8s-node05.deroris.local   Ready    <none>   70m   v1.15.3   172.31.3.105   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.2
node03.k8s.deroris.local   Ready    <none>   73m   v1.15.3   172.31.3.103   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.2
node04.k8s.deroris.local   Ready    <none>   70m   v1.15.3   172.31.3.104   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.2
```

```
# kubectl label node k8s-node02.deroris.local node-role.kubernetes.io/router=
# kubectl label node node03.k8s.deroris.local node-role.kubernetes.io/worker=
# kubectl label node node04.k8s.deroris.local node-role.kubernetes.io/worker=
# kubectl label node k8s-node05.deroris.local node-role.kubernetes.io/worker=
```

ラベルがついた。ラベルだけなので別に意味はない。


```
[root@k8s-node01 ~]# kubectl get node -o wide
NAME                     STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-node01.deroris.local   Ready    master   81m   v1.15.3   172.31.3.101   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.1
k8s-node02.deroris.local   Ready    router   78m   v1.15.3   172.31.3.102   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.1
k8s-node05.deroris.local   Ready    worker   70m   v1.15.3   172.31.3.105   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.2
node03.k8s.deroris.local   Ready    worker   73m   v1.15.3   172.31.3.103   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.2
node04.k8s.deroris.local   Ready    worker   70m   v1.15.3   172.31.3.104   <none>        CentOS Linux 7 (Core)   3.10.0-957.27.2.el7.x86_64   docker://19.3.2
```

# Taintを設定

router nodeにTaintをつけて、特定のノードにPodが配備されるようにする。（routerノードにネットワーク系を集約する。）
Taintは <Key>=<Value>:<Event>という書式です。

Valueに何か入れたほうがいい気がするけど今回は空で。

```
[root@k8s-node01 ~]# kubectl describe node k8s-node01.deroris.local | grep Taints
Taints:             node-role.kubernetes.io/master:NoSchedule
[root@k8s-node01 ~]# kubectl describe node k8s-node02.deroris.local | grep Taints
Taints:             <none>
[root@k8s-node01 ~]# kubectl taint nodes k8s-node02.deroris.local node-role.kubernetes.io/router=:NoSchedule
node/k8s-node02.deroris.local tainted
[root@k8s-node01 ~]# kubectl describe node k8s-node02.deroris.local | grep Taints
Taints:             node-role.kubernetes.io/router:NoSchedule
```


# MetalLBを入れる

デフォルトのyamlを編集して上記のTaintがついたノードにだけプロビジョニングされるようにする。
https://raw.githubusercontent.com/google/metallb/master/manifests/metallb.yaml

```
[root@k8s-node01 ~]# diff metallb.yaml metallb-onlyrouternode.yaml 
302c302,303
<         key: node-role.kubernetes.io/master
---
>         key: node-role.kubernetes.io/router
>         operator: Equal
353a355,358
>       tolerations:
>       - key: "node-role.kubernetes.io/router"
>         operator: Equal
>         effect: NoSchedule
\ No newline at end of file
```

metallbをL2モードで使い、使うIPアドレスの範囲を指定してあげる。範囲内のIPアドレスをIngressなどのtype: LoadBalancerのPodsが利用することになる。外からルーティングできるIPアドレスを設定しよう。今回の構成の場合、ノードの直のサブネット内にした。

```
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: my-ip-space
      protocol: layer2
      addresses:
      - 172.31.3.111-172.31.3.120 # 使う外側のIPアドレスの範囲を指定する

EOF
```

# nginx-ingress-controller インストール

ingress-controllerもrouter Taintがついてるノードのにみする。

https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

```
[root@k8s-node01 ~]# diff mandatory.yaml mandatory-onlyrouternode.yaml
275c275,278
< 
---
>       tolerations:
>       - key: "node-role.kubernetes.io/router"
>         operator: "Equal"
>         effect: "NoSchedule"
```


↑のマニフェストを食わせるとこんな感じになる。

```
[root@k8s-node01 ~]# kubectl get all -o wide -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE     IP          NODE                     NOMINATED NODE   READINESS GATES
pod/nginx-ingress-controller-5659d54c8d-66ssh   1/1     Running   0          3d17h   10.13.2.3   node03.k8s.deroris.local   <none>           <none>


NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE     SELECTOR
service/ingress-nginx   LoadBalancer   10.108.113.11   172.31.3.111   80:31403/TCP,443:31350/TCP   3d17h   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx


NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS                 IMAGES                                                                  SELECTOR
deployment.apps/nginx-ingress-controller   1/1     1            1           3d17h   nginx-ingress-controller   quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx

NAME                                                  DESIRED   CURRENT   READY   AGE     CONTAINERS                 IMAGES                                                                  SELECTOR
replicaset.apps/nginx-ingress-controller-5659d54c8d   1         1         1       3d17h   nginx-ingress-controller   quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx,pod-template-hash=5659d54c8d
```


# 確認

試しにこんな感じのマニフェストを食わせる。
/apacheでapacheに、/nginxでnginxに行く。

```
cat <<EOF | kubectl create -f -

# namespace
apiVersion: v1
kind: Namespace
metadata:
  name: ns-test
---
# ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: lb
  namespace: ns-test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
    - host: ns-test.172.31.3.111.nip.io # ここのIPアドレスはすでに付与されているものを入れたが指定したい場合は？アノテーションとかに書くのかな。
      http:
        paths:
          - path: /apache
            backend:
              serviceName: apache-svc
              servicePort: 80
          - path: /nginx
            backend:
              serviceName: nginx-svc
              servicePort: 80
          - path: /
            backend:
              serviceName: blackhole
              servicePort: 80
---
# apache
apiVersion: v1
kind: Service
metadata:
  name: apache-svc
  namespace: ns-test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpd
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - image: httpd:alpine
        name: httpd
        ports:
        - containerPort: 80
---
# nginx
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: ns-test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:alpine
        name: nginx
        ports:
        - containerPort: 80

EOF
```


ちゃんと動いた。


```
[root@k8s-node01 ~]# kubectl get all,ingress -A -o wide
NAMESPACE        NAME                                                 READY   STATUS    RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES
ingress-nginx    pod/nginx-ingress-controller-5659d54c8d-66ssh        1/1     Running   0          3d17h   10.13.2.3      node03.k8s.deroris.local   <none>           <none>
kube-system      pod/canal-6zxqp                                      2/2     Running   0          3d18h   172.31.3.104   node04.k8s.deroris.local   <none>           <none>
kube-system      pod/canal-7pqsp                                      2/2     Running   0          3d18h   172.31.3.105   k8s-node05.deroris.local   <none>           <none>
kube-system      pod/canal-grk7c                                      2/2     Running   0          3d18h   172.31.3.102   k8s-node02.deroris.local   <none>           <none>
kube-system      pod/canal-lw5nh                                      2/2     Running   0          3d18h   172.31.3.101   k8s-node01.deroris.local   <none>           <none>
kube-system      pod/canal-rbsxv                                      2/2     Running   0          3d18h   172.31.3.103   node03.k8s.deroris.local   <none>           <none>
kube-system      pod/coredns-5c98db65d4-5cmnn                         1/1     Running   0          3d19h   10.13.1.2      k8s-node02.deroris.local   <none>           <none>
kube-system      pod/coredns-5c98db65d4-zprpl                         1/1     Running   0          3d19h   10.13.0.2      k8s-node01.deroris.local   <none>           <none>
kube-system      pod/etcd-k8s-node01.deroris.local                      1/1     Running   0          3d19h   172.31.3.101   k8s-node01.deroris.local   <none>           <none>
kube-system      pod/kube-apiserver-k8s-node01.deroris.local            1/1     Running   0          3d19h   172.31.3.101   k8s-node01.deroris.local   <none>           <none>
kube-system      pod/kube-controller-manager-k8s-node01.deroris.local   1/1     Running   0          3d19h   172.31.3.101   k8s-node01.deroris.local   <none>           <none>
kube-system      pod/kube-proxy-525d9                                 1/1     Running   0          3d19h   172.31.3.104   node04.k8s.deroris.local   <none>           <none>
kube-system      pod/kube-proxy-62svl                                 1/1     Running   0          3d19h   172.31.3.101   k8s-node01.deroris.local   <none>           <none>
kube-system      pod/kube-proxy-6j95z                                 1/1     Running   0          3d19h   172.31.3.103   node03.k8s.deroris.local   <none>           <none>
kube-system      pod/kube-proxy-s7llh                                 1/1     Running   0          3d19h   172.31.3.105   k8s-node05.deroris.local   <none>           <none>
kube-system      pod/kube-proxy-zd8qt                                 1/1     Running   0          3d19h   172.31.3.102   k8s-node02.deroris.local   <none>           <none>
kube-system      pod/kube-scheduler-k8s-node01.deroris.local            1/1     Running   0          3d19h   172.31.3.101   k8s-node01.deroris.local   <none>           <none>
metallb-system   pod/controller-7d9775b7bd-9mt2s                      1/1     Running   0          3d18h   10.13.2.2      node03.k8s.deroris.local   <none>           <none>
metallb-system   pod/speaker-b5kj9                                    1/1     Running   0          3d18h   172.31.3.105   k8s-node05.deroris.local   <none>           <none>
metallb-system   pod/speaker-cjkdx                                    1/1     Running   0          3d18h   172.31.3.104   node04.k8s.deroris.local   <none>           <none>
metallb-system   pod/speaker-q969g                                    1/1     Running   0          3d18h   172.31.3.103   node03.k8s.deroris.local   <none>           <none>
metallb-system   pod/speaker-shmq4                                    1/1     Running   0          3d18h   172.31.3.102   k8s-node02.deroris.local   <none>           <none>
ns-test          pod/httpd-77655b8cf7-sdknm                           1/1     Running   0          44s     10.13.2.4      node03.k8s.deroris.local   <none>           <none>
ns-test          pod/nginx-5c69f5ccbf-zx8dv                           1/1     Running   0          44s     10.13.2.5      node03.k8s.deroris.local   <none>           <none>


NAMESPACE       NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE     SELECTOR
default         service/kubernetes      ClusterIP      10.96.0.1       <none>         443/TCP                      3d19h   <none>
ingress-nginx   service/ingress-nginx   LoadBalancer   10.108.113.11   172.31.3.111   80:31403/TCP,443:31350/TCP   3d17h   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
kube-system     service/kube-dns        ClusterIP      10.96.0.10      <none>         53/UDP,53/TCP,9153/TCP       3d19h   k8s-app=kube-dns
ns-test         service/apache-svc      NodePort       10.97.207.142   <none>         80:31192/TCP                 44s     app=httpd
ns-test         service/nginx-svc       NodePort       10.101.222.70   <none>         80:30933/TCP                 44s     app=nginx

NAMESPACE        NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE     CONTAINERS                 IMAGES                                              SELECTOR
kube-system      daemonset.apps/canal        5         5         5       5            5           beta.kubernetes.io/os=linux   3d18h   calico-node,kube-flannel   calico/node:v3.9.2,quay.io/coreos/flannel:v0.11.0   k8s-app=canal
kube-system      daemonset.apps/kube-proxy   5         5         5       5            5           beta.kubernetes.io/os=linux   3d19h   kube-proxy                 k8s.gcr.io/kube-proxy:v1.15.5                       k8s-app=kube-proxy
metallb-system   daemonset.apps/speaker      4         4         4       4            4           beta.kubernetes.io/os=linux   3d18h   speaker                    metallb/speaker:master                              app=metallb,component=speaker

NAMESPACE        NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS                 IMAGES                                                                  SELECTOR
ingress-nginx    deployment.apps/nginx-ingress-controller   1/1     1            1           3d17h   nginx-ingress-controller   quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
kube-system      deployment.apps/coredns                    2/2     2            2           3d19h   coredns                    k8s.gcr.io/coredns:1.3.1                                                k8s-app=kube-dns
metallb-system   deployment.apps/controller                 1/1     1            1           3d18h   controller                 metallb/controller:master                                               app=metallb,component=controller
ns-test          deployment.apps/httpd                      1/1     1            1           44s     httpd                      httpd:alpine                                                            app=httpd
ns-test          deployment.apps/nginx                      1/1     1            1           44s     nginx                      nginx:alpine                                                            app=nginx

NAMESPACE        NAME                                                  DESIRED   CURRENT   READY   AGE     CONTAINERS                 IMAGES                                                                  SELECTOR
ingress-nginx    replicaset.apps/nginx-ingress-controller-5659d54c8d   1         1         1       3d17h   nginx-ingress-controller   quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx,p
od-template-hash=5659d54c8d
kube-system      replicaset.apps/coredns-5c98db65d4                    2         2         2       3d19h   coredns                    k8s.gcr.io/coredns:1.3.1                                                k8s-app=kube-dns,pod-template-hash=5c98db65d4
metallb-system   replicaset.apps/controller-7d9775b7bd                 1         1         1       3d18h   controller                 metallb/controller:master                                               app=metallb,component=controller,pod-template-hash=7d9775b7bd
ns-test          replicaset.apps/httpd-77655b8cf7                      1         1         1       44s     httpd                      httpd:alpine                                                            app=httpd,pod-template-hash=77655b8cf7
ns-test          replicaset.apps/nginx-5c69f5ccbf                      1         1         1       44s     nginx                      nginx:alpine                                                            app=nginx,pod-template-hash=5c69f5ccbf





NAMESPACE   NAME                    HOSTS                         ADDRESS   PORTS   AGE
ns-test     ingress.extensions/lb   ns-test.172.31.3.111.nip.io             80      44s
```

```
murata:~ $ curl http://ns-test.172.31.3.111.nip.io/nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```


172.31.3.111のIPアドレスを持っているノードがいるはずである。
むむ。node02をrouterにしたはずなのに、Ingressはnode03にいるぞ・・・

```
[root@k8s-node01 ~]# kubectl describe service/ingress-nginx -n ingress-nginx
Name:                     ingress-nginx
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/par...
Selector:                 app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type:                     LoadBalancer
IP:                       10.108.113.11
LoadBalancer Ingress:     172.31.3.111
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31403/TCP
Endpoints:                10.13.2.3:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31350/TCP
Endpoints:                10.13.2.3:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31060
Events:
  Type    Reason        Age                    From             Message
  ----    ------        ----                   ----             -------
  Normal  nodeAssigned  14m (x343 over 3d18h)  metallb-speaker  announcing from node "node03.k8s.deroris.local"
```


一度deployment.apps/nginx-ingress-controllerを消してマニフェストを食わせなおしたらnode02にデプロイされた。(順番間違えたかな。
結局のところ、metallbのpod/controller-xxxとingress-nginx    pod/nginx-ingress-controller-xxxが当該のノードに上がっていればよしということ。

```
[root@k8s-node01 ~]# kubectl get all,ing  -A -o wide | grep node02
ingress-nginx    pod/nginx-ingress-controller-5659d54c8d-x4svj        1/1     Running   0          59m     10.13.1.4      k8s-node02.deroris.local   <none>           <none>
kube-system      pod/canal-grk7c                                      2/2     Running   0          3d21h   172.31.3.102   k8s-node02.deroris.local   <none>           <none>
kube-system      pod/coredns-5c98db65d4-5cmnn                         1/1     Running   0          3d22h   10.13.1.2      k8s-node02.deroris.local   <none>           <none>
kube-system      pod/kube-proxy-zd8qt                                 1/1     Running   0          3d22h   172.31.3.102   k8s-node02.deroris.local   <none>           <none>
metallb-system   pod/controller-7d9775b7bd-v25q5                      1/1     Running   0          78m     10.13.1.3      k8s-node02.deroris.local   <none>           <none>
metallb-system   pod/speaker-shmq4                                    1/1     Running   0          3d21h   172.31.3.102   k8s-node02.deroris.local   <none>           <none>
```


# まとめ

Router nodeを固定できると外側に出すサーバを固定できるということになるので、ノードの配置が楽になるはずです。
devとprodなどのTaintをつけて運用する場合もあるみたいだけど、物理的に分けてしまえばいいじゃんって思う。EKSやGKE等の場合はよいのかしら。