今更ながらRaspberry Piでkubernetes構築（クラスタ編）

# これは2019/09/19時点での話です

[前回](https://qiita.com/yyojiro/items/0cd7ec7e2a39610be53f)はRaspberry Piにkubernetesをシングル構成で構築した。
3台構成での残り2台分の資材が届いたので早速クラスタ構築開始。
最終的に以下のような構成にする。

| ホスト名 | 役割 | IP |
|:-:|:-:|:-:|
| k8s01 | Master | 192.168.11.10 |
| k8s02 | Worker | 192.168.11.11 |
| k8s03 | Worker | 192.168.11.12 |

（続き）
[Raspberry PiのKubernetesにmetrics-serverを入れる](https://qiita.com/yyojiro/items/febfaeadabd2fe8eed08)
[Raspberry PiのKubernetesにDashboardを入れた話](https://qiita.com/yyojiro/items/527188bb24783f9376b2)

# 構築手順
前回構築したやつをMasterとし、今回構築するやつはWorkerにする。

## OSインストールとか
OSインストールや基本設定は前回と同様なので省略。
[前回手順](https://qiita.com/yyojiro/items/0cd7ec7e2a39610be53f#%E3%83%91%E3%83%83%E3%82%B1%E3%83%BC%E3%82%B8%E3%81%AE%E3%82%A4%E3%83%B3%E3%82%B9%E3%83%88%E3%83%BC%E3%83%AB)をそのまま実行。


## kubeadm joinでクラスタに組み込み
Worker用のノードに入れるもんいれたら、Masterでjoin用のtoken発行を行う。

```
root@k8s01:~# kubeadm token create --print-join-command
kubeadm join 192.168.11.10:6443 --token 1flgzl.4x70omgdnigie8yn     --discovery-token-ca-cert-hash sha256:76ea8013fa291ca1350d633343f2cfba9b4ece246f39fb3048ac4458c1f3e96c 
```

上記出力結果をコピーしてWorker側でそのまま実行。

```
root@k8s02:~#  kubeadm join 192.168.11.10:6443 --token 1flgzl.4x70omgdnigie8yn     --discovery-token-ca-cert-hash sha256:76ea8013fa291ca1350d633343f2cfba9b4ece246f39fb3048ac4458c1f3e96c 
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.2. Latest validated version: 18.09
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

root@k8s02:~# 
```

ノード追加がうまくいったか確認する。

```
root@k8s01:~# kubectl get nodes
NAME    STATUS     ROLES    AGE     VERSION
k8s01   Ready      master   27h     v1.15.3
k8s02   NotReady   <none>   5m20s   v1.16.0
k8s03   NotReady   <none>   19s     v1.16.0
root@k8s01:~# 
```

あれ？？？
なんかバージョン違うし、待っててもNotReadyからReadyにならないし。。。
運が悪いことに、Master構築した時と最新バージョンが違っていた模様。
Workerノードをいったんクラスタから切り離し。

```
root@k8s02:~# kubeadm reset
```


## 紆余曲折を経てv1.15.3に統一した
Workerノードで下記を実行してkubeletをv.15.3に統一した。

```
root@k8s02:~# apt-get install kubelet=1.15.3*
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '1.15.3-00' (kubernetes-xenial:kubernetes-xenial [armhf]) for 'kubelet'
The following packages will be DOWNGRADED:
  kubelet
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 56 not upgraded.
Need to get 17.7 MB of archives.
After this operation, 2,905 kB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf kubelet armhf 1.15.3-00 [17.7 MB]
Fetched 17.7 MB in 4s (4,115 kB/s)  
dpkg: warning: downgrading kubelet from 1.16.0-00 to 1.15.3-00
(Reading database ... 92015 files and directories currently installed.)
Preparing to unpack .../kubelet_1.15.3-00_armhf.deb ...
Unpacking kubelet (1.15.3-00) over (1.16.0-00) ...
Setting up kubelet (1.15.3-00) ...
root@k8s02:~# 
```

念のためOS再起動してからkubeadm join実行したら今度はうまくいった。

```
root@k8s01:~# kubectl get nodes
NAME    STATUS   ROLES    AGE     VERSION
k8s01   Ready    master   7m18s   v1.15.3
k8s02   Ready    <none>   2m54s   v1.15.3
k8s03   Ready    <none>   2m54s   v1.15.3
root@k8s01:~# 
root@k8s01:~# kubectl get pods --all-namespaces
NAMESPACE        NAME                            READY   STATUS    RESTARTS   AGE
kube-system      coredns-5644d7b6d9-sz2t8        1/1     Running   0          19m
kube-system      coredns-5644d7b6d9-vnxxf        1/1     Running   0          19m
kube-system      etcd-k8s01                      1/1     Running   0          18m
kube-system      kube-apiserver-k8s01            1/1     Running   0          19m
kube-system      kube-controller-manager-k8s01   1/1     Running   1          19m
kube-system      kube-flannel-ds-arm-426sl       1/1     Running   0          17m
kube-system      kube-flannel-ds-arm-c7gpj       1/1     Running   8          15m
kube-system      kube-flannel-ds-arm-v5w64       1/1     Running   8          15m
kube-system      kube-proxy-2j2ds                1/1     Running   1          15m
kube-system      kube-proxy-75lpk                1/1     Running   0          19m
kube-system      kube-proxy-8dsf2                1/1     Running   1          15m
kube-system      kube-scheduler-k8s01            1/1     Running   0          19m
metallb-system   controller-6bcfdfd677-7dvpv     1/1     Running   0          19m
metallb-system   speaker-5pdrx                   1/1     Running   1          15m
metallb-system   speaker-62zk4                   1/1     Running   0          17m
metallb-system   speaker-fh48h                   1/1     Running   1          15m
root@k8s01:~# 
```

## 動作確認

ノードが3つになったので、replicasを3にしてnginxサービスを構築。

```yaml:nginx3.deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer
```

サービスデプロイ。

```
root@k8s01:~# kubectl create -f nginx3.deployment.yaml 
```

しばらくすると3か所でnginxのコンテナが作られてることが確認できる。

```
root@k8s01:~# kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
nginx-deployment-85ff79dd56-8rdr2   1/1     Running   0          19m   10.244.1.2   k8s03   <none>           <none>
nginx-deployment-85ff79dd56-cq6gw   1/1     Running   0          19m   10.244.2.2   k8s02   <none>           <none>
nginx-deployment-85ff79dd56-lmdm5   1/1     Running   0          19m   10.244.0.5   k8s01   <none>           <none>
root@k8s01:~# 
```

このままだと面白みがないので、それぞれのコンテナの中に入ってnginxのindex.htmlを変えてやる。

```
root@k8s01:~# kubectl exec -it nginx-deployment-85ff79dd56-lmdm5 -- /bin/bash
root@nginx-deployment-85ff79dd56-lmdm5:/# 
root@nginx-deployment-85ff79dd56-lmdm5:/# cd /usr/share/nginx/html/
root@nginx-deployment-85ff79dd56-lmdm5:/usr/share/nginx/html# ls
50x.html  index.html
root@nginx-deployment-85ff79dd56-lmdm5:/usr/share/nginx/html# echo "node1" > index.html 
root@nginx-deployment-85ff79dd56-lmdm5:/usr/share/nginx/html# exit
```

他のノードも同様。
curlしてみた結果、ロードバランスされてることを確認。

```
root@k8s01:~# curl 192.168.11.200
node3
root@k8s01:~# curl 192.168.11.200
node1
root@k8s01:~# curl 192.168.11.200
node2
```

忘れてたがラベルもつけてあげよう。

```
root@k8s01:~# kubectl get nodes
NAME    STATUS   ROLES    AGE     VERSION
k8s01   Ready    master   12m     v1.15.3
k8s02   Ready    <none>   4m19s   v1.15.3
k8s03   Ready    <none>   4m4s    v1.15.3
root@k8s01:~# 
root@k8s01:~# kubectl label node k8s02 node-role.kubernetes.io/worker=worker
node/k8s02 labeled
root@k8s01:~# kubectl label node k8s03 node-role.kubernetes.io/worker=worker
node/k8s03 labeled
root@k8s01:~# 
root@k8s01:~# kubectl get nodes
NAME    STATUS   ROLES    AGE     VERSION
k8s01   Ready    master   14m     v1.15.3
k8s02   Ready    worker   6m22s   v1.15.3
k8s03   Ready    worker   6m7s    v1.15.3
root@k8s01:~# 
```

# （参考）Masterのバージョンをv1.16.0にしてみた場合
**これは結局失敗した手順だが、何が起こったかただの参考情報として残す。**
バージョン合わせてやるためにMasterノードを更新。

```
root@k8s01:~# apt-get update
root@k8s01:~# apt-get install -y kubelet kubeadm kubectl 
``` 

リセット、再構築

```
root@k8s02:~# kubeadm reset
root@k8s02:~# kubeadm init --pod-network-cidr=10.244.0.0/16
root@k8s02:~# cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
root@k8s02:~# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
root@k8s02:~# kubectl taint nodes --all node-role.kubernetes.io/master-
```

**flannelが動かないいいい**

```
root@k8s01:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                            READY   STATUS             RESTARTS   AGE
kube-system   coredns-5644d7b6d9-k8qgx        0/1     Pending            0          13m
kube-system   coredns-5644d7b6d9-xg7nt        0/1     Pending            0          13m
kube-system   etcd-k8s01                      1/1     Running            0          13m
kube-system   kube-apiserver-k8s01            1/1     Running            0          14m
kube-system   kube-controller-manager-k8s01   1/1     Running            0          13m
kube-system   kube-flannel-ds-arm-8vfb8       0/1     CrashLoopBackOff   3          80s
kube-system   kube-flannel-ds-arm-fjl64       0/1     CrashLoopBackOff   3          80s
kube-system   kube-flannel-ds-arm-l5h42       0/1     CrashLoopBackOff   3          80s
kube-system   kube-proxy-2zrl6                1/1     Running            0          13m
kube-system   kube-proxy-49qsj                1/1     Running            0          11m
kube-system   kube-proxy-fzj6h                1/1     Running            0          12m
kube-system   kube-scheduler-k8s01            1/1     Running            0          13m
root@k8s01:~# 
```

describeで確認してみるが、動かない理由はよくわからんかった。

```
root@k8s01:~# kubectl describe pod  kube-flannel-ds-arm-8vfb8   -n kube-system 
Name:         kube-flannel-ds-arm-8vfb8
Namespace:    kube-system
Priority:     0
Node:         k8s03/192.168.11.12
Start Time:   Thu, 19 Sep 2019 09:13:30 +0100
Labels:       app=flannel
              controller-revision-hash=7d7b8f7d47
              pod-template-generation=1
              tier=node
Annotations:  <none>
Status:       Running
IP:           192.168.11.12
IPs:
  IP:           192.168.11.12
Controlled By:  DaemonSet/kube-flannel-ds-arm
Init Containers:
  install-cni:
    Container ID:  docker://6d53a05527e7cbfe07b1876360578e7464b61a5fa8a7830eb6853191fa670a27
    Image:         quay.io/coreos/flannel:v0.11.0-arm
    Image ID:      docker-pullable://quay.io/coreos/flannel@sha256:c3d2c9c54eadeacd9d74717cfcd73a12782773d864e7be4686fa8bd9ae2c4f42
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
    Args:
      -f
      /etc/kube-flannel/cni-conf.json
      /etc/cni/net.d/10-flannel.conflist
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 19 Sep 2019 09:13:32 +0100
      Finished:     Thu, 19 Sep 2019 09:13:32 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/cni/net.d from cni (rw)
      /etc/kube-flannel/ from flannel-cfg (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-46hvn (ro)
Containers:
  kube-flannel:
    Container ID:  docker://e5fe0253df90e0b7343707b0c19487eed36fa70f9bc7d75be31dacd78de2ebe7
    Image:         quay.io/coreos/flannel:v0.11.0-arm
    Image ID:      docker-pullable://quay.io/coreos/flannel@sha256:c3d2c9c54eadeacd9d74717cfcd73a12782773d864e7be4686fa8bd9ae2c4f42
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/bin/flanneld
    Args:
      --ip-masq
      --kube-subnet-mgr
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 19 Sep 2019 09:15:12 +0100
      Finished:     Thu, 19 Sep 2019 09:15:14 +0100
    Ready:          False
    Restart Count:  4
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:       kube-flannel-ds-arm-8vfb8 (v1:metadata.name)
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/flannel from run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-46hvn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  run:
    Type:          HostPath (bare host directory volume)
    Path:          /run/flannel
    HostPathType:  
  cni:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-flannel-cfg
    Optional:  false
  flannel-token-46hvn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  flannel-token-46hvn
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/arch=arm
Tolerations:     :NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  <unknown>            default-scheduler  Successfully assigned kube-system/kube-flannel-ds-arm-8vfb8 to k8s03
  Normal   Pulled     2m54s                kubelet, k8s03     Container image "quay.io/coreos/flannel:v0.11.0-arm" already present on machine
  Normal   Created    2m54s                kubelet, k8s03     Created container install-cni
  Normal   Started    2m53s                kubelet, k8s03     Started container install-cni
  Normal   Pulled     73s (x5 over 2m53s)  kubelet, k8s03     Container image "quay.io/coreos/flannel:v0.11.0-arm" already present on machine
  Normal   Created    73s (x5 over 2m53s)  kubelet, k8s03     Created container kube-flannel
  Normal   Started    73s (x5 over 2m52s)  kubelet, k8s03     Started container kube-flannel
  Warning  BackOff    70s (x7 over 2m47s)  kubelet, k8s03     Back-off restarting failed container
root@k8s01:~# 
```

結局、これを調べるよりv1.15.3にしたほうがよさそうな気がしたので、v1.15.3にダウングレードした。
そのうちflannelもv1.16.0で使えるようになるだろう。たぶん。