MetalLBでKubernetesのtype:LoadBalancerを有効化させるときに嵌まったこと

タイトルについては最下部あたりに見出し付きで記述（本記事自体は構築手順メモ）。
[MetalLB](https://metallb.universe.tf)をfirewalld有効下で使おうとしたところ少し詰まったので今後のために。

# 構成
## ハードウェア

|type|spec|remarks  |
|:------------|:-------------------------------|:---------------------------------------------------------|
| Masterノード | 2コア、8GB RAM、64GB HDD × ３台 | **m1** `10.0.3.1` / **m2** `10.0.3.2` / **m3** `10.0.3.3` |
| Workerノード | 2コア、8GB RAM、64GB HDD × ３台 | **w1** `10.0.3.4` / **w2** `10.0.3.5` / **w3** `10.0.3.6` |
| LBノード | １コア、2GB RAM、16GB HDD × ２台 | **lb1** `10.0.3.11` / **lb2** `10.0.3.12` |
 
## ソフトウェア
- CentOS 7.6
- Kubernetes v1.13.1
- Weave v1.10

## ネットワーク

|range|remarks|
|:--|:--|
|10.0.0.0/16|existing private network|
|10.0.3.0/24|Kubernetes infrastructure|
|10.0.4.0/24|MetalLB External IP|
* DHCP, DNSサーバ側でホスト名、IPは固定済

## 参考にした記事、URL等
  * MetalLB関連
      - https://blog.web-apps.tech/type-loadbalancer_by_metallb/
      - https://metallb.universe.tf
  * HA関連
      - https://kubernetes.io/docs/setup/independent/high-availability/
      - https://mseeeen.msen.jp/build-cluster-with-keepalived-on-centos-7/
      - https://www.server-world.info/query?os=CentOS_7&p=haproxy
      - https://knowledge.sakura.ad.jp/8084/
  * Kubernetes構築関連
      - https://docs.docker.com/install/linux/docker-ce/centos/
      - https://kubernetes.io/docs/setup/independent/install-kubeadm/

以降、[公式手順書](https://kubernetes.io/docs/setup/independent/install-kubeadm/)/[HA](https://kubernetes.io/docs/setup/independent/high-availability/)に具体的手順が記載されていなかった部分や、やや相違ある部分は<font color=red>★</font>マークつける。

# Kubernetes構築

## LB構築 <font color=red>★</font>

まずはkubeapi endpointの冗長化から。[このサイト](https://mseeeen.msen.jp/build-cluster-with-keepalived-on-centos-7/)が分かりやすい。
※今回はkeepalivedを用いたものの、[ifcfgを弄る方法](https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c00946805)でも良いんじゃないかと思った

HAProxyについては[こちら](https://www.server-world.info/query?os=CentOS_7&p=haproxy)や[この辺](https://knowledge.sakura.ad.jp/8084/)を参考に設定した。

下記はLBノード（**lb1** `10.0.3.11`, **lb2** `10.0.3.12`）に対しての設定例。

```
yum -y update
yum -y install ntpdate haproxy keepalived
# 時間がサーバによってずれているとTLSハンドシェイクが失敗したりするので
ntpdate ntp.nict.jp
systemctl enable ntpdate
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
# keepalived
mv /etc/keepalived/keepalived.conf{,.org}
cat <<EOF > /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
   }
}

vrrp_instance keepalive {
    state BACKUP
    interface ens192
    virtual_router_id 10
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass arbitraryPassword
    }
    virtual_ipaddress {
        10.0.3.10
    }
}
EOF
# haproxy
mv /etc/haproxy/haproxy.cfg{,.org}
cat <<EOF > /etc/haproxy/haproxy.cfg
 global
    log         127.0.0.1 local2 info
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     256
    user        haproxy
    group       haproxy
    daemon

defaults
    mode               tcp
    log                global
    option             tcplog
    timeout connect    10s
    timeout client     30s
    timeout server     30s

frontend kubeapi6443
    bind *:6443
    default_backend    backend6443

frontend kubeapi10250
    bind *:10250
    default_backend backend10250

backend backend6443
    option redispatch
    retries 1
    balance            roundrobin
    server             m1 10.0.3.1:6443 check
    server             m2 10.0.3.2:6443 check
    server             m3 10.0.3.3:6443 check

backend backend10250
    option redispatch
    retries 1
    balance            roundrobin
    server             m1 10.0.3.1:10250 check
    server             m2 10.0.3.2:10250 check
    server             m3 10.0.3.3:10250 check
EOF
systemctl start keepalived haproxy
systemctl enable keepalived haproxy
firewall-cmd --zone=public --add-port=6443/tcp --add-port=10250/tcp --permanent
firewall-cmd --reload
```

構築後、`nc -v 10.0.3.10 6443`で、**lb** `10.0.3.10`まで疎通確認（Masterノード構築まではちゃんとは繋がらない）。

## Master/Workerノード共通

ほぼ[公式手順書](https://kubernetes.io/docs/setup/independent/install-kubeadm/)のまま。
Dockerの導入とipv4.ip_forwardの追加くらい。

```
yum -y update
yum -y install ntpdate yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
ntpdate ntp.nict.jp
systemctl start ntpdate docker
systemctl enable ntpdate docker
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet
```

## Masterノード(**m1** `10.0.3.1`) <font color=red>★</font>

[公式手順書](https://kubernetes.io/docs/setup/independent/install-kubeadm/)/[HA](https://kubernetes.io/docs/setup/independent/high-availability/)では開放対象ポートが足りなかったり、CNI(Weave)のURLがやや古かったり。

```
for x in {m,w}{1,2,3}; do echo for $x; ssh $x yes y ¥| ssh-keygen -q -t rsa -N '' ¥> /dev/null; ssh $x "cat >> .ssh/authorized_keys" < .ssh/id_rsa.pub; ssh $x firewall-cmd --zone=public --add-port={6443/tcp,2379/tcp,2380/tcp,10250/tcp,10251/tcp,10252/tcp,30000-32767/tcp,6783/tcp,6783/udp,6784/udp,53/udp} --add-masquerade --permanent; firewall-cmd --reload; systemctl start kubelet; done
cat <<EOF > kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "lb"
controlPlaneEndpoint: "lb:6443"
EOF
kubeadm init --config=kubeadm-config.yaml
kubectl apply -f https://cloud.weave.works/k8s/v1.10/net.yaml
kubectl get pod -n kube-system -w
cat <<EOF > copy.sh
USER=root # customizable
CONTROL_PLANE_IPS="m2 m3"
for host in ${CONTROL_PLANE_IPS}; do
    ssh $host mkdir -p /etc/kubernetes/pki/etcd
    scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/ca.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/sa.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:/etc/kubernetes/pki/etcd/
    scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/
    scp /etc/kubernetes/admin.conf "${USER}"@$host:/etc/kubernetes/
    ssh $host mkdir -p .kube
    ssh $host cp /etc/kubernetes/admin.conf .kube/config
done
EOF
chmod a+x copy.sh
./copy.sh
```

途中、`kubeadm init ...`実行時に得られた`kubeadm join lb:6443 ...`はメモしておく。

## Masterノード（その他）

```
kubeadm join lb:6443 ... --experimental-control-plane
```

## Workerノード

```
kubeadm join lb:6443 ...
```

## 構築後確認

`kubectl get pods -n kube-system`でRunning以外のステータスが上がる場合は正常に起動できていない。
`describe`で原因詳細を確認し対応するか、面倒なら当該Masterノード上で

```
systemctl stop kubelet
docker stop `docker ps | awk '{print $1}' | grep -v C`
docker rm `docker ps -a | awk '{print $1}' | grep -v C`
systemctl start kubelet
```

を叩くなどして一旦kubeletの初期化を試みてみる。

# MetalLB導入

```
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml
cat <<EOF > configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.4.1-10.0.4.254
EOF
kubectl apply -f configmap.yaml
kubectl get pods -n metallb-system -w
```

## controllerがContainerCreatingで止まる

手順に沿えば難しくないはずなのだが、controllerが下記`ContainerCreating`で止まる現象に悩まされた。

```
[root@m1 ~]# kubectl get pods -n metallb-system
NAME                          READY   STATUS              RESTARTS   AGE
controller-7cc9c87cfb-9q8rj   0/1     ContainerCreating   0          2m13s
speaker-b7k5m                 1/1     Running             0          2m13s
speaker-c9kb7                 1/1     Running             0          2m13s
speaker-qv8xb                 1/1     Running             0          2m13s
```

しばらく放置するとEventsに`rpc error: code = DeadlineExceeded desc = context deadline exceeded`の文字が。
`yum install docker`でCentOS公式リポジトリからdockerをインストールしていたのが原因だった。恥ずかしい。

```
[root@m1 ~]# kubectl describe pod controller-7d94c7b8c5-plrxh -n metallb-system
Name:               controller-7d94c7b8c5-plrxh
Namespace:          metallb-system
Priority:           0
PriorityClassName:  <none>
Node:               w2/10.0.3.5
Start Time:         Sun, 30 Dec 2018 17:21:37 +0900
Labels:             app=metallb
                    component=controller
                    pod-template-hash=7d94c7b8c5
Annotations:        prometheus.io/port: 7472
                    prometheus.io/scrape: true
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/controller-7d94c7b8c5
Containers:
  controller:
    Container ID:  
    Image:         metallb/controller:v0.7.2
    Image ID:      
    Port:          7472/TCP
    Host Port:     0/TCP
    Args:
      --port=7472
      --config=config
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from controller-token-m7x87 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  controller-token-m7x87:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  controller-token-m7x87
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age   From                      Message
  ----     ------                  ----  ----                      -------
  Normal   Scheduled               4m3s  default-scheduler         Successfully assigned metallb-system/controller-7d94c7b8c5-plrxh to w2
  Warning  FailedCreatePodSandBox  2s    kubelet, w2  Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   SandboxChanged          2s    kubelet, w2  Pod sandbox changed, it will be killed and re-created.
```

その他にも健全に起動していた*と思っていた*`kube-system + weave`も、下記コマンドを実行するとやたらに`dial tcp4 :0->10.0.3.1:6783: connect: no route to host, retry...`を吐く。

```
for i in $(kubectl get pods -n kube-system | grep weave | awk '{ print $1}'); do kubectl get pods $i -o wide -n kube-system; kubectl exec -n kube-system $i -c weave -- /home/weave/weave --local status connections; done
```

こちらも[Weave側のサイト](https://www.weave.works/docs/net/latest/install/using-weave/)に下記注記があったので自身の確認不足。

>
```
Note: If there is a firewall between $HOST1 and $HOST2, you must permit traffic to flow through TCP 6783 and UDP 6783/6784, which are Weave’s control and data ports.
```

## いつまで経ってもExternalIPがアサインされない

全てRunningになったのを確認した後、チュートリアルのnginxをデプロイしてみるも、いつまでたってもserviceにExternalIPが割り当てられず、pendingのままとなった。

下記コマンドを実行したところ53/udp宛通信破棄や`no route to host...`を確認したため、53/udpへのアクセスとmasquerade許可設定を追加したところ、解消した。

```
# firewall-cmd --set-log-denied all
# tail -100 /var/log/messages | grep REJECT | tail -1
Dec 30 20:17:55 m1 kernel: FINAL_REJECT: IN=weave OUT=weave PHYSIN=vethwe-bridge PHYSOUT=vethwepl51affd6 MAC=ee:f0:1f:bd:27:85:06:ae:ab:d3:75:25:08:00 SRC=10.0.3.4 DST=10.32.0.13 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=44486 PROTO=UDP SPT=39203 DPT=53 LEN=62 
# kubectl logs -l component=controller -n metallb-system
{"caller":"reflector.go:205","level":"error","msg":"go.universe.tf/metallb/internal/k8s/k8s.go:237: Failed to list *v1.ConfigMap: Get https://10.96.0.1:443/api/v1/namespaces/metallb-system/configmaps?fieldSelector=metadata.name%3Dconfig\u0026limit=500\u0026resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host","ts":"2018-12-30T11:05:13.674244Z"}
{"caller":"reflector.go:205","level":"error","msg":"go.universe.tf/metallb/internal/k8s/k8s.go:231: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500\u0026resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host","ts":"2018-12-30T11:05:13.674466Z"}
```