[06] オンプレ環境で VM 2台を使って 1マスタ・1ノードの kubernetes を構築してみる (L2 MetalLB の導入)



# 概要

下表の構成で、オンプレ上に k8s を構築したときの記録である.
本記事では「**L2 MetalLB の導入**」を記す.


|No|用途|ノード名|形態|公開IP|内部IP|OS|備考|
|:--|:--|:--|:--|:--|:--|:--|:--|
|1|k8sマスタ|master01|VM|192.168.1.91|172.24.20.11|Ubuntu18.04||
|2|k8sノード|node01|VM|192.168.1.92|172.24.20.12|Ubuntu18.04||


# 参考にしたサイトおよび書籍

|URL|備考|
|:--|:--|
|[実践 Vagrant](https://www.oreilly.co.jp/books/9784873116655/)||
|[15Stepで習得 Dockerから入るKubernetes](http://www.ric.co.jp/book/contents/book_1161.html)|K8s だけでなく、Ansible, Vagrant, GlusterFS のことなども学べる.|
|https://github.com/takara9/vagrant-k8s|[『15Stepで習得 Dockerから入るKubernetes』](http://www.ric.co.jp/book/contents/book_1161.html)の著者が公開されている GitHub.<br/>Vagrant や Ansible コードを公開してくださっている.|
|https://github.com/takara9/vagrant-kubernetes|同上|
|https://github.com/takara9/codes_for_lessons|同上|
|https://nextpublishing.jp/book/12197.html|『解体kubeadm　フェーズから読み解くKubernetesクラスタ構築ツールの全貌』を参考にして 1マスタ・1ノードを構築した.|
|https://www.kaitoy.xyz/2020/10/31/metallb/||



# 環境

## 物理PC および 仮想PC の OS

```bash
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
```

## 物理PC

下記ソフトを導入済みであること.

- Vagrant 
- VirtualBox



# 手順

## 1. master01 に ssh ログインする

```bash
$ vagrant ssh master01
```

## 2. MetalLB マニフェストを入手する

現時点 (2021.05.26) で最新バージョンである 0.9.6 の MetalLB マニフェストを入手する

```bash
vagrant@master01:~$ wget https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
vagrant@master01:~$ wget https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
```

## 3. MetalLB マニフェストを適用する

ここの手順を実行しても、まだ MetalLB は動かない.

```bash
vagrant@master01:~$ kubectl apply -f namespace.yaml
namespace/metallb-system created
```
```bash
vagrant@master01:~$ kubectl apply -f metallb.yaml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
```
```bash
vagrant@master01:~$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created
```

## 4. MetalLB の L2 用マニフェストを作成する
 
[MetalLB 公式サイトの Layer 2 Configuration](https://metallb.universe.tf/configuration/#layer-2-configuration) から次のように マニフェストを作成する

### metallb-config.yml

今回は、172.24.20.245 - 250 までの 6個の LB が導入できるようにしておく.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.24.20.245-172.24.20.250
```

## 5. MetalLB の L2 用マニフェストを適用する

Pod「MetalLB」を作成する

```bash
vagrant@master01:~$ kubectl apply -f metallb-config.yml
configmap/config created
```

Pod「MetalLB」の状態を見る (下記🛑)

```bash
vagrant@master01:~$ kubectl get pod -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
kube-system            coredns-558bd4d5db-86gdp                     1/1     Running   0          94m
kube-system            coredns-558bd4d5db-9wj98                     1/1     Running   0          94m
kube-system            etcd-master01                                1/1     Running   0          94m
kube-system            kube-apiserver-master01                      1/1     Running   0          94m
kube-system            kube-controller-manager-master01             1/1     Running   1          94m
kube-system            kube-flannel-ds-mvlx9                        1/1     Running   2          90m
kube-system            kube-flannel-ds-n9zbw                        1/1     Running   0          94m
kube-system            kube-proxy-h9hwt                             1/1     Running   0          90m
kube-system            kube-proxy-qrng2                             1/1     Running   0          94m
kube-system            kube-scheduler-master01                      1/1     Running   1          94m
kube-system            metrics-server-755675c897-h86pc              1/1     Running   0          94m
kubernetes-dashboard   cluster-admin-68f7c6c68f-l8hr4               1/1     Running   0          94m
kubernetes-dashboard   dashboard-metrics-scraper-778b77d469-9xzl9   1/1     Running   0          94m
metallb-system 🛑      controller-64f86798cc-8f6s4                  1/1     Running   0          9m6s
metallb-system 🛑      speaker-tshfh                                1/1     Running   0          9m6s
metallb-system 🛑      speaker-xv26j                                1/1     Running   0          9m6s
```


## 6. MetalLB を使ってみる(1/2) ... Pod「Nginx」を起動させる

外部からアクセスしたい Pod「Nginx」を立てる
　
### 6-1. test-metallb-1.yml を作成する

https://www.kaitoy.xyz/2020/10/31/metallb/ からの転記である.

```yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: psp:priv
rules:
- apiGroups:
  - policy
  resourceNames:
  - privileged
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default:psp:privileged
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: psp:priv
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.1-alpine
        ports:
        - containerPort: 80
```

### 6-2. test-metallb-1.yml を適用する

```bash
vagrant@master01:~$ kubectl apply -f test-metallb-1.yml
role.rbac.authorization.k8s.io/psp:priv created
rolebinding.rbac.authorization.k8s.io/default:psp:privileged created
deployment.apps/nginx-deployment created
```


### 6-3. Pod「Nginx」が稼動していることを見る

```bash
vagrant@master01:~/metallb$ kubectl get pod -A
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
default                nginx-deployment-79d8d59989-q4sff            0/1     ContainerCreating   0          9s  🆗
default                nginx-deployment-79d8d59989-t2gxk            0/1     ContainerCreating   0          9s  🆗
kube-system            coredns-558bd4d5db-86gdp                     1/1     Running             0          95m
kube-system            coredns-558bd4d5db-9wj98                     1/1     Running             0          95m
kube-system            etcd-master01                                1/1     Running             0          95m
kube-system            kube-apiserver-master01                      1/1     Running             0          95m
kube-system            kube-controller-manager-master01             1/1     Running             1          95m
kube-system            kube-flannel-ds-mvlx9                        1/1     Running             2          92m
kube-system            kube-flannel-ds-n9zbw                        1/1     Running             0          95m
kube-system            kube-proxy-h9hwt                             1/1     Running             0          92m
kube-system            kube-proxy-qrng2                             1/1     Running             0          95m
kube-system            kube-scheduler-master01                      1/1     Running             1          95m
kube-system            metrics-server-755675c897-h86pc              1/1     Running             0          95m
kubernetes-dashboard   cluster-admin-68f7c6c68f-l8hr4               1/1     Running             0          95m
kubernetes-dashboard   dashboard-metrics-scraper-778b77d469-9xzl9   1/1     Running             0          95m
metallb-system         controller-64f86798cc-8f6s4                  1/1     Running             0          10m
metallb-system         speaker-tshfh                                1/1     Running             0          10m
metallb-system         speaker-xv26j                                1/1     Running             0          10m
```

`-o wide` を指定して確認してみる.

```bash
vagrant@master01:~/metallb$ kubectl get pod -A -o wide
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
default 🆗             nginx-deployment-79d8d59989-q4sff            1/1     Running   0          55s   10.244.1.4     node01     <none>           <none>
default 🆗             nginx-deployment-79d8d59989-t2gxk            1/1     Running   0          55s   10.244.1.5     node01     <none>           <none>
kube-system            coredns-558bd4d5db-86gdp                     1/1     Running   0          96m   10.244.0.3     master01   <none>           <none>
kube-system            coredns-558bd4d5db-9wj98                     1/1     Running   0          96m   10.244.0.2     master01   <none>           <none>
kube-system            etcd-master01                                1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-apiserver-master01                      1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-controller-manager-master01             1/1     Running   1          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-flannel-ds-mvlx9                        1/1     Running   2          92m   172.24.20.12   node01     <none>           <none>
kube-system            kube-flannel-ds-n9zbw                        1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-proxy-h9hwt                             1/1     Running   0          92m   172.24.20.12   node01     <none>           <none>
kube-system            kube-proxy-qrng2                             1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-scheduler-master01                      1/1     Running   1          96m   172.24.20.11   master01   <none>           <none>
kube-system            metrics-server-755675c897-h86pc              1/1     Running   0          96m   10.244.1.2     node01     <none>           <none>
kubernetes-dashboard   cluster-admin-68f7c6c68f-l8hr4               1/1     Running   0          96m   10.244.0.4     master01   <none>           <none>
kubernetes-dashboard   dashboard-metrics-scraper-778b77d469-9xzl9   1/1     Running   0          96m   10.244.0.5     master01   <none>           <none>
metallb-system         controller-64f86798cc-8f6s4                  1/1     Running   0          11m   10.244.1.3     node01     <none>           <none>
metallb-system         speaker-tshfh                                1/1     Running   0          11m   172.24.20.12   node01     <none>           <none>
metallb-system         speaker-xv26j                                1/1     Running   0          11m   172.24.20.11   master01   <none>           <none>
```

## 7. MetalLB を使ってみる(2/2) ... LoadBalancer Service を動かす

### 7-1. Service「MetallLB」用のマニフェスト「metallb-service.yml」を作成する

```yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80
  type: LoadBalancer
```


### 7-2. 現在の Service の状態を見ておく

```bash
vagrant@master01:~/metallb$ kubectl get svc -A -o wide 
NAMESPACE              NAME                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE    SELECTOR
default                kubernetes                  ClusterIP   10.32.0.1     <none>        443/TCP                  100m   <none>
kube-system            kube-dns                    ClusterIP   10.32.0.10    <none>        53/UDP,53/TCP,9153/TCP   100m   k8s-app=kube-dns
kube-system            metrics-server              ClusterIP   10.32.0.51    <none>        443/TCP                  100m   k8s-app=metrics-server
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.32.0.215   <none>        8000/TCP                 100m   k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.32.0.70    <none>        443/TCP                  100m   k8s-app=kubernetes-dashboard
```

### 7-3. Service「MetallLB」用のマニフェストを適用する

```bash
vagrant@master01:~$ kubectl apply -f metallb-service.yml
service/nginx created
```

### 7-4. 適用後の Service の状態を見ておく

LoadBalancer「nginx」が追加された

```bash
vagrant@master01:~/metallb$ kubectl get svc -A -o wide 
NAMESPACE              NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                  AGE    SELECTOR
default                kubernetes                  ClusterIP      10.32.0.1     <none>          443/TCP                  101m   <none>
default 🆗             nginx                       LoadBalancer   10.32.0.188   172.24.20.245   8080:32005/TCP           3s     app=nginx
kube-system            kube-dns                    ClusterIP      10.32.0.10    <none>          53/UDP,53/TCP,9153/TCP   101m   k8s-app=kube-dns
kube-system            metrics-server              ClusterIP      10.32.0.51    <none>          443/TCP                  101m   k8s-app=metrics-server
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP      10.32.0.215   <none>          8000/TCP                 101m   k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard        ClusterIP      10.32.0.70    <none>          443/TCP                  101m   k8s-app=kubernetes-dashboard
```

**describe を使って詳細を見てみる.**

下記 🆗 の通り、LoadBalancer Service「nginx」に対して、
EXTERNAL-IP `172.24.20.245` が割り当てられた.

```bash
vagrant@master01:~/metallb$ kubectl describe svc nginx
Name:                     nginx
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.32.0.188
IPs:                      10.32.0.188
LoadBalancer Ingress:     172.24.20.245    🆗
Port:                     <unset>  8080/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  32005/TCP
Endpoints:                10.244.1.4:80,10.244.1.5:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age   From                Message
  ----    ------        ----  ----                -------
  Normal  IPAllocated   54s   metallb-controller  Assigned IP "172.24.20.245"
  Normal  nodeAssigned  53s   metallb-speaker     announcing from node "node01"
```

　

## 8. 動作確認をする

### 8-1. Service「LoadBalancer」の状態を見る

次の通り クラスタ外に対して 172.24.20.245 が公開できた

```bash
vagrant@master01:~/metallb$ kubectl get svc -A -o wide 
NAMESPACE              NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                  AGE    SELECTOR
default                kubernetes                  ClusterIP      10.32.0.1     <none>          443/TCP                  101m   <none>
default 🆗             nginx                       LoadBalancer   10.32.0.188   172.24.20.245   8080:32005/TCP           3s     app=nginx
kube-system            kube-dns                    ClusterIP      10.32.0.10    <none>          53/UDP,53/TCP,9153/TCP   101m   k8s-app=kube-dns
kube-system            metrics-server              ClusterIP      10.32.0.51    <none>          443/TCP                  101m   k8s-app=metrics-server
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP      10.32.0.215   <none>          8000/TCP                 101m   k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard        ClusterIP      10.32.0.70    <none>          443/TCP                  101m   k8s-app=kubernetes-dashboard
```


### 8-2. クラスタ外 (ここでは物理ホスト 192.168.10.115) から 1 の EXTERNAL-IP の 8080 にアクセスする

#### クラスタ外から wget を実行してみる

ここでは物理ホスト上での操作である.
L4 LoadBalancer の IP:PORT は 172.24.20.245:8080 である.

```bash
$ wget http://172.24.20.245:8080 -O -
--2021-05-29 11:30:29--  http://172.24.20.245:8080/
Connecting to 172.24.20.245:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: ‘STDOUT’

-                                                     0%[                                                                                                                   ]       0  --.-KB/s               <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
-                                                   100%[==================================================================================================================>]     612  --.-KB/s    in 0s      

2021-05-29 11:30:29 (103 MB/s) - written to stdout [612/612]
```

#### Web Browser からのアクセスをしてみる

・`http://172.24.20.245` にアクセスし、Nginx のトップページが表示された.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1246056/269c35d4-46f1-56d5-871b-39416618a8f0.png)