オンプレでもIngressする

L7のロードバランサをオンプレ(ベアメタル)のk8sで使うには、いろいろ設定せねばならぬ。k8s本体にはそんな機能はないんだよ。
CNIにCanal, metallb(L2 modeとして), L7LBにnginx-ingress-controllerを使う。

nginx-ingress-controllerを入れて設定しておかないとKind Ingressのyamlを食わせてもうまく動かないので注意。
※クラウドとかでEKSやGKEなどを使っている場合には良しなにやってくれるので必要なし。
※minikubeの人はアドオンがあるので別途ググってください。簡単です。

# TL;DR

v1.19.2で試しています。
Calico for policy and flannel (aka Canal), metal-lb をインストールして Ingress nginxを設定して動作確認しています。



# 初期セットアップ



https://qiita.com/murata-tomohide/items/be2171ac459f8f7d54f0 を参考にセットアップしておきます。 https://qiita.com/murata-tomohide/items/be2171ac459f8f7d54f0#%E5%8B%95%E4%BD%9C%E7%A2%BA%E8%AA%8D この辺りまでできていればおｋ。

Canalまで終わったところだとこんな感じ。

```
murata:~ $ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}


murata:~ $ kubectl get all -A
NAMESPACE     NAME                                                    READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-kube-controllers-c9784d67d-49g6t             1/1     Running   0          2m20s
kube-system   pod/canal-wwmvf                                         2/2     Running   0          2m20s
kube-system   pod/coredns-f9fd979d6-ccgm5                             1/1     Running   0          7m43s
kube-system   pod/coredns-f9fd979d6-ztwgs                             1/1     Running   0          7m43s
kube-system   pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running   0          7m53s
kube-system   pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running   0          7m53s
kube-system   pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running   0          7m53s
kube-system   pod/kube-proxy-rfd7c                                    1/1     Running   0          7m43s
kube-system   pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running   0          7m53s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  8m2s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   8m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/canal        1         1         1       1            1           kubernetes.io/os=linux   2m21s
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   8m

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           2m21s
kube-system   deployment.apps/coredns                   2/2     2            2           8m

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-c9784d67d   1         1         1       2m21s
kube-system   replicaset.apps/coredns-f9fd979d6                   2         2         2       7m43s
```

corednsのPodが動いていることと、ホストのNICが増えていることを確認します。(CanalはFlannnelとcaricoを使うらしい（よくわかってない）

```
murata:~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:81:7f:82 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 172.30.203.203/24 brd 172.30.203.255 scope global noprefixroute ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe81:7f82/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:ec:bb:c4:47 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether ca:7e:00:d4:d8:c7 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::c87e:ff:fed4:d8c7/64 scope link
       valid_lft forever preferred_lft forever
7: cali700ca5a0c12@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
8: cali70c270b17c5@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
9: cali7867aceb346@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
```


# MetalLB インストール

MetalLBについてはこちら
[KubernetesロードバランサーのMetalLBを導入した話(Necoプロジェクト体験入部)](https://blog.cybozu.io/entry/2019/03/25/093000)

要するにTypeがLoadBalancerの場合にIPアドレスを付与してくれるやつ。

https://metallb.universe.tf/installation/
の通りにやっていきます。

## kube-proxyの設定を変える

editで変えても下にあるシェルを流してもOKっぽいです。今回はシェルを流しました。

```
murata:~ $ # see what changes would be made, returns nonzero returncode if different
ube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl diff -f - -n kube-systemmurata:~ $ kubectl get configmap kube-proxy -n kube-system -o yaml | \
> sed -e "s/strictARP: false/strictARP: true/" | \
> kubectl diff -f - -n kube-system
diff -u -N /tmp/LIVE-375254102/v1.ConfigMap.kube-system.kube-proxy /tmp/MERGED-128212413/v1.ConfigMap.kube-system.kube-proxy
--- /tmp/LIVE-375254102/v1.ConfigMap.kube-system.kube-proxy     2020-10-05 16:55:11.169956454 +0900
+++ /tmp/MERGED-128212413/v1.ConfigMap.kube-system.kube-proxy   2020-10-05 16:55:11.175956620 +0900
@@ -30,7 +30,7 @@
       excludeCIDRs: null
       minSyncPeriod: 0s
       scheduler: ""
-      strictARP: false
+      strictARP: true
       syncPeriod: 0s
       tcpFinTimeout: 0s
       tcpTimeout: 0s
@@ -79,7 +79,6 @@
     fieldsV1:
       f:data:
         .: {}
-        f:config.conf: {}
         f:kubeconfig.conf: {}
       f:metadata:
         f:annotations:
@@ -91,6 +90,14 @@
     manager: kubeadm
     operation: Update
     time: "2020-10-05T07:44:04Z"
+  - apiVersion: v1
+    fieldsType: FieldsV1
+    fieldsV1:
+      f:data:
+        f:config.conf: {}
+    manager: kubectl-client-side-apply
+    operation: Update
+    time: "2020-10-05T07:55:11Z"
   name: kube-proxy
   namespace: kube-system
   resourceVersion: "191"
murata:~ $ kubectl get configmap kube-proxy -n kube-system -o yaml | \
> sed -e "s/strictARP: false/strictARP: true/" | \
> kubectl apply -f - -n kube-system
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap/kube-proxy configured
```

## マニフェストを食う

完全にコピペです。

```
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
```

こんな感じです。

```
murata:~ $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
y -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
namespace/metallb-system created
murata:~ $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
murata:~ $ # On first install only
murata:~ $ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created
```




## 利用するIPアドレスのconfigmapを作ります。

https://metallb.universe.tf/configuration/#layer-2-configuration
Layer 2 Configuration の部分を行います。

addressの部分は設定時にバインドできるIPアドレスにします。今回はノードのIPアドレスの帯域で空いているところ。
※そのIPアドレスでクライアントPCからアクセスします。

この場合ノードサーバのIPアドレスは172.30.203.203なので、空いていた172.30.203.203-172.30.203.205を設定しました。（必ず使っていないことを確認すること！＆自分自身はいけるみたい？
アドレスのところ、うまくバリデーションしてくれないので注意してください。変にコメント入れると動かない気がします。

````
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.30.203.203-172.30.203.205

EOF
```


現段階ではこんな感じになる。

```
murata:~ $ kubectl get all,configmaps -A
NAMESPACE        NAME                                                    READY   STATUS    RESTARTS   AGE
kube-system      pod/calico-kube-controllers-c9784d67d-49g6t             1/1     Running   0          14m
kube-system      pod/canal-wwmvf                                         2/2     Running   0          14m
kube-system      pod/coredns-f9fd979d6-ccgm5                             1/1     Running   0          19m
kube-system      pod/coredns-f9fd979d6-ztwgs                             1/1     Running   0          19m
kube-system      pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running   0          19m
kube-system      pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running   0          19m
kube-system      pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running   0          19m
kube-system      pod/kube-proxy-rfd7c                                    1/1     Running   0          19m
kube-system      pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running   0          19m
metallb-system   pod/controller-fb659dc8-75t6h                           1/1     Running   0          6m18s
metallb-system   pod/speaker-jb8rb                                       1/1     Running   0          6m18s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  19m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   19m

NAMESPACE        NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system      daemonset.apps/canal        1         1         1       1            1           kubernetes.io/os=linux        14m
kube-system      daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux        19m
metallb-system   daemonset.apps/speaker      1         1         1       1            1           beta.kubernetes.io/os=linux   6m18s

NAMESPACE        NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system      deployment.apps/calico-kube-controllers   1/1     1            1           14m
kube-system      deployment.apps/coredns                   2/2     2            2           19m
metallb-system   deployment.apps/controller                1/1     1            1           6m18s

NAMESPACE        NAME                                                DESIRED   CURRENT   READY   AGE
kube-system      replicaset.apps/calico-kube-controllers-c9784d67d   1         1         1       14m
kube-system      replicaset.apps/coredns-f9fd979d6                   2         2         2       19m
metallb-system   replicaset.apps/controller-fb659dc8                 1         1         1       6m18s

NAMESPACE        NAME                                           DATA   AGE
kube-public      configmap/cluster-info                         2      19m
kube-system      configmap/canal-config                         6      14m
kube-system      configmap/coredns                              1      19m
kube-system      configmap/extension-apiserver-authentication   6      19m
kube-system      configmap/kube-proxy                           2      19m
kube-system      configmap/kubeadm-config                       2      19m
kube-system      configmap/kubelet-config-1.19                  1      19m
metallb-system   configmap/config                               1      2m37s
```


# nginx-ingress-controller インストール

下記のマニフェストを利用します。BareMetal用だとNordPortになってしまうためちゃんと動きません。
https://kubernetes.github.io/ingress-nginx/deploy/#docker-for-mac


```
murata:~ $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.1/deploy/static/provider/cloud/deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
```


Ingressが産まれています。
Serviceの `ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.99.216.235   172.30.203.203   80:31980/TCP,443:32708/TCP   30s` 
でEXTERNAL-IPにちゃんとIPアドレスが入っているかを確認してください。

```
murata:~ $ kubectl get all,ing,configmaps -A
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE        NAME                                                    READY   STATUS      RESTARTS   AGE
ingress-nginx    pod/ingress-nginx-admission-create-hfs2f                0/1     Completed   0          30s
ingress-nginx    pod/ingress-nginx-admission-patch-xdg52                 0/1     Completed   1          30s
ingress-nginx    pod/ingress-nginx-controller-59859f77c7-xnvcf           1/1     Running     0          30s
kube-system      pod/calico-kube-controllers-c9784d67d-49g6t             1/1     Running     0          16m
kube-system      pod/canal-wwmvf                                         2/2     Running     0          16m
kube-system      pod/coredns-f9fd979d6-ccgm5                             1/1     Running     0          22m
kube-system      pod/coredns-f9fd979d6-ztwgs                             1/1     Running     0          22m
kube-system      pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running     0          22m
kube-system      pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running     0          22m
kube-system      pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running     0          22m
kube-system      pod/kube-proxy-rfd7c                                    1/1     Running     0          22m
kube-system      pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running     0          22m
metallb-system   pod/controller-fb659dc8-75t6h                           1/1     Running     0          8m56s
metallb-system   pod/speaker-jb8rb                                       1/1     Running     0          8m56s

NAMESPACE       NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default         service/kubernetes                           ClusterIP      10.96.0.1       <none>           443/TCP                      22m
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.99.216.235   172.30.203.203   80:31980/TCP,443:32708/TCP   30s
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.99.223.96    <none>           443/TCP                      30s
kube-system     service/kube-dns                             ClusterIP      10.96.0.10      <none>           53/UDP,53/TCP,9153/TCP       22m

NAMESPACE        NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system      daemonset.apps/canal        1         1         1       1            1           kubernetes.io/os=linux        16m
kube-system      daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux        22m
metallb-system   daemonset.apps/speaker      1         1         1       1            1           beta.kubernetes.io/os=linux   8m56s

NAMESPACE        NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx    deployment.apps/ingress-nginx-controller   1/1     1            1           30s
kube-system      deployment.apps/calico-kube-controllers    1/1     1            1           16m
kube-system      deployment.apps/coredns                    2/2     2            2           22m
metallb-system   deployment.apps/controller                 1/1     1            1           8m56s

NAMESPACE        NAME                                                  DESIRED   CURRENT   READY   AGE
ingress-nginx    replicaset.apps/ingress-nginx-controller-59859f77c7   1         1         1       30s
kube-system      replicaset.apps/calico-kube-controllers-c9784d67d     1         1         1       16m
kube-system      replicaset.apps/coredns-f9fd979d6                     2         2         2       22m
metallb-system   replicaset.apps/controller-fb659dc8                   1         1         1       8m56s

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           2s         30s
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           4s         30s

NAMESPACE        NAME                                           DATA   AGE
ingress-nginx    configmap/ingress-controller-leader-nginx      0      17s
ingress-nginx    configmap/ingress-nginx-controller             0      30s
kube-public      configmap/cluster-info                         2      22m
kube-system      configmap/canal-config                         6      16m
kube-system      configmap/coredns                              1      22m
kube-system      configmap/extension-apiserver-authentication   6      22m
kube-system      configmap/kube-proxy                           2      22m
kube-system      configmap/kubeadm-config                       2      22m
kube-system      configmap/kubelet-config-1.19                  1      22m
metallb-system   configmap/config                               1      5m15s
```





# 動作確認

試しにこんな感じのマニフェストを食わせる。
/apacheでapacheに、/nginxでnginxに行く。

※ Kubernetes 1.19からIngressAPIがGAとなりました。併せて書式がちょっと変わっているので注意してください。
https://qiita.com/tkusumi/items/ee1e100d34465ff2d764#whats-new-%E6%96%B0%E6%83%85%E5%A0%B1

↓は1.18までのものです。

```
cat <<EOF | kubectl apply -f -

# namespace
apiVersion: v1
kind: Namespace
metadata:
  name: ns-test
---
# ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: lb
  namespace: ns-test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
    - host: ns-test.172.30.203.203.nip.io
      http:
        paths:
          - path: /apache
            backend:
              serviceName: apache-svc
              servicePort: 80
          - path: /nginx
            backend:
              serviceName: nginx-svc
              servicePort: 80
          - path: /
            backend:
              serviceName: blackhole
              servicePort: 80
---
# apache
apiVersion: v1
kind: Service
metadata:
  name: apache-svc
  namespace: ns-test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpd
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - image: httpd:alpine
        name: httpd
        ports:
        - containerPort: 80
---
# nginx
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: ns-test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:alpine
        name: nginx
        ports:
        - containerPort: 80

EOF

```


ServiceとIngressのところにIPアドレスが入っています。

```
murata:~ $ kubectl get all,ing -A
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE        NAME                                                    READY   STATUS      RESTARTS   AGE
ingress-nginx    pod/ingress-nginx-admission-create-hfs2f                0/1     Completed   0          5m28s
ingress-nginx    pod/ingress-nginx-admission-patch-xdg52                 0/1     Completed   1          5m28s
ingress-nginx    pod/ingress-nginx-controller-59859f77c7-xnvcf           1/1     Running     0          5m28s
kube-system      pod/calico-kube-controllers-c9784d67d-49g6t             1/1     Running     0          21m
kube-system      pod/canal-wwmvf                                         2/2     Running     0          21m
kube-system      pod/coredns-f9fd979d6-ccgm5                             1/1     Running     0          27m
kube-system      pod/coredns-f9fd979d6-ztwgs                             1/1     Running     0          27m
kube-system      pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running     0          27m
kube-system      pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running     0          27m
kube-system      pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running     0          27m
kube-system      pod/kube-proxy-rfd7c                                    1/1     Running     0          27m
kube-system      pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running     0          27m
metallb-system   pod/controller-fb659dc8-75t6h                           1/1     Running     0          13m
metallb-system   pod/speaker-jb8rb                                       1/1     Running     0          13m
ns-test          pod/httpd-6cd65b68c6-r4cpx                              1/1     Running     0          2m17s
ns-test          pod/nginx-7fb7fd49b4-wq7zl                              1/1     Running     0          2m17s

NAMESPACE       NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default         service/kubernetes                           ClusterIP      10.96.0.1       <none>           443/TCP                      27m
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.99.216.235   172.30.203.203   80:31980/TCP,443:32708/TCP   5m28s
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.99.223.96    <none>           443/TCP                      5m28s
kube-system     service/kube-dns                             ClusterIP      10.96.0.10      <none>           53/UDP,53/TCP,9153/TCP       27m
ns-test         service/apache-svc                           NodePort       10.103.168.10   <none>           80:32324/TCP                 2m17s
ns-test         service/nginx-svc                            NodePort       10.104.26.107   <none>           80:32278/TCP                 2m17s

NAMESPACE        NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system      daemonset.apps/canal        1         1         1       1            1           kubernetes.io/os=linux        21m
kube-system      daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux        27m
metallb-system   daemonset.apps/speaker      1         1         1       1            1           beta.kubernetes.io/os=linux   13m

NAMESPACE        NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx    deployment.apps/ingress-nginx-controller   1/1     1            1           5m28s
kube-system      deployment.apps/calico-kube-controllers    1/1     1            1           21m
kube-system      deployment.apps/coredns                    2/2     2            2           27m
metallb-system   deployment.apps/controller                 1/1     1            1           13m
ns-test          deployment.apps/httpd                      1/1     1            1           2m17s
ns-test          deployment.apps/nginx                      1/1     1            1           2m17s

NAMESPACE        NAME                                                  DESIRED   CURRENT   READY   AGE
ingress-nginx    replicaset.apps/ingress-nginx-controller-59859f77c7   1         1         1       5m28s
kube-system      replicaset.apps/calico-kube-controllers-c9784d67d     1         1         1       21m
kube-system      replicaset.apps/coredns-f9fd979d6                     2         2         2       27m
metallb-system   replicaset.apps/controller-fb659dc8                   1         1         1       13m
ns-test          replicaset.apps/httpd-6cd65b68c6                      1         1         1       2m17s
ns-test          replicaset.apps/nginx-7fb7fd49b4                      1         1         1       2m17s

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           2s         5m28s
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           4s         5m28s

NAMESPACE   NAME                    CLASS    HOSTS                           ADDRESS          PORTS   AGE
ns-test     ingress.extensions/lb   <none>   ns-test.172.30.203.203.nip.io   172.30.203.203   80      2m17s
```




ほかのPCから動作確認。

```
murata:~ $ curl http://ns-test.172.30.203.203.nip.io/apache
<html><body><h1>It works!</h1></body></html>
murata:~ $ curl http://ns-test.172.30.203.203.nip.io/nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
```

apacheはapacheに、nginxはnginxにバランシングされています。（ヘッダーはIngressのものになってしまうのでペイロード部分で確認した



# まとめ

nginx-ingress-controllerはDaemonSetでありServiceでもIngressでもないので気を付けて。
複数台のノードでクラスタを組んでいる場合には特定のノード(router node)にIngressコントローラーを配置したかったりするかもしれません。その場合にはyamlファイルを編集すればOKみたいです。(Router Node)
SSL設定などもありますので、↑をそのまま使うのはまだまだかな。


それでは！