Calico と CoreDNS で Kubernetes Cluster 外から サービス名でアクセス


#####● 何をしたいのか。
1. Calico で Pod と同じように Service を扱いたい。
2. CoreDNS の名前解決を外部から使いたい。

#####● 事前準備
● Kubernetes 環境
KVM環境下に Master 1 , Node 2 で構築
CNI は、 Calico

```
$ kubectl get nodes
NAME                 STATUS   ROLES    AGE    VERSION
070master.testk.jp   Ready    master   5d2h   v1.16.1
071node01.testk.jp   Ready    <none>   5d1h   v1.16.1
072node02.testk.jp   Ready    <none>   5d1h   v1.16.1

```

● Nginx を Deployment で配置し、サービスを登録

```
$ kubectl get deployment -n apptest
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
deployment01   2/2     2            2           43h
$ kubectl get pod -n apptest
NAME                            READY   STATUS    RESTARTS   AGE
deployment01-7b4b9477fb-9z8ws   1/1     Running   2          43h
deployment01-7b4b9477fb-ff652   1/1     Running   2          43h
$ kubectl get svc -n apptest
NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service01   ClusterIP   10.20.71.66     <none>        80/TCP    43h

```

● kube-dns を確認

```
$ kubectl get svc -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.20.0.10   <none>        53/UDP,53/TCP,9153/TCP   5d3h
```

● Advertising Kubernetes services over BGP
下記のページを参照し、設定
[Advertising Kubernetes services over BGP](https://docs.projectcalico.org/v3.9/networking/service-advertisement#about-advertising-kubernetes-services-over-bgp)
kubeadm init 実行時のオプション
--service-cidr=10.20.0.0/16

```
$ kubectl describe ds/calico-node -n kube-system | grep CALICO_ADVERTISE_CLUSTER_IPS
      CALICO_ADVERTISE_CLUSTER_IPS:       10.20.0.0/16
```

● BGP ルータ 2 を用意

```
router1
# birdc show status
BIRD 2.0.4 ready.
BIRD 2.0.4
Router ID is 10.0.0.51
Current server time is 2019-10-19 17:48:01.962
Last reboot on 2019-10-19 12:07:21.365
Last reconfiguration on 2019-10-19 12:07:21.365
Daemon is up and running

router2
# birdc show status
BIRD 2.0.4 ready.
BIRD 2.0.4
Router ID is 10.0.0.52
Current server time is 2019-10-19 17:48:19.236
Last reboot on 2019-10-19 12:08:14.849
Last reconfiguration on 2019-10-19 12:08:14.849
Daemon is up and running

```

● /etc/resolv.conf に nameserver の追加
router2 は、router1 を指定

```
router1
# cat /etc/resolv.conf

nameserver 10.20.0.10
nameserver 192.168.122.1

router2
# cat /etc/resolv.conf

nameserver 10.0.0.51
nameserver 192.168.122.1

```

● router1 に Dnsmasq をインストール

```
# systemctl status dnsmasq
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; disabled; vendor preset: disabled)
   Active: active (running) since 土 2019-10-19 12:52:09 JST; 4h 59min ago
 Main PID: 1286 (dnsmasq)
   CGroup: /system.slice/dnsmasq.service
           └─1286 /usr/sbin/dnsmasq -k --auth-peer=10.20.0.10

```

● BGP Peer の確認

```
Kubernetes側
$ calicoctl get BGPPeer -o wide
NAME        PEERIP      NODE                 ASN
bgppeer00   10.0.0.51   070master.testk.jp   65051
bgppeer01   10.0.0.70   071node01.testk.jp   65400
bgppeer02   10.0.0.70   072node02.testk.jp   65400

router1 kube が Kubernetes側との接続を示す。
# birdc show protocol
BIRD 2.0.4 ready.
Name       Proto      Table      State  Since         Info
kube       BGP        ---        up     12:07:21.852  Established
master     BGP        ---        up     12:08:20.255  Established

router2
# birdc show protocol
BIRD 2.0.4 ready.
Name       Proto      Table      State  Since         Info
master     BGP        ---        up     12:08:19.359  Established

```

これで事前準備は終わり

#####● 調査開始
調査項目
router1 と router2 で下記を確認
1. service01 の CLUSTER-IP からサービス名を取得
2. service01 の サービス名から CLUSTER-IP を取得
3. service01 の CLUSTER-IP へ curl で HTTPリクエスト
4. service01 の サービス名 へ curl で HTTPリクエスト

```
router1
1.
# dig -4 -x 10.20.71.66

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> -4 -x 10.20.71.66
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6203
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;66.71.20.10.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
66.71.20.10.in-addr.arpa. 30	IN	PTR	service01.apptest.svc.cluster.local.

;; Query time: 1 msec
;; SERVER: 10.20.0.10#53(10.20.0.10)
;; WHEN: 土 10月 19 18:25:44 JST 2019
;; MSG SIZE  rcvd: 126

2.
# dig service01.apptest.svc.cluster.local

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> service01.apptest.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40918
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;service01.apptest.svc.cluster.local. IN	A

;; ANSWER SECTION:
service01.apptest.svc.cluster.local. 30	IN A	10.20.71.66

;; Query time: 0 msec
;; SERVER: 10.20.0.10#53(10.20.0.10)
;; WHEN: 土 10月 19 18:26:22 JST 2019
;; MSG SIZE  rcvd: 115

3.
# curl -I 10.20.71.66
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:26:52 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

4.
# curl -I service01.apptest.svc.cluster.local
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:27:49 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

```
router1 に間してはすべてできました。

```
router2
1.
# dig -4 -x 10.20.71.66

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> -4 -x 10.20.71.66
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 23998
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;66.71.20.10.in-addr.arpa.	IN	PTR

;; Query time: 0 msec
;; SERVER: 10.0.0.51#53(10.0.0.51)
;; WHEN: 土 10月 19 18:25:29 JST 2019
;; MSG SIZE  rcvd: 53

2.
# dig service01.apptest.svc.cluster.local

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> service01.apptest.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63847
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;service01.apptest.svc.cluster.local. IN	A

;; ANSWER SECTION:
service01.apptest.svc.cluster.local. 30	IN A	10.20.71.66

;; Query time: 0 msec
;; SERVER: 10.0.0.51#53(10.0.0.51)
;; WHEN: 土 10月 19 18:29:33 JST 2019
;; MSG SIZE  rcvd: 115

3.
# curl -I 10.20.71.66
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:30:08 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

4.
# curl -I service01.apptest.svc.cluster.local
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:30:38 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

```

router2 では、1. service01 の CLUSTER-IP からサービス名を取得 でうまくいきません。
しかし、やりたかったことは出来たようです。
router1 に Envoy を追加すれば、もっと面白いかな。