kopsでamazon-vpc-cni-k8sを利用する

kopsでamazon-vpc-cni-k8sを利用してみました。kopsの利用方法自体は参考URLを参考にしてください。
本記事では省略しています。podがVPC内のIPを利用するのでk8sクラスタ外のEC2がpodにNodePortなどを利用せずにアクセスできることがポイントです。

# 参考URL
- [kopsを使ってKubernetesクラスタをAWS上で構成 | Amazon Web Services ブログ](https://aws.amazon.com/jp/blogs/news/configure-kubernetes-cluster-on-aws-by-kops/)
- [aws/amazon-vpc-cni-k8s: Networking plugin repository for pod networking in Kubernetes using Elastic Network Interfaces on AWS](https://github.com/aws/amazon-vpc-cni-k8s)
- [kops/networking.md at master · kubernetes/kops](https://github.com/kubernetes/kops/blob/master/docs/networking.md#amazon-vpc-backend)


## kops create cluster
オプションで`--networking amazon-vpc-routed-eni`を指定。

```
$ kops create cluster \
  --name paper2-cni.cluster.k8s.local \
  --zones us-east-1a,us-east-1c \
  --master-size="t2.micro" --node-size="t2.micro" \
  --networking amazon-vpc-routed-eni \
  --yes
I0729 16:55:35.622015   51569 create_cluster.go:1318] Using SSH public key: /Users/paper2/.ssh/id_rsa.pub
I0729 16:55:37.923294   51569 create_cluster.go:472] Inferred --cloud=aws from zone "us-east-1a"
I0729 16:55:39.634860   51569 subnets.go:184] Assigned CIDR 172.20.32.0/19 to subnet us-east-1a
I0729 16:55:39.634927   51569 subnets.go:184] Assigned CIDR 172.20.64.0/19 to subnet us-east-1c
I0729 16:55:45.861707   51569 apply_cluster.go:456] Gossip DNS: skipping DNS validation
I0729 16:55:47.768441   51569 executor.go:91] Tasks: 0 done / 81 total; 31 can run
I0729 16:55:50.086276   51569 vfs_castore.go:731] Issuing new certificate: "apiserver-aggregator-ca"
I0729 16:55:50.101474   51569 vfs_castore.go:731] Issuing new certificate: "ca"
I0729 16:55:52.641220   51569 executor.go:91] Tasks: 31 done / 81 total; 25 can run
I0729 16:55:57.438697   51569 vfs_castore.go:731] Issuing new certificate: "kubelet-api"
I0729 16:55:57.524915   51569 vfs_castore.go:731] Issuing new certificate: "kops"
I0729 16:55:57.553610   51569 vfs_castore.go:731] Issuing new certificate: "kube-controller-manager"
I0729 16:55:57.621873   51569 vfs_castore.go:731] Issuing new certificate: "kube-proxy"
I0729 16:55:57.656320   51569 vfs_castore.go:731] Issuing new certificate: "apiserver-aggregator"
I0729 16:55:57.847304   51569 vfs_castore.go:731] Issuing new certificate: "kubelet"
I0729 16:55:58.005469   51569 vfs_castore.go:731] Issuing new certificate: "kubecfg"
I0729 16:55:58.200209   51569 vfs_castore.go:731] Issuing new certificate: "kube-scheduler"
I0729 16:55:58.528400   51569 vfs_castore.go:731] Issuing new certificate: "apiserver-proxy-client"
I0729 16:56:00.733555   51569 executor.go:91] Tasks: 56 done / 81 total; 21 can run
I0729 16:56:00.902358   51569 logging_retryer.go:60] Retryable error (RequestError: send request failed
caused by: Post https://ec2.us-east-1.amazonaws.com/: EOF) from ec2/DescribeSecurityGroups - will retry after delay of 56ms
I0729 16:56:09.965480   51569 logging_retryer.go:60] Retryable error (RequestError: send request failed
caused by: Post https://elasticloadbalancing.us-east-1.amazonaws.com/: EOF) from elasticloadbalancing/DescribeTags - will retry after delay of 30ms
I0729 16:56:37.412321   51569 executor.go:91] Tasks: 77 done / 81 total; 3 can run
I0729 16:56:40.602256   51569 vfs_castore.go:731] Issuing new certificate: "master"
I0729 16:56:41.881540   51569 executor.go:91] Tasks: 80 done / 81 total; 1 can run
I0729 16:56:42.818618   51569 executor.go:91] Tasks: 81 done / 81 total; 0 can run
I0729 16:56:43.495941   51569 update_cluster.go:291] Exporting kubecfg for cluster
kops has set your kubectl context to paper2-cni.cluster.k8s.local

Cluster is starting.  It should be ready in a few minutes.

Suggestions:
 * validate cluster: kops validate cluster
 * list nodes: kubectl get nodes --show-labels
 * ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.paper2-cni.cluster.k8s.local
 * the admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
 * read about installing addons at: https://github.com/kubernetes/kops/blob/master/docs/addons.md.
```

# podをデプロイしてみる


```deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 5
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.12.1
        ports:
        - containerPort: 80
        - containerPort: 443
```

```sh
$ kubectl apply -f deployment.yaml
deployment.extensions/nginx-deployment created
```

# podにVPCのIPが割り振られているか確認
```sh
$ kubectl get all -o wide
NAME                                    READY     STATUS    RESTARTS   AGE       IP              NODE
pod/nginx-deployment-7f786f684b-4cgqh   1/1       Running   0          31s       172.20.80.188   ip-172-20-95-59.ec2.internal
pod/nginx-deployment-7f786f684b-6fpq2   1/1       Running   0          32s       172.20.40.146   ip-172-20-36-86.ec2.internal
pod/nginx-deployment-7f786f684b-bgd2j   1/1       Running   0          32s       172.20.81.121   ip-172-20-95-59.ec2.internal
pod/nginx-deployment-7f786f684b-cz848   1/1       Running   0          32s       172.20.59.80    ip-172-20-36-86.ec2.internal
pod/nginx-deployment-7f786f684b-xbjfr   1/1       Running   0          32s       172.20.51.46    ip-172-20-36-86.ec2.internal

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE       SELECTOR
service/kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   7m        <none>

NAME                                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS   IMAGES         SELECTOR
deployment.extensions/nginx-deployment   5         5         5            5           32s       nginx        nginx:1.12.1   app=nginx

NAME                                                DESIRED   CURRENT   READY     AGE       CONTAINERS   IMAGES         SELECTOR
replicaset.extensions/nginx-deployment-7f786f684b   5         5         5         32s       nginx        nginx:1.12.1   app=nginx,pod-template-hash=3934292406

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS   IMAGES         SELECTOR
deployment.apps/nginx-deployment   5         5         5            5           32s       nginx        nginx:1.12.1   app=nginx

NAME                                          DESIRED   CURRENT   READY     AGE       CONTAINERS   IMAGES         SELECTOR
replicaset.apps/nginx-deployment-7f786f684b   5         5         5         32s       nginx        nginx:1.12.1   app=nginx,pod-template-hash=3934292406
```

podのIPに`172.20/16`が割り振られていますね。

# k8sクラスタに所属しないEC2からアクセスしてみる
通常k8sのpodへはingressやNodePortを使用しない限りk8sクラスタの外部からはアクセスできません。しかしamazon-vpc-cni-k8sを利用するとpodのIPはVPCのIPになるので、同じVPC内のインスタンスからpodにアクセスできます。

それではEC2をk8sクラスタのサブネット上に作成してアクセスしてみます。
kopsはセキュリティグループでアクセスを制限しているので作成したEC2にk8sクラスタを構築するEC2にアクセスできるセキュリティグループを割り当てる必要があります。

```
[ec2-user@ip-172-20-89-195 ~]$ ip route
default via 172.20.64.1 dev eth0
169.254.169.254 dev eth0
172.20.64.0/19 dev eth0 proto kernel scope link src 172.20.89.195
[ec2-user@ip-172-20-89-195 ~]$  curl 172.20.80.188
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```

k8sクラスタ外のEC2からpodにアクセスできました。