AWS EKS上でNGINX Ingress Controller + external-dns で複数環境のDNS (Route53) を管理する

# 背景

随分まえから Nginx Ingress Controllerとexternal-dnsで複数環境のDNSレコードとサービスを結びつけるのやりたかったが、1年越しでやっとできた

# 全体イメージ

![ingress-external-dns.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/34740eb8-9ab6-a3bf-7d9b-d3356671b537.png)



# Steps

## 1. Nginx Ingress Controllerをインストール

https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/

deploy.yamlをダウンロード

```
curl -O https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/aws/deploy.yaml
```

今回は、以下のannotationをつけてssl終端をLBでやるように設定

```yaml：deploy.yaml
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-northeast-1:xxxxxxxx:certificate/xxxxxxxxxxxxxxxxxxxx # ACM for your-domain.com
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
```

Apply

```
kubectl apply -f deploy.yaml
```

## 2. テストアプリのDeploy

https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/


namespace `naka` を作成

```
kubectl create ns naka
```

appleというappをデプロイ

```
kubectl apply -n naka -f https://raw.githubusercontent.com/cornellanthony/nlb-nginxIngress-eks/master/apple.yaml
```

bananaというappをデプロイ

```
kubectl apply -n naka -f https://raw.githubusercontent.com/cornellanthony/nlb-nginxIngress-eks/master/banana.yaml
```


## 3. Ingressをデプロイ


```yaml:ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: apple.your-domain.com
    http:
      paths:
        - path: /
          backend:
            serviceName: apple-service
            servicePort: 5678
  - host: banana.your-domain.com
    http:
      paths:
        - path: /
          backend:
            serviceName: banana-service
            servicePort: 5678
```

```
kubectl apply -n naka -f ingress.yaml
```

この時点で、 `apple.your-domain.com` と `banana.your-domain.com` のDNS Recordを Nginx Ingress controllerのLBに向ければ、すでに使用が可能。

ここから先は、DNSの設定もKubernetesのリソースとして管理するためのステップ。

## 4. IAM role を作成

https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md#iam-policy

EKSを使っている場合 IAM roleをservice accountに紐付けるので、以下のようにする

```terraform:eks-external-dns-service-account.tf
data "aws_iam_policy_document" "eks-external-dns-service-account" {
  statement {
    sid     = "AllowAssumeRoleWithWebIdentity"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "oidc.eks.ap-northeast-1.amazonaws.com/id/xxxxxx:sub"
      values = [
        "system:serviceaccount:external-dns:external-dns" # system:serviceaccount:<namespace>:<service-account-name>
      ]
    }

    principals {
      identifiers = ["arn:aws:iam::xxxxxx:oidc-provider/oidc.eks.ap-northeast-1.amazonaws.com/id/xxxxxxxxxxxxxxxxx"]
      type        = "Federated"
    }
  }
}

resource "aws_iam_role" "eks-external-dns-service-account" {
  assume_role_policy = data.aws_iam_policy_document.eks-external-dns-service-account.json
  name               = "eks-external-dns-service-account"
}

data "aws_iam_policy_document" "route53-your-domain" {
  statement {
    sid = "ChangeResourceRecordSetsYourDomain"
    actions = [
      "route53:ChangeResourceRecordSets"
    ]

    resources = [
      "arn:aws:route53:::hostedzone/xxxxxxxxxxxx" # your-domain.com
    ]
  }

  statement {
    sid = "ListRoute53"
    actions = [
      "route53:ListHostedZones",
      "route53:ListResourceRecordSets",
    ]

    resources = [
      "*"
    ]
  }
}

resource "aws_iam_policy" "route53-your-domain" {
  name        = "route53-your-domain"
  path        = "/"
  description = "Allow to change Route53 recordsets for your-domain.com"
  policy      = data.aws_iam_policy_document.route53-your-domain.json
}

resource "aws_iam_role_policy_attachment" "role-policy-attachment" {
  role       = aws_iam_role.eks-external-dns-service-account.name
  policy_arn = aws_iam_policy.route53-your-domain.arn
}
```

バックエンドの設定は必要だが今回は省略

```
terraform apply
```

## 5. external-dnsをインストール

https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md

```yaml:external-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  # If you're using Amazon EKS with IAM Roles for Service Accounts, specify the following annotation.
  # Otherwise, you may safely omit it.
  annotations:
    # Substitute your account ID and IAM service role name below.
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT-ID:role/IAM-SERVICE-ROLE-NAME # 上のセクションで作成したrole
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
      # If you're using kiam or kube2iam, specify the following annotation.
      # Otherwise, you may safely omit it.
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::ACCOUNT-ID:role/IAM-SERVICE-ROLE-NAME
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.7.3
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=your-domain.com # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
        - --provider=aws
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
        - --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
        - --registry=txt
        - --txt-owner-id=my-hostedzone-identifier
      securityContext:
        fsGroup: 65534 # For ExternalDNS to be able to read Kubernetes and AWS token files
```

今回は `external-dns` というnamespaceにデプロイするのでNamespaceを作成

```
kubectl create ns external-dns
```

```
kubectl apply -f external-dns.yaml -n external-dns
```

## 6. 確認

banana

```
curl https://banana.your-domain.com
banana
```

apple

```
curl https://apple.your-domain.com
apple
```

# 参照

- [ingress-nginx](https://kubernetes.github.io/ingress-nginx)
- [external-dns](https://github.com/kubernetes-sigs/external-dns)
    - [Setting up ExternalDNS for Services on AWS](https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md)
- [Using a Network Load Balancer with the NGINX Ingress Controller on Amazon EKS](https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks)