GKEでもNGINXのIngressは使える

# GKEでもNGINXのIngressは使える

## はじめに

GKEでIngressといったらGoogle Cloud Load Blancingと思っていたが、Nginxも使えたみたいなので、ちょっと触ってみる。


## NginxのIngressについて

↓に記載されている
https://cloud.google.com/community/tutorials/nginx-ingress-gke

NginxのIngressはKubernetesコミュニティが管理している[kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx)とNginx社管理の[nginxinc/kubernetes-ingress](https://github.com/nginxinc/kubernetes-ingress)は別物である。

主な違いは以下を参照

https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/nginx-ingress-controllers.md

| Aspect or Feature | kubernetes/ingress-nginx | nginxinc/kubernetes-ingress with NGINX | nginxinc/kubernetes-ingress with NGINX Plus |
|:--|:--|:--|:--|
| Fundamental |  |  |  |
| Authors | Kubernetes community | NGINX Inc and community | NGINX Inc and community |
| NGINX version | Custom NGINX build that includes several third-party modules | NGINX official mainline build | NGINX Plus |
| Commercial support | N/A | N/A | Included |
| Load balancing configuration via the Ingress resource |  |  |  |
| Merging Ingress rules with the same host | Supported | Supported via Mergeable Ingresses | Supported via Mergeable Ingresses |
| HTTP load balancing extensions - Annotations | See the supported annotations | See the supported annotations | See the supported annotations |
| HTTP load balancing extensions -- ConfigMap | See the supported ConfigMap keys | See the supported ConfigMap keys | See the supported ConfigMap keys |
| TCP/UDP | Supported via a ConfigMap | Supported via a ConfigMap with native NGINX configuration | Supported via a ConfigMap with native NGINX configuration |
| Websocket | Supported | Supported via an annotation | Supported via an annotation |
| TCP SSL Passthrough | Supported via a ConfigMap | Not supported | Not supported |
| JWT validation | Not supported | Not supported | Supported |
| Session persistence | Supported via a third-party module | Not supported | Supported |
| Canary testing (by header, cookie, weight) | Supported via annotations | Supported via custom resources | Supported via custom resources |
| Configuration templates *1 | See the template | See the templates | See the templates |
| Load balancing configuration via Custom Resources |  |  |  |
| HTTP load balancing | Not supported | See VirtualServer and VirtualServerRoute resources. | See VirtualServer and VirtualServerRoute resources. |
| Deployment |  |  |  |
| Command-line arguments *2 | See the arguments | See the arguments | See the arguments |
| TLS certificate and key for the default server | Required as a command-line argument/ auto-generated | Required as a command-line argument | Required as a command-line argument |
| Helm chart | Supported | Supported | Supported |
| Operational |  |  |  |
| Reporting the IP address(es) of the Ingress controller into Ingress resources | Supported | Supported | Supported |
| Extended Status | Supported via a third-party module | Not supported | Supported |
| Prometheus Integration | Supported | Supported | Supported |
| Dynamic reconfiguration of endpoints (no configuration reloading) | Supported with a third-party Lua module | Not supported | Supported |


## 環境構築

### 前提条件

* GKEクラスタに使用するSubnetは作成済みとする

### GKEクラスタ作成


```
gcloud beta container --project "PROJECT-NAME" clusters create "nginx-ingress" \
--zone "asia-northeast1-a" --node-locations "asia-northeast1-a","asia-northeast1-b" \
--no-enable-basic-auth --release-channel "regular" --machine-type "n1-standard-1" \
--image-type "COS" --disk-type "pd-standard" --disk-size "100" --metadata disable-legacy-endpoints=true \
--scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
--num-nodes "1" --enable-stackdriver-kubernetes --enable-ip-alias \
--network "projects/PROJECT-NAME/global/networks/sakon-network" --subnetwork "projects/PROJECT-NAME/regions/asia-northeast1/subnetworks/sakon-gke-subnet-1" \
--cluster-secondary-range-name "sakon-gkesub1-2nd-range-2" --services-secondary-range-name "sakon-gkesub1-2nd-range-1" \
--enable-intra-node-visibility --default-max-pods-per-node "110" --addons HorizontalPodAutoscaling,HttpLoadBalancing \
--enable-autoupgrade --enable-autorepair
```

```
$ kubectl version --short
Client Version: v1.14.7
Server Version: v1.14.3-gke.11
```

### Tillerのインストール

```
$ kubectl create serviceaccount --namespace kube-system tiller
serviceaccount/tiller created
```

```
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created
```

```
$ helm init --service-account tiller
$HELM_HOME has been configured at /Users/sakonju/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
```

tillerがインストールされてるか確認
```
$ kubectl get deploy -n kube-system
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
event-exporter-v0.2.5                      1/1     1            1           34h
fluentd-gcp-scaler                         1/1     1            1           34h
heapster-v1.6.1                            1/1     1            1           34h
kube-dns                                   2/2     2            2           34h
kube-dns-autoscaler                        1/1     1            1           34h
l7-default-backend                         1/1     1            1           34h
metrics-server-v0.3.1                      1/1     1            1           34h
stackdriver-metadata-agent-cluster-level   1/1     1            1           34h
tiller-deploy                              1/1     1            1           34h
```

### GKEにアプリケーションをデプロイ

```
$ kubectl run --restart=Always hello-app --image=gcr.io/google-samples/hello-app:1.0 --port=8080
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/hello-app created
```

serviceを作って外部公開する
```
$ kubectl expose deployment hello-app
service/hello-app exposed

$ kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE   SELECTOR
hello-app    ClusterIP   10.121.227.201   <none>        8080/TCP   32s   run=hello-app
kubernetes   ClusterIP   10.120.0.1       <none>        443/TCP    34h   <none>
```

### Helmを使用してNGINX Ingress Controllerをdeploy

以下の様な構成となる

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/33101/9636de7f-cd27-8cd5-c183-26f5d989c4b6.png)

`nginx-ingress`を[Helm](https://github.com/helm/charts/tree/master/stable/nginx-ingress)でdeployする

```
$ helm install --name nginx-ingress stable/nginx-ingress --set rbac.create=true --set controller.publishService.enabled=true
NAME:   nginx-ingress
LAST DEPLOYED: Sun Oct  6 14:23:12 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                                            READY  STATUS             RESTARTS  AGE
nginx-ingress-controller-6cb795cdc5-r9j5c       0/1    ContainerCreating  0         1s
nginx-ingress-default-backend-6b8dc9d88f-hfv54  0/1    ContainerCreating  0         1s

==> v1/Service
NAME                           TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.122.205.4  <pending>    80:31003/TCP,443:32313/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.121.57.91  <none>       80/TCP                      1s

==> v1/ServiceAccount
NAME                   SECRETS  AGE
nginx-ingress          1        1s
nginx-ingress-backend  1        1s

==> v1beta1/ClusterRole
NAME           AGE
nginx-ingress  1s

==> v1beta1/ClusterRoleBinding
NAME           AGE
nginx-ingress  1s

==> v1beta1/Deployment
NAME                           READY  UP-TO-DATE  AVAILABLE  AGE
nginx-ingress-controller       0/1    1           0          1s
nginx-ingress-default-backend  0/1    1           0          1s

==> v1beta1/Role
NAME           AGE
nginx-ingress  1s

==> v1beta1/RoleBinding
NAME           AGE
nginx-ingress  1s


NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls
```

↓L4ロードバランサが作成される

```
==> v1/Service
NAME                           TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.122.205.4  <pending>    80:31003/TCP,443:32313/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.121.57.91  <none>       80/TCP                      1s
```

しばらくするとEXTERNAL-IPも取得されている。

```
$ kubectl get service
NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
hello-app                       ClusterIP      10.121.227.201   <none>         8080/TCP                     19h
kubernetes                      ClusterIP      10.120.0.1       <none>         443/TCP                      2d5h
nginx-ingress-controller        LoadBalancer   10.122.205.4     34.84.181.58   80:31003/TCP,443:32313/TCP   2m44s
nginx-ingress-default-backend   ClusterIP      10.121.57.91     <none>         80/TCP                       2m44s
```

![スクリーンショット 2019-10-06 14.28.48.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/33101/b33425a3-3470-36eb-a2b9-09b81c5a60f4.png)

2番目のサービスである、 `nginx-ingress-default-backend`に注目。default-backendは全てのURLパスを処理し、NGINXコントローラが理解できないホスト(つまり、Ingressリソースにマッピングされていない全てのリクエスト)をホストするサービス。デフォルトのバックエンドは2つのURLを公開する。

* `/healthz` 200を返す
* `/` 404を返す

```
$ curl -v http://34.84.181.58/healthz
*   Trying 34.84.181.58...
* TCP_NODELAY set
* Connected to 34.84.181.58 (34.84.181.58) port 80 (#0)
> GET /healthz HTTP/1.1
> Host: 34.84.181.58
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Sun, 06 Oct 2019 06:32:28 GMT
< Content-Type: text/html
< Content-Length: 0
< Connection: keep-alive
<
```

```
$ curl -v http://34.84.181.58/
*   Trying 34.84.181.58...
* TCP_NODELAY set
* Connected to 34.84.181.58 (34.84.181.58) port 80 (#0)
> GET / HTTP/1.1
> Host: 34.84.181.58
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: openresty/1.15.8.1
< Date: Sun, 06 Oct 2019 06:41:45 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
<
* Connection #0 to host 34.84.181.58 left intact
```

### NGINX Ingress Controllerを使用するためのIngressリソースの構成

GCPの場合、annotationで定義が無い場合はGCLBが作成される。

NGINXの場合
`annotations: kubernetes.io/ingress.class: nginx`
GCLBの場合
`annotations: kubernetes.io/ingress.class: gce`

```yaml:ingress-resource.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /hello
        backend:
          serviceName: hello-app
          servicePort: 8080
```

applyする

```
$ kubectl apply -f ingress-resource.yaml
ingress.networking.k8s.io/ingress-resource created
```

作成されたリソースを確認

```
$ kubectl describe ing
Name:             ingress-resource
Namespace:        default
Address:          34.84.181.58
Default backend:  default-http-backend:80 (10.124.0.9:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /hello   hello-app:8080 (10.124.1.11:8080)
Annotations:
  kubernetes.io/ingress.class:                       nginx
  nginx.ingress.kubernetes.io/ssl-redirect:          false
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"ingress-resource","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"hello-app","servicePort":8080},"path":"/hello"}]}}]}}

Events:  <none>
```

アクセスしてみる

`/hello`ではバックエンドのwebアプリケーションにアクセス出来るようになっている！

```
$ curl http://34.84.181.58/hello
Hello, world!
Version: 1.0.0
Hostname: hello-app-84d9648f89-vzh5v
```

`/hello`以外だと、`404`を返すので、デフォルトバックエンドサービスが適切に機能している。

```
$ curl http://34.84.181.58/test
default backend - 404
```