OpenshiftにNGINX Ingress Controllerを追加してみる

## 目的
Openshiftでデフォルトで提供されるRouterの他に、NGINX Ingress Controllerも使いたい。


## 環境
Openshift 4.6

- Openshift自体はInternet 接続なし
- bastion nodeのみInternet接続可能
- IBM CloudのVirtual Private Cloudを利用
- コンテナImageを管理するプライベートレジストリにIBM Cloud Container Registryを利用


## 参考情報

- [IBM Cloud > Virtual Private Cloud について](https://cloud.ibm.com/docs/vpc-on-classic?topic=vpc-on-classic-about&locale=ja)

- [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/)

- [OpenShift > Configuring Ingress Controller sharding by using route labels](https://docs.openshift.com/container-platform/4.6/networking/ingress-operator.html#nw-ingress-sharding-route-labels_configuring-ingress)

- [DNS サブドメインへの VPC ロード・バランサーのホスト名の登録](https://cloud.ibm.com/docs/containers?topic=containers-vpc-lbaas&locale=ja)

## NGINX Ingress Controller を Openshiftにデプロイ

helm v3でNGINX Ingress ControllerをOpenshiftにデプロイする

### 1. bastion nodeにhelmコマンドを用意

```
[user01@bastion-node ~]$ curl -o helm.tar.gz -k https://get.helm.sh/helm-v3.4.2-linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.7M  100 12.7M    0     0  7776k      0  0:00:01  0:00:01 --:--:-- 7778k
[user01@bastion-node ~]$ tar xf helm.tar.gz
[user01@bastion-node ~]$ mv linux-amd64/helm ~/.local/bin/.
[user01@bastion-node ~]$ PATH=$PATH:~/.local/bin
[user01@bastion-node ~]$ helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}
[user01@bastion-node ~]$
```

### 2. ingress-nginx chartをダウンロード

```
[user01@bastion-node ~]$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
[user01@bastion-node ~]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "ibm-helm" chart repository
Update Complete. ⎈Happy Helming!⎈
[user01@bastion-node ~]$
[user01@bastion-node ~]$ helm repo list
NAME            URL
ibm-helm        https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm
ingress-nginx   https://kubernetes.github.io/ingress-nginx
[user01@bastion-node ~]$
[user01@bastion-node ~]$ helm search repo ingress-nginx/ingress-nginx
WARNING: Repo "ibm-helm" is corrupt or missing. Try 'helm repo update'.
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
ingress-nginx/ingress-nginx     3.15.2          0.41.2          Ingress controller for Kubernetes using NGINX a...
[user01@bastion-node ~]$
[user01@bastion-node ~]$ mkdir ~/work/ingress-nginx
[user01@bastion-node ~]$ cd ~/work/ingress-nginx
[user01@bastion-node ingress-nginx]$ helm pull ingress-nginx/ingress-nginx
[user01@bastion-node ingress-nginx]$ tar xf ingress-nginx-3.15.2.tgz
[user01@bastion-node ingress-nginx]$ ls
ingress-nginx  ingress-nginx-3.15.2.tgz
[user01@bastion-node ingress-nginx]$ 
```

### 3. 必要なイメージをプライベートレジストリに用意

- helmチャート内のファイルからデプロイに使うimageを確認。

```
[user01@bastion-node ingress-nginx]$ grep image: ingress-nginx/values.yaml -A2
  image:
    repository: k8s.gcr.io/ingress-nginx/controller
    tag: "v0.41.2"
--
  #    image: nginx:latest
  #  - name: lemonldap-ng-controller
  #    image: lemonldapng/lemonldap-ng-controller:0.2.0
  #    args:
  #      - /lemonldap-ng-controller
--
  #   image: busybox
  #   command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']

--
      image:
        repository: docker.io/jettech/kube-webhook-certgen
        tag: v1.5.0
--
  image:
    repository: k8s.gcr.io/defaultbackend-amd64
    tag: "1.5"
[user01@bastion-node ingress-nginx]$
```

対象imageは以下
  - k8s.gcr.io/ingress-nginx/controller:v0.41.2
  - docker.io/jettech/kube-webhook-certgen:v1.5.0
  - k8s.gcr.io/defaultbackend-amd64:1.5


- bastion nodeでimageをpullして、tagを付与し、プライベートレジストリにpush

```
docker pull k8s.gcr.io/defaultbackend-amd64:1.5
docker tag k8s.gcr.io/defaultbackend-amd64:1.5 jp.icr.io/test/defaultbackend-amd64:1.5
docker push jp.icr.io/test/defaultbackend-amd64:1.5
```

上記の手順と同様に他のimageもプライベートレジストリにpushする

- クラスターがプライベートレジストリにアクセスできるようにPullシークレットをOpenshiftに追加
※pull secretの中身（user id, password）を用意する手順は省略

```
[user01@bastion-node ingress-nginx]$ oc -n user01-test create secret docker-registry containerregistry-test --docker-server=jp.icr.io --docker-username=iamapikey --docker-password=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --docker-email=e34559@jp.ibm.com
secret/containerregistry-test created
[user01@bastion-node ingress-nginx]$
```


### 4. helmチャートのファイルを更新

internet接続のない環境なのでimageなど一部を変更

#### ingress-nginx/values.yaml

annotationsの部分の記載は、NGINX Ingress Controllerの前段のロードバランサーをIBM Cloudで作るためのもの。

```
[user01@bastion-node ingress-nginx]$ cp ingress-nginx/values.yaml ingress-nginx/values.yaml.org
[user01@bastion-node ingress-nginx]$ vi ingress-nginx/values.yaml
[user01@bastion-node ingress-nginx]$ diff ingress-nginx/values.yaml ingress-nginx/values.yaml.org
13c13
<     repository: jp.icr.io/test/ingress-nginx/controller
---
>     repository: k8s.gcr.io/ingress-nginx/controller
15c15
<     #digest: sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de
---
>     digest: sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de
269a270
>
352,353c353
<     annotations:
<       service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
---
>     annotations: {}
481c481
<         repository: jp.icr.io/test/jettech/kube-webhook-certgen
---
>         repository: docker.io/jettech/kube-webhook-certgen
490,491c490
<       #runAsUser: 2000
<       runAsUser: 1000622000
---
>       runAsUser: 2000
597c596
<     repository: jp.icr.io/test/defaultbackend-amd64
---
>     repository: k8s.gcr.io/defaultbackend-amd64
714,715c713
< imagePullSecrets:
< - name: containerregistry-test
---
> imagePullSecrets: []
[user01@bastion-node ingress-nginx]$
```

#### ingress-nginx/templates/controller-role.yaml

以下追記。

```
[user01@bastion-node ingress-nginx]$ vi ingress-nginx/templates/controller-role.yaml
[user01@bastion-node ingress-nginx]$ tail -n 10 ingress-nginx/templates/controller-role.yaml
{{- end }}
  - apiGroups:
    - security.openshift.io
    resourceNames:
    - privileged
    resources:
    - securitycontextconstraints
    verbs:
    - use

[user01@bastion-node ingress-nginx]$
```
※ templates以下のファイルは全て読み込まれてしまうので、元ファイルを別名でコピーして同じディレクトリに置いたりしないこと。


### 5. NGINX Ingress Controllerのデプロイ実行

- ocコマンドでopenshiftにログインし、NGINX Ingress Controllerを導入するproject(namespace)に切り替える

```
oc login -u user01
oc project user01-test
```

- デプロイ実行

```
[user01@bastion-node ingress-nginx]$ helm install ingress-nginx-test ./ingress-nginx
NAME: ingress-nginx-test
LAST DEPLOYED: Mon Dec 21 16:53:25 2020
NAMESPACE: user01-test
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace user01-test get services -o wide -w ingress-nginx-test-controller'

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc get all
NAME                                                READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-test-controller-cfc699764-lpm5t   1/1     Running   0          19m

NAME                                              TYPE           CLUSTER-IP      EXTERNAL-IP                          PORT(S)                      AGE
service/ingress-nginx-test-controller             LoadBalancer   172.21.22.87    d1b930bf-jp-tok.lb.appdomain.cloud   80:31140/TCP,443:32343/TCP   19m
service/ingress-nginx-test-controller-admission   ClusterIP      172.21.199.70   <none>                               443/TCP                      19m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-test-controller   1/1     1            1           19m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-test-controller-cfc699764   1         1         1       19m
[user01@bastion-node ingress-nginx]$
```


## DNS サブドメインへの VPC ロード・バランサーのホスト名の登録

```
[user01@bastion-node ~]$ ibmcloud ks nlb-dns create vpc-gen2 --cluster tmpcluster --lb-host d1b930bf-jp-tok.lb.appdomain.cloud --type private
Creating NLB DNS...
OK
NLB hostname was created as test-cluster-xx1.jp-tok.containers.appdomain.cloud
[user01@bastion-node ~]$
[user01@bastion-node ~]$ ibmcloud ks nlb-dns ls --cluster tmpcluster
OK
Subdomain                                                                            Load Balancer Hostname               SSL Cert Status   SSL Cert Secret Name                               Secret Namespace
test-cluster-xx0.jp-tok.containers.appdomain.cloud   818aee49-jp-tok.lb.appdomain.cloud   created           test-cluster-xx0   openshift-ingress
test-cluster-xx1.jp-tok.containers.appdomain.cloud   d1b930bf-jp-tok.lb.appdomain.cloud   created           test-cluster-xx1   default
[user01@bastion-node ~]$
```




## shardingを設定

このままだと、NGINX Ingress Controllerでサービスを公開すると、デフォルトのRouterにも変更が加わる。（そんなに問題はないのですが。）
デフォルトのRouterには変更を行わないようにするために追加設定を行う。

- ingress controllerに設定を追加

```
oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"routeSelector":{"matchExpressions":[{"key":"nginxingress", "operator":"DoesNotExist"}]}}}' --type=merge
```


## NGINX Ingress Controllerのテスト

- テスト用イメージ用意

```
docker pull openshift/hello-openshift:latest
docker tag openshift/hello-openshift:latest jp.icr.io/test/test/openshift/hello-openshift:latest
docker push jp.icr.io/test/test/openshift/hello-openshift:latest
```

- テスト用podをデプロイ

```
oc new-app --docker-image=jp.icr.io/test/test/openshift/hello-openshift:latest --name user01-test -o yaml > test_deploy.yaml
vi test_deploy.yaml
# 以下の様にpull secretや envを追記
---
      spec:
        imagePullSecrets:
        - name: containerregistry-test
        containers:
        - image: ' '
          name: user01-test
          ports:
          - containerPort: 8080
            protocol: TCP
          - containerPort: 8888
            protocol: TCP
          resources: {}
          env:
          - name: RESPONSE
            value: Hello World!
~~
---

oc create -f test_deploy.yaml
```

- サービスをingressで公開

```
[user01@bastion-node ingress-nginx]$ cat test_ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-myservicea
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud
    http:
      paths:
      - path: /
        backend:
          serviceName: user01-test
          servicePort: 8080
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc create -f test_ingress.yaml
ingress.networking.k8s.io/ingress-user01-test created
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc get ingress
NAME                  CLASS    HOSTS                                                                                                        ADDRESS                              PORTS   AGE
ingress-user01-test   <none>   user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud   d1b930bf-jp-tok.lb.appdomain.cloud   80      46s
[user01@bastion-node ingress-nginx]$
```


- 接続確認

```
[user01@bastion-node ingress-nginx]$ curl http://user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud
Hello World!
[user01@bastion-node ingress-nginx]$
```


以上