ROSAにIngress Controllerを追加してアレコレ

# はじめに
こちらは[OpenShift Advent Calendar 2023](https://qiita.com/advent-calendar/2023/openshift) 12/12の記事です。

ROSA（Red Hat OpenShift Service on AWS）のv4.14からデフォルトIngressがCLBからNLBに変更になりました。また合わせてCustom DomainがOpenShift Ingress Controllerで管理されることになり、Custom Domain Operatorは非推奨になるとのこと。

そんなROSA v4.14でIngress Controllerを追加してみたので、結果を紹介します。
また、合わせて証明書の設定やアクセスログ設定も実施したので、その内容についても記載しています。

# 構築検証

設定作業は[こちら](https://access.redhat.com/documentation/en-us/red_hat_openshift_service_on_aws/4/html/networking/configuring-ingress#nw-ingress-setting-internal-lb_configuring-ingress)のマニュアルに沿って進めていきます。

## 前提

ROSA：
　version 4.14.4
　Private Linkモードで構築
　Classic Mode（HCPではない）


## 構築前の状態

```
[user@host ~]$ oc get ingresscontrollers -n openshift-ingress-operator
NAME      AGE
default   3d20h　　←”default” Ingress Controllerが1つ存在
[user@ip-10-0-17-126 ~]$ oc get all -n openshift-ingress
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                                 READY   STATUS    RESTARTS   AGE
pod/router-default-bf4775545-bzhzq   1/1     Running   0          3d20h　　←router Podが2つ
pod/router-default-bf4775545-z2b2j   1/1     Running   0          3d20h　　←router Podが2つ

NAME                              TYPE           CLUSTER-IP       EXTERNAL-IP
           PORT(S)                      AGE
service/router-default            LoadBalancer   172.30.173.240   a43d81xxxxx.elb.us-east-1.amazonaws.com   80:31416/TCP,443:32051/TCP   3d20h　　←NLBを利用
service/router-internal-default   ClusterIP      172.30.129.181   <none>
           80/TCP,443/TCP,1936/TCP      3d20h

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/router-default   2/2     2            2           3d20h

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/router-default-5754dd447b   0         0         0       3d20h
replicaset.apps/router-default-58564b7469   0         0         0       3d20h
replicaset.apps/router-default-5db588b9d9   0         0         0       3d20h
replicaset.apps/router-default-bf4775545    2         2         2       3d20h
[user@ip-10-0-17-126 ~]$
```

## 構築

Ingress Controller用のカスタムリソースを準備します。

まずは参考情報として、以下マニュアル記載のままの状態で設定を適用してみます。

```
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  namespace: openshift-ingress-operator
  name: new-ingress-controller
spec:
  domain: new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
  endpointPublishingStrategy:
    type: LoadBalancerService
    loadBalancer:
      scope: Internal
```

ドメインについては新規でドメインを準備できなかったので、デフォルトで作成されるROSAのドメインのサブドメインとして定義しました。

設定適用後の状態

```
[user@host home]$ oc apply -f ingresscontroller.yaml
ingresscontroller.operator.openshift.io/new-ingress-controller created
[user@host home]$
[user@host home]$ oc get ingresscontrollers.operator.openshift.io -n openshift-ingress-operator
NAME                     AGE
default                  4d2h
new-ingress-controller   10s
```


<details><summary>詳細情報</summary>



```yaml
[user@host home]$ oc get ingresscontrollers.operator.openshift.io -n openshift-ingress-operator new-ingress-controller -o yaml
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operator.openshift.io/v1","kind":"IngressController","metadata":{"annotations":{},"name":"new-ingress-controller","namespace":"openshift-ingress-operator"},"spec":{"domain":"new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com","endpointPublishingStrategy":{"loadBalancer":{"scope":"Internal"},"type":"LoadBalancerService"}}}
  creationTimestamp: "2023-12-12T07:48:42Z"
  finalizers:
  - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
  generation: 2
  name: new-ingress-controller
  namespace: openshift-ingress-operator
  resourceVersion: "3892710"
  uid: c5f7abbc-6416-4c8e-9049-a53b526f291c
spec:
  clientTLS:
    clientCA:
      name: ""
    clientCertificatePolicy: ""
  domain: new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
  endpointPublishingStrategy:
    loadBalancer:
      dnsManagementPolicy: Managed
      scope: Internal
    type: LoadBalancerService
  httpCompression: {}
  httpEmptyRequestsPolicy: Respond
  httpErrorCodePages:
    name: ""
  tuningOptions:
    reloadInterval: 0s
  unsupportedConfigOverrides: null
status:
  availableReplicas: 0
  conditions:
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    reason: Valid
    status: "True"
    type: Admitted
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: 'The deployment has Available status condition set to False (reason:
      MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.'
    reason: DeploymentUnavailable
    status: "False"
    type: DeploymentAvailable
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: 0/2 of replicas are available, max unavailable is 1
    reason: DeploymentMinimumReplicasNotMet
    status: "False"
    type: DeploymentReplicasMinAvailable
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: 0/2 of replicas are available
    reason: DeploymentReplicasNotAvailable
    status: "False"
    type: DeploymentReplicasAllAvailable
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: |
      Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available...
    reason: DeploymentRollingOut
    status: "True"
    type: DeploymentRollingOut
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: The endpoint publishing strategy supports a managed load balancer
    reason: WantedByEndpointPublishingStrategy
    status: "True"
    type: LoadBalancerManaged
  - lastTransitionTime: "2023-12-12T07:48:46Z"
    message: The LoadBalancer service is provisioned
    reason: LoadBalancerProvisioned
    status: "True"
    type: LoadBalancerReady
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: LoadBalancer is not progressing
    reason: LoadBalancerNotProgressing
    status: "False"
    type: LoadBalancerProgressing
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: DNS management is supported and zones are specified in the cluster DNS
      config.
    reason: Normal
    status: "True"
    type: DNSManaged
  - lastTransitionTime: "2023-12-12T07:48:46Z"
    message: The record is provisioned in all reported zones.
    reason: NoFailedZones
    status: "True"
    type: DNSReady
  - lastTransitionTime: "2023-12-12T07:48:46Z"
    message: 'One or more status conditions indicate unavailable: DeploymentAvailable=False
      (DeploymentUnavailable: The deployment has Available status condition set to
      False (reason: MinimumReplicasUnavailable) with message: Deployment does not
      have minimum availability.)'
    reason: IngressControllerUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: |-
      One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available...
      )
    reason: IngressControllerProgressing
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-12-12T07:49:12Z"
    message: 'One or more other status conditions indicate a degraded state: DeploymentAvailable=False
      (DeploymentUnavailable: The deployment has Available status condition set to
      False (reason: MinimumReplicasUnavailable) with message: Deployment does not
      have minimum availability.)'
    reason: DegradedConditions
    status: "True"
    type: Degraded
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: IngressController is upgradeable.
    reason: Upgradeable
    status: "True"
    type: Upgradeable
  - lastTransitionTime: "2023-12-12T07:48:42Z"
    message: No evaluation condition is detected.
    reason: NoEvaluationCondition
    status: "False"
    type: EvaluationConditionsDetected
  domain: new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
  endpointPublishingStrategy:
    loadBalancer:
      dnsManagementPolicy: Managed
      providerParameters:
        aws:
          type: NLB
        type: AWS
      scope: Internal
    type: LoadBalancerService
  observedGeneration: 2
  selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=new-ingress-controller
  tlsProfile:
    ciphers:
    - ECDHE-ECDSA-AES128-GCM-SHA256
    - ECDHE-RSA-AES128-GCM-SHA256
    - ECDHE-ECDSA-AES256-GCM-SHA384
    - ECDHE-RSA-AES256-GCM-SHA384
    - ECDHE-ECDSA-CHACHA20-POLY1305
    - ECDHE-RSA-CHACHA20-POLY1305
    - DHE-RSA-AES128-GCM-SHA256
    - DHE-RSA-AES256-GCM-SHA384
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    minTLSVersion: VersionTLS12
```
</details>


解説

spec.endpointPublishingStrategyの箇所でLoadBalancerServiceとして定義するとstatusとして下記のようにAWSのNLBが作成されます。

```
status:
:
  endpointPublishingStrategy:
    loadBalancer:
      dnsManagementPolicy: Managed
      providerParameters:
        aws:
          type: NLB
        type: AWS
      scope: Internal
    type: LoadBalancerService
:
```

:::note info
v4.14.2でIngress Controllerを作成した時はyamlでproviderParametersを定義しないとCLBが作成されたのですが、v4.14.4では定義しなくてもデフォルトNLBが作成されるように変わったようです。
:::



Pod、Serviceの状態は以下の通り。

```
[user@host home]$ oc get pod -n openshift-ingress
NAME                                            READY   STATUS    RESTARTS   AGE
router-default-bf4775545-bzhzq                  1/1     Running   0          4d2h
router-default-bf4775545-z2b2j                  1/1     Running   0          4d2h
router-new-ingress-controller-d4795df69-d5rld   1/1     Running   0          2m2s
router-new-ingress-controller-d4795df69-tjtx4   1/1     Running   0          2m2s
[user@host home]$
[user@host home]$ oc get svc -n openshift-ingress
NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP
                  PORT(S)                      AGE
router-default                           LoadBalancer   172.30.173.240   a43d8xxxxx.elb.us-east-1.amazonaws.com   80:31416/TCP,443:32051/TCP   4d2h
router-internal-default                  ClusterIP      172.30.129.181   <none>
                  80/TCP,443/TCP,1936/TCP      4d2h
router-internal-new-ingress-controller   ClusterIP      172.30.163.0     <none>
                  80/TCP,443/TCP,1936/TCP      2m10s
router-new-ingress-controller            LoadBalancer   172.30.131.253   aba1cxxxxx.elb.us-east-1.amazonaws.com   80:31168/TCP,443:30129/TCP   2m10s
```

router用PodとLoadBalancerタイプのServiceが作成されます。



AWSマネジメントコンソールで確認するとaba1cxxxxxというNLBが作成されています。a43dxxxxxはデフォルトのNLBであり、同じ設定となっています。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/142407/e8f38de3-95a5-b695-bf25-4635d5de69d5.png)

Listener設定はこちら。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/142407/86f98234-9ad6-bbbd-6c25-7275a032f649.png)

タグを見ると`owned`タグが付いていることが確認できます。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/142407/943caa6c-f5a8-e45c-82bf-bb3b28e98512.png)

Target Goupをみるとrouter podが稼働するノードのみhealthyステータスとなっています。（こちらもデフォルトのものと同じ仕様）
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/142407/b4dfcc0a-edab-5ae4-f2b6-6431c6401c19.png)


Route53の設定をAWSマネジメントコンソールで確認すると、`*.new-apps.rosa-～`のレコードが追加で登録されています。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/142407/c0bff958-dfa0-2704-a7d1-cbf183a968f4.png)


## 検証

Podを作成してアクセス確認を行います。

こちらもマニュアルの手順に従います。

<details><summary>検証用環境準備</summary>


```
[user@host home]$ oc new-project test01
Now using project "test01" on server "https://api.rosa-sd-2023dec.5q0v.p1.openshiftapps.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname

[user@host home]$
[user@host home]$ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json
Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "hello-openshift" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "hello-openshift" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "hello-openshift" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "hello-openshift" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/hello-openshift created
[user@host home]$
[user@host home]$ oc get all
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                  READY   STATUS    RESTARTS   AGE
pod/hello-openshift   1/1     Running   0          10s
[user@host home]$
[user@host home]$ oc expose pod/hello-openshift
service/hello-openshift exposed
[user@host home]$
[user@host home]$ oc get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hello-openshift   ClusterIP   172.30.53.183   <none>        8080/TCP   13s
[user@host home]$
[user@host home]$ vi hello-openshift-route.yaml
[user@host home]$ cat hello-openshift-route.yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    type: new-ingress-controller
  name: hello-openshift-edge　　←追加したIngress Controllerを指定する
spec:
  subdomain: hello-openshift
  tls:
    termination: edge
  to:
    kind: Service
    name: hello-openshift
[user@host home]$
[user@host home]$ oc apply -f hello-openshift-route.yaml
route.route.openshift.io/hello-openshift-edge created
[user@host home]$
```
</details>


作成されたrouteの詳細は以下となります。

```
[user@host home]$ oc get route hello-openshift-edge -o yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations:
：
spec:
  subdomain: hello-openshift
  tls:
    termination: edge
  to:
    kind: Service
    name: hello-openshift
    weight: 100
  wildcardPolicy: None
status:
  ingress:
  - conditions:
    - lastTransitionTime: "2023-12-12T08:51:33Z"
      status: "True"
      type: Admitted
    host: hello-openshift.new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
    routerCanonicalHostname: router-new-ingress-controller.new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
    routerName: new-ingress-controller
    wildcardPolicy: None
  - conditions:
    - lastTransitionTime: "2023-12-12T08:51:33Z"
      status: "True"
      type: Admitted
    host: hello-openshift.apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
    routerCanonicalHostname: router-default.apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
    routerName: default
    wildcardPolicy: None
```

hostとして追加したIngress Controllerに紐づく`hello-openshift.new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com`だけでなくデフォルトのIngress Controllerに紐づく`hello-openshift.apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com`の2つが登録されます。
それぞれRoute53では別のNLBに関連付けされています。



それぞれのhostに対して接続確認を行います。

```
[user@host home]$ curl -k https://hello-openshift.new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
Hello OpenShift!
[user@host home]$ curl -k https://hello-openshift.apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
Hello OpenShift!
```

→接続OK。


# おまけ その1（証明書を追加）

新規追加したIngress Controllerに証明書を設定します。

証明書は検証用途ですので、オレオレ証明書を準備します。

## 準備


<details><summary>オレオレ証明書準備</summary>


```command
[user@host home]$ openssl genrsa 2048 > server.key
Generating RSA private key, 2048 bit long modulus
......+++
.....................................................+++
e is 65537 (0x10001)
[user@host home]$ openssl req -new -key server.key > server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:jp
State or Province Name (full name) []:Tokyo
Locality Name (eg, city) [Default City]:Hoge City
Organization Name (eg, company) [Default Company Ltd]:Fuga Org
Organizational Unit Name (eg, section) []:HogeFuga Unit
Common Name (eg, your name or your server's hostname) []:new-ingress-controller
Email Address []:hogefuga@abc.efg

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[user@host home]$ openssl x509 -days 3650 -req -sha256 -signkey server.key < server.csr > server.crt
Signature ok
subject=/C=jp/ST=Tokyo/L=Hoge City/O=Fuga Org/OU=HogeFuga Unit/CN=new-ingress-controller/emailAddress=hogefuga@abc.efg
Getting Private key
[user@host home]$ ls -l
total 20
-rw-rw-r-- 1 user user 1354 Dec 12 09:23 server.crt
-rw-rw-r-- 1 user user 1078 Dec 12 09:22 server.csr
-rw-rw-r-- 1 user user 1679 Dec 12 09:21 server.key
[user@host home]$
```
</details>


## 証明書適用

手順は[こちら](https://access.redhat.com/documentation/en-us/red_hat_openshift_service_on_aws/4/html/networking/configuring-ingress#nw-ingress-setting-a-custom-default-certificate_configuring-ingress)を参照します。

Secret登録

```
[user@host home]$ oc get secret -n openshift-ingress
NAME                                          TYPE                                  DATA   AGE
builder-dockercfg-mdqbc                       kubernetes.io/dockercfg               1      4d4h
builder-token-t5lpx                           kubernetes.io/service-account-token   4      4d4h
default-dockercfg-nvkn9                       kubernetes.io/dockercfg               1      4d4h
default-token-zvjz7                           kubernetes.io/service-account-token   4      4d4h
deployer-dockercfg-d8tgj                      kubernetes.io/dockercfg               1      4d4h
deployer-token-xchqn                          kubernetes.io/service-account-token   4      4d4h
rosa-sd-2023dec-primary-cert-bundle-secret    kubernetes.io/tls                     2      4d3h
router-certs-new-ingress-controller           kubernetes.io/tls                     2      99m
router-dockercfg-qnp5k                        kubernetes.io/dockercfg               1      4d4h
router-metrics-certs-default                  kubernetes.io/tls                     2      4d4h
router-metrics-certs-new-ingress-controller   kubernetes.io/tls                     2      99m
router-stats-default                          Opaque                                2      4d4h
router-stats-new-ingress-controller           Opaque                                2      99m
router-token-mqpw6                            kubernetes.io/service-account-token   4      4d4h
[user@host home]$
[user@host home]$ oc create secret tls new-ingress-controller-cert --cert=server.crt --key=server.key -n openshift-ingress
secret/new-ingress-controller-cert created
[user@host home]$
[user@host home]$ oc get secret -n openshift-ingress
NAME                                          TYPE                                  DATA   AGE
builder-dockercfg-mdqbc                       kubernetes.io/dockercfg               1      4d4h
builder-token-t5lpx                           kubernetes.io/service-account-token   4      4d4h
default-dockercfg-nvkn9                       kubernetes.io/dockercfg               1      4d4h
default-token-zvjz7                           kubernetes.io/service-account-token   4      4d4h
deployer-dockercfg-d8tgj                      kubernetes.io/dockercfg               1      4d4h
deployer-token-xchqn                          kubernetes.io/service-account-token   4      4d4h
new-ingress-controller-cert                   kubernetes.io/tls                     2      9s
rosa-sd-2023dec-primary-cert-bundle-secret    kubernetes.io/tls                     2      4d4h
router-certs-new-ingress-controller           kubernetes.io/tls                     2      101m
router-dockercfg-qnp5k                        kubernetes.io/dockercfg               1      4d4h
router-metrics-certs-default                  kubernetes.io/tls                     2      4d4h
router-metrics-certs-new-ingress-controller   kubernetes.io/tls                     2      101m
router-stats-default                          Opaque                                2      4d4h
router-stats-new-ingress-controller           Opaque                                2      101m
router-token-mqpw6                            kubernetes.io/service-account-token   4      4d4h
[user@host home]$
```

追加したSecretをIngress Controllerに設定

```
[user@host home]$ oc patch --type=merge -n openshift-ingress-operator ingresscontrollers/new-ingress-controller --patch '{"spec":{"defaultCertificate":{"name":"new-ingress-controller-cert"}}}'
ingresscontroller.operator.openshift.io/new-ingress-controller patched
```

設定確認（抜粋）

```
[user@host home]$ oc get ingresscontroller -n openshift-ingress-operator new-ingress-controller -o yaml
apiVersion: operator.openshift.io/v1
kind: IngressController
：
spec:
  clientTLS:
    clientCA:
      name: ""
    clientCertificatePolicy: ""
  defaultCertificate:
    name: new-ingress-controller-cert　　←こちらが設定された
  domain: new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
  endpointPublishingStrategy:
    loadBalancer:
      dnsManagementPolicy: Managed
      scope: Internal
    type: LoadBalancerService
```

設定後router Podが作り直されます。

```
[user@host home]$ oc get pod -n openshift-ingress
NAME                                             READY   STATUS        RESTARTS   AGE
router-default-bf4775545-bzhzq                   1/1     Running       0          4d3h
router-default-bf4775545-z2b2j                   1/1     Running       0          4d3h
router-new-ingress-controller-74756b76db-59d6v   1/1     Running       0          41s
router-new-ingress-controller-74756b76db-bg6x8   1/1     Running       0          41s
router-new-ingress-controller-d4795df69-d5rld    1/1     Terminating   0          103m
router-new-ingress-controller-d4795df69-tjtx4    0/1     Terminating   0          103m
```

## 証明書確認

アクセス確認

```
[user@host home]$ curl -k https://hello-openshift.new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
Hello OpenShift!
```

証明書確認

```
[user@host home]$ echo Q | openssl s_client -connect hello-openshift.new-apps.rosa-sd-2023dec.5q0v.p1.openshif
tapps.com:443 -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -enddate
subject= /C=jp/ST=Tokyo/L=Hoge City/O=Fuga Org/OU=HogeFuga Unit/CN=new-ingress-controller/emailAddress=hogefuga@abc.efg
issuer= /C=jp/ST=Tokyo/L=Hoge City/O=Fuga Org/OU=HogeFuga Unit/CN=new-ingress-controller/emailAddress=hogefuga@abc.efg
notAfter=Dec  9 09:23:13 2033 GMT
```

作成した証明書の情報が確認できた。


# おまけ その2（アクセスログ出力設定）

せっかくなのでアクセスログも有効化してみます。この設定によりrouter podでのアクセスログが出力されるようになります。

手順書は[こちら](https://access.redhat.com/documentation/en-us/red_hat_openshift_service_on_aws/4/html/networking/configuring-ingress#nw-configure-ingress-access-logging_configuring-ingress)を参照します。

## 準備

Ingress Controllerの定義にアクセスログをサイドカーコンテナに出力するよう設定します。

```yaml
[user@host home]$ cat ingresscontroller_log.yaml
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  namespace: openshift-ingress-operator
  name: new-ingress-controller
spec:
  domain: new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
  endpointPublishingStrategy:
    type: LoadBalancerService
    loadBalancer:
      scope: Internal
  logging:
    access:
      destination:
        type: Container
```

適用すると、新規Podが起動します。

```
[user@host home]$ oc apply -f ingresscontroller_log.yaml
ingresscontroller.operator.openshift.io/new-ingress-controller configured
[user@host home]$
[user@host home]$ oc get pod -n openshift-ingress
NAME                                             READY   STATUS        RESTARTS   AGE
router-default-bf4775545-bzhzq                   1/1     Running       0          4d4h
router-default-bf4775545-z2b2j                   1/1     Running       0          4d4h
router-new-ingress-controller-6966f6c87c-72tk4   2/2     Running       0          32s
router-new-ingress-controller-6966f6c87c-vvj7z   2/2     Running       0          32s
router-new-ingress-controller-74756b76db-59d6v   1/1     Terminating   0          13m
router-new-ingress-controller-74756b76db-bg6x8   0/1     Terminating   0          13m
```

## 確認

アクセス確認

```
[user@host home]$ curl -k https://hello-openshift.new-apps.rosa-sd-2023dec.5q0v.p1.openshiftapps.com
Hello OpenShift!
```

アクセスログ確認

```
[user@host home]$  oc -n openshift-ingress logs deployment.apps/router-new-ingress-controller -c logs
Found 2 pods, using pod/router-new-ingress-controller-6966f6c87c-72tk4
rsyslogd 8.2102.0-7.el8_6.1: running as pid 1, enabling container-specific defaults, press ctl-c to terminate rsyslog
2023-12-12T09:44:07.138656+00:00 router-new-ingress-controller-6966f6c87c-72tk4 router-new-ingress-controller-6966f6c87c-72tk4 haproxy[19]: 127.0.0.1:37026 [12/Dec/2023:09:44:07.138] public openshift_default/<NOSRV> 0/-1/-1/-1/0 503 152 - - SC-- 1/1/0/0/0 0/0 "HEAD / HTTP/1.1"
2023-12-12T09:46:48.997514+00:00 router-new-ingress-controller-6966f6c87c-72tk4 router-new-ingress-controller-6966f6c87c-72tk4 haproxy[19]: 10.0.17.126:42864 [12/Dec/2023:09:46:48.994] fe_sni~ be_edge_http:test01:hello-openshift-edge/pod:hello-openshift:hello-openshift::10.130.2.58:8080 0/0/2/0/2 200 137 - - --NI 2/1/0/0/0 0/0 "GET / HTTP/1.1"
2023-12-12T09:46:48.998136+00:00 router-new-ingress-controller-6966f6c87c-72tk4 router-new-ingress-controller-6966f6c87c-72tk4 haproxy[19]: 10.0.17.126:42864 [12/Dec/2023:09:46:48.991] public_ssl be_sni/fe_sni 1/0/6 1783 -- 1/1/0/0/0 0/0
```

アクセスしたログが確認できました。



# まとめ

ROSA v4.14からIngress Controllerの設定変更の自由度が高まりました。本記事で紹介したようにIngress Controllerの新規作成、証明書設定、アクセスログ設定を合わせて実施して、問題なく動作することが確認できました。