Red Hat OpenShift on IBM Cloudに自作Helm Operatorをデプロイする

# 目的

過去記事では、Operator SDKを使って自作のAnsible Operatorを開発しました。今回はHelm Operatorを開発してみます。

https://qiita.com/teruq/items/4fc4193a5e1de2b3802b

https://qiita.com/teruq/items/34d9120ffeba75052b09

なお、今回は簡単のためにコンテナレジストリはDocker Hubを使用します。

# 手順

## プロジェクトとAPIひな型の生成

```
$ mkdir hello-helm-operator
$ cd hello-helm-operator/
$ operator-sdk init --plugins helm --domain teruq.example.com
$ operator-sdk create api --group example --version v1alpha1 --kind Hello
```

## Operator Deploymentマニフェストの修正（オプション）

必須ではありませんが、トライアンドエラーするのに便利ですので``ImagePullPolicy: Always``を追加します。

```config/manager/manager.yaml
      containers:
      - args:
        - --leader-elect
        - --leader-election-id=hello-helm-operator
        image: controller:latest
        imagePullPolicy: Always  # 追加
        name: manager
        securityContext:
          allowPrivilegeEscalation: false
```

## CustomResourceDefinitionの修正

``replicaCount``プロパティを必須指定します。このプロパティ名は後述する``values.yaml``のキーと一致させる必要があります。

```config/crd/bases/example.teruq.example.com_hellos.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: hellos.example.teruq.example.com
spec:
  group: example.teruq.example.com
  names:
    kind: Hello
    listKind: HelloList
    plural: hellos
    singular: hello
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Hello is the Schema for the hellos API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: Spec defines the desired state of Hello
            type: object
            #x-kubernetes-preserve-unknown-fields: true                   # 削除
            properties:                                                   # 追加
              replicaCount:                                               # 追加
                description: The number of replicas of Hello deployment.  # 追加
                type: integer                                             # 追加
                default: 1                                                # 追加
            required:                                                     # 追加
            - replicaCount                                                # 追加
          status:
            description: Status defines the observed state of Hello
            type: object
            x-kubernetes-preserve-unknown-fields: true
        type: object
    served: true
    storage: true
    subresources:
      status: {}
```

## Helmテンプレートの修正

``containerPortが80番固定になっていますが、今回使うコンテナは8080で起動しますので、8080に修正します。

```helm-charts/hello/templates/deployment.yaml
          ports:
            - name: http
              containerPort: 80
              →
              containerPort: 8080
```

## values.yamlの修正

3か所修正します。

* image.repository
* image.tag
* serviceAccount.create
* service.port

values.yaml内の``replicaCount``は、先ほどCustomResourceDefinitionで指定したプロパティ名と名前が一致しています。

```helm-charts/hello/values.yaml
# Default values for hello.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: docker.io/ibmcom/hello  # nginxから変更
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: latest                         # タグを付与

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: false                       # trueから変更
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

podAnnotations: {}

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 8080                           # 80から変更

ingress:
  enabled: false
  className: ""
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}
```

## Operatorのビルドとプッシュ

Operatorイメージをビルドしプッシュします。今回は簡単のためにDocker Hubを使用します。

```
$ make docker-build docker-push IMG=docker.io/teruq/hello-helm-operator:0.0.1
```

## Operatorの手動デプロイ

クラスターにOperatorをデプロイします。

```
$ make deploy IMG=docker.io/teruq/hello-helm-operator:0.0.1
cd config/manager && /.../hello-helm-operator/bin/kustomize edit set image controller=docker.io/teruq/hello-helm-operator:0.0.1
/.../hello-helm-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/hello-helm-operator-system created
customresourcedefinition.apiextensions.k8s.io/hellos.example.teruq.example.com created
serviceaccount/hello-helm-operator-controller-manager created
role.rbac.authorization.k8s.io/hello-helm-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/hello-helm-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/hello-helm-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/hello-helm-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/hello-helm-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/hello-helm-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/hello-helm-operator-proxy-rolebinding created
configmap/hello-helm-operator-manager-config created
service/hello-helm-operator-controller-manager-metrics-service created
deployment.apps/hello-helm-operator-controller-manager created
```

Operator Podが起動したことを確認します。

```
$ oc get pods -n hello-helm-operator-system
NAME                                                      READY   STATUS    RESTARTS   AGE
hello-helm-operator-controller-manager-68d85df8bb-bvtpq   2/2     Running   1          50s
```

## 動作確認

### CustomResourceの作成

``replicaCount``を2にしてHello CustomResourceを作成します。Helm経験者なら想像がつくと思いますが、values.yamlを--valuesで上書きするイメージです。

```sample-cr.yaml
apiVersion: example.teruq.example.com/v1alpha1
kind: Hello
metadata:
  name: sample
spec:
  replicaCount: 2
```

### CustomResourceの登録

CustomResourceを登録します。

```
$ oc create -f sample-cr.yaml -n qiita
```

Podが2つ起動しました。

```
$ oc get pods -n qiita
NAME                                    READY   STATUS    RESTARTS   AGE
sample-hello-5dc7dd9ddb-6vjnm           1/1     Running   0          13s
sample-hello-5dc7dd9ddb-7rl84           1/1     Running   0          13s
```

### CustomResourceの変更

``replicaCount``を3に変更します。

```
$ oc patch hello sample --type merge -p '{"spec": {"replicaCount": 3}}' -n qiita
hello.example.teruq.example.com/sample patched
```

Pod数が3個になりました。

```
$ oc get pods -n qiita
NAME                                    READY   STATUS    RESTARTS   AGE
sample-hello-5dc7dd9ddb-6vjnm           1/1     Running   0          5m42s
sample-hello-5dc7dd9ddb-7rl84           1/1     Running   0          5m42s
sample-hello-5dc7dd9ddb-rgz9f           1/1     Running   0          20s
```

### Serviceを削除

Service名を確認します。

```
$ oc get svc -n qiita
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
sample-hello     ClusterIP   172.21.235.140   <none>        8080/TCP   6m29s
```

Serviceを削除します。

```
$ oc delete svc sample-hello -n qiita
service "sample-hello" deleted
```

Operatorにより自動的に復活しました。

```
$ oc get svc -n qiita
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
sample-hello     ClusterIP   172.21.80.121   <none>        8080/TCP   7s
```

### クリーンアップ

次の作業のためにOperatorとCustomResourceを削除します。

```
$ oc delete -f sample-cr.yaml
$ make undeploy
```

## OperatorのOLMインストール

### Bundleイメージのビルドとプッシュ

Bundleを作成します。

```
$ make bundle IMG=docker.io/teruq/hello-helm-operator-bundle:v0.0.1
operator-sdk generate kustomize manifests -q

Display name for the operator (required):
> Hello Sample

Description for the operator (required):
> Helm Operator for hello app

Provider's name for the operator (required):
> teruq

Any relevant URL for the provider name (optional):
> https://qiita.com/teruq

Comma-separated list of keywords for your operator (required):
> hello,helm

Comma-separated list of maintainers and their emails (e.g. 'name1:email1, name2:email2') (required):
> teruq
cd config/manager && /.../hello-helm-operator/bin/kustomize edit set image controller=docker.io/teruq/hello-helm-operator:0.0.1
/.../hello-helm-operator/bin/kustomize build config/manifests | operator-sdk generate bundle -q --overwrite --version 0.0.1
INFO[0003] Creating bundle.Dockerfile
INFO[0003] Creating bundle/metadata/annotations.yaml
INFO[0003] Bundle metadata generated suceessfully
operator-sdk bundle validate ./bundle
INFO[0000] All validation tests have completed successfully
```

Bundleイメージをビルドしプッシュします。

```
$ make bundle-build bundle-push BUNDLE_IMG=docker.io/teruq/hello-helm-operator-bundle:v0.0.1
```

### OperatorのOLMインストール

BundleをOLMでインストールします。

```
$ operator-sdk run bundle docker.io/teruq/hello-helm-operator-bundle:v0.0.1 -n qiita-operators
INFO[0014] Successfully created registry pod: docker-io-teruq-hello-helm-operator-bundle-v0-0-1
INFO[0014] Created CatalogSource: hello-helm-operator-catalog
INFO[0015] OperatorGroup "operator-sdk-og" created
INFO[0015] Created Subscription: hello-helm-operator-v0-0-1-sub
INFO[0025] Approved InstallPlan install-r7v8m for the Subscription: hello-helm-operator-v0-0-1-sub
INFO[0025] Waiting for ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" to reach 'Succeeded' phase
INFO[0025]   Waiting for ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" to appear
INFO[0032]   Found ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" phase: Pending
FATA[0120] Failed to run bundle: error waiting for CSV to install: timed out waiting for the condition
```

エラーが発生しました。状況を確認します。

```
$ oc get csv hello-helm-operator.v0.0.1 -o yaml
...
  requirementStatus:
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    message: CRD is present and Established condition is true
    name: hellos.example.teruq.example.com
    status: Present
    uuid: 049ca9e6-1523-4255-949b-3178f0b2422b
    version: v1
  - group: ""
    kind: ServiceAccount
    message: Service account does not exist
    name: hello-helm-operator-controller-manager
    status: NotPresent
    version: v1
```

ServiceAccountがないということです。Ansible Operatorのときは自動で作成されたので、なぜ今回これがないかは不明です。別途調査することにして、今回は手動でServiceAccountを作成します。

```
$ oc create sa hello-helm-operator-controller-manager -n qiita-operators
serviceaccount/hello-helm-operator-controller-manager created
```

クリーンアップして再度インストールします。

```
$ operator-sdk cleanup hello-helm-operator
INFO[0000] subscription "hello-helm-operator-v0-0-1-sub" deleted
INFO[0000] customresourcedefinition "hellos.example.teruq.example.com" deleted
INFO[0000] clusterserviceversion "hello-helm-operator.v0.0.1" deleted
INFO[0000] catalogsource "hello-helm-operator-catalog" deleted
INFO[0000] operatorgroup "operator-sdk-og" deleted
INFO[0000] Operator "hello-helm-operator" uninstalled

$ operator-sdk run bundle docker.io/teruq/hello-helm-operator-bundle:v0.0.1 -n qiita-operators
INFO[0014] Successfully created registry pod: docker-io-teruq-hello-helm-operator-bundle-v0-0-1
INFO[0014] Created CatalogSource: hello-helm-operator-catalog
INFO[0014] OperatorGroup "operator-sdk-og" created
INFO[0014] Created Subscription: hello-helm-operator-v0-0-1-sub
INFO[0019] Approved InstallPlan install-sntr2 for the Subscription: hello-helm-operator-v0-0-1-sub
INFO[0019] Waiting for ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" to reach 'Succeeded' phase
INFO[0019]   Waiting for ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" to appear
INFO[0026]   Found ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" phase: Pending
INFO[0028]   Found ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" phase: Installing
INFO[0052]   Found ClusterServiceVersion "qiita-operators/hello-helm-operator.v0.0.1" phase: Succeeded
INFO[0052] OLM has successfully installed "hello-helm-operator.v0.0.1"
```

今度は成功したので、CustomResourceを登録します。

```
$ oc create -f sample-cr.yaml -n qiita
hello.example.teruq.example.com/sample created
```

無事Podが起動しました。

```
$ oc get pods -n qiita
NAME                                    READY   STATUS    RESTARTS   AGE
sample-hello-5dc7dd9ddb-fzztp           1/1     Running   0          12s
sample-hello-5dc7dd9ddb-q4cjv           1/1     Running   0          12s
```

### クリーンアップ

```
$ operator-sdk cleanup hello-helm-operator
```