sample-controllerを動かしてOperatorについて学ぶ

## はじめに
Operatorの説明を読んでもOperatorとHelmの違いがよく分からん、ということでサンプルを動かしながら学んでみました。

## 参考
sample-controllerのソースコード：https://github.com/kubernetes/sample-controller
Dockerfile：https://github.com/uzimihsr/sample-controller/blob/in-cluster-config/Dockerfile
手順参考：https://blog.uzimihsr.com/post/2021-07-13-kubernetes-run-sample-controller-as-deployment/

## 用語
| 用語 | 意味 |
|:-:|:-:|
|`CRD`|Kubernetesの拡張機能である`CR`を利用するための定義   |
|`CR`|  `CRD`で定義した後、通常のリソースと同じように作成できるようになる。`spec`にどのようなフィールドを設定するかは作成者が決められる |
|`Custom Contoroller`|   CRの管理を行うController.Reconciliation Loopを実行するコンポーネント|
|`Operotor`|  CRDとCustom Controllerのセット。特定のソフトウェアの管理を自動化するためのソフトウェア。|

## Sample Controllerについて
client-goとcode-generatorを使ったKubernetes Wayで実装されたサンプルのcontroller。

Deploymentの上位リソースである`Foo`を管理するController。
ReplicasetオブジェクトをDeploymentが管理するように、DeploymentオブジェクトをFooが管理できるようにする。
Foo ControllerはNginxイメージを使ったDeploymentを管理し、Fooが管理しているDeploymentオブジェクトを追加・削除されると即時にDeploymentを望ましい状態に調整する。

### OpenShift上で動かしてみる
1.まずはソースコードを手元にclone
```
$ git clone https://github.com/kubernetes/sample-controller
Cloning into 'sample-controller'...
remote: Enumerating objects: 27266, done.
remote: Counting objects: 100% (6353/6353), done.
remote: Compressing objects: 100% (896/896), done.
remote: Total 27266 (delta 5618), reused 5589 (delta 5439), pack-reused 20913
Receiving objects: 100% (27266/27266), 11.84 MiB | 3.29 MiB/s, done.
Resolving deltas: 100% (19121/19121), done.

$ cd sample-controller/

$ ls
artifacts/          CONTRIBUTING.md  controller_test.go  go.mod  hack/    main.go  pkg/       SECURITY_CONTACTS
code-of-conduct.md  controller.go    docs/               go.sum  LICENSE  OWNERS   README.md

$ vi Dockerfile

$ ls
CONTRIBUTING.md  LICENSE  README.md          artifacts           controller.go       docs    go.sum  main.go
Dockerfile       OWNERS   SECURITY_CONTACTS  code-of-conduct.md  controller_test.go  go.mod  hack    pkg
```
2.openshiftにログイン後、イメージをビルドする
```
$ oc new-project sample-controller
Now using project "sample-controller" on server "https://XXX.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname

$ oc new-build --binary --strategy=docker --name sample
-controller
    * A Docker build using binary input will be created
      * The resulting image will be pushed to image stream tag "sample-controller:latest"
      * A binary build was created, use 'oc start-build --from-dir' to trigger a new build

--> Creating resources with label build=sample-controller ...
    imagestream.image.openshift.io "sample-controller" created
    buildconfig.build.openshift.io "sample-controller" created
--> Success
$ oc start-build sample-controller --from-dir .
Uploading directory "." as binary input for the build ...
..
Uploading finished
build.build.openshift.io/sample-controller-1 started
```

3.CRDを作成する
```
$ cd artifacts/examples/
$ ls
crd-status-subresource.yaml  crd.yaml  example-foo.yaml
$ oc apply -f crd.yaml
customresourcedefinition.apiextensions.k8s.io/foos.samplecontroller.k8s.io created

$ oc get crd | grep samplecontroller
foos.samplecontroller.k8s.io

$ oc api-resources | grep foo
foos
 samplecontroller.k8s.io/v1alpha1              true         Foo
```
4.CRを作成する
```
$ oc apply -f example-foo.yaml
foo.samplecontroller.k8s.io/example-foo created
$ oc get foo
NAME          AGE
example-foo   60s
$ oc get foo -o yaml
apiVersion: v1
items:
- apiVersion: samplecontroller.k8s.io/v1alpha1
  kind: Foo
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"samplecontroller.k8s.io/v1alpha1","kind":"Foo","metadata":{"annotations":{},"name":"example-foo","namespace":"sample-controller"},"spec":{"deploymentName":"example-foo","replicas":1}}
    creationTimestamp: "2024-08-01T07:21:53Z"
    generation: 1
    name: example-foo
    namespace: sample-controller
    resourceVersion: "2618365"
    uid: 02da87b8-b7e2-4aed-bf30-59127c7ecb82
  spec:
    deploymentName: example-foo
    replicas: 1
kind: List
metadata:
  resourceVersion: ""
  ```
:::note info
CRDとCRはcontrollerがなくても作成することが可能。
しかしCRDとCRというリソースが作成されただけで、**何の効果もないオブジェクトが存在するだけ**の状態。
:::

4.controllerをdeploymentとしてデプロイする
```
$ oc create deploy sample-controller
 --image=image-registry.openshift-image-registry.svc:5000/sample-controller/sample-controller
deployment.apps/sample-controller created
$ oc get pod
NAME                                 READY   STATUS      RESTARTS   AGE
sample-controller-1-build            0/1     Completed   0          16m
sample-controller-5cb7844bc9-6dk8f   1/1     Running     0          12s
```

5.deploymentとfooを参照する権限がないとエラーが出てくる
```
$ oc logs sample-controller-5cb7844b
c9-6dk8f
・
・
・
E0801 07:28:26.879984       1 reflector.go:158] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.0.0-20240727175048-71959c526d54/tools/cache/reflector.go:243: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:sample-controller:default\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope" logger="UnhandledError"
W0801 07:28:34.472872       1 reflector.go:561] pkg/mod/k8s.io/client-go@v0.0.0-20240727175048-71959c526d54/tools/cache/reflector.go:243: failed to list *v1alpha1.Foo: foos.samplecontroller.k8s.io is forbidden: User "system:serviceaccount:sample-controller:default" cannot list resource "foos" in API group "samplecontroller.k8s.io" at the cluster scope
E0801 07:28:34.472919       1 reflector.go:158] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.0.0-20240727175048-71959c526d54/tools/cache/reflector.go:243: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: foos.samplecontroller.k8s.io is forbidden: User \"system:serviceaccount:sample-controller:default\" cannot list resource \"foos\" in API group \"samplecontroller.k8s.io\" at the cluster scope" logger="UnhandledError"
```

6.権限をつける
```
$ oc create clusterrole foo-control --verb="*" --resource=foo,deployment
clusterrole.rbac.authorization.k8s.io/foo-control created

foo-control   2024-08-01T07:32:45Z
$ oc get clusterrole foo-control -o
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2024-08-01T07:32:45Z"
  name: foo-control
  resourceVersion: "2624839"
  uid: 18688cce-1c13-4576-bc9e-f381eeb643b5
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - '*'
- apiGroups:
  - samplecontroller.k8s.io
  resources:
  - foos
  verbs:
  - '*'

$ oc create clusterrolebinding foo-control-binding --clusterrole=foo-control --serviceaccount=sample-controller:default
clusterrolebinding.rbac.authorization.k8s.io/foo-control-binding created:q:
```

しかしエラー
```
oc logs sample-controller-5cb7844bc9-n5vqj
・
・
・
E0801 07:57:44.677381       1 controller.go:228] "Error syncing; requeuing for later retry" err="deployments.apps \"example-foo\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>" logger="UnhandledError" objectReference="sample-controller/example-foo"
```

7.取り敢えず動かしたいだけなので、権限が強すぎるが以下のclusterroleに変更
（この辺りを調べる学ぶ必要がありそうhttps://kubernetes.io/ja/docs/concepts/overview/working-with-objects/owners-dependents/ ）
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2024-08-01T07:32:45Z"
  name: foo-control
  resourceVersion: "2639689"
  uid: 18688cce-1c13-4576-bc9e-f381eeb643b5
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
```
fooで管理されるdeploymentができたことを確認
```
$ oc get deploy
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
example-foo         1/1     1            1           8m10s
sample-controller   1/1     1            1           12m
```

8.fooのreplica数を1から5に変更
```diff
apiVersion: samplecontroller.k8s.io/v1alpha1
kind: Foo
metadata:
  name: example-foo
  namespace: sample-controller
spec:
  deploymentName: example-foo
- replicas: 1
+ replicas: 5
```

deploymentが5つに増えていることを確認
```
$ oc get deploy
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
example-foo         5/5     5            5           10m
sample-controller   1/1     1            1           15m
```