Repository: canonical/microk8s

Issue:
CSINode is not yet initialized after enabling RBAC 
<!--
   Thank you for submitting an issue. Please fill in the template below
   information about the bug you encountered.
-->

#### Summary
Enabled RBAC add-on and one of three nodes shows *CSINode is not yet initialized*

#### What Should Happen Instead?
Expected all nodes to be Ready after enabling RBAC addon

#### Reproduction Steps

1. Install Ubuntu Core 22 on 3x Intel NUC 11 systems2. 
2. Install microk8s with `snap install microk8s --channel=latest/edge/strict` 
3. Verify all nodes online with `kubectl get nodes -owide`
4. Enable RBAC with `microk8s enable rbac`

[inspection-report-20220622_185138.tar.gz](https://github.com/canonical/microk8s/files/8960620/inspection-report-20220622_185138.tar.gz)

#### Can you suggest a fix?
In my case, k8s-node-03 shows CSINode is not yet initialized preventing the node from coming ready. Possibly related to RBAC permissions not allowing the node to update a resource?





Discussion:
Hi @scottrus

I am not able to reproduce this issue. The strict branch is under development so it is possible this issue is fixed as I remember there was something similar on the classic branch. Where do you see the `CSINode is not yet initialized`. Could you share the respective log file and a dmesg output?
Is there a more stable branch that can be used on Core 22? I’m not able to run `--classic`.

I removed MicroK8s snap and used `hostnamectl` to change the system name from `k8s-node-03.localdomain` to just `k8s-node-03`. After that I could not recreate the issue either. 

It’s not clear if something changed in the strict branch or the hostname change resolved it.
> Is there a more stable branch that can be used on Core 22?

On Core you need a strictly confined snap and right now the latest/edge/strict branch provides the only such build of MicroK8s. With 1.25 we will have a dedicated track.
We are facing the same issue on a node with `v1.21.13-3+cbc10c94808907`. It causes the node to be unready with the following conditions:

```
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Sun, 20 Nov 2022 00:52:25 +0100   Sun, 20 Nov 2022 00:52:25 +0100   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Sun, 27 Nov 2022 09:21:50 +0100   Sun, 27 Nov 2022 09:21:50 +0100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Sun, 27 Nov 2022 09:21:50 +0100   Sun, 27 Nov 2022 09:21:50 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Sun, 27 Nov 2022 09:21:50 +0100   Sun, 27 Nov 2022 09:21:50 +0100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                False   Sun, 27 Nov 2022 09:21:50 +0100   Sun, 27 Nov 2022 09:21:50 +0100   KubeletNotReady              CSINode is not yet initialized
```
We ended up using the following workaround:

Add 

```
--feature-gates=CSIMigration=false
```

to the end of `/var/snap/microk8s/current/args/kubelet` and then restart `kubelite` with 

```
sudo systemctl restart snap.microk8s.daemon-kubelite
```

We are still waiting on final confirm whether the node can start all pods again, but currently the node is in ready state:

```
Ready                True    Sun, 27 Nov 2022 09:47:24 +0100   Sun, 27 Nov 2022 09:47:24 +0100   KubeletReady                 kubelet is posting ready status. AppArmor enabled
```

Edit: We still had problems with pods not being allowed to create service accounts, so we ultimately ended up disabling rbac again with

```
microk8s disable rbac
```
Confirm still having this problem on v1.26.0 revision 4390.
Have to disable RBAC to work around ATM.
Had the same problem on my laptop, hostname had an uppercase and that created the following permission problems (from `snap logs -f microk8s`):

```
2023-01-05T14:26:35+01:00 microk8s.daemon-kubelite[10253]: W0105 14:26:35.588382   10253 reflector.go:424] object-"kube-system"/"kube-root-ca.crt": failed to list *v1.ConfigMap: configmaps "kube-root-ca.crt" is forbidden: User "system:node:Luhman16" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'Luhman16' and this object
2023-01-05T14:26:35+01:00 microk8s.daemon-kubelite[10253]: E0105 14:26:35.588426   10253 reflector.go:140] object-"kube-system"/"kube-root-ca.crt": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kube-root-ca.crt" is forbidden: User "system:node:Luhman16" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'Luhman16' and this object
2023-01-05T14:26:35+01:00 microk8s.daemon-kubelite[10253]: W0105 14:26:35.588611   10253 reflector.go:424] object-"kube-system"/"kubernetes-services-endpoint": failed to list *v1.ConfigMap: configmaps "kubernetes-services-endpoint" is forbidden: User "system:node:Luhman16" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'Luhman16' and this object
2023-01-05T14:26:35+01:00 microk8s.daemon-kubelite[10253]: W0105 14:26:35.588639   10253 reflector.go:424] object-"kube-system"/"calico-config": failed to list *v1.ConfigMap: configmaps "calico-config" is forbidden: User "system:node:Luhman16" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'Luhman16' and this object
2023-01-05T14:26:35+01:00 microk8s.daemon-kubelite[10253]: E0105 14:26:35.588660   10253 reflector.go:140] object-"kube-system"/"kubernetes-services-endpoint": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kubernetes-services-endpoint" is forbidden: User "system:node:Luhman16" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'Luhman16' and this object
2023-01-05T14:26:35+01:00 microk8s.daemon-kubelite[10253]: E0105 14:26:35.588667   10253 reflector.go:140] object-"kube-system"/"calico-config": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "calico-config" is forbidden: User "system:node:Luhman16" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'Luhman16' and this object
```

Fixed the problem by:
- changing the hostname of my laptop to lower case
- `snap remove --purge microk8s`
- reboot
- `snap install microk8s`
- `microk8s enable rbac`
@michael-mri 
Thank you. I confirm that change to lower case fixed the issue.
Now my cluster are up and running without any issue.
As much as I appreciate the effort @KagariSan I would like to politely disagree with the idea that this issue can be closed as completed @ktsakalozos. The shell script changes in #3681 offer a workaround, but the idea that something as vital as RBAC does not appear to work with hostnames with upper-case letters in them should be resolved, not warned, I believe. @ktsakalozos could you be so kind as to reopen the issue?

Resolving patch:
diff --git a/scripts/wrappers/join.py b/scripts/wrappers/join.py
--- a/scripts/wrappers/join.py
+++ b/scripts/wrappers/join.py
@@ -153,7 +153,7 @@ def get_connection_info(
         if cluster_type == "dqlite":
             req_data = {
                 "token": token,
-                "hostname": socket.gethostname(),
+                "hostname": socket.gethostname().lower(),
                 "port": cluster_agent_port,
                 "worker": worker,
             }
@@ -162,7 +162,7 @@ def get_connection_info(
         else:
             req_data = {
                 "token": token,
-                "hostname": socket.gethostname(),
+                "hostname": socket.gethostname().lower(),
                 "port": cluster_agent_port,
                 "callback": callback_token,
             }
@@ -450,7 +450,7 @@ def update_cert_auth_kubelet(token, ca, master_ip, master_port):
     """
     kubelet_token = "{}-kubelet".format(token)
     traefik_port = get_traefik_port()
-    kubelet_user = "system:node:{}".format(socket.gethostname())
+    kubelet_user = "system:node:{}".format(socket.gethostname().lower())
     cert = get_client_cert(
         master_ip, master_port, "kubelet", kubelet_token, kubelet_user, "system:nodes"
     )


Test patch: