Repository: canonical/microk8s

Issue:
Leaving a worker node always leads to a broken state.
#### Summary
MicroK8S 1.26: doing a "microk8s leave" command on a worker node that has previously joined a cluster, always leave microk8s in a broken state. MicroK8S will regenerate certificates and will try to restart without success.

#### What Should Happen Instead?
MicroK8S status should give a "microk8s is running" message.

#### Reproduction Steps
Create a microk8s cluster with 3 master nodes and 2 worker nodes with "microk8s add-node" command.
Finally, when everything is up and running, on both worker nodes do a "microk8s leave" command.
You won't be able to restart microk8s anymore on worker nodes.

#### Introspection Report
"microk8s inspect" will trigger the following error:
FAIL:  Service snap.microk8s.daemon-kubelite is not running

The command "snap logs microk8s.daemon-kubelite" will print:

2023-01-30T00:14:43Z microk8s.daemon-kubelite[234081]: W0130 00:14:43.528692  234081 authentication.go:520] AnonymousAuth is not allowed with the AlwaysAllow authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
2023-01-30T00:14:43Z microk8s.daemon-kubelite[234081]: Error: failed to parse service-account-issuer-key-file: error reading private key file /var/snap/microk8s/4390/certs/serviceaccount.key: data does not contain a valid RSA or ECDSA private key
2023-01-30T00:14:43Z microk8s.daemon-kubelite[234081]: F0130 00:14:43.528708  234081 daemon.go:67] API Server exited failed to parse service-account-issuer-key-file: error reading private key file /var/snap/microk8s/4390/certs/serviceaccount.key: data does not contain a valid RSA or ECDSA private key
2023-01-30T00:14:43Z systemd[1]: snap.microk8s.daemon-kubelite.service: Main process exited, code=exited, status=255/EXCEPTION
2023-01-30T00:14:43Z systemd[1]: snap.microk8s.daemon-kubelite.service: Failed with result 'exit-code'.
2023-01-30T00:14:43Z systemd[1]: snap.microk8s.daemon-kubelite.service: Scheduled restart job, restart counter is at 5.
2023-01-30T00:14:43Z systemd[1]: Stopped Service for snap application microk8s.daemon-kubelite.
2023-01-30T00:14:43Z systemd[1]: snap.microk8s.daemon-kubelite.service: Start request repeated too quickly.
2023-01-30T00:14:43Z systemd[1]: snap.microk8s.daemon-kubelite.service: Failed with result 'exit-code'.
2023-01-30T00:14:43Z systemd[1]: Failed to start Service for snap application microk8s.daemon-kubelite.

Report attached:
[inspection-report-20230130_002026.tar.gz](https://github.com/canonical/microk8s/files/10531264/inspection-report-20230130_002026.tar.gz)



Discussion:
Thank you for reporting this @mjordan79 . It seems to be an issue in the `leave` script [1]. Other than the obvious workaround of re installing the node that left the cluster we do not have another immediate solution. We should have news on this soon.

[1] https://github.com/canonical/microk8s/blob/master/scripts/wrappers/leave.py

Resolving patch:
diff --git a/scripts/wrappers/join.py b/scripts/wrappers/join.py
--- a/scripts/wrappers/join.py
+++ b/scripts/wrappers/join.py
@@ -826,7 +826,6 @@ def join_dqlite_worker_node(info, master_ip, master_port, token):
         exit(1)
 
     store_remote_ca(info["ca"])
-    store_cert("serviceaccount.key", info["service_account_key"])
 
     store_base_kubelet_args(info["kubelet_args"])
     update_kubelet_node_ip(info["kubelet_args"], hostname_override)


Test patch:
diff --git a/tests/test-cluster.py b/tests/test-cluster.py
--- a/tests/test-cluster.py
+++ b/tests/test-cluster.py
@@ -458,6 +458,65 @@ def test_worker_node(self):
         kubelet = vm.run("cat /var/snap/microk8s/current/credentials/kubelet.config")
         assert "127.0.0.1" in kubelet.decode()
 
+    def test_worker_node_leave(self):
+        """
+        Test when a worker node leaves the cluster
+        """
+        print("Setting up a worker node")
+        vm = VM(backend)
+        vm.setup(channel_to_test)
+        self.VM.append(vm)
+
+        # Form cluster
+        vm_master = self.VM[0]
+        print("Adding machine {} to cluster".format(vm.vm_name))
+        add_node = vm_master.run("/snap/bin/microk8s.add-node")
+        endpoint = [ep for ep in add_node.decode().split() if ":25000/" in ep]
+        vm.run("/snap/bin/microk8s.join {} --worker".format(endpoint[0]))
+
+        # Wait for nodes to be ready
+        print("Waiting for node to register")
+        attempt = 0
+        while attempt < 10:
+            try:
+                connected_nodes = vm_master.run("/snap/bin/microk8s.kubectl get no")
+                if "NotReady" in connected_nodes.decode():
+                    time.sleep(5)
+                    continue
+                print(connected_nodes.decode())
+                break
+            except ChildProcessError:
+                time.sleep(10)
+                attempt += 1
+                if attempt == 10:
+                    raise
+
+        # Leave the worker node from the cluster
+        print("Leaving the worker node {} from the cluster".format(vm.vm_name))
+        vm.run("/snap/bin/microk8s.leave")
+
+        # Wait for worker node to leave the cluster
+        attempt = 0
+        while attempt < 10:
+            try:
+                connected_nodes = vm_master.run("/snap/bin/microk8s.kubectl get no")
+                if "NotReady" in connected_nodes.decode():
+                    print(connected_nodes.decode())
+                    break
+                time.sleep(5)
+                continue
+            except ChildProcessError:
+                time.sleep(10)
+                attempt += 1
+                if attempt == 10:
+                    raise
+
+        # Check that the worker node is Ready
+        print("Checking that the worker node {} is working and Ready".format(vm.vm_name))
+        worker_node = vm.run("/snap/bin/microk8s status --wait-ready")
+        print(worker_node.decode())
+        assert "microk8s is running" in worker_node.decode()
+
     def test_no_cert_reissue_in_nodes(self):
         """
         Test that each node has the cert no-reissue lock.