Repository: canonical/microk8s

Issue:
Connection failed. The joining node is already known to dqlite.
<!--
   Thank you for submitting an issue. Please fill in the template below
   information about the bug you encountered.
-->

#### Summary
If, joining a new node fails for any reason, the joining node gets added to .dqlite, and the cluster has to be fully shutdown to remediate.

#### What Should Happen Instead?
Either, `microk8s remove-node (broken ip)` should work, or there should be a less impactful way to remediate.

#### Reproduction Steps
Have not tried reproducing.

BUT, On a production cluster running 1.25, I provisioned a new server, added microk8s, and `microk8s add-node` followed by join.

I failed to specify the correct channel, so, the new node now running 1.26 failed to join the cluster. 

Not a problem, let me remove microk8s, and install the correct channel.

```
Contacting cluster at 10.100.5.100
Connection failed. The joining node (10.100.5.102) is already known to dqlite (504).
```

https://microk8s.io/docs/restore-quorum

Is supposedly the method needed to resolve this issue, however, this is highly impactful to workloads, as the entire cluster has to be shutdown.

#### Related Tickets

https://github.com/canonical/microk8s/issues/2212


#### A few more details

```
root@kube01:~/Kubernetes/namespaces/readymotorsports.net# cat /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml
- Address: 10.100.5.105:19001
  ID: 3297041220608546238
  Role: 0
- Address: 10.100.5.100:19001
  ID: 6578832437747111275
  Role: 0
- Address: 10.100.5.106:19001
  ID: 16212301467739305109
  Role: 0
- Address: 10.100.5.102:19001
  ID: 17023170999447087085
  Role: 1
root@kube01:~/Kubernetes/namespaces/readymotorsports.net# microk8s remove-node 10.100.5.102
Error from server (NotFound): nodes "10.100.5.102" not found
Node 10.100.5.102 does not exist in Kubernetes.
root@kube01:~/Kubernetes/namespaces/readymotorsports.net# kubectl get node -o wide
NAME     STATUS   ROLES                  AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kube06   Ready    <none>                 83d    v1.25.5   10.100.5.106   <none>        Ubuntu 22.04.1 LTS   5.15.0-58-generic   containerd://1.6.8
kube01   Ready    control-plane,master   95d    v1.25.5   10.100.5.100   <none>        Ubuntu 22.04.1 LTS   5.15.0-58-generic   containerd://1.6.8
kube05   Ready    <none>                 108d   v1.25.5   10.100.5.105   <none>        Ubuntu 22.04.1 LTS   5.15.0-58-generic   containerd://1.6.8
```




Discussion:
I will note,

Just changing the IP/hostname of the joining node, allows it to join.

```
root@kube04:~# microk8s join redacted
Contacting cluster at 10.100.5.100
Waiting for this node to finish joining the cluster. .. .. ..
root@kube04:~# kubectl get nodes
NAME     STATUS     ROLES                  AGE    VERSION
kube05   Ready      <none>                 108d   v1.25.5
kube06   Ready      <none>                 83d    v1.25.5
kube01   Ready      control-plane,master   95d    v1.25.5
kube04   NotReady   <none>                 6s     v1.25.6
root@kube04:~#
```
I had similar problem. Reboot + `microk8s leave` and re-joining cluster did the trick for me.
You could also try to do a `microk8s remove-node <node-ip> --force` to make sure dqlite has removed this IP from the nodes it knows. We fail adding nodes with IPs dqlite already knows about so we do not risk corrupting the datastore.
Guys-

I don't need a work-around. I have work-arounds!

This ticket, is so that hopefully somebody fixes the underlying issue causing this.
Apologies for the misunderstanding @XtremeOwnageDotCom. The situation where a node is on dqlite but not registered with k8s indicates that something failed during the join process or the node removal. In such situation the cluster should not need to be shutdown nor should the [recovery from lost quorum](https://microk8s.io/docs/restore-quorum) be followed.

In the case described in this issue we have a cluster to which we accidentally added a node of a version we do not want. If we want to fix this by removing the node and adding a new one we have to do the following.
 - on the node call `microk8s leave`. This will stop the node.
 - on any of the remaining nodes in the cluster. You call `microk8s remove-node <node>` so that the cluster will know that the node that left will not ever return to the cluster. If for any reason the node is on dqlite but not on the k8s cluster call `microk8s remove-node <node-ip> --force`
 - snap install a new node with the right version
 - join the node to the cluster

The remove node process is described in https://microk8s.io/docs/clustering

To reproduce this issue and to fully understand how we got into this situation I need more information on what happened during:

> Not a problem, let me remove microk8s, and install the correct channel.


Steps to reproduce after creating a 3 CP node cluster:
1. `microk8s leave` on one of the nodes.
2. `microk8s remove-node <IP>` on the same node (not the correct way to do remove-node)

After this, the node gets removed
```
# k get no
No resources found
```

But it still is not removed from `/var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml`. 
Doing a `microk8s remove-node <node-id> --force` from a different node resolves it.
The `<node-id>` is found in `/var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml` and has to match the node that cannot be removed.

Resolving patch:
diff --git a/scripts/wrappers/remove_node.py b/scripts/wrappers/remove_node.py
--- a/scripts/wrappers/remove_node.py
+++ b/scripts/wrappers/remove_node.py
@@ -8,13 +8,15 @@
 import click
 import netifaces
 
+from ipaddress import ip_address, IPv4Address
+
 from common.cluster.utils import (
     try_set_file_permissions,
     is_node_running_dqlite,
 )
 
-snapdata_path = os.environ.get("SNAP_DATA")
 snap_path = os.environ.get("SNAP")
+snapdata_path = os.environ.get("SNAP_DATA")
 callback_tokens_file = "{}/credentials/callback-tokens.txt".format(snapdata_path)
 
 cluster_dir = "{}/var/kubernetes/backend".format(snapdata_path)
@@ -22,6 +24,22 @@
 
 def remove_dqlite_node(node, force=False):
     try:
+        # If node is an IP address, find the node name.
+        if type(ip_address(node)) is IPv4Address:
+            node_info = subprocess.check_output(
+                "{}/microk8s-kubectl.wrapper get no -o json".format(snap_path).split()
+            )
+            info = json.loads(node_info.decode())
+            found = False
+            for n in info["items"]:
+                if found:
+                    break
+                for a in n["status"]["addresses"]:
+                    if a["type"] == "InternalIP" and a["address"] == node:
+                        node = n["metadata"]["name"]
+                        found = True
+                        break
+
         # Make sure this node exists
         node_info = subprocess.check_output(
             "{}/microk8s-kubectl.wrapper get no {} -o json".format(snap_path, node).split()


Test patch: