Repository: canonical/microk8s

Issue:
Joining HA cluster fails when `--advertise-address` or `--bind-address` are configured on the kube-apiserver
#### Summary
Joining a HA cluster fails when `--advertise-address` or `--bind-address` are configured on the *kube-apiserver* (in `/var/snap/microk8s/current/args/kube-apiserver`) on the joining node. Joining procedure does not generate certificates and `csrf.conf` on joining node.

#### What Should Happen Instead?
The node with configured `--advertise-address` or `--bind-address` on the *kube-apiserver* should be able to successfully join the cluster.

#### Reproduction Steps
1. install *microk8s* on two nodes (*nodeA* & *nodeB*)
2. add an `--advertise-address` or `--bind-address` option to the *kube-apiserver* on *nodeB*; example for address on default-route interface:
    `echo --advertise-address=$(ip -j r s default | jq -r '.[0].prefsrc') >> /var/snap/microk8s/current/args/kube-apiserver`
3. restart *microk8s* on *nodeB* (`sudo snap restart microk8s`)
4. *nodeA*: `microk8s add-node`
5. *nodeB*: `microk8s join <ADDRESS_ON_DEFAULT_INTERFACE_nodeA>:25000/<TOKEN>/<TOKEN>`
6. Wait & validate failing join operation:
```bash
nodeA:~$ microk8s kubectl get no
NAME               STATUS   ROLES    AGE   VERSION
u2204srv-mk8s-01   Ready    <none>   20m   v1.26.1
```

```
nodeB:~$ microk8s kubectl get no
E0221 11:55:27.414876   24107 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")
E0221 11:55:27.418325   24107 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")
E0221 11:55:27.421624   24107 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")
E0221 11:55:27.425385   24107 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")
E0221 11:55:27.428299   24107 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")
```
<!-- #### Introspection Report /-->

#### Analysis
When joining a cluster in HA mode the `csr.conf` file gets invalidated (content: `changeme`), after which all *microk8s* services will be restarted, from `update_dqlite(...)` in `join.py`:
https://github.com/canonical/microk8s/blob/73b5ed2be0b2102a19ecaa01369c091e1889b9ee/scripts/wrappers/join.py#L730-L733

(When restarting) The *apiservice-kicker* is responsible for generating CSR config, CSR and certificates. In case `--advertise-address` or `--bind-address` are configured on the *kube-apiserver*, rendering and copying of the CSR config file and certificate generation is skipped:
https://github.com/canonical/microk8s/blob/73b5ed2be0b2102a19ecaa01369c091e1889b9ee/microk8s-resources/wrappers/apiservice-kicker#L48-L50
The invald `certs/csr.conf` further prevents manual generation of certificates, with the new CA, using `microk8s refresh-certs -e <...>`.

**Note:** The guard against certificate regeneration on clusters - provided by the `${SNAP_DATA}/var/lock/no-cert-reissue` lock-file - is not relevant to the joining behavior, as `mark_no_cert_reissue()` is only called after the service restart (*apiserver-kicker* execution) and return from `update_dqlite()`:
https://github.com/canonical/microk8s/blob/73b5ed2be0b2102a19ecaa01369c091e1889b9ee/scripts/wrappers/join.py#L882-L887

##### Workarounds
https://github.com/canonical/microk8s/issues/2123#issuecomment-1209995093 outlines a working workaround, by copying the `certs/csrf.conf.rendered` to `certs/csr.conf` and regenerating `server.crt` and `front-proxy-client.crt`. #3740 should eliminate the step of copying the csrf, but will not fix the cluster joining issue in the first place.

#### Related issues
* #2123
* #3321
* #3068
* loosely #3241

#### Can you suggest a fix?
##### Considerations
###### Who/Responsibility
First, the question of **who** should be in charge for generating CSR config, CSR and certificates has to be answered:

Per introduction of the *apiserver-kicker* in #177 it was mainly intended to regenerate certificates on IP changes, but now seems to be responsible for all CSR/certificate generation: Indicated by the implementation in the joining code using a similar strategy, to indicate a pending regeneration of the required resources, as the utility functions used by the *apiserver-kicker*; see below.
https://github.com/canonical/microk8s/blob/73b5ed2be0b2102a19ecaa01369c091e1889b9ee/scripts/wrappers/join.py#L730-L731
https://github.com/canonical/microk8s/blob/73b5ed2be0b2102a19ecaa01369c091e1889b9ee/microk8s-resources/actions/common/utils.sh#L629-L631
Explicitly generating all resources in the joining code would offer the benefit of knowing the **intent**, while **the current implementation suggests the *apiserver-kicker* to be responsible**.

###### What
Second, it shall be considered what resources, apart from the actual certificates should be generated. In my opinion, these two are valid, dependent, assumptions about the current configuration of a user using custom `--advertise-address` or `--bind-address` option on the *kube-apiserver*:
* a custom `csrf.conf.template` may exist
* a custom `csrf.conf.rendered` may exist

no custom `csrf.conf` may exist currently, as it is overwritten by the joining logic.

##### Proposed Fix
Rough outline:
* let the *apiserver-kicker* be responsible for all certificate related resource generation
* validate the `server.crt` against the `ca.crt` and trigger certificate regeneration if they do not match
* validate the contents of the CSR configuration files in order of `csr.conf`, `csrf.conf.rendered`, `csrf.conf.template`, each replacing with (rendered variant) of the next, if invalid configuration files

This proposed behavior would make:
1. the *apiserver-kicker* to regenerate certificates independent of any configuration and cluster state, in the case the certificates do not match the root CA
2. allow users to keep manually edited CSR config files to any "depth" they decide, by keeping or deleting/invalidating them

#### Are you interested in contributing with a fix?
yes ✔

I can provide a fix, as most of the proposed procedure has already been tested locally; CSR config and certificate validation are straight forward and possible to implement with likely negligible overhead in the *apiserver-kicker*.

Before providing code and a PR, I would like a review/discussion of the proposed fix, as well as the applied assumptions.


Discussion:
Sorry for the late response @bitmeal. You have done a superb job identifying the problem and figuring out the right path forward. I have no objections to your proposed approach. I think it would clear a number of ambiguous cases. Should we continue our discussion over a PR review?  

Resolving patch:
diff --git a/scripts/wrappers/join.py b/scripts/wrappers/join.py
--- a/scripts/wrappers/join.py
+++ b/scripts/wrappers/join.py
@@ -682,6 +682,7 @@ def update_dqlite(cluster_cert, cluster_key, voters, host):
     """
     service("stop", "apiserver")
     service("stop", "k8s-dqlite")
+    # will allow for apiservice-kicker to generate new certificates @5s loop rate
     time.sleep(10)
     shutil.rmtree(cluster_backup_dir, ignore_errors=True)
     shutil.move(cluster_dir, cluster_backup_dir)
@@ -727,9 +728,6 @@ def update_dqlite(cluster_cert, cluster_key, voters, host):
             waits -= 1
     print(" ")
 
-    with open("{}//certs/csr.conf".format(snapdata_path), "w") as f:
-        f.write("changeme")
-
     restart_all_services()
 
 
@@ -913,6 +911,24 @@ def join_etcd(connection_parts, verify=True):
     mark_no_cert_reissue()
 
 
+def mark_join_in_progress():
+    """
+    Mark node as currently being in the process of joining a cluster.
+    """
+    lock_file = "{}/var/lock/join-in-progress".format(snapdata_path)
+    open(lock_file, "a").close()
+    os.chmod(lock_file, 0o700)
+
+
+def unmark_join_in_progress():
+    """
+    Unmark as joining cluster; join operation has finished.
+    """
+    lock_file = "{}/var/lock/join-in-progress".format(snapdata_path)
+    if os.path.exists(lock_file):
+        os.unlink(lock_file)
+
+
 @click.command(
     context_settings={"ignore_unknown_options": True, "help_option_names": ["-h", "--help"]}
 )
@@ -970,10 +986,14 @@ def join(connection, worker, skip_verify, disable_low_memory_guard):
         )
         sys.exit(1)
 
+    mark_join_in_progress()
+
     if is_node_running_dqlite():
         join_dqlite(connection_parts, verify, worker)
     else:
         join_etcd(connection_parts, verify)
+
+    unmark_join_in_progress()
     sys.exit(0)
 
 


Test patch: