Repository: canonical/microk8s

Issue:
microk8s v1.24 - refresh-certs doesn't seem to refresh certs 
<!--
   Thank you for submitting an issue. Please fill in the template below
   information about the bug you encountered.
-->

#### Summary
i have microk8s 1.24 and i need to grant access to the cluster via FQDN ..
I edited csr.conf.template  and i refreshed the certificates as follow

```sudo microk8s refresh-certs --cert server.crt```
```sudo microk8s refresh-certs --cert front-proxy-client.crt```

i didn't refresh the ca.ctr  because apparently i would have to rebuild the cluster
" Warning: refreshing the root CA requires nodes to leave and re-join the cluster"
unfortunately it's not yet working... when i connect to the API server via the FQDN i get connection rejected

```
Unable to connect to the server: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster, kubernetes.default.svc.cluster.local, not [xxxxx.com](http://xxxxx.com/)
```

i was only able to make it work, after @neoaggelos  suggested as workaround, to run ```sudo snap set microk8s dummy="$(date)"```

#### What Should Happen Instead?
refresh certs should work

#### Reproduction Steps
<!-- Are you able to consistently reproduce the issue? Please add a list of steps that lead to the bug. -->

add a DNS entry to csr.conf.template and refresh certs

#### Introspection Report
<!-- Please run `microk8s inspect` and attach the generated tarball. -->
[inspection-report-20220612_111958.tar.gz](https://github.com/canonical/microk8s/files/8885549/inspection-report-20220612_111958.tar.gz)


#### Can you suggest a fix?
<!-- (This section is optional). How do you propose that the issue be fixed? -->

#### Are you interested in contributing with a fix?
<!-- yes/no, or @mention maintainers. Community contributions are welcome. -->

<!-- Thank you for making MicroK8s better -->



Discussion:
Best way to check if certificates were refreshed is to check file modification time. Did you try to restart API server?
https://microk8s.io/docs/services-and-ports document states that DNS entries for the server can be added to `/var/snap/microk8s/current/certs/csr.conf.template` and `microk8s refresh-certs` should be run afterwards.

However, `microk8s refresh-certs` now requires `--cert` argument and I concur @anubisg1 observation that it does not change the X509v3 Subject Alternative Name entries in API server certificate. `server.csr` and `server.ctr` files are modified but contain only the original `kubernetes...` SAN entries.

I've noticed that adding DNS.n entries `csr.conf` file directly and running `microk8s refresh-certs --cert server.crt` DOES update SAN entries in `server.crt`. Further, I've noticed that timestamp on `csr.conf.rendered` file does not change after running `microk8s refresh-certs`. So it seems that `csr.conf.template` is used only during cluster installation and not when `microk8s refresh-certs` is run.

Seeing this issue also on v1.23
> Seeing this issue also on v1.23

Same. I had to downgrade to v1.22 to get certs which reflect the entries in `/var/snap/microk8s/current/certs/csr.conf.template`.
Same on v1.25
@neoaggelos is there a reasonable workaround for this? Without this it's not possible to have the microk8s control plane behind a load balancer.
> @neoaggelos is there a reasonable workaround for this? Without this it's not possible to have the microk8s control plane behind a load balancer.

The workaround is `sudo snap set microk8s dummy="$(date)" ` as described in my initial report 
Thanks @anubisg1, I missed that.
For others stumbling across this I needed to perform the following in order for MicroK8s nodes to work across my wireguard mesh.

**Before** setting up the cluster, on each node prep them as follows:

* edit `/var/snap/microk8s/current/args/kube-apiserver` and add `--advertise-address={{wireguard_ip}}`
* edit `/var/snap/microk8s/current/args/kubelet` and add `--node-ip={{wireguard_ip}}`
* edit `/var/snap/microk8s/current/certs/csr.conf.template` and add `IP.100 = {{wireguard_ip}}` just before the `#MOREIPS` comment
* refresh the certificates with `sudo microk8s refresh-certs --cert ca.crt --cert server.crt --cert front-proxy-client.crt`
* restart microk8s

Create the cluster...

**After** nodes have joined (and you get the x509 certificate error when trying to run `microk8s kubectl get nodes -o wide`). On each node apply the fix:

* `sudo snap set microk8s dummy="$(date)"`

Hopefully that helps someone else (or me in 6months time when I've forgotten...)


Resolving patch:
diff --git a/scripts/wrappers/refresh_certs.py b/scripts/wrappers/refresh_certs.py
--- a/scripts/wrappers/refresh_certs.py
+++ b/scripts/wrappers/refresh_certs.py
@@ -128,10 +128,15 @@ def reproduce_front_proxy_client_cert():
     Produce the front proxy client certificate
     """
     subprocess.check_call("rm -rf {}/certs/front-proxy-client.crt".format(snapdata_path).split())
-    p = subprocess.Popen(
-        ["bash", "-c", ". {}/actions/common/utils.sh; gen_proxy_client_cert".format(snap_path)]
+    subprocess.check_call(
+        [
+            "bash",
+            "-c",
+            ". {}/actions/common/utils.sh; refresh_csr_conf; gen_proxy_client_cert".format(
+                snap_path
+            ),
+        ]
     )
-    p.communicate()
     subprocess.check_call("rm -rf .slr".split())
     restart("kubelite")
 
@@ -141,10 +146,13 @@ def reproduce_server_cert():
     Produce the server certificate
     """
     subprocess.check_call("rm -rf {}/certs/server.crt".format(snapdata_path).split())
-    p = subprocess.Popen(
-        ["bash", "-c", ". {}/actions/common/utils.sh; gen_server_cert".format(snap_path)]
+    subprocess.check_call(
+        [
+            "bash",
+            "-c",
+            ". {}/actions/common/utils.sh; refresh_csr_conf; gen_server_cert".format(snap_path),
+        ]
     )
-    p.communicate()
     subprocess.check_call("rm -rf .slr".split())
     restart("kubelite")
     restart("cluster-agent")


Test patch:
diff --git a/tests/unit/test_refresh_certs.py b/tests/unit/test_refresh_certs.py
--- a/tests/unit/test_refresh_certs.py
+++ b/tests/unit/test_refresh_certs.py
@@ -44,54 +44,42 @@ def test_reproduce_all_root_ca_certs(self, mock_check_call, mock_subproc_popen):
         assert self.is_argument_in_call(mock_subproc_popen, "update_configs")
 
     @patch("refresh_certs.restart")
-    @patch("subprocess.Popen")
     @patch("subprocess.check_call")
-    def test_reproduce_front_proxy_client_cert(
-        self, mock_check_call, mock_subproc_popen, mock_restart
-    ):
-        process_mock = Mock()
-        mock_subproc_popen.return_value = process_mock
-
-        reproduce_front_proxy_client_cert()
-
+    def test_reproduce_front_proxy_client_cert(self, mock_check_call, mock_restart):
         """
         Make sure we:
         1. remove the front-proxy-client.crt
         2. call gen_proxy_client_cert
         3. restart microk8s
         """
+        reproduce_front_proxy_client_cert()
+
         snapdata_path = os.environ.get("SNAP_DATA")
         cmd = "rm -rf {}/certs/front-proxy-client.crt".format(snapdata_path).split()
         assert call(cmd) in mock_check_call.call_args_list
         assert mock_check_call.called
-        assert mock_subproc_popen.called
         assert mock_restart.called
-        assert self.is_argument_in_call(mock_subproc_popen, "gen_proxy_client_cert")
+        assert self.is_argument_in_call(mock_check_call, "gen_proxy_client_cert")
 
     @patch("refresh_certs.restart")
-    @patch("subprocess.Popen")
     @patch("subprocess.check_call")
-    def test_reproduce_server_cert(self, mock_check_call, mock_subproc_popen, mock_restart):
-        process_mock = Mock()
-        mock_subproc_popen.return_value = process_mock
-
-        reproduce_server_cert()
-
+    def test_reproduce_server_cert(self, mock_check_call, mock_restart):
         """
         Make sure we:
         1. remove the server.crt
         2. call gen_server_cert
         3. restart microk8s
         """
+        reproduce_server_cert()
+
         snapdata_path = os.environ.get("SNAP_DATA")
         assert (
             call("rm -rf {}/certs/server.crt".format(snapdata_path).split())
             in mock_check_call.call_args_list
         )
         assert mock_check_call.called
-        assert mock_subproc_popen.called
         assert mock_restart.called
-        assert self.is_argument_in_call(mock_subproc_popen, "gen_server_cert")
+        assert self.is_argument_in_call(mock_check_call, "gen_server_cert")
 
     def is_argument_in_call(self, mock_function, argument_substring):
         """Search for a substring in the list of arguments in all calls of a mocked function"""