Repository: canonical/microk8s

Issue:
snap.microk8s.daemon-containerd.service times out on longer containerd startup times
<!--
   Thank you for submitting an issue. Please fill in the template below
   information about the bug you encountered.
-->

#### Summary

Occasionally, `snap install` fails with (line breaks added for clarity):
```
$ sudo snap install microk8s --classic --channel=1.24
error: cannot perform the following tasks:

- Start snap "microk8s" (4023) services 
  (systemctl command [start snap.microk8s.daemon-containerd.service] failed with exit status 1: 
  Job for snap.microk8s.daemon-containerd.service failed because a timeout was exceeded.

See "systemctl status snap.microk8s.daemon-containerd.service" 
and "journalctl -xeu snap.microk8s.daemon-containerd.service" for details.
```

The status and journalctl log for `snap.microk8s.daemon-containerd.service` show it has `exec`'d `containerd`,
but it didn't print anything, and apparently didn't reach the point to notify systemd (per `daemon: notify`)
within the default start timeout of 90 seconds.

If you retry, `snap install` usually works.

...

However, the issue might happen later on `snap start microk8s`,
or even _worse_: on reboot, during systemd services initialization.

The issue is worse on reboot because there's no `snap` command
handling to abort starting the other services if the issue happens --
`snap` starts/checks each service sequentially, but on system boot,
all of the snap's services are started in parallel.

This might cause the `snap.microk8s.daemon-containerd.service`
to repeatedly time out (due to `containerd` startup time) and keep
being restarted (`restart-condition: always`).

...

Something similar impacts `snap.microk8s.daemon-kubelite.service`,
as `run-kubelite-with-args` has a `for`-loop to wait for the containerd socket.

The loop eventually finishes, and kubelite starts but then fails, so the service is restarted.
It keeps being restarted as well, not because it has a similar `restart-condition`,
but apparently the duration and frequency of its executions don't exceed the
systemd restart burst threshold.


#### What Should Happen Instead?

The unit `snap.microk8s.daemon-containerd.service` should tolerate
longer startup times from `containerd`, instead of timeout and fail on
`snap install/start microk8s`, or continuously restarting on reboot.

The unit `snap.microk8s.daemon-kubelite.service` should not wait 
for a fixed amount of time until the containerd socket shows up
(as it might not be sufficient) -- let's start after `daemon-containerd`.




#### Reproduction Steps

Use a wrapper to simulate `containerd` taking more than 90 seconds to notify systemd it's ready:

Create wrapper:
```
$ sudo cp /snap/microk8s/current/bin/containerd /usr/local/bin/containerd.orig

$ cat <<EOF | sudo tee /usr/local/bin/containerd
#!/bin/sh
/usr/bin/sleep 95
exec /usr/local/bin/containerd.orig "\$@"
EOF

$ sudo chmod +x /usr/local/bin/containerd
```
Setup on reboot:
```
$ DIR=/etc/systemd/system/snap.microk8s.daemon-containerd.service.d
$ sudo mkdir -p $DIR
$ cat <<EOF | sudo tee $DIR/hack.conf
[Service]
ExecStartPre=/usr/bin/mount --bind /usr/local/bin/containerd /snap/microk8s/current/bin/containerd
EOF

$ sudo reboot
```

Note the number of restarts of both services within a few minutes after reboot:
```
$ systemctl show snap.microk8s.daemon-containerd.service | grep Restart
Restart=always
RestartUSec=100ms
NRestarts=4
RestartKillSignal=15

$ systemctl show snap.microk8s.daemon-kubelite.service | grep Restart
Restart=on-failure
RestartUSec=100ms
NRestarts=16
RestartKillSignal=15
```

#### Introspection Report
<!-- Please run `microk8s inspect` and attach the generated tarball. -->

Not available from the customer environment that reproduced this.

#### Can you suggest a fix?

We haven't gotten to a successful instrumentation of containerd under
the snap to precisely understand why `containerd` took so long to start
(it keeps busy before even initializing its logging mechanism for debug;
and modifying the service's `snap run` with `--strace` changes behavior).

However, apparently these 2 changes would improve both services:
1) `daemon-containerd`: an increased start timeout.
2) `daemon-kubelite`: start after `daemon-containerd`

#### Are you interested in contributing with a fix?

Yes. I'll send a PR.

With the patched snap, the problem doesn't happen anymore (same reproducer):
```
$ sudo snap install microk8s_v1.24.10_amd64.snap --classic --dangerous
microk8s v1.24.10 installed

$ sudo reboot

< a few minutes>

$ systemctl status snap.microk8s.daemon-containerd.service
...
    Drop-In: /etc/systemd/system/snap.microk8s.daemon-containerd.service.d
             └─hack.conf
     Active: active (running) since ...; 4min 47s ago
...

$ systemctl status snap.microk8s.daemon-kubelite.service
...
     Active: active (running) since ...; 2min 54s ago
...
```

<!-- Thank you for making MicroK8s better -->



Discussion:

Resolving patch:
diff --git a/scripts/wrappers/join.py b/scripts/wrappers/join.py
--- a/scripts/wrappers/join.py
+++ b/scripts/wrappers/join.py
@@ -153,7 +153,7 @@ def get_connection_info(
         if cluster_type == "dqlite":
             req_data = {
                 "token": token,
-                "hostname": socket.gethostname(),
+                "hostname": socket.gethostname().lower(),
                 "port": cluster_agent_port,
                 "worker": worker,
             }
@@ -162,7 +162,7 @@ def get_connection_info(
         else:
             req_data = {
                 "token": token,
-                "hostname": socket.gethostname(),
+                "hostname": socket.gethostname().lower(),
                 "port": cluster_agent_port,
                 "callback": callback_token,
             }
@@ -450,7 +450,7 @@ def update_cert_auth_kubelet(token, ca, master_ip, master_port):
     """
     kubelet_token = "{}-kubelet".format(token)
     traefik_port = get_traefik_port()
-    kubelet_user = "system:node:{}".format(socket.gethostname())
+    kubelet_user = "system:node:{}".format(socket.gethostname().lower())
     cert = get_client_cert(
         master_ip, master_port, "kubelet", kubelet_token, kubelet_user, "system:nodes"
     )
diff --git a/scripts/wrappers/refresh_certs.py b/scripts/wrappers/refresh_certs.py
--- a/scripts/wrappers/refresh_certs.py
+++ b/scripts/wrappers/refresh_certs.py
@@ -128,10 +128,15 @@ def reproduce_front_proxy_client_cert():
     Produce the front proxy client certificate
     """
     subprocess.check_call("rm -rf {}/certs/front-proxy-client.crt".format(snapdata_path).split())
-    p = subprocess.Popen(
-        ["bash", "-c", ". {}/actions/common/utils.sh; gen_proxy_client_cert".format(snap_path)]
+    subprocess.check_call(
+        [
+            "bash",
+            "-c",
+            ". {}/actions/common/utils.sh; refresh_csr_conf; gen_proxy_client_cert".format(
+                snap_path
+            ),
+        ]
     )
-    p.communicate()
     subprocess.check_call("rm -rf .slr".split())
     restart("kubelite")
 
@@ -141,10 +146,13 @@ def reproduce_server_cert():
     Produce the server certificate
     """
     subprocess.check_call("rm -rf {}/certs/server.crt".format(snapdata_path).split())
-    p = subprocess.Popen(
-        ["bash", "-c", ". {}/actions/common/utils.sh; gen_server_cert".format(snap_path)]
+    subprocess.check_call(
+        [
+            "bash",
+            "-c",
+            ". {}/actions/common/utils.sh; refresh_csr_conf; gen_server_cert".format(snap_path),
+        ]
     )
-    p.communicate()
     subprocess.check_call("rm -rf .slr".split())
     restart("kubelite")
     restart("cluster-agent")


Test patch:
diff --git a/tests/unit/test_refresh_certs.py b/tests/unit/test_refresh_certs.py
--- a/tests/unit/test_refresh_certs.py
+++ b/tests/unit/test_refresh_certs.py
@@ -44,54 +44,42 @@ def test_reproduce_all_root_ca_certs(self, mock_check_call, mock_subproc_popen):
         assert self.is_argument_in_call(mock_subproc_popen, "update_configs")
 
     @patch("refresh_certs.restart")
-    @patch("subprocess.Popen")
     @patch("subprocess.check_call")
-    def test_reproduce_front_proxy_client_cert(
-        self, mock_check_call, mock_subproc_popen, mock_restart
-    ):
-        process_mock = Mock()
-        mock_subproc_popen.return_value = process_mock
-
-        reproduce_front_proxy_client_cert()
-
+    def test_reproduce_front_proxy_client_cert(self, mock_check_call, mock_restart):
         """
         Make sure we:
         1. remove the front-proxy-client.crt
         2. call gen_proxy_client_cert
         3. restart microk8s
         """
+        reproduce_front_proxy_client_cert()
+
         snapdata_path = os.environ.get("SNAP_DATA")
         cmd = "rm -rf {}/certs/front-proxy-client.crt".format(snapdata_path).split()
         assert call(cmd) in mock_check_call.call_args_list
         assert mock_check_call.called
-        assert mock_subproc_popen.called
         assert mock_restart.called
-        assert self.is_argument_in_call(mock_subproc_popen, "gen_proxy_client_cert")
+        assert self.is_argument_in_call(mock_check_call, "gen_proxy_client_cert")
 
     @patch("refresh_certs.restart")
-    @patch("subprocess.Popen")
     @patch("subprocess.check_call")
-    def test_reproduce_server_cert(self, mock_check_call, mock_subproc_popen, mock_restart):
-        process_mock = Mock()
-        mock_subproc_popen.return_value = process_mock
-
-        reproduce_server_cert()
-
+    def test_reproduce_server_cert(self, mock_check_call, mock_restart):
         """
         Make sure we:
         1. remove the server.crt
         2. call gen_server_cert
         3. restart microk8s
         """
+        reproduce_server_cert()
+
         snapdata_path = os.environ.get("SNAP_DATA")
         assert (
             call("rm -rf {}/certs/server.crt".format(snapdata_path).split())
             in mock_check_call.call_args_list
         )
         assert mock_check_call.called
-        assert mock_subproc_popen.called
         assert mock_restart.called
-        assert self.is_argument_in_call(mock_subproc_popen, "gen_server_cert")
+        assert self.is_argument_in_call(mock_check_call, "gen_server_cert")
 
     def is_argument_in_call(self, mock_function, argument_substring):
         """Search for a substring in the list of arguments in all calls of a mocked function"""