Repository: kubernetes/kops (go)

Issue:
All nodes have warning events when stood up with kOps 1.30
/kind bug

**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

```
$ kops version     
Client version: 1.30.0 (git-v1.30.0)
```

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

```
Server Version: v1.30.4
```

**3. What cloud provider are you using?**

```
AWS
```

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

The cluster is stood up via a manifest that contains:

```yaml
spec:
  nodeProblemDetector:
    enabled: true
```

**5. What happened after the commands executed?**

The cluster comes up fine and all pods are healthy however when using `kubectl describe node` against any of the nodes (control-plane or worker) they have the following Warning events:
```
  Warning  InvalidDiskCapacity      19m                kubelet                invalid capacity 0 on image filesystem
  Warning  ContainerdStart          15m                systemd-monitor        Starting containerd container runtime...
  Warning  ContainerdUnhealthy      15m                health-checker         Node condition ContainerRuntimeUnhealthy is now: True, reason: ContainerdUnhealthy, message: "cri:containerd was found unhealthy; repair flag : true"
  Warning  KubeletUnhealthy         15m                health-checker         Node condition KubeletUnhealthy is now: True, reason: KubeletUnhealthy, message: "kubelet:kubelet was found unhealthy; repair flag : true"
```

These events do not clear over time.
The `InvalidDiskCapacity` event is what leads to the `KubeletUnhealthy` event, and the `ContainerdStart` event is what leads to the `ContainerdUnhealthy` event.

**6. What did you expect to happen?**

All events on the nodes to have a `Normal` status.

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

```
Skipping this for now since it's the same manifest I've used for many generations of kOps clusters.
```

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

**9. Anything else do we need to know?**

I believe that `node-problem-detector` is at fault.
It can't find the `crictl` or `systemctl` commands.

Here are some logs from `node-problem-detector` related to the `ContainerdUnhealthy` status:
```
I0820 02:06:49.633546       1 log_monitor.go:159] New status generated: &{Source:systemd-monitor Events:[{Severity:warn Timestamp:2024-08-20 02:05:43.239889 +0000 UTC Reason:ContainerdStart Message:Starting containerd container runtime...}] Conditions:[]}
I0820 02:06:50.213247       1 plugin.go:281] Start logs from plugin {Type:permanent Condition:ContainerRuntimeUnhealthy Reason:ContainerdUnhealthy Path:/home/kubernetes/bin/health-checker Args:[--component=cri --enable-repair=true --cooldown-time=2m --health-check-timeout=60s] TimeoutString:0xc000060bb0 Timeout:3m0s}
 I0820 02:06:50.212369      18 health_checker.go:180] command /usr/bin/crictl --timeout=2s --runtime-endpoint=unix:///var/run/containerd/containerd.sock pods --latest failed: fork/exec /usr/bin/crictl: no such file or directory, 
I0820 02:06:50.212821      18 health_checker.go:180] command systemctl show containerd --property=InactiveExitTimestamp failed: exec: "systemctl": executable file not found in $PATH,
I0820 02:06:50.212835      18 health_checker.go:86] error in getting uptime for cri: exec: "systemctl": executable file not found in $PATH
I0820 02:06:50.213334       1 custom_plugin_monitor.go:283] New status generated: &{Source:health-checker Events:[{Severity:warn Timestamp:2024-08-20 02:06:50.213306946 +0000 UTC m=+8.109772440 Reason:ContainerdUnhealthy Message:Node condition ContainerRuntimeUnhealthy is now: True, reason: ContainerdUnhealthy, message: "cri:containerd was found unhealthy; repair flag : true"}] Conditions:[{Type:ContainerRuntimeUnhealthy Status:True Transition:2024-08-20 02:06:50.213306946 +0000 UTC m=+8.109772440 Reason:ContainerdUnhealthy Message:cri:containerd was found unhealthy; repair flag : true}]}
```
And logs for the `KubeletUnhealthy` status:
```
I0820 02:06:50.213530       1 plugin.go:281] Start logs from plugin {Type:permanent Condition:KubeletUnhealthy Reason:KubeletUnhealthy Path:/home/kubernetes/bin/health-checker Args:[--component=kubelet --enable-repair=true --cooldown-time=1m --loopback-time=0 --health-check-timeout=10s] TimeoutString:0xc000060c90 Timeout:3m0s}
 I0820 02:06:50.213073      15 health_checker.go:180] command systemctl show kubelet --property=InactiveExitTimestamp failed: exec: "systemctl": executable file not found in $PATH,
I0820 02:06:50.213109      15 health_checker.go:86] error in getting uptime for kubelet: exec: "systemctl": executable file not found in $PATH
I0820 02:06:50.213564       1 plugin.go:282] End logs from plugin {Type:permanent Condition:KubeletUnhealthy Reason:KubeletUnhealthy Path:/home/kubernetes/bin/health-checker Args:[--component=kubelet --enable-repair=true --cooldown-time=1m --loopback-time=0 --health-check-timeout=10s] TimeoutString:0xc000060c90 Timeout:3m0s}
I0820 02:06:50.213602       1 custom_plugin_monitor.go:283] New status generated: &{Source:health-checker Events:[{Severity:warn Timestamp:2024-08-20 02:06:50.213581121 +0000 UTC m=+8.110046618 Reason:KubeletUnhealthy Message:Node condition KubeletUnhealthy is now: True, reason: KubeletUnhealthy, message: "kubelet:kubelet was found unhealthy; repair flag : true"}] Conditions:[{Type:KubeletUnhealthy Status:True Transition:2024-08-20 02:06:50.213581121 +0000 UTC m=+8.110046618 Reason:KubeletUnhealthy Message:kubelet:kubelet was found unhealthy; repair flag : true}]}
```

I was able to stop the errors by just disabling the checks for them by making the following changes to the `node-problem-detector` daemonset and then rolling the nodes afterwards (since ones marked with the errors didn't clear):
```diff
@ -36,7 +36,7 @@
         - /node-problem-detector
         - --logtostderr
         - --config.system-log-monitor=/config/kernel-monitor.json,/config/systemd-monitor.json
-        - --config.custom-plugin-monitor=/config/kernel-monitor-counter.json,/config/systemd-monitor-counter.json,/config/health-checker-containerd.json,/config/health-checker-kubelet.json
+        - --config.custom-plugin-monitor=/config/kernel-monitor-counter.json,/config/systemd-monitor-counter.json
         - --config.system-stats-monitor=/config/system-stats-monitor.json
         env:
         - name: NODE_NAME
```

I also tried this instead to see if I could fix the problems so that it had access to the `crictl` and `systemctl` commands and the containerd socket:
```diff
@@ -76,6 +76,14 @@
         - mountPath: /var/run/dbus/
           mountPropagation: Bidirectional
           name: dbus
+        - mountPath: /usr/bin/crictl
+          name: crictl
+          readOnly: true
+        - mountPath: /usr/bin/systemctl
+          name: systemctl
+          readOnly: true
+        - mountPath: /var/run/containerd/containerd.sock
+          name: containerd-sock
       dnsPolicy: ClusterFirst
       priorityClassName: system-node-critical
       restartPolicy: Always
@@ -115,6 +123,18 @@
           path: /var/run/dbus/
           type: ""
         name: dbus
+      - hostPath:
+          path: /usr/local/bin/crictl
+          type: File
+        name: crictl
+      - hostPath:
+          path: /usr/bin/systemctl
+          type: File
+        name: systemdctl
+      - hostPath:
+          path: /var/run/containerd/containerd.sock
+          type: Socket
+        name: containerd-sock
   updateStrategy:
     rollingUpdate:
       maxSurge: 0
```
The above fixed the `ContainerdStart` and `ContainerdUnhealthy` problems but the `InvalidDiskCapacity` and `KubeletUnhealthy` problems still persist with the warning message still saying: `invalid capacity 0 on image filesystem`.
I haven't worked out how to fix it... I'm thinking perhaps something from the host still needs to be mounted into the container somewhere (wherever the image filesystem lives).
I tried running the `/home/kubernetes/bin/health-checker --component=kubelet --enable-repair=true --cooldown-time=1m --loopback-time=0 --health-check-timeout=10s` command manually in the container and passing in the `-v` option with various levels to try and get more verbosity, but that all just returned messages like the following no matter what verbosity level I tried.
```
I0820 06:38:38.371286    1509 health_checker.go:89] kubelet is unhealthy, component uptime: 19.371223472s
kubelet:kubelet was found unhealthy; repair flag : true
```

Here's a link to a Slack thread I created yesterday when I discovered and was working through the problem.
https://kubernetes.slack.com/archives/C3QUFP0QM/p1724120394886979

Pull request:
Disable node-problem-detector containerd and kubelet checks
Closes #16763

Patch:
diff --git a/upup/models/cloudup/resources/addons/node-problem-detector.addons.k8s.io/k8s-1.17.yaml.template b/upup/models/cloudup/resources/addons/node-problem-detector.addons.k8s.io/k8s-1.17.yaml.template
index 37b10b055a..4cdfc53e19 100644
--- a/upup/models/cloudup/resources/addons/node-problem-detector.addons.k8s.io/k8s-1.17.yaml.template
+++ b/upup/models/cloudup/resources/addons/node-problem-detector.addons.k8s.io/k8s-1.17.yaml.template
@@ -55,7 +55,7 @@ spec:
         - /node-problem-detector
         - --logtostderr
         - --config.system-log-monitor=/config/kernel-monitor.json,/config/systemd-monitor.json
-        - --config.custom-plugin-monitor=/config/kernel-monitor-counter.json,/config/systemd-monitor-counter.json,/config/health-checker-containerd.json,/config/health-checker-kubelet.json
+        - --config.custom-plugin-monitor=/config/kernel-monitor-counter.json,/config/systemd-monitor-counter.json
         - --config.system-stats-monitor=/config/system-stats-monitor.json
         image: {{ .Image }}
         securityContext:


Test patch:
diff --git a/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-bootstrap_content b/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-bootstrap_content
index 7f58f6d3cd..232414f5b6 100644
--- a/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-bootstrap_content
@@ -163,7 +163,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.17
     manifest: node-problem-detector.addons.k8s.io/k8s-1.17.yaml
-    manifestHash: c5efa7accf480cced0a3610be297e7109361d2a8c101e3dce086d256ec7e706f
+    manifestHash: 94b31d90ea983e0cb64dc233c11174684a0b0fb18ea9cd7cbc93013881e1a230
     name: node-problem-detector.addons.k8s.io
     prune:
       kinds:
diff --git a/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-node-problem-detector.addons.k8s.io-k8s-1.17_content b/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-node-problem-detector.addons.k8s.io-k8s-1.17_content
index 8bfe13aeae..93926cbe34 100644
--- a/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-node-problem-detector.addons.k8s.io-k8s-1.17_content
+++ b/tests/integration/update_cluster/many-addons/data/aws_s3_object_many-addons.example.com-addons-node-problem-detector.addons.k8s.io-k8s-1.17_content
@@ -69,7 +69,7 @@ spec:
         - /node-problem-detector
         - --logtostderr
         - --config.system-log-monitor=/config/kernel-monitor.json,/config/systemd-monitor.json
-        - --config.custom-plugin-monitor=/config/kernel-monitor-counter.json,/config/systemd-monitor-counter.json,/config/health-checker-containerd.json,/config/health-checker-kubelet.json
+        - --config.custom-plugin-monitor=/config/kernel-monitor-counter.json,/config/systemd-monitor-counter.json
         - --config.system-stats-monitor=/config/system-stats-monitor.json
         env:
         - name: NODE_NAME