Repository: k3s-io/k3s

Issue #7130: pods with `runtimeClassName: nvidia` get killed after a few seconds
Created: 2023-03-21T12:17:01Z
Environmental Info:
Tested K3s Versions:  v1.25.6 and v1.25.7
k3s version v1.25.6+k3s1 (9176e03c)
go version go1.19.5

Node(s) CPU architecture, OS, and Version:
Linux agent 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
3 servers, 1 agent, connected via wireguard mesh
kubectl get nodes -o wide
NAME              STATUS     ROLES                       AGE   VERSION        INTERNAL-IP       EXTERNAL-IP       OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
control-plane-1   Ready      control-plane,etcd,master   38m   v1.25.6+k3s1   100.121.166.220   100.121.166.220   Ubuntu 22.04.2 LTS   5.15.0-56-generic   containerd://1.6.15-k3s1
control-plane-2   Ready      control-plane,etcd,master   37m   v1.25.6+k3s1   100.121.227.93    100.121.227.93    Ubuntu 22.04.2 LTS   5.15.0-56-generic   containerd://1.6.15-k3s1
control-plane-3   Ready      control-plane,etcd,master   36m   v1.25.6+k3s1   100.121.146.219   100.121.146.219   Ubuntu 22.04.2 LTS   5.15.0-56-generic   containerd://1.6.15-k3s1
agent             Ready      <none>                      32m   v1.25.6+k3s1   100.121.42.124    100.121.42.124    Ubuntu 22.04.2 LTS   5.15.0-67-generic   containerd://1.6.15-k3s1
Describe the bug:
I've installed k3s-agent on a machine that has GPUs built-in. I followed all the steps outlined here.
I also installed nvidia-device-plugin with:
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: "nvidia"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      runtimeClassName: nvidia
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      # Mark this pod as a critical add-on; when enabled, the critical add-on
      # scheduler reserves resources for critical add-on pods so that they can
      # be rescheduled after a failure.
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      priorityClassName: "system-node-critical"
      containers:
        - image: nvcr.io/nvidia/k8s-device-plugin:v0.13.0
          name: nvidia-device-plugin-ctr
          env:
            - name: FAIL_ON_INIT_ERROR
              value: "false"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
kubectl -n kube-system logs nvidia-device-plugin-daemonset-xtt9q
2023/03/21 11:52:11 Starting FS watcher.
2023/03/21 11:52:11 Starting OS watcher.
2023/03/21 11:52:11 Starting Plugins.
2023/03/21 11:52:11 Loading configuration.
2023/03/21 11:52:11 Updating config with default resource matching patterns.
2023/03/21 11:52:11 
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": false,
    "nvidiaDriverRoot": "/",
    "gdsEnabled": false,
    "mofedEnabled": false,
    "plugin": {
      "passDeviceSpecs": false,
      "deviceListStrategy": "envvar",
      "deviceIDStrategy": "uuid"
    }
  },
  "resources": {
    "gpus": [
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ]
  },
  "sharing": {
    "timeSlicing": {}
  }
}
2023/03/21 11:52:11 Retreiving plugins.
2023/03/21 11:52:11 Detected NVML platform: found NVML library
2023/03/21 11:52:11 Detected non-Tegra platform: /sys/devices/soc0/family file not found
2023/03/21 11:52:17 Starting GRPC server for 'nvidia.com/gpu'
2023/03/21 11:52:17 Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia-gpu.sock
2023/03/21 11:52:17 Registered device plugin for 'nvidia.com/gpu' with Kubelet
The nvidia-device-plugin is running correctly without any restarts.
I'm able to create nvidia-enabled pods that output correctly the GPUs:
apiVersion: v1
kind: Pod
metadata:
 name: nvidia-smi
spec:
 runtimeClassName: "nvidia"
 containers:
    - name: nvidia-smi
      image: "nvidia/cuda:12.1.0-base-ubuntu18.04"
      command:
        - "nvidia-smi"
      resources:
        requests:
          nvidia.com/gpu: "1"
        limits:
          nvidia.com/gpu: "1"
The output is as expected (kubectl logs nvidia-smi):
Tue Mar 21 12:03:05 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:40:00.0 Off |                  N/A |
| 27%   29C    P8     8W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
Here comes the weird part
Let's now run e.g. nginx with runtimeClassName: nvidia:
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: denkjobs
spec:
  restartPolicy: "Never"
  containers:
  - name: nginx
    image: "nginx"
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
  runtimeClassName: nvidia
After 0-30 seconds the pod gets killed:
NAME         READY   STATUS      RESTARTS   AGE
nginx        0/1     Error       0          13m
or
NAME         READY   STATUS      RESTARTS   AGE
nginx        0/1     Completed       0          13m
Name:                nginx
Namespace:           denkjobs
Priority:            0
Runtime Class Name:  nvidia
Service Account:     default
Node:                agent/100.121.42.124
Start Time:          Tue, 21 Mar 2023 12:56:25 +0100
Labels:              <none>
Annotations:         <none>
Status:              Failed
IP:                  10.42.3.7
IPs:
  IP:  10.42.3.7
Containers:
  nginx:
    Container ID:  containerd://084bd9cb01e8de9c2e9e193f6c511be492de489a2a51ae7dcecc8f634219f672
    Image:         nginx
    Image ID:      docker.io/library/nginx@sha256:aa0afebbb3cfa473099a62c4b32e9b3fb73ed23f2a75a65ce1d4b4f55a5c2ef2
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
    Args:
      infinity
    State:          Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 21 Mar 2023 12:56:32 +0100
      Finished:     Tue, 21 Mar 2023 12:57:03 +0100
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        500m
      memory:     512Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rdwhv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-rdwhv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  13m   default-scheduler  Successfully assigned denkjobs/nginx to denkmaschine3
  Normal  Pulling    13m   kubelet            Pulling image "nginx"
  Normal  Pulled     13m   kubelet            Successfully pulled image "nginx" in 6.850461587s (6.850472207s including waiting)
  Normal  Created    13m   kubelet            Created container nginx
  Normal  Started    13m   kubelet            Started container nginx
  Normal  Killing    13m   kubelet            Stopping container nginx
Same pod without runtimeClussName: nvidia keeps running:
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          2m
Additional Info
nvidia-enabled containers directly run with ctr run --gpus 0 ... are not getting killed
journalctl -fu k3s-agent during the relevant lifetime of the nginx pod:
Mar 21 12:13:31 agent k3s[185245]: I0321 12:13:31.877137  185245 topology_manager.go:205] "Topology Admit Handler"
Mar 21 12:13:31 agent k3s[185245]: E0321 12:13:31.877228  185245 cpu_manager.go:394] "RemoveStaleState: removing container" podUID="6e7c7c7f-2901-4bdc-a9dc-25efa296db46" containerName="nvidia-smi"
Mar 21 12:13:31 agent k3s[185245]: I0321 12:13:31.877275  185245 memory_manager.go:345] "RemoveStaleState removing state" podUID="6e7c7c7f-2901-4bdc-a9dc-25efa296db46" containerName="nvidia-smi"
Mar 21 12:13:31 agent k3s[185245]: I0321 12:13:31.964635  185245 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bqx\" (UniqueName: \"kubernetes.io/projected/b6968b99-f11d-40f6-b483-8132f90748e4-kube-api-access-r8bqx\") pod \"nginx\" (UID: \"b6968b99-f11d-40f6-b483-8132f90748e4\") " pod="denkjobs/nginx"
Mar 21 12:13:34 agent k3s[185245]: I0321 12:13:34.399208  185245 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="205f3c0abcfcfd53bf62da65ac80ed05acf882ca77f11147d4134e50cfaa17c0"
Mar 21 12:13:35 agent k3s[185245]: I0321 12:13:35.489268  185245 reconciler.go:211] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8bqx\" (UniqueName: \"kubernetes.io/projected/b6968b99-f11d-40f6-b483-8132f90748e4-kube-api-access-r8bqx\") pod \"b6968b99-f11d-40f6-b483-8132f90748e4\" (UID: \"b6968b99-f11d-40f6-b483-8132f90748e4\") "
Mar 21 12:13:35 agent k3s[185245]: I0321 12:13:35.491411  185245 operation_generator.go:890] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6968b99-f11d-40f6-b483-8132f90748e4-kube-api-access-r8bqx" (OuterVolumeSpecName: "kube-api-access-r8bqx") pod "b6968b99-f11d-40f6-b483-8132f90748e4" (UID: "b6968b99-f11d-40f6-b483-8132f90748e4"). InnerVolumeSpecName "kube-api-access-r8bqx". PluginName "kubernetes.io/projected", VolumeGidValue ""
Mar 21 12:13:35 agent k3s[185245]: I0321 12:13:35.589435  185245 reconciler.go:399] "Volume detached for volume \"kube-api-access-r8bqx\" (UniqueName: \"kubernetes.io/projected/b6968b99-f11d-40f6-b483-8132f90748e4-kube-api-access-r8bqx\") on node \"agent\" DevicePath \"\""
Mar 21 12:13:40 agent k3s[185245]: I0321 12:13:40.591952  185245 scope.go:115] "RemoveContainer" containerID="084bd9cb01e8de9c2e9e193f6c511be492de489a2a51ae7dcecc8f634219f672"
cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true


[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
  runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

/var/log/syslog after creating the nginx pod with nvidia runtime class

here you can see that the pod is killed immediately after startup

Mar 21 13:23:23 agent k3s[1541]: I0321 13:23:23.125548    1541 topology_manager.go:205] "Topology Admit Handler"
Mar 21 13:23:23 agent k3s[1541]: E0321 13:23:23.125647    1541 cpu_manager.go:394] "RemoveStaleState: removing container" podUID="669c3b19-2aa4-4ee0-b9b6-0f1de6214ddd" containerName="nginx"
Mar 21 13:23:23 agent k3s[1541]: I0321 13:23:23.125712    1541 memory_manager.go:345] "RemoveStaleState removing state" podUID="669c3b19-2aa4-4ee0-b9b6-0f1de6214ddd" containerName="nginx"
Mar 21 13:23:23 agent systemd[1]: Created slice libcontainer container kubepods-burstable-pod6972d491_5270_4f99_a28a_691f0d6f90bf.slice.
Mar 21 13:23:23 agent k3s[1541]: I0321 13:23:23.177399    1541 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcf9r\" (UniqueName: \"kubernetes.io/projected/6972d491-5270-4f99-a28a-691f0d6f90bf-kube-api-access-mcf9r\") pod \"nginx\" (UID: \"6972d491-5270-4f99-a28a-691f0d6f90bf\") " pod="denkjobs/nginx"
Mar 21 13:23:23 agent systemd-udevd[8856]: Using default interface naming scheme 'v249'.
Mar 21 13:23:23 agent systemd-networkd[1364]: vethb9798904: Link UP
Mar 21 13:23:23 agent networkd-dispatcher[1392]: WARNING:Unknown index 17 seen, reloading interface list
Mar 21 13:23:23 agent kernel: [ 3513.105032] cni0: port 3(vethb9798904) entered blocking state
Mar 21 13:23:23 agent kernel: [ 3513.105037] cni0: port 3(vethb9798904) entered disabled state
Mar 21 13:23:23 agent kernel: [ 3513.105107] device vethb9798904 entered promiscuous mode
Mar 21 13:23:23 agent systemd-networkd[1364]: vethb9798904: Gained carrier
Mar 21 13:23:23 agent kernel: [ 3513.109075] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Mar 21 13:23:23 agent kernel: [ 3513.109089] IPv6: ADDRCONF(NETDEV_CHANGE): vethb9798904: link becomes ready
Mar 21 13:23:23 agent kernel: [ 3513.109092] cni0: port 3(vethb9798904) entered blocking state
Mar 21 13:23:23 agent kernel: [ 3513.109095] cni0: port 3(vethb9798904) entered forwarding state
Mar 21 13:23:24 agent systemd-networkd[1364]: vethb9798904: Link DOWN
Mar 21 13:23:24 agent systemd-networkd[1364]: vethb9798904: Lost carrier
Mar 21 13:23:24 agent kernel: [ 3513.834318] cni0: port 3(vethb9798904) entered disabled state
Mar 21 13:23:24 agent kernel: [ 3513.835028] device vethb9798904 left promiscuous mode
Mar 21 13:23:24 agent kernel: [ 3513.835032] cni0: port 3(vethb9798904) entered disabled state
Mar 21 13:23:24 agent systemd[1]: run-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-04265b8965b24c6b11467e691d58b531f3c7d42a4f7aaeef51726464cda1e822-rootfs.mount: Deactivated successfully.
Mar 21 13:23:24 agent systemd[1]: run-k3s-containerd-io.containerd.grpc.v1.cri-sandboxes-04265b8965b24c6b11467e691d58b531f3c7d42a4f7aaeef51726464cda1e822-shm.mount: Deactivated successfully.
Mar 21 13:23:24 agent systemd[1]: run-netns-cni\x2dff2367b4\x2dd457\x2d349f\x2dc460\x2dbbdac8f453d9.mount: Deactivated successfully.
Mar 21 13:23:24 agent k3s[1541]: I0321 13:23:24.863403    1541 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="04265b8965b24c6b11467e691d58b531f3c7d42a4f7aaeef51726464cda1e822"
Mar 21 13:23:25 agent k3s[1541]: I0321 13:23:25.998077    1541 reconciler.go:211] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcf9r\" (UniqueName: \"kubernetes.io/projected/6972d491-5270-4f99-a28a-691f0d6f90bf-kube-api-access-mcf9r\") pod \"6972d491-5270-4f99-a28a-691f0d6f90bf\" (UID: \"6972d491-5270-4f99-a28a-691f0d6f90bf\") "
Mar 21 13:23:26 agent k3s[1541]: I0321 13:23:26.000564    1541 operation_generator.go:890] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6972d491-5270-4f99-a28a-691f0d6f90bf-kube-api-access-mcf9r" (OuterVolumeSpecName: "kube-api-access-mcf9r") pod "6972d491-5270-4f99-a28a-691f0d6f90bf" (UID: "6972d491-5270-4f99-a28a-691f0d6f90bf"). InnerVolumeSpecName "kube-api-access-mcf9r". PluginName "kubernetes.io/projected", VolumeGidValue ""
Mar 21 13:23:26 agent systemd[1]: var-lib-kubelet-pods-6972d491\x2d5270\x2d4f99\x2da28a\x2d691f0d6f90bf-volumes-kubernetes.io\x7eprojected-kube\x2dapi\x2daccess\x2dmcf9r.mount: Deactivated successfully.
Mar 21 13:23:26 agent k3s[1541]: I0321 13:23:26.099100    1541 reconciler.go:399] "Volume detached for volume \"kube-api-access-mcf9r\" (UniqueName: \"kubernetes.io/projected/6972d491-5270-4f99-a28a-691f0d6f90bf-kube-api-access-mcf9r\") on node \"agent\" DevicePath \"\""
Mar 21 13:23:26 agent systemd[1]: Removed slice libcontainer container kubepods-burstable-pod6972d491_5270_4f99_a28a_691f0d6f90bf.slice.

Anyone having any ideas how to debug this further?
What is killing my pods?
I'm completely lost...

Duplicate issue #8318: K3s GPU Agent Node has all pods constantly restart
Created: 2023-09-07T20:36:27Z
Environmental Info:
K3s Version:

adammc@isaic:~$ k3s -v
k3s version v1.27.4+k3s1 (36645e73)
go version go1.20.6

Node(s) CPU architecture, OS, and Version:

adammc@isaic:~$ uname -a
Linux isaic.mcaq.me 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

1 Server, 1 GPU Agent, configured using https://github.com/alexellis/k3sup that are on different clouds
Describe the bug:

Essentially, I cannot run anything on the agent node because after 1-2mins, the pods restart.
Steps To Reproduce:
I doubt this is re-produceable, and instead think its caused by something being configured differently where I am hosting the GPU agent node (On Openstack through https://isaic.ca/), but litrally just running k3sup join on my system creates this problem.
Expected behavior:

Default Pods (Node Exporter and Traefik) should come up and run fine
Actual behavior:

Pods constantly restart. The Traefik Pods have an error, the node exporter does not.
+ '[' '!' -e /pause ]
+ mkfifo /pause
/usr/bin/entry: line 84: can't open /pause: Interrupted system call

Additional context / logs:

adam@vmi1447591:~$ kubectl describe node isaic.mcaq.me
Name:               isaic.mcaq.me
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=isaic.mcaq.me
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=k3s
Annotations:        flannel.alpha.coreos.com/backend-data: {"PublicKey":"wxseHSBRKRJfuJYDL4eN4IZUZHurSJ9L3WRPbTZAkEk="}
                    flannel.alpha.coreos.com/backend-type: wireguard
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.1.56
                    k3s.io/hostname: isaic.mcaq.me
                    k3s.io/internal-ip: 192.168.1.56
                    k3s.io/node-args: ["agent"]
                    k3s.io/node-config-hash: FDWWD5WXW4EHUAZ3J63RGNVLRP6JN4DAMGK52EUDOPFYEN3XN6XQ====
                    k3s.io/node-env:
                      {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/dc43f496a0a9ac19d3b2444d390db38e0cfb38e672721f838b075422b8734994","K3S_TOKEN":"********","K3S_U...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 07 Sep 2023 13:09:36 -0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  isaic.mcaq.me
  AcquireTime:     <unset>
  RenewTime:       Thu, 07 Sep 2023 13:32:05 -0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.1.56
  Hostname:    isaic.mcaq.me
Capacity:
  cpu:                16
  ephemeral-storage:  1016057248Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             64293428Ki
  pods:               110
Allocatable:
  cpu:                16
  ephemeral-storage:  988420490080
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             64293428Ki
  pods:               110
System Info:
  Machine ID:                 64f675d95c8d411fadf4c3e37c78b08c
  System UUID:                1bf3eda3-a208-46af-98c1-8b6fbd7233b8
  Boot ID:                    f38f6db9-40eb-4cb7-a2fe-d104d2e11ba2
  Kernel Version:             5.15.0-83-generic
  OS Image:                   Ubuntu 22.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.1-k3s1
  Kubelet Version:            v1.27.4+k3s1
  Kube-Proxy Version:         v1.27.4+k3s1
PodCIDR:                      10.42.8.0/24
PodCIDRs:                     10.42.8.0/24
ProviderID:                   k3s://isaic.mcaq.me
Non-terminated Pods:          (2 in total)
  Namespace                   Name                            CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                            ------------  ----------  ---------------  -------------  ---
  kube-system                 svclb-traefik-49c11ff9-p76gc    0 (0%)        0 (0%)      0 (0%)           0 (0%)         10m
  lens-metrics                node-exporter-ndk8c             10m (0%)      200m (1%)   24Mi (0%)        100Mi (0%)     10m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                10m (0%)   200m (1%)
  memory             24Mi (0%)  100Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)

Some things in here appear strange to me. flannel.alpha.coreos.com/public-ip: 192.168.1.56 is set to the public IP, and from Lens https://k8slens.dev/ the memory is marked as N/A https://photos.app.goo.gl/xudHwja6YXhLZZ5r7
I have verified that the container runtime is okay
sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml
sudo ctr image pull docker.io/nvidia/cuda:11.6.2-base-ubuntu20.04
sudo ctr run --rm -t --runc-binary=/usr/bin/nvidia-container-runtime --env NVIDIA_VISIBLE_DEVICES=all docker.io/nvidia/cuda:11.6.2-base-ubuntu20.04 cuda-11.6.2-base-ubuntu20.04 nvidia-smi
runtime = "nvidia-container-runtime"
adammc@isaic:~$ sudo ctr run --rm -t --runc-binary=/usr/bin/nvidia-container-runtime --env NVIDIA_VISIBLE_DEVICES=all docker.io/nvidia/cuda:11.6.2-base-ubuntu20.04 cuda-11.6.2-base-ubuntu20.04 nvidia-smi
Thu Sep  7 20:35:39 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe          On  | 00000000:00:06.0 Off |                   On |
| N/A   31C    P0              41W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| MIG devices:                                                                          |
+------------------+--------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                   Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                     BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                |        ECC|                       |
|==================+================================+===========+=======================|
|  No MIG devices found                                                                 |
+---------------------------------------------------------------------------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

Maybe one interesting thing to note, is other times I have configured k3s, and on the master node in this case, the internal node IP is always set to the external node IP, see
NAME                           STATUS   ROLES                       AGE   VERSION        INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
isaic.mcaq.me                  Ready    <none>                      29m   v1.27.4+k3s1   192.168.1.56      <none>        Ubuntu 22.04.3 LTS   5.15.0-83-generic   containerd://1.7.1-k3s1
vmi1447591.contaboserver.net   Ready    control-plane,etcd,master   2d    v1.27.4+k3s1   144.126.146.150   <none>        Ubuntu 20.04.6 LTS   5.4.0-162-generic   containerd://1.7.1-k3s1

The contabo nodes internal IP is actually its external IP. I wonder if that has anything to do with this?
I also got a cgroup systemd error originally when setting this up, so every time I reboot I also run
sudo mkdir /sys/fs/cgroup/systemd
sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Also, legacy iptables were not originally installed, so I fixed that with
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Again, not sure if that info is relevant but thought it would be useful.

Closing comment (2023-09-07T20:43:23Z): I cannot run anything on the agent node because after 1-2mins, the pods restart.


This looks like a duplicate of #7130.

I'm not sure that it was ever resolved; but it's not something that K3s or containerd would do. Best I can tell something in the GPU operator is terminating the pods, but I have no idea why it would do that.