Repository: kubernetes/kubernetes (language: go)

Issue:
ci-kubernetes-e2e-kind-rootless began to fail on 2024-05-24: `Network plugin returns error: cni plugin not initialized`, caused due to `Turning off swap in unprivileged tmpfs mounts unsupported`
### Which jobs are failing?

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kind-rootless

### Which tests are failing?

Everything

### Since when has it been failing?

2024-05-24

### Testgrid link

https://testgrid.k8s.io/sig-testing-kind#kind-rootless

### Reason for failure (if possible)

```
  I0527 01:48:02.282662 2585 util.go:506] >>> kubeConfig: /home/prow/go/src/k8s.io/kubernetes/_rundir/c90e98a0-ee29-4890-b77b-c4642a035a4f/kubeconfig
  I0527 01:48:02.284039 2585 helper.go:48] Waiting up to 30m0s for all (but 0) nodes to be schedulable
  I0527 01:48:02.448443 2585 resource.go:131] Condition Ready of node kind-control-plane is false instead of true. Reason: KubeletNotReady, message: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  I0527 01:48:02.448486 2585 wait.go:256] Unschedulable nodes= 1, maximum value for starting tests= 0
  I0527 01:48:02.448546 2585 wait.go:258] 	-> Node kind-control-plane [[[ Ready=false, Network(available)=false, Taints=[{node.kubernetes.io/not-ready  NoSchedule <nil>}], NonblockingTaints=node-role.kubernetes.io/control-plane ]]]
  I0527 01:48:02.448556 2585 wait.go:270] ==== node wait: 0 out of 1 nodes are ready, max notReady allowed 0.  Need 1 more before starting.
```

### Anything else we need to know?

_No response_

### Relevant SIG(s)

/sig testing



Discussion / hints:
/sig node
kubelet logs
```
May 27 06:38:17 kind-control-plane kubelet[721]: Output: mount: /var/lib/kubelet/pods/0eed826e-a932-4683-ba0f-f95e26e9f7f9/volumes/kubernetes.io~projected/kube-api-access-k2t9w: wrong fs type, bad option, bad superblock on tmpfs, missing codepage or helper program, or other error.
May 27 06:38:17 kind-control-plane kubelet[721]:        dmesg(1) may have more information after failed mount system call.
May 27 06:38:17 kind-control-plane kubelet[721]: E0527 06:38:17.627285     721 mount_linux.go:282] Mount failed: exit status 32
May 27 06:38:17 kind-control-plane kubelet[721]: Mounting command: mount
May 27 06:38:17 kind-control-plane kubelet[721]: Mounting arguments: -t tmpfs -o size=16720162816,noswap tmpfs /var/lib/kubelet/pods/d0fc5891-2a86-4fb6-9f14-e67b806ec0e3/volumes/kubernetes.io~projected/kube-api-access-sk697
May 27 06:38:17 kind-control-plane kubelet[721]: Output: mount: /var/lib/kubelet/pods/d0fc5891-2a86-4fb6-9f14-e67b806ec0e3/volumes/kubernetes.io~projected/kube-api-access-sk697: wrong fs type, bad option, bad superblock on tmpfs, missing codepage or helper program, or other error.
May 27 06:38:17 kind-control-plane kubelet[721]:        dmesg(1) may have more information after failed mount system call.
May 27 06:38:17 kind-control-plane kubelet[721]: E0527 06:38:17.627407     721 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0fc5891-2a86-4fb6-9f14-e67b806ec0e3-kube-api-access-sk697 podName:d0fc5891-2a86-4fb6-9f14-e67b806ec0e3 nodeName:}" failed. No retries permitted until 2024-05-27 06:38:49.627386266 +0000 UTC m=+69.060375548 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sk697" (UniqueName: "kubernetes.io/projected/d0fc5891-2a86-4fb6-9f14-e67b806ec0e3-kube-api-access-sk697") pod "kube-proxy-f2bs2" (UID: "d0fc5891-2a86-4fb6-9f14-e67b806ec0e3") : mount failed: exit status 32
May 27 06:38:17 kind-control-plane kubelet[721]: Mounting command: mount
May 27 06:38:17 kind-control-plane kubelet[721]: Mounting arguments: -t tmpfs -o size=16720162816,noswap tmpfs /var/lib/kubelet/pods/d0fc5891-2a86-4fb6-9f14-e67b806ec0e3/volumes/kubernetes.io~projected/kube-api-access-sk697
May 27 06:38:17 kind-control-plane kubelet[721]: Output: mount: /var/lib/kubelet/pods/d0fc5891-2a86-4fb6-9f14-e67b806ec0e3/volumes/kubernetes.io~projected/kube-api-access-sk697: wrong fs type, bad option, bad superblock on tmpfs, missing codepage or helper program, or other error.
May 27 06:38:17 kind-control-plane kubelet[721]:        dmesg(1) may have more information after failed mount system call.
```

dmesg:
```
[ 8260.902617] tmpfs: Turning off swap in unprivileged tmpfs mounts unsupported
```
https://github.com/torvalds/linux/blob/v6.8/mm/shmem.c#L4004-L4011
Seems a regression in
- https://github.com/kubernetes/kubernetes/pull/124060

Fix patch:
diff --git a/pkg/kubelet/util/swap/swap_util.go b/pkg/kubelet/util/swap/swap_util.go
index 2160820b6f19a..c12f65b4d279e 100644
--- a/pkg/kubelet/util/swap/swap_util.go
+++ b/pkg/kubelet/util/swap/swap_util.go
@@ -26,6 +26,7 @@ import (
 
 	"k8s.io/apimachinery/pkg/util/version"
 	"k8s.io/klog/v2"
+	"k8s.io/kubernetes/pkg/kubelet/userns/inuserns"
 	utilkernel "k8s.io/kubernetes/pkg/util/kernel"
 	"k8s.io/mount-utils"
 )
@@ -46,6 +47,14 @@ func IsTmpfsNoswapOptionSupported(mounter mount.Interface, mountPath string) boo
 			return false
 		}
 
+		if inuserns.RunningInUserNS() {
+			// Turning off swap in unprivileged tmpfs mounts unsupported
+			// https://github.com/torvalds/linux/blob/v6.8/mm/shmem.c#L4004-L4011
+			// https://github.com/kubernetes/kubernetes/issues/125137
+			klog.InfoS("Running under a user namespace - tmpfs noswap is not supported")
+			return false
+		}
+
 		kernelVersion, err := utilkernel.GetVersion()
 		if err != nil {
 			klog.ErrorS(err, "cannot determine kernel version, unable to determine is tmpfs noswap is supported")


Test patch: