Change:
* You need to be able to exec into pods * Feature gate `UserNamespacesStatelessPodsSupport` need to be enabled. <del> In addition, support is needed in the {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} to use this feature with Kubernetes stateless pods: </del> <ins> The cluster that you're using **must** include at least one node that meets the [requirements](/docs/concepts/workloads/pods/user-namespaces/#before-you-begin) for using user namespaces with Pods. </ins> <del> * CRI-O: v1.25 has support for user namespaces. </del> <ins> If you have a mixture of nodes and only some of the nodes provide user namespace support for Pods, you also need to ensure that the user namespace Pods are [scheduled](/docs/concepts/scheduling-eviction/assign-pod-node/) to suitable nodes. </ins> Please note that **if your container runtime doesn't support user namespaces, the new `pod.spec` field will be silently ignored and the pod will be created without

Discussion:
CONDITION was running with UserNamespacesStatelessPodsSupport=true -apiserver was running with UserNamespacesStatelessPodsSupport=true busybox-user configured hostUsers: false deployed successfully: file created: containerd log, no UserNameSpace Option in spec.Spec struct (I modified the log to highlight NamespaceOptions part): SYMPTOM No Pod user namespace created:container was running as root:no id mapping set in uid_map: Pod user namespace created with command was running as non-root user feature gate for kubelet and apiserver Pod configuration and apply it user namespace with I have tested directly with and to confirm runc support user namespace. Here is part of feature output, so I think containerd and runc part is OK: It seems kubelet did not provide the correct user namespace options in request of RunPodSandbox. <details</details<detailsno </details<details</details<details</details<details</details<detailsNot relavent </details
/sig node
can you please attach the for this. First, it will confirm that kubelet picked up the configuration and second it may have some failures that can give a clue what is happening /cc /triage needs-information
any chance you can try the same set up on an OS with newer kernel version? Just to confirm if this is OS specific or not
Am I reading the report correctly and you are using containerd 1.6? I mean, it says wrongly formatted this: In that case, this is completely expected. It is documented here too, see the end of this section in bold letter: https://v1- What this means is: support for userns was in containerd 1.7. Older versions of containerd, for backwards compatibility, just ignore the request of userns from the kubelet.
Also, in the original report, you pasted twice the api-server in the output of :)
if confirms this is running containerd 1.6, I think we can close this :)
1.7 supports it, correct?

Can you send an update for https://v1-
/remove-kind bug /kind documentation /triage accepted /retitle "document that containerd 1.6 does not support userns feature (KEP 127)"
/triage accepted
/retitle "document that containerd 1.7 is needed to use the userns feature (KEP 127)"
In which case, I think the place to explain that is /remove-triage needs-information
/transfer website
/language en /sig docs
thanks, done! Please note, though, that the link I pasted (https://v1-) mentions also that support is needed from the container runtime (if you don't have support what was reported here happens) and the only runtime supported is CRIO. I think maybe that was not so clear, and now containerd 1.7 was released a few days ago, so I opened a PR to mention containerd 1.7 and hopefully clarify the runtime support
Thank you for kind reply. containerd version is 1.6.8 in my env. I'll try to update it to 1.7.0 and deploy the rootless pod again. Besides, I have already tried to debug run the kubelet from src with dlv and confirmed that the API request contained userns options(idmapping). So the problem likely exists on containerd side.
Additionally, the information provided by kubelet's log is so limited. That makes me hard to locate the problem.
Here is the update after I upgrade containerd 1.6.8 -1.7.0 After I deploy the rootless pod: userns created: is running as the rootless user: mapping is correct: Thank you for the great help!
And what about OCI runtime version (eg. runc)? I think it is helpful to clarify its version requisition.
cool, great that it is working now! Regarding the OCI runtime, each containerd version mandates that at least runc version X should be used, so we should be safe with that. I think distros honor it too, but if you know some example where it is not honored, let us know!