Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We saw this several times: after restarting kubelet, kubelet didn't creating new Pods. The healthz endpoint showed that the was stuck: From the stack trace, the goroutine was stuck in the call of when doing , so no other updates can be processed: Listing the volumes of orphaned Pods on this Node, there was an unresponsive NFS volume, stat it will hang forever, so the above goroutine should be stuck in cleaning up this Pod: The NFS server IP is a service IP, whose backend Pod happened to be scheduled to this Node. However, as was stuck, it didn't handle Pod Add/Update from apiserver and didn't create the backend Pod, hence a dead lock. Searched PR and issue history, it seems there were several ones fixing NFS volume issues but looks like not covering this case: (however reverted) (still open) Is there anyone still working on it? The main syncloop of kubelet shouldn't hang due to an unresponsive NFS volume, leading to no Pods can be created. Kubernetes version (use ): v1.20.5
/sig storage /sig node similar with and And the problem code is in According to and 's comments in , it will be removed later, but I checked the new code in mount-utils. The func is not changed: Possible fix to this would be something like below:
/assign
Hey Quick question: the links that you attached that could be possible fixes, were really helpful! Since the approach to check for wouldn't help here, a timeout based approach seems best, I'm a little unsure about what this timeout period could be for this case, any suggestions on the same?
I have no idea. you may ask sig-node guys in the slack or weekly meetings. Or send a PR and discuss it in the PR. The only nfs related timeout that I can found is 1 minute in
hit again, it‘s easy to reproduce if there is nfs service in the cluster.
/assign /triage accepted
/area node-lifecycle
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue New contributor can follow some advice in .
Hey ! I'm wondering if this issue is still being worked on? I'd like to help if there's room to do so!
I'm not working on it. Thanks for the help.
awesome!
/assign
Hi are you still working on this? I'm interested in this and please let me know if I can give it a try, Thanks! : )
I'm actually still on it but if you're open to it we can team up? I don't mind sharing
Thanks for reply, I am vary glad to hear that, I think you can continue your work and I would take a look at other issues, if there is anything I can help, please feel free to ping me : )
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Is someone still working on this or I am free to take a stab at this? I have ran into this a few times and trying to fix this cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /remove-label good-first-issue This isn't a great first issue to work on. It requires having a NFS server and instrumenting it to be down to test the fix.
The label(s) /remove-label good-first-issue api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feedlabels -additionallabelslabels -? <detailsIn response to : /remove-lifecycle rotten /remove-label good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-good-first-issue
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the PR is closed You can: Mark this PR as fresh with Close this PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Does this problem still exist?
/triage needs-information
/remove-lifecycle stale
/triage accepted cancel
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted cancel Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
remove-triage accepted
/remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
statx with ATSTATXDONT_SYNC flag will not stuck in syncLoop. how about replace stat with statx?

Discussion:
"time" "github.com/moby/sys/mountinfo" <ins> "golang.org/x/sys/unix" </ins> "k8s.io/klog/v2" utilexec "k8s.io/utils/exec"

errNotMounted = "not mounted" ) <ins> var ( // Error statx support since Linux 4.11, https://man7.org/linux/man-pages/man2/statx.2.html errStatxNotSupport = errors.New("the statx syscall is not supported. At least Linux kernel 4.11 is needed") ) </ins> // Mounter provides the default implementation of mount.Interface // for the linux platform.  This implementation assumes that the // kubelet is running in the host's root mount namespace.

return ListProcMounts(procMountsPath) } <del> // IsLikelyNotMountPoint determines if a directory is not a mountpoint. // It is fast but not necessarily ALWAYS correct. If the path is in fact // a bind mount from one part of a mount to another it will not be detected. // It also can not distinguish between mountpoints and symbolic links. // mkdir /tmp/a /tmp/b; mount --bind /tmp/a /tmp/b; IsLikelyNotMountPoint("/tmp/b") // will return true. When in fact /tmp/b is a mount point. If this situation // is of interest to you, don't use this function... func (mounter *Mounter) IsLikelyNotMountPoint(file string) (bool, error) { </del> <ins> func statx(file string) (unix.Statx_t, error) { var stat unix.Statx_t if err := unix.Statx(0, file, unix.AT_STATX_DONT_SYNC, 0, &stat); err != nil { if err == unix.ENOSYS { return stat, errStatxNotSupport } return stat, err } return stat, nil } func (mounter *Mounter) isLikelyNotMountPointStat(file string) (bool, error) { </ins> stat, err := os.Stat(file) if err != nil { return true, err

return true, nil } <ins> func (mounter *Mounter) isLikelyNotMountPointStatx(file string) (bool, error) { var stat, rootStat unix.Statx_t var err error if stat, err = statx(file); err != nil { return true, err } if stat.Attributes_mask != 0 { if stat.Attributes_mask&unix.STATX_ATTR_MOUNT_ROOT != 0 { if stat.Attributes&unix.STATX_ATTR_MOUNT_ROOT != 0 { // file is a mountpoint return false, nil } else { // no need to check rootStat if unix.STATX_ATTR_MOUNT_ROOT supported return true, nil } } } root := filepath.Dir(strings.TrimSuffix(file, "/")) if rootStat, err = statx(root); err != nil { return true, err } return (stat.Dev_major == rootStat.Dev_major && stat.Dev_minor == rootStat.Dev_minor), nil } // IsLikelyNotMountPoint determines if a directory is not a mountpoint. // It is fast but not necessarily ALWAYS correct. If the path is in fact // a bind mount from one part of a mount to another it will not be detected. // It also can not distinguish between mountpoints and symbolic links. // mkdir /tmp/a /tmp/b; mount --bind /tmp/a /tmp/b; IsLikelyNotMountPoint("/tmp/b") // will return true. When in fact /tmp/b is a mount point. If this situation // is of interest to you, don't use this function... func (mounter *Mounter) IsLikelyNotMountPoint(file string) (bool, error) { notMountPoint, err := mounter.isLikelyNotMountPointStatx(file) if errors.Is(err, errStatxNotSupport) { // fall back to isLikelyNotMountPointStat return mounter.isLikelyNotMountPointStat(file) } return notMountPoint, err } </ins> // CanSafelySkipMountPointCheck relies on the detected behavior of umount when given a target that is not a mount point. func (mounter *Mounter) CanSafelySkipMountPointCheck() bool { return mounter.withSafeNotMountedBehavior