Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s- Which test(s) are failing: should unmount if pod is force deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 11/07/18 15:02 Testgrid link: https://k8s- Reason for failure: The test case stops and starts kubelet, and it seems like kubelet is unable to start again Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
cc
Thanks for opening this Looking at , looks like could be the cause? /assign can you please look into this as priority. This is failing critical release blocking tests. /cc /sig cli /priority critical-urgent /milestone v1.13
It looks like kubelet keeps restarting. The error right before restarting is:
/assign
This bug could have been exposed by the newly tests in
Hmm, so I see two errors: Which is a Fatal error and would prevent Kubelet from starting anyways. In which case this error is handled correctly by the pluginwatcher, but I'm not sure preventing Kubelet from starting is the desired behavior. Should we log this error and skip it? There also seem to be a cleanup issue here on that file is this expected? This one is clearly a bug and not really handled correctly. It's due to the fact that we encounter an error at start, cleanup but don't wait for the gorountines to finish before cleaning up. I can send in a fix pretty easily for the second issue, basically add a waitgroup in the . For the first error, what do we think is the expected behavior?
This was mentioned in as well. /cc
Is this related to the issue where plugin watcher is recursively scanning for sockets?
Also, it should be noted that the path it's erroring on is not a socket type.
That's a bit different, as it is intended to scan recursively. Also note that the error here is produced by . What should be our behavior in such a case? Ignore that error?
I'm not sure if we want to be recursively scanning. The volume directories are mount points, which means there could be lots of files/directories existing in those volumes.
maybe we can put a limit on depth of plugin directory for on the same PR:
Ideally the directory that the plugin sockets are dropped in should be different from the directory where mounts happens. Currently the plugin socket directory is: ) The mounts directory is (so far so good). Problem is the "global mount" directory is also :( So options are: recursively scanning like suggested, and say plugins will can only be directly under This will break existing plugins, for example, the GCP plugin is at (see ). recursively scanning but change the plugin socket directory to ? This will also break existing plugins, like the GCP one pointed out above. the plugin dir the same, and leave recursion in, but put a limit on depth of plugin directory as suggested by This is ugly but may be the cleanest solution.
This conversation is completely different from the bug surfaced by this issue. Let's move that to the corresponding issue: I have a fix ready to go out for the panic, we should figure out what behavior we expect Kubelet to have when encountering this error from filepath.Walk: Should we ignore it?
Ignoring the error (with a warning maybe) instead of failing all of kubelet seems reasonable to me. If the plugin did not setup their registration socket correctly, then the end result should be that the plugin fails to register, but all other plugins should continue to work correctly.
the latest runs of this job no longer timesout - Instead I am seeing a new CSI related failure Is this expected. Do we want to track that failure in this bug now or do we have a separate issue?
the latest test logs indicate some test-infra issue:
the same shows up as a timeout on that particular test. Lets wait for the next run. but somehow the timeout seems to have gone. Are we still planning to merge and associated with this issue?
should be merged as it fixes a bug. can you help with approval? Thanks!
are you missing the right PR# in the comment above?
this is the correct PR -
/reopen Keeping it open until we can see the job pass a couple of rounds
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I see just one CSI test timing out in serial job Is that something the fixes here attempted to resolve? if not I can open a new issue if it persists and close this one out. Thanks.
The test mentioned in my comment above was a flake.. closing this issue now. will open a new issue for the flake
I'm getting the following related error on a busy Node: In the above case it's a full filesystem, but I imagine the same error would pop up with a RO filesystem as well. Looking at the patch, I'd expect kubelet would still be failing to start when it tries to watch it. (above was from a v1.12.3 cluster) , what do you reckon?
plugin watcher fixes for 1.12 are expected to be released with 1.12.5:
, I understand that I'm missing that patch. The patch code, however, would not make a difference in this case, if I read it correctly. It would not prevent the recursive dive into mounted volumes, as the errors happening at the deeper levels of that dive.
The patch blacklists the plugin watcher from searching the directory
OK, I was looking at the wrong patch after all (). Thanks for the clarifications.

Discussion:
} // Walks through the plugin directory discover any existing plugin sockets. <ins> // Goroutines started here will be waited for in Stop() before cleaning up. // Ignore all errors except root dir not being walkable </ins> func (w *Watcher) traversePluginDir(dir string) error { return w.fs.Walk(dir, func(path string, info os.FileInfo, err error) error { if err != nil { <del> return fmt.Errorf("error accessing path: %s error: %v", path, err) </del> <ins> if path == dir { return fmt.Errorf("error accessing path: %s error: %v", path, err) } glog.Errorf("error accessing path: %s error: %v", path, err) return nil </ins> } switch mode := info.Mode(); {

return fmt.Errorf("failed to watch %s, err: %v", path, err) } case mode&os.ModeSocket != 0: <ins> w.wg.Add(1) </ins> go func() { <ins> defer w.wg.Done() </ins> w.fsWatcher.Events <- fsnotify.Event{ Name: path, Op:   fsnotify.Create,