Change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Flakes: Full failure: Which test(s) are failing: Since when has it been failing: Since merging Testgrid link: Reason for failure: Serial slow tests are failing with: and the parallel tests are flaking with: Anything else we need to know: I would not expect kubelet to crash if the node if the plugin is not present. /sig windows cc:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It seems to me that it is not related to a plugin being not present, but that it cannot open the endpoint where potential plugins could register to. In other words, I suspect this error happens independently from any plugins. If we can trust this message , then this would indicate that it tries to open that listening socket twice. This would mean that is called twice. If this is the case, the question would be: Why? It would be interesting to see if shows twice.
There is an issue on some Windows not updated. When you restart kubelet and the socket file So the workarround, is to remove before restart kubeblet
And why tests on the PR didn't trigger this error ?
Question: Is it legal that is called twice? It seems that the function is called from , which is analogous to the Linux implementation Calling twice looks fishy to me. It seems to me that this problem is not caused by, but only exposed by the device plugin change
You mean that kubelet is initialized twice on the same computer, causing this error the second time? How is this handled on Linux? If this is allowed we could simply delete if present before opening it again.
Are you sure it's call twice ? because on windows is not present but replace by you can see the first line comment and . A simple test is: kubelet and check the socket file is still present kubelet you got the error, stop it socket file kubelet again, normally it's working Maybe the fix can be add in
I think your explanation is very plausible: That it is not called twice within a single kubelet initialization, but kubelet is initialized again for some reason on the same node. (It still means that it is called twice, but I didn't think about the possibility that kubelet is initialized twice.)
There are various ways of fixing it. But I would be interested in learning how this is handled under Linux. To choose the same path for Windows, too.
Thanks for your quick response and analysis :+1: It would be interesting to see if shows twice. You can find the kubelet logs (and many others) for the windows nodes in the "artifacts" tab on This set of tests runs configuration for GMSA tests which as part of the set up of the domain controller, causes a node resets. This explains why one of the nodes seems ok (it isn't randomly selected as part of the domain machines) and why the parallel tests don't cause a full failure (they don't cause node resets). Not all clusters use that script. This set of tests don't use them either (uses aks-engine which has a different start up script). IMHO this is a breaking change scince any cluster that has a node restart would fail to come back online. and kubelet should be able to restart (because of node restart or crash for another reason, etc.) It looks like there is code to clean up old sockets but it is not working on windows: Given there is clean up code there and it's a breaking change I think this this should be fixed in kubelet.
Oh yes, I missed that! (I should have looked only a few lines above the line causing the error.) It at least looks exactly like the code we need. The function is implemented in the same file: But why it is failing to work under Windows? Do you have an idea? Yes, sounds good.
Ok so on windows failed here
So this exemple reproduce the error. the failed only on socket file not on regular file ! Link to
It's bug on windows 1903 not present on windows 1809 The fix can be replace by
Very cool! Just for interest, how did you find
I hate Windows ;-) and I googled
It's bug on windows 1903 golang/go (comment) not present on windows 1809 This set of tests are running on 2019 LTSC (also known as 1809). If this is only an issue on 1903 is there something else happening?
Sorry, I read to quickly, there is a change of behaviour between 1809 and 1903 about unix socket, If my understanding is good on 1903 unix socket is like regular file but not on 1809, so it's why os.Stat failed on 1809. but I tested on Windows 10 20H2 and the issue still here with
So in summary, the key point is: PR fixes this issue

Discussion:
if filePath == m.checkpointFile() { continue } <del> stat, err := os.Stat(filePath) </del> <ins> // TODO: Until the bug - https://github.com/golang/go/issues/33357 is fixed, os.stat wouldn't return the // right mode(socket) on windows. Hence deleting the file, without checking whether // its a socket, on windows. stat, err := os.Lstat(filePath) </ins> if err != nil { klog.Errorf("Failed to stat file %s: %v", filePath, err) continue