Change:
What happened: When I restart pod with my driver and driver-registrar, node looses NodeID annotation. How to reproduce it (as minimally and precisely as possible): Kubernetes without any alpha features (local-up- in this case) HostPath driver for everything to be Running the node has annotation the node pod (deleting all pods here, we don't care about attacher/provisioner). will start a new node pod with new registrar. Wait until the new pod gets Running. the node does not have the annotation It's 80% reproducible in my local cluster. New driver-registrar deletes socket of the old registrar and creates new one. I can see in logs: So the old socket was deregistered very shortly before new one got registered. IMO these two operations collide with each other and the deregistration "wins" somehow. Environment: Kubernetes version (use ): today's master /kind bug /sig storage
Yeah, InstallCSIDriver runs in parallel with UninstallCSIDriver. UninstallCSIDriver updates the node while InstallCSIDriver is getting the current value. In most cases, InstallCSIDriver gets the old value (still with annotations) and does not update the node. Some locks are missing there. But even with the locks it will be racy. We must make sure that UninstallCSIDriver grabs the lock before InstallCSIDriver, so the annotation is always in the end.
We should see if we can handle this at the layer so that it works for everyone. We could use where key is the CSI driver name to ensure that two operations are not started in parallel for the same driver. But that's not the hard part, like you mentioned, the tricky part is ensuring the correct order of operations between create/delete events.
/milestone v1.13 /priority critical-urgent
I am investigating this today. Will update with fix/finding on a PR.
A best practice for the driver should be to remove the registration sockets when the driver is uninstalled: this would generally remedy this problem for that particular driver in normal uninstall-install cases. We do it for GCE PD here: I still think the underlying problem should be resolved though :+1:
Is there an incoming PR for this?
has been working on this today. He believes that he has the problem understood and is able to repro it. He is working on a fix/verification (which takes time since it requires multiple runs to repro/verify). He will add more information soon as he can.
After spending a good amount of time chasing the cause for this issue, it came down to the pluginwatcher code that can handle fsnotify CREATE and fsnotify DELETE events concurrently here: The at that line number causes both CREATE and DELETE events to be handled concurrently. This can create race conditions in plugins by calling and concurrently. This is fixed in this PR (see for detail of fix).
that is a great suggestion, will make not of it for documentation.
hi,I test in v1.13 and the problem existed! my csi plugin running,but the node annotation can not labeled! [root kubectl get po -n kube-system -o wide -w NAME READY STATUS RESTARTS AGE IP NODE csi-attacher-cinderplugin-0 2/2 Running 0 4d 111.111.111.116 111.111.111.116 csi-nodeplugin-cinderplugin-cpq7m 2/2 Running 0 14s 111.111.111.118 111.111.111.118 csi-nodeplugin-cinderplugin-h48dh 2/2 Running 0 14s 111.111.111.116 111.111.111.116 Annotations: then cause my pod can not attached I0409 01:50:59. 1 ] Saving attach error to "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e" I0409 01:50:59. 1 ] Ignoring VolumeAttachment "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e" change I0409 01:50:59. 1 ] Saved attach error to "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e" I0409 01:50:59. 1 ] Error processing "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e": failed to attach: node "111.111.111.116" has no NodeID annotation
AttachVolume.Attach failed for volume "pvc--758e-11e9-bfc0-" : node "rancher196" has no NodeID annotation
Docker 18.9.6 Kubelet v1.12.0 CentOS Linux 7 3.10.0-957.10.1.el7.x86_64 ttachVolume.Attach failed for volume "pvc--75ed-11e9-bfc0-" : node "crawler203" has no NodeID annotation Dashboard/volume/pvc--75ed-11e9-bfc0- Volume Details State:Detached Health: Frontend:Block Device Attached Node & Endpoint : Size:2 Gi Actual Size:0 Bi Base Image: Engine Image:rancher/longhorn-engine:v0.4.1 Created:10 minutes ago Latest Backup:
I don't find code in kubelet 1.12.0 about this problem. strange!

Discussion:
//TODO: Handle errors by taking corrective measures w.wg.Add(1) <del> go func() { </del> <ins> func() { </ins> defer w.wg.Done() if event.Op&fsnotify.Create == fsnotify.Create {