Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Quote I took goroutine dump of controller-managers that were stuck in volume provisioning and I think we have a deadlock in code: I think we have a problem introduced when we merged - and what is happening is, when in is called it acquires a lock on , but same function ends up calling which acquires another lock on object . Now the problem is, function gets called in a loop and it will try to acquire lock for each node it processes and hence two operations could result in deadlock. This results into inability to provision a with a default StorageClass. The volume does not get to call the API endpoint, just hangs after call, which is clear in the goroutine dump. What you expected to happen: 1) Provision a from successfully. 2) Don't be blocked by credentials secret change. How to reproduce it (as minimally and precisely as possible): 1) Provision a vSphere cluster with credentials secret set 2) Create volume 3) Wait... Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.0-rc.2+-dirty Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux Kernel (e.g. ): GNU/Linux Install tools: openshift-install (kubeadm) Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/vmware
This should be a blocker bug for 1.19 at least as far as vsphere cloudprovider integration is concerned, because it breaks cloudprovider integration/volume provisioning. cc
cc:
/milestone v1.19
Something makes me think this issue existed before, and just escalated to a larger scope after adding informers, demanding refresh for all objects. This is used within plugin in and it is mounting the volume if it is required, calling - it is locking on all the nodes at once. If the pod was just restarted (say just changed, or the health probes demanded a restart), it would process all existing at the same time as trying to mount/create the volume. And both threads will clang on the same lock, as the would probably not be populated at that moment, so will go the path of - same which is used in . So we get lost volume data, pending etc... And the links I'm sharing are from Kubernetes, where issues with similar footprint randomly arose.

Discussion:
return *node, nil } <ins> func (nm *NodeManager) getNodes() map[string]*v1.Node { nm.registeredNodesLock.RLock() defer nm.registeredNodesLock.RUnlock() registeredNodes := make(map[string]*v1.Node, len(nm.registeredNodes)) for nodeName, node := range nm.registeredNodes { registeredNodes[nodeName] = node } return registeredNodes } </ins> func (nm *NodeManager) addNode(node *v1.Node) { nm.registeredNodesLock.Lock() nm.registeredNodes[node.ObjectMeta.Name] = node

// // This method is a getter but it can cause side-effect of updating NodeInfo objects. func (nm *NodeManager) GetNodeDetails() ([]NodeDetails, error) { <del> nm.registeredNodesLock.Lock() defer nm.registeredNodesLock.Unlock() </del> var nodeDetails []NodeDetails <del> for nodeName, nodeObj := range nm.registeredNodes { </del> <ins> for nodeName, nodeObj := range nm.getNodes() { </ins> nodeInfo, err := nm.GetNodeInfoWithNodeObject(nodeObj) if err != nil { return nil, err

} func (nm *NodeManager) refreshNodes() (errList []error) { <del> nm.registeredNodesLock.Lock() defer nm.registeredNodesLock.Unlock() for nodeName := range nm.registeredNodes { </del> <ins> for nodeName := range nm.getNodes() { </ins> nodeInfo, err := nm.getRefreshedNodeInfo(convertToK8sType(nodeName)) if err != nil { errList = append(errList, err)