Change:
on my scenescreate same cronjob use same sfs(the nfs storage on Huawei Cloud) type storage by pvc to do same training work. At the same time, a node will schedule many pods using the same pvc, and there will be many mount and umount tasks about this pvc executed at the same time. And we find all pods on a node at container creating state, because of waiting pvc to mount. check log, we can't find any we can't find any operationExecutor.MountVolume started or failed log like "operationExecutor.MountVolume started for" and also we check volumemanagertotalvolumes metrics, volume is add to dsw. analyze code, if isExpectedError, kubelet not log any thing and operationExecutor will generate operation, add call pendingOperations Run, pendingOperations Run will check operation slice, to find whether has same operation exists, and if operation exist, will check operation's operationPending status, if equal true, means same operation now is excuting, and return AlreadyExists. operationKey composed of volumeName, podName, nodeName so MountVolume operationKey {volumeName, EmptyUniquePodName,EmptyNodeName}, use EmptyUniquePodName to avoid executing mount/map from multiple pods referencing the same volume in parallel soUnmountVolume operationKey {volumeName, podUID,EmptyNodeName}, because All volume plugins can execute unmount/unmap for multiple pods referencing the same volume in parallel now see isOperationExists func, if previousOp.podName equals to EmptyUniquePodName or current operation podName equals to EmptyUniquePodName, will return podNameMatch, mount operation can find umount operation or umount operation can find mount operation because of isOperationExists, this mean if a volume now is excuting mount, a umount can't add to operations, and if a umount now is excuting, a mount operation can't add to operations, but multiple pods referencing the same volume can add multiple umount operation into operation, may lead operation leak. t0 two pod do umount volume operation t1 pod1 umount failed, update pending to false t2 pod3 add, operation excuter add mount operation, will use index 0, override umount operation t3 mount operation failed , update pending to false t4 pod2 umount operation add, will find index 0, add override opertation, now ,we have two same umount operation, now we have two goroutines to do pod2 umount operations t5 first goroutine pod2 umount failed, will update index 0 operation, update pending to false t6 second goroutine pod2 umount success, will delete index 0 operation, finally lead a umount opertion in cache, and this operation will lead all mount opeartion return AlreadyExistsError, and not cause any csi call. and i think there exists some other scene can trigger this leak. and also, when operation success, deleteOperation, will use tail element to replace, and may increase the probalility of occurrence. because isOperationExists will always pick first match one. and also override mechanism may lead exponentialBackOff invalid for example mount and umount take turns to excute. pv can mount , pod can running reproduce step: create same cronjob that use same pvc(use csi sfs) and csi driver restart periodic No response_ <details</details23 <detailsHuaweiCloud </details<details</details<details</details<details</details<details</detailsCSI
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
+1 meet the same problem
I don't know much about volumeManager. It seems that there is a problem with the override mechanism. Can mount and unmount of the same volume be executed at the same time? If it can, mount should not override any unmount operation. Otherwise, the mount operation should override all matched operations, not just the first one.
also don't know much about design history background, but comments in volume manager seems to conflict with each other among UmountDevice and UnmountVolume. need sameone who familar with it to explain Otherwise, the mount operation should override all matched operations, not just the first one. yes, must discuss rules for the existence of an operation
cc
Suppose there are the following cases: volume's "volumemount" should block the next same "volumemount" or "umountdevice" and all related pods's "volumeumount" exp1: , then could not be , then could be volume's "volumeumount" should block the next same "volumeumount" and it's "volumemount" or "umountdevice", but it should not block it's related other pods's "volumeumount" exp2: , then could not be , then could not be , then could not be , then could be volume's "umountdevice" should block the next same "umountdevice" or "volumemount" and all related pods's "volumeumount" exp3: , then could not be , then could be So we should improve the isOperationExists to satify case 2, that is, the intermediate state should not occur. Maybe since this pr: , this logic remains the same. /cc
/remove-sig node for sig storage to triage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

Discussion:
return false, -1 } <ins> opIndex := -1 </ins> for previousOpIndex, previousOp := range grm.operations { volumeNameMatch := previousOp.key.volumeName == key.volumeName

key.podName == EmptyUniquePodName || previousOp.key.podName == key.podName <ins> podNameExactMatch := previousOp.key.podName == key.podName </ins> nodeNameMatch := previousOp.key.nodeName == EmptyNodeName || key.nodeName == EmptyNodeName || previousOp.key.nodeName == key.nodeName <ins> nodeNameExactMatch := previousOp.key.nodeName == key.nodeName </ins> if volumeNameMatch && podNameMatch && nodeNameMatch { <del> return true, previousOpIndex </del> <ins> // nonExactMatch pending first if previousOp.operationPending { return true, previousOpIndex } // nonExactMatch with no pending, set opIndex to the first nonExactMatch // exactMatch can override opIndex to expected if opIndex == -1 || (podNameExactMatch && nodeNameExactMatch) { opIndex = previousOpIndex } </ins> } } <ins> return opIndex != -1, opIndex </ins> <del> return false, -1 </del> } func (grm *nestedPendingOperations) getOperation(key operationKey) (uint, error) {

return fmt.Errorf("timeout after %v", timeout) } } <ins> func Test_NestedPendingOperations_OperationExists_PendingFirst(t *testing.T) { // Arrange grm := NewNestedPendingOperations(true /* exponentialBackOffOnError */) volumeName := v1.UniqueVolumeName("test-volume") podName1 := volumetypes.UniquePodName("pod1") podName2 := volumetypes.UniquePodName("pod2") podName3 := volumetypes.UniquePodName("pod3") podName4 := EmptyUniquePodName nodeName := EmptyNodeName // delay after an operation is signaled to finish to ensure it actually // finishes before running the next operation. delay := 50 * time.Millisecond // fake operation1 for pod1 failed operation1DoneCh := make(chan interface{}) operation1 := generateWaitWithErrorFunc(operation1DoneCh) err1 := grm.Run(volumeName, podName1, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation1, OperationName: "umount"}) if err1 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err1) } // fake operation2 for pod2 fails operation2DoneCh := make(chan interface{}) operation2 := generateWaitWithErrorFunc(operation2DoneCh) err2 := grm.Run(volumeName, podName2, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation2, OperationName: "umount"}) if err2 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err2) } // fake operation3 for pod3 pending operation3DoneCh := make(chan interface{}) operation3 := generateWaitFunc(operation3DoneCh) defer func() { close(operation3DoneCh) }() err3 := grm.Run(volumeName, podName3, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation3, OperationName: "umount"}) if err3 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err3) } operation1DoneCh <- true operation2DoneCh <- true time.Sleep(delay) // fake operation4 for EmptyUniquePodName should be rejected as operation3 is still pending operation4DoneCh := make(chan interface{}) operation4 := generateWaitFunc(operation4DoneCh) defer func() { close(operation4DoneCh) }() err4 := grm.Run(volumeName, podName4, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation4, OperationName: "mount"}) // Assert if err4 == nil { t.Errorf("NestedPendingOperations did not fail. Expected an operation to already exist") } if !IsAlreadyExists(err4) { t.Errorf("NestedPendingOperations did not return alreadyExistsError, got: %v", err4) } } func Test_NestedPendingOperations_OperationExists_ExactMatchFirstNoPending(t *testing.T) { // Arrange grm := NewNestedPendingOperations(true /* exponentialBackOffOnError */) volumeName := v1.UniqueVolumeName("test-volume") podName1 := volumetypes.UniquePodName("pod1") podName2 := volumetypes.UniquePodName("pod2") podName3 := volumetypes.UniquePodName("pod3") podName4 := EmptyUniquePodName nodeName := EmptyNodeName // delay after an operation is signaled to finish to ensure it actually // finishes before running the next operation. delay := 50 * time.Millisecond backoffDelay := 500 * time.Millisecond // fake operation1 for pod1 fails operation1DoneCh := make(chan interface{}) operation1 := generateWaitWithErrorFunc(operation1DoneCh) err1 := grm.Run(volumeName, podName1, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation1, OperationName: "umount"}) if err1 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err1) } // fake operation2 for pod2 fails operation2DoneCh := make(chan interface{}) operation2 := generateWaitWithErrorFunc(operation2DoneCh) err2 := grm.Run(volumeName, podName2, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation2, OperationName: "umount"}) if err2 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err2) } // fake operation3 for pod3 fails operation3DoneCh := make(chan interface{}) operation3 := generateWaitWithErrorFunc(operation3DoneCh) err3 := grm.Run(volumeName, podName3, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation3, OperationName: "umount"}) if err3 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err3) } operation1DoneCh <- true operation2DoneCh <- true operation3DoneCh <- true time.Sleep(delay) // fake operation4 with EmptyUniquePodName fails operation4DoneCh := make(chan interface{}) operation4 := generateWaitWithErrorFunc(operation4DoneCh) err4 := grm.Run(volumeName, podName4, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation4, OperationName: "mount"}) if err4 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err4) } operation4DoneCh <- true // operation for pod2 retry time.Sleep(backoffDelay) operation5 := noopFunc err5 := grm.Run(volumeName, podName2, nodeName, volumetypes.GeneratedOperations{OperationFunc: operation5, OperationName: "umount"}) if err5 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err5) } time.Sleep(delay) // Assert // Operation5 will override operation2, since we successfully finished unmount operation on pod2, it should be removed from operations array grm.(*nestedPendingOperations).lock.Lock() defer grm.(*nestedPendingOperations).lock.Unlock() for _, op := range grm.(*nestedPendingOperations).operations { if op.key.podName == podName2 { t.Errorf("NestedPendingOperations failed. Operation for pod2 should be removed") } } } </ins>