Change:
now := kl.clock.Now() if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) && now.Before(kl.lastStatusReportTime.Add(kl.nodeStatusReportFrequency)) { if !podCIDRChanged && !nodeStatusHasChanged(&originalNode.Status, &node.Status) { <ins> // We must mark the volumes as ReportedInUse in volume manager's dsw even // if no changes were made to the node status (no volumes were added or removed // from the VolumesInUse list). // // The reason is that on a kubelet restart, the volume manager's dsw is // repopulated and the volume ReportedInUse is initialized to false, while the // VolumesInUse list from the Node object still contains the state from the // previous kubelet instantiation. // // Once the volumes are added to the dsw, the ReportedInUse field needs to be // synced from the VolumesInUse list in the Node.Status. // // The MarkVolumesAsReportedInUse() call cannot be performed in dsw directly // because it does not have access to the Node object. // This also cannot be populated on node status manager init because the volume // may not have been added to dsw at that time. kl.volumeManager.MarkVolumesAsReportedInUse(node.Status.VolumesInUse) </ins> return nil } }

Discussion:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 2019-01-15 (when they were ) Testgrid link: Reason for failure: They all seem to be failing with: The failures seem to start with .
/sig storage
Summoning and This seems to have been failing since it was in master-blocking.
This is the same as
Thanks closing this one. /close
Closing this issue. <detailsIn response to : Thanks closing this one. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen I took a look at the latest test failures and I think it is a different issue than
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at one test failure here: Sequence of events: stopped, pod deleted: started again: volume to dsw started and failed because it's not in the node's volumesinuse list. This repeats for a while. also started, but we skipped cleaning up because the volume is in dsw so we want to let volume manager gracefully cleanup don't remove volume from dsw because actual state has not been updated? So it seems like there's a problem with populating the node's volumesinuse after a kubelet restart.
cc
/assign will continue investigating.

i will try repro-ing at without different commits to try to figure out which pr introduced the regression
/priority important-soon
I ran these tests against a cluster with the NodeLease feature disabled, and the tests pass. So I think the issue is related to the new NodeLease feature enabled a month ago: cc
I discussed with VolumesInUse is updated periodically in the Node.Status. So we need to make sure this flow is working well with the NodeLease feature.
and I walked through the code. The NodeLease feature introduced an early exit to the node status update function, so our theory is we need to add a call to . I will make the change and test it out