Change:
<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package volumemanager import ( "k8s.io/api/core/v1" "k8s.io/kubernetes/pkg/kubelet/config" "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/volume/util/types" ) // FakeVolumeManager is a test implementation that just tracks calls type FakeVolumeManager struct { volumes       map[v1.UniqueVolumeName]bool reportedInUse map[v1.UniqueVolumeName]bool } // NewFakeVolumeManager creates a new VolumeManager test instance func NewFakeVolumeManager(initialVolumes []v1.UniqueVolumeName) *FakeVolumeManager { volumes := map[v1.UniqueVolumeName]bool{} for _, v := range initialVolumes { volumes[v] = true } return &FakeVolumeManager{ volumes:       volumes, reportedInUse: map[v1.UniqueVolumeName]bool{}, } } // Run is not implemented func (f *FakeVolumeManager) Run(sourcesReady config.SourcesReady, stopCh <-chan struct{}) { } // WaitForAttachAndMount is not implemented func (f *FakeVolumeManager) WaitForAttachAndMount(pod *v1.Pod) error { return nil } // GetMountedVolumesForPod is not implemented func (f *FakeVolumeManager) GetMountedVolumesForPod(podName types.UniquePodName) container.VolumeMap { return nil } // GetExtraSupplementalGroupsForPod is not implemented func (f *FakeVolumeManager) GetExtraSupplementalGroupsForPod(pod *v1.Pod) []int64 { return nil } // GetVolumesInUse returns a list of the initial volumes func (f *FakeVolumeManager) GetVolumesInUse() []v1.UniqueVolumeName { inuse := []v1.UniqueVolumeName{} for v := range f.volumes { inuse = append(inuse, v) } return inuse } // ReconcilerStatesHasBeenSynced is not implemented func (f *FakeVolumeManager) ReconcilerStatesHasBeenSynced() bool { return true } // VolumeIsAttached is not implemented func (f *FakeVolumeManager) VolumeIsAttached(volumeName v1.UniqueVolumeName) bool { return false } // MarkVolumesAsReportedInUse adds the given volumes to the reportedInUse map func (f *FakeVolumeManager) MarkVolumesAsReportedInUse(volumesReportedAsInUse []v1.UniqueVolumeName) { for _, reportedVolume := range volumesReportedAsInUse { if _, ok := f.volumes[reportedVolume]; ok { f.reportedInUse[reportedVolume] = true } } } // GetVolumesReportedInUse is a test function only that returns a list of volumes // from the reportedInUse map func (f *FakeVolumeManager) GetVolumesReportedInUse() []v1.UniqueVolumeName { inuse := []v1.UniqueVolumeName{} for reportedVolume := range f.reportedInUse { inuse = append(inuse, reportedVolume) } return inuse } </ins>

Discussion:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 2019-01-15 (when they were ) Testgrid link: Reason for failure: They all seem to be failing with: The failures seem to start with .
/sig storage
Summoning and This seems to have been failing since it was in master-blocking.
This is the same as
Thanks closing this one. /close
Closing this issue. <detailsIn response to : Thanks closing this one. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen I took a look at the latest test failures and I think it is a different issue than
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at one test failure here: Sequence of events: stopped, pod deleted: started again: volume to dsw started and failed because it's not in the node's volumesinuse list. This repeats for a while. also started, but we skipped cleaning up because the volume is in dsw so we want to let volume manager gracefully cleanup don't remove volume from dsw because actual state has not been updated? So it seems like there's a problem with populating the node's volumesinuse after a kubelet restart.
cc
/assign will continue investigating.

i will try repro-ing at without different commits to try to figure out which pr introduced the regression
/priority important-soon
I ran these tests against a cluster with the NodeLease feature disabled, and the tests pass. So I think the issue is related to the new NodeLease feature enabled a month ago: cc
I discussed with VolumesInUse is updated periodically in the Node.Status. So we need to make sure this flow is working well with the NodeLease feature.
and I walked through the code. The NodeLease feature introduced an early exit to the node status update function, so our theory is we need to add a call to . I will make the change and test it out