Change:
kubeletmetrics.VolumeStatsInodesFreeKey, kubeletmetrics.VolumeStatsInodesUsedKey, } <ins> // Poll kubelet metrics waiting for the volume to be picked up // by the volume stats collector var kubeMetrics metrics.KubeletMetrics waitErr := wait.Poll(30*time.Second, 5*time.Minute, func() (bool, error) { framework.Logf("Grabbing Kubelet metrics") // Grab kubelet metrics from the node the pod was scheduled on var err error kubeMetrics, err = metricsGrabber.GrabFromKubelet(pod.Spec.NodeName) if err != nil { framework.Logf("Error fetching kubelet metrics") return false, err } key := volumeStatKeys[0] kubeletKeyName := fmt.Sprintf("%s_%s", kubeletmetrics.KubeletSubsystem, key) if !findVolumeStatMetric(kubeletKeyName, pvc.Namespace, pvc.Name, kubeMetrics) { return false, nil } return true, nil }) Expect(waitErr).NotTo(HaveOccurred(), "Error finding volume metrics : %v", waitErr) </ins> for _, key := range volumeStatKeys { kubeletKeyName := fmt.Sprintf("%s_%s", kubeletmetrics.KubeletSubsystem, key) <del> verifyVolumeStatMetric(kubeletKeyName, pvc.Namespace, pvc.Name, kubeMetrics) </del> <ins> found := findVolumeStatMetric(kubeletKeyName, pvc.Namespace, pvc.Name, kubeMetrics) Expect(found).To(BeTrue(), "PVC %s, Namespace %s not found for %s", pvc.Name, pvc.Namespace, kubeletKeyName) </ins> } <ins> framework.Logf("Deleting pod %q/%q", pod.Namespace, pod.Name) framework.ExpectNoError(framework.DeletePodWithWait(f, c, pod)) </ins> }) })

Discussion:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: Test should not flake /sig storage /cc
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
- You won't be able to assign this to me (not a kubernetes member). Am taking a look.
volumemetrics.go_ could/should have more logging, either calling or more calls to .
The test appears to fail on this metric: "kubeletvolumestatsused_bytes". So samplePVC != pvcName and/or sampleNS != namespace. It would be nice if the sample name and ns were also logged. Also something is weird here: the framework's is called before the local . Note in the log:
- will add a couple of Logf() statements to log the samples - you're right that will help. Once the volume is attached, volume stats are grabbed every interval. The thing I noticed in the logs is that we issue a delete for the pod at but the volume only got attached/bound successfully at The test assumes blocks till the pod is running and PVC is attached but maybe that's not a valid assumption?
Nevermind - That log message has 2 timestamps in it - the volume was provisioned at .
Pushed The working theory being that volume stats were not collected within the wait earlier being used.
did you figure out if I was correct in thinking that the framework was invoked before the local ? That seems very, very unlikely but that is how it looked in the log.
- I believe you're right that the framework is being invoked before the local one.

I observe the same in a test as well, so I don't think it's specific to this call. Based on the , nested blocks should be invoked inner-outer. It's not clear, however - whether the local is nested - or at the same level as the framework block when I look at