Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow] Since when has it been failing: 11/13 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: This is consistently failing in 1.13 branch and is release blocking. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test /sig storage /priority critical-urgent /milestone v1.13 /assign /cc
The test case is failing because it wrote 2MB instead of the expected 1MB. It's also very odd that the test is failing in soak, but passing in regular CI suites.
There's a few problems: .GetVolumeSource returns "/tmp" for everything. This means all tests will share the same directory. This should be probably use some unique directory instead volume IO test generates a unique mountPath directory, however that doesn't mean a unique directory is actually created on the host. Ie, a mountPath of "/a/b/c" on a hostpath volume at "/tmp" doesn't mean directory "/tmp/a/b/c" is created. io test file name is not unique seems like the IO test is not cleaning up the files it creates, so the files are getting reused on subsequent tests due to the non-unique naming issues.
The other thing I can't explain is why it's failing only now
Fix is here:
As per discussion in 1.13 burndown this morning, this is a test job issue and its not release blocking at this point. So we can take this fix in 1.14 as opposed now during code freeze let us know if you feel otherwise
/milestone v1.14 /priority important-soon
/remove-priority critical-urgent

Discussion:
import ( "fmt" "math" <del> "path" </del> <ins> "path/filepath" </ins> "strconv" "strings" "time"

testpatterns.FileSizeLarge:  "8d763edc71bd16217664793b5a15e403", } <ins> const mountPath = "/opt" </ins> type volumeIOTestSuite struct { tsInfo TestSuiteInfo }

} // Return the plugin's client pod spec. Use an InitContainer to setup the file i/o test env. <del> func makePodSpec(config framework.VolumeTestConfig, dir, initCmd string, volsrc v1.VolumeSource, podSecContext *v1.PodSecurityContext) *v1.Pod { volName := fmt.Sprintf("%s-%s", config.Prefix, "io-volume") </del> <ins> func makePodSpec(config framework.VolumeTestConfig, initCmd string, volsrc v1.VolumeSource, podSecContext *v1.PodSecurityContext) *v1.Pod { </ins> var gracePeriod int64 = 1 <ins> volName := fmt.Sprintf("io-volume-%s", config.Namespace) </ins> return &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind:       "Pod",

VolumeMounts: []v1.VolumeMount{ { Name:      volName, <del> MountPath: dir, </del> <ins> MountPath: mountPath, </ins> }, }, },

// Note: `fsizes` values are enforced to each be at least `MinFileSize` and a multiple of `MinFileSize` //   bytes. func testVolumeIO(f *framework.Framework, cs clientset.Interface, config framework.VolumeTestConfig, volsrc v1.VolumeSource, podSecContext *v1.PodSecurityContext, file string, fsizes []int64) (err error) { <del> dir := path.Join("/opt", config.Prefix, config.Namespace) ddInput := path.Join(dir, "dd_if") </del> <ins> ddInput := filepath.Join(mountPath, fmt.Sprintf("%s-%s-dd_if", config.Prefix, config.Namespace)) </ins> writeBlk := strings.Repeat("abcdefghijklmnopqrstuvwxyz123456", 32) // 1KiB value loopCnt := testpatterns.MinFileSize / int64(len(writeBlk)) // initContainer cmd to create and fill dd's input file. The initContainer is used to create

// used to create a 1MiB file in the target directory. initCmd := fmt.Sprintf("i=0; while [ $i -lt %d ]; do echo -n %s >>%s; let i+=1; done", loopCnt, writeBlk, ddInput) <del> clientPod := makePodSpec(config, dir, initCmd, volsrc, podSecContext) </del> <ins> clientPod := makePodSpec(config, initCmd, volsrc, podSecContext) </ins> By(fmt.Sprintf("starting %s", clientPod.Name)) podsNamespacer := cs.CoreV1().Pods(config.Namespace)

return fmt.Errorf("failed to create client pod %q: %v", clientPod.Name, err) } defer func() { <del> // note the test dir will be removed when the kubelet unmounts it </del> <ins> deleteFile(clientPod, ddInput) </ins> By(fmt.Sprintf("deleting client pod %q...", clientPod.Name)) e := framework.DeletePodWithWait(f, cs, clientPod) if e != nil {

if math.Mod(float64(fsize), float64(testpatterns.MinFileSize)) != 0 { fsize = fsize/testpatterns.MinFileSize + testpatterns.MinFileSize } <del> fpath := path.Join(dir, fmt.Sprintf("%s-%d", file, fsize)) </del> <ins> fpath := filepath.Join(mountPath, fmt.Sprintf("%s-%d", file, fsize)) defer func() { deleteFile(clientPod, fpath) }() </ins> if err = writeToFile(clientPod, fpath, ddInput, fsize); err != nil { return err } if err = verifyFile(clientPod, fpath, fsize, ddInput); err != nil { return err } <del> deleteFile(clientPod, fpath) </del> } return