Change:
The "Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content" e2e test is failing on our 2k-node gce clusters (https://k8s-) with the following error: cc
Could you take a look into it or reassign as apt? This is part of the correctness suite and large cluster tests are release-blocking.
/assign
is this run using the kubemark environment? I'll also take a look. It could be that the commands this test needs to execute may not work with kubemark. /assign
cc
No, this is not kubemark. This is on a real 2000-node cluster.
Also all other volume-related e2e tests are passing, except this one.
Looking at the log I see that the nfs test fails trying to verify the test file's content. The test exec's into the nfs client pod and executes: The return code from above is 127. The test then deletes the nfs client pod calling which works. The nfs-server pod is deleted in the AfterEach. For some reason the reported error is 137 but the returned err 127. The same command is exec'd for all of the tested storage plugins so not sure why it's failing just for nfs. Error 127 might be "command not found".
The other strange thing is that it's only failing in this suite and not the others.
When I look at the kubelet log, I see the exec command coming through: I also see these suspicious kubelet soon afterwards: So I think the grep command is causing the node to run out of memory and get evicted. I think we can fix this either by: Making the grep command more memory efficient. I'm guessing it's reading the whole file into memory or, Make this test Serial so that it doesn't run in parallel with other pods, since it consumes a lot of memory.
An unrelated issue, is that the node allocatable feature didn't actually evict the pod. Instead this pod got a system OOM kill. /cc
related to the memcg notifications, not node allocatable. Thanks!
any theories why we see this only for the nfs-io tests, and not the glusterfs-io test for instance?
the gluster test didn't run on this in this suite. I'm suspecting that the reason why we didn't see a problem with this test on other suites is that the load could be higher here. I've asked to see if we can reduce the memory usage of the grep call.
pointed out that there are no newlines in the generated file, so when grep is called it could buffer the entire file in memory. Adding a newline at the end of the dd input file should do it.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scalability sig/storage: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
It seems like even if the generated file is broken up by newlines, grep could still use on the order of hundreds of MB of memory for a 100MiB file (even with the -m1 and -F options). We could check the file line by line, but it would take a long time. A better solution would be to check the MD5 file hash. It's fast and uses little memory. The expected hashes will be hard-coded into the test, and if different file sizes are in the future, a hash needs to be computed and as well.
Thanks a lot for fixing this!
Ref

Discussion:
"k8s.io/kubernetes/test/e2e/framework" ) <del> const minFileSize = 1 * framework.MiB </del> <ins> const ( minFileSize = 1 * framework.MiB fileSizeSmall  = 1 * framework.MiB fileSizeMedium = 100 * framework.MiB fileSizeLarge  = 1 * framework.GiB ) // MD5 hashes of the test file corresponding to each file size. // Test files are generated in testVolumeIO() // If test file generation algorithm changes, these must be recomputed. var md5hashes = map[int64]string{ fileSizeSmall:  "5c34c2813223a7ca05a3c2f38c0d1710", fileSizeMedium: "f2fa202b1ffeedda5f3a58bd1ae81104", fileSizeLarge:  "8d763edc71bd16217664793b5a15e403", } </ins> // Return the plugin's client pod spec. Use an InitContainer to setup the file i/o test env. func makePodSpec(config framework.VolumeTestConfig, dir, initCmd string, volsrc v1.VolumeSource, podSecContext *v1.PodSecurityContext) *v1.Pod {

return fmt.Errorf("size of file %s is %d, expected %d", fpath, size, expectSize) } <del> By("verifying file content") // use `grep ... -f` rather than the expected content in a variable to reduce logging rtnstr, err = podExec(pod, fmt.Sprintf("grep -c -m1 -f %s %s", dd_input, fpath)) </del> <ins> By("verifying file hash") rtnstr, err = podExec(pod, fmt.Sprintf("md5sum %s | cut -d' ' -f1", fpath)) </ins> if err != nil { <del> return fmt.Errorf("unable to test file content via `grep %s`: %v", fpath, err) </del> <ins> return fmt.Errorf("unable to test file hash via `md5sum %s`: %v", fpath, err) </ins> } <del> foundCnt, err := strconv.Atoi(strings.TrimSuffix(rtnstr, "n")) if err != nil { return fmt.Errorf("unable to convert string %q to int: %v", rtnstr, err) </del> <ins> actualHash := strings.TrimSuffix(rtnstr, "n") expectedHash, ok := md5hashes[expectSize] if !ok { return fmt.Errorf("File hash is unknown for file size %d. Was a new file size added to the test suite?", expectSize) </ins> } <del> if foundCnt == 0 { rtnstr, err = podExec(pod, fmt.Sprintf("cat %s", dd_input)) if err != nil || len(rtnstr) == 0 { return fmt.Errorf("string not found in file %s and unable to read dd's input file %s: %v", fpath, dd_input, err) } return fmt.Errorf("string %q not found in file %s", rtnstr, fpath) </del> <ins> if actualHash != expectedHash { return fmt.Errorf("MD5 hash is incorrect for file %s with size %d. Expected: `%s`; Actual: `%s`", fpath, expectSize, expectedHash, actualHash) </ins> } return nil

}) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB, 1 * framework.GiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium, fileSizeLarge} </ins> err := testVolumeIO(f, cs, config, volSource, &podSec, testFile, fileSizes) Expect(err).NotTo(HaveOccurred()) })

}) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> err := testVolumeIO(f, cs, config, volSource, nil /*no secContext*/, testFile, fileSizes) Expect(err).NotTo(HaveOccurred()) })

}) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> fsGroup := int64(1234) podSec := v1.PodSecurityContext{ FSGroup: &fsGroup,