Code change:
return fmt.Errorf("size of file %s is %d, expected %d", fpath, size, expectSize) } <del> By("verifying file content") // use `grep ... -f` rather than the expected content in a variable to reduce logging rtnstr, err = podExec(pod, fmt.Sprintf("grep -c -m1 -f %s %s", dd_input, fpath)) </del> <ins> By("verifying file hash") rtnstr, err = podExec(pod, fmt.Sprintf("md5sum %s | cut -d' ' -f1", fpath)) </ins> if err != nil { <del> return fmt.Errorf("unable to test file content via `grep %s`: %v", fpath, err) </del> <ins> return fmt.Errorf("unable to test file hash via `md5sum %s`: %v", fpath, err) </ins> } <del> foundCnt, err := strconv.Atoi(strings.TrimSuffix(rtnstr, "n")) if err != nil { return fmt.Errorf("unable to convert string %q to int: %v", rtnstr, err) </del> <ins> actualHash := strings.TrimSuffix(rtnstr, "n") expectedHash, ok := md5hashes[expectSize] if !ok { return fmt.Errorf("File hash is unknown for file size %d. Was a new file size added to the test suite?", expectSize) </ins> } <del> if foundCnt == 0 { rtnstr, err = podExec(pod, fmt.Sprintf("cat %s", dd_input)) if err != nil || len(rtnstr) == 0 { return fmt.Errorf("string not found in file %s and unable to read dd's input file %s: %v", fpath, dd_input, err) } return fmt.Errorf("string %q not found in file %s", rtnstr, fpath) </del> <ins> if actualHash != expectedHash { return fmt.Errorf("MD5 hash is incorrect for file %s with size %d. Expected: `%s`; Actual: `%s`", fpath, expectSize, expectedHash, actualHash) </ins> } return nil

Related discussion:
text: The "Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content" e2e test is failing on our 2k-node gce clusters (https://k8s-) with the following error: cc
Could you take a look into it or reassign as apt? This is part of the correctness suite and large cluster tests are release-blocking.
/assign
is this run using the kubemark environment? I'll also take a look. It could be that the commands this test needs to execute may not work with kubemark. /assign
cc
No, this is not kubemark. This is on a real 2000-node cluster.
Also all other volume-related e2e tests are passing, except this one.
Looking at the log I see that the nfs test fails trying to verify the test file's content. The test exec's into the nfs client pod and executes: The return code from above is 127. The test then deletes the nfs client pod calling which works. The nfs-server pod is deleted in the AfterEach. For some reason the reported error is 137 but the returned err 127. The same command is exec'd for all of the tested storage plugins so not sure why it's failing just for nfs. Error 127 might be "command not found".
The other strange thing is that it's only failing in this suite and not the others.
When I look at the kubelet log, I see the exec command coming through: I also see these suspicious kubelet soon afterwards: So I think the grep command is causing the node to run out of memory and get evicted. I think we can fix this either by: Making the grep command more memory efficient. I'm guessing it's reading the whole file into memory or, Make this test Serial so that it doesn't run in parallel with other pods, since it consumes a lot of memory.
An unrelated issue, is that the node allocatable feature didn't actually evict the pod. Instead this pod got a system OOM kill. /cc
related to the memcg notifications, not node allocatable. Thanks!
any theories why we see this only for the nfs-io tests, and not the glusterfs-io test for instance?
the gluster test didn't run on this in this suite.