Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: CRI-O job in our CI using kubernetes master tests Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with mount options Since when has it been failing: since this PR has been merged: Testgrid link: link to our job is here: https://openshift-gce- https://openshift-gce- https://openshift-gce- Reason for failure: we don't know the reason of the failure but it doesn't seem related to CRI-O. It's constantly reproducible also. Anything else we need to know: /cc <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
/sig node /sig storage
/cc
Curiously, it does not seem to have flaked in core kubernetes CI
I'm still looking at it on our side
This test is newly in this commit. It tests that nfs external dynamic provisioner can provision a volume with specific mount options and the volume is properly mounted in the pod. The mount options specified in this test are "vers=4.1", "proto=tcp", and "nosuid". () The test seems to fail to create a pod with the volume and other tests that don't specify these mount options don't fail. So, I guess that this is caused because this environment doesn't support either of the option above. Is it possible to try testing it manually by changing these options?
yes, I have a box where I can change them and re-test, do you have suggestions on what to use to test?
(sorry fat fingers)
If it was an unsupported mount option, then the test should fail every time, not just flake?
yes, it's failing every time for us
Are there kubelet node logs we can look at?
yes, we have artifacts for each job at https://gcsweb-
As a starting point, I would recommend deleting only "proto=tcp", and check if it still fails, first. Then moving back "proto=tcp" and deleting only "nosuid", and check what happens. And continue testing with combinations of options. Or, if we can find harmless mount options that can be specified for nfs, we will be able to replace the options with certainly harmless ones. Note that tests for AWS and GCE uses "debug" and "nouid32", which don't seem to be valid. for nfs
The container is failing with exit-code 1 with: I'll try to see which one is the bad one
do you think we could make the test easier to debug and figure out which mount option doesn't work?
yes, I'm trying to add something to that command to at least have some logs if that fails cause right now, if you do kubectl get logs on the failing pod you don't get anything
looks like removing makes the container happy and test as well. I'm not sure why it fails on rhel and fedora though, I'll dig into that, but yeah, we could use something different if possible
confirmed that if I remove nosuid the test works fine, I"m modifying the test code to output something useful in case it fails
Thank you for your quick investigation and improvement!
I'm going to open a PR for the test output to have something useful () and understand why it fails with
sorry, I am the one who wrote that hideous bash one-liner. TY for any patches
so, isn't shown: is that required for the test? can we have another option checked instead?
maybe we can use ? who is setting and where?
PTAL at
do you guys have any insight on this?
As for this e2e test, we will be able to use any options which are valid for nfs e2e test, because the purpose of the test is to check that the volume mount options work well for provisioning and the options are properly reflected to the mount options inside the pod. (Actually, I chose for nfs test just because I found it first.) As for issue for nfs dynamic provisioner with this environment, it might be still worth looking into, if this option can be specified by users. However, this could be out of the scope of this issue.
exactly, thanks, I'm going to investigate it not as part of this PR
Since this test failure is non-blocking for release, setting priority approprately. /priority important-longterm

Discussion:
SupportedFsType: sets.NewString( "", // Default fsType ), <del> SupportedMountOption: sets.NewString("proto=tcp", "nosuid"), </del> <ins> SupportedMountOption: sets.NewString("proto=tcp", "relatime"), </ins> RequiredMountOption:  sets.NewString("vers=4.1"), IsPersistent:         true, IsFsGroupSupported:   false,

// Get entry, get mount options at 6th word, replace brackets with commas command += fmt.Sprintf(" && ( mount | grep 'on /mnt/test' | awk '{print $6}' | sed 's/^(/,/; s/)$/,/' | grep -q ,%s, )", option) } <ins> command += " || (mount | grep 'on /mnt/test'; false)" </ins> runInPodWithVolume(client, claim.Namespace, claim.Name, t.NodeName, command) By("checking the created volume is readable and retains data")

pod, err := c.CoreV1().Pods(ns).Create(pod) framework.ExpectNoError(err, "Failed to create pod: %v", err) defer func() { <ins> body, err := c.CoreV1().Pods(ns).GetLogs(pod.Name, &v1.PodLogOptions{}).Do().Raw() if err != nil { framework.Logf("Error getting logs for pod %s: %v", pod.Name, err) } else { framework.Logf("Pod %s has the following logs: %s", pod.Name, body) } </ins> framework.DeletePodOrFail(c, ns, pod.Name) }() framework.ExpectNoError(framework.WaitForPodSuccessInNamespaceSlow(c, pod.Name, pod.Namespace))