Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial) Which test(s) are failing: Various CSI Driver tests. Since when has it been failing: Earliest observed: 11/17 Consistently failing since: 11/29 Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.17 /sig storage /priority critical-urgent
/assign to investigate. are the expansion test flakes supposed to be fixed?
It appears as though some of the failures are also due to tests owned by sig-scheduling: /sig scheduling
The only failures I see in CSI+expansion is from gce-pd CSI driver. We need to fix for that.
We should move "SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow]" out of e2e tests. It relies on a particular state of the cluster before the test start that is hard to validate. It should be more than enough to have an integration test for this, even a unit test (which we already hav ) should be fine .
will you be willing to work on that? it's flaking A LOT right now, definitely more than 50%
I created , should be addressed soon.
There are several tests flaky. Summarize them as below expansion test See comments from and a few other tests The cause of test failure is node is OutOfPod resource. See one of the log message {kubelet test- This is due to the SchedulerPredicates test mentioned above , which creates more than 100 Pods at the same time. We need to move this test out to make sure other tests have enough resources.
I assume this was not cherry-picked to 1.17, right?
/reopen this is still an issue on 1.17. we should probably consider cherry-picking as this is creating a lot of noise in the dashboards
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
correct, it was not cherry-picked to 1.17.
/retitle [Flaking Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial)
/retitle [Flaky Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial)
/milestone v1.18
/cc
/assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test fix needs to be cherry-picked. Assigning to sig-scheduler to track it. /remove-sig storage
/unassign
Those labels are not set on the issue: <detailsIn response to : /assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test needs to be cherry-picked. Assigning to sig-scheduler to track it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test fix needs to be cherry-picked. Assigning to sig-scheduler to track it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17
/retitle [Failing Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial) /remove-kind flake /remove-sig scheduling /sig storage
/unassign /assign Looks like the cherry pick did not cause this to bounce back. Any ideas why that would be? cc
/milestone v1.18
Is this failure specifically for 1.17 (because we only cherry picked into 1.18)?
Looks like the 1.17 serial test grid is looking fine? (slightly flaky)
Can we also use a new bug to track the quay related failures which are unrelated to the original issue?
This job is now green

Discussion:
var _ = SIGDescribe("SchedulerPredicates [Serial]", func() { var cs clientset.Interface var nodeList *v1.NodeList <del> var totalPodCapacity int64 </del> var RCName string var ns string f := framework.NewDefaultFramework("sched-pred")

}) <del> // This test verifies that max-pods flag works as advertised. It assumes that cluster add-on pods stay stable // and cannot be run in parallel with any other test that touches Nodes or Pods. It is so because to check // if max-pods is working we need to fully saturate the cluster and keep it in this state for few seconds. // // Slow PR #13315 (8 min) ginkgo.It("validates MaxPods limit number of pods that are allowed to run [Slow]", func() { totalPodCapacity = 0 for _, node := range nodeList.Items { framework.Logf("Node: %v", node) podCapacity, found := node.Status.Capacity[v1.ResourcePods] framework.ExpectEqual(found, true) totalPodCapacity += podCapacity.Value() } WaitForPodsToBeDeleted(cs) currentlyScheduledPods := WaitForStableCluster(cs, masterNodes) podsNeededForSaturation := int(totalPodCapacity) - currentlyScheduledPods ginkgo.By(fmt.Sprintf("Starting additional %v Pods to fully saturate the cluster max pods and trying to start another one", podsNeededForSaturation)) // As the pods are distributed randomly among nodes, // it can easily happen that all nodes are satured // and there is no need to create additional pods. // StartPods requires at least one pod to replicate. if podsNeededForSaturation > 0 { framework.ExpectNoError(testutils.StartPods(cs, podsNeededForSaturation, ns, "maxp", *initPausePod(f, pausePodConfig{ Name:   "", Labels: map[string]string{"name": ""}, }), true, framework.Logf)) } podName := "additional-pod" WaitForSchedulerAfterAction(f, createPausePodAction(f, pausePodConfig{ Name:   podName, Labels: map[string]string{"name": "additional"}, }), ns, podName, false) verifyResult(cs, podsNeededForSaturation, 1, ns) }) </del> // This test verifies we don't allow scheduling of pods in a way that sum of local ephemeral storage limits of pods is greater than machines capacity. // It assumes that cluster add-on pods stay stable and cannot be run in parallel with any other test that touches Nodes or Pods. // It is so because we need to have precise control on what's running in the cluster.